Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too strict invariant for celery deployment storage #2436

Closed
mrdrprofuroboros opened this issue May 8, 2020 · 1 comment
Closed

Too strict invariant for celery deployment storage #2436

mrdrprofuroboros opened this issue May 8, 2020 · 1 comment
Assignees

Comments

@mrdrprofuroboros
Copy link
Contributor

mrdrprofuroboros commented May 8, 2020

I'm trying to set up a dockerized celery deployment on a single machine:

  • I made a doker-compose with rabbitmq, dagit master and dagster celery workers containers
  • I had to use custom celery_config.yaml to overwrite broker url for internal docker network address
  • But I still want to use filesystem storage through mapping all my containers to a single shared volume on disk

And when I try to launch my pipeline I get
dagster.check.CheckError: Invariant failed. Description: Must use S3 or GCS storage with non-local Celery broker: amqp://guest:guest@cube_rabbitmq:5672// and backend: rpc://

@schrockn schrockn self-assigned this May 8, 2020
@mgasner
Copy link
Contributor

mgasner commented May 8, 2020

it would be nice if our local filesystem-related failures were early -- right now, we will fail when a solid looks for an upstream intermediate. maybe we could have a sentinel file written to the filesystem that sub-executions look for (though they would need to know they were sub-executions which right now they don't). it's complicated b/c distributed disjoint executions without dependencies don't actually need a shared filesystem at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants