Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only spin up resources when needed #1937

Open
schrockn opened this issue Nov 25, 2019 · 1 comment
Assignees
Labels

Comments

@schrockn
Copy link
Member

@schrockn schrockn commented Nov 25, 2019

Now that we are going to start pushing stuff like https://dagster.phacility.com/D1349 and also supporting larger pipelines and pushing solid subselection more in the tools, we are going to need to start spinning up resources on-demand only. As written for example a spark context would be brought up in every execution step regardless of whether or not it was used.

@mgasner mgasner added the core label Nov 29, 2019
@schrockn

This comment has been minimized.

Copy link
Member Author

@schrockn schrockn commented Dec 2, 2019

@schrockn schrockn self-assigned this Dec 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.