Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

agent start CLI command now allows for kwargs #1737

Merged
merged 8 commits into from
Nov 13, 2019
Merged

Conversation

joshmeek
Copy link

@joshmeek joshmeek commented Nov 13, 2019

Thanks for contributing to Prefect!

Please describe your work and make sure your PR:

  • adds new tests (if appropriate)
  • updates CHANGELOG.md (if appropriate)
  • updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

Note that your PR will not be reviewed unless all three boxes are checked.

What does this PR change?

Closes #1734
prefect agent start now accepts kwargs. This is useful for users when providing the configuration options for the Fargate Agent.

e.g.

$ prefect agent start fargate cpu=$CPU memory=$MEMORY networkConfiguration=$NETWORK_CONFIGURATION

Why is this PR important?

Having to provide kwargs in the boto3 camel casing convention could potentially lead to conflicts in environment variable names when using external tooling (e.g. Terraform) to deploy the Agent.

Instead of having to do:

$ export memory=512
$ prefect agent start fargate

There is now the option to use custom names:

$ export AGENT_MEMORY=512
$ prefect agent start fargate memory=$AGENT_MEMORY

@joshmeek joshmeek added component:cli Related to the Prefect CLI component:agent Related to the flow submitting agent labels Nov 13, 2019
@joshmeek
Copy link
Author

Interesting, looks like a few tests are failing on 3.5 for unrelated reasons:

FAILED tests/tasks/google/test_bigquery.py
FAILED tests/tasks/google/test_gcs_copy.py
FAILED tests/tasks/google/test_gcs_upload_download.py

@cicdw
Copy link
Member

cicdw commented Nov 13, 2019

Looks like google made some python releases very recently: https://github.com/googleapis/google-cloud-python/releases/tag/storage-1.23.0

You can either:

  • investigate the changes and whether, e.g., support for 3.5 has been dropped
  • bound our google packages above in requirements to temporarily avoid the error

@joshmeek
Copy link
Author

joshmeek commented Nov 13, 2019

Going to change the bounds now. Doesn't appear they dropped support for 3.5 but this issue googleapis/google-cloud-python#9786 is relevant. Not sure why it's only affecting the tests for 3.5 though.

@codecov
Copy link

codecov bot commented Nov 13, 2019

Codecov Report

Merging #1737 into master will increase coverage by 0.01%.
The diff coverage is 100%.

docs/cloud/agent/fargate.md Show resolved Hide resolved
src/prefect/cli/agent.py Show resolved Hide resolved
Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome, i think this is a significant improvement

@joshmeek joshmeek merged commit 9a6ac52 into master Nov 13, 2019
@joshmeek joshmeek deleted the agent_options branch November 13, 2019 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:agent Related to the flow submitting agent component:cli Related to the Prefect CLI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add CLI flags for all Fargate Agent keyword arguments
2 participants