Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add waiter config params to emr.add_job_flow_steps #28464

Merged
merged 10 commits into from
Jan 5, 2023
Merged

Add waiter config params to emr.add_job_flow_steps #28464

merged 10 commits into from
Jan 5, 2023

Conversation

ShelRoman
Copy link
Contributor

A small tweak related to #27286

 WaiterConfig={
     "Delay": 5,
     "MaxAttempts": 100,
}

The hardcoded config isn't usable with jobs that are running longer than ~500 seconds.
Moved those values into default arguments of the add_job_flow_steps() function which won't change default behaviour but will add some flexibility for use cases with longer emr steps.

Also considered option to pass whole config dict instead of waiter_delay and waiter_max_attempts, but these changes are smaller.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon-aws AWS/Amazon - related issues labels Dec 19, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 19, 2022

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@ShelRoman ShelRoman requested review from Taragolis and removed request for eladkal January 3, 2023 07:27
@ShelRoman
Copy link
Contributor Author

@eladkal could you please take a look also? I didn't intend to remove you from the reviewers but it happened

@ShelRoman
Copy link
Contributor Author

@Taragolis could you please merge this one?
I don't have write access to do that.

@potiuk
Copy link
Member

potiuk commented Jan 5, 2023

@Taragolis could you please merge this one? I don't have write access to do that.

Just to make sure to set you expectations. Please do not ping people like that. It's quite selfish.

Be patient. There are 100s of PR being possibly looked at in parallel and merged in due time. You are looking at 1 PR while people who merge them at many @ShelRoman. be mindful for people time and accept the fact that people here work in their free time - and do not expect "immediate" reactions.

Watch my talk https://www.youtube.com/watch?v=G6VjYvKr2wQ&list=PLGudixcDaxY2LxjeHpZRtzq7miykjjFOn&index=55 about empathy to understand why.

@ShelRoman
Copy link
Contributor Author

@potiuk thanks for the clarification and the link. :)
I didn't expect an immediate reaction, which might be commercial deformation kicks in, since it's my first PR in open-source.

@Taragolis Taragolis merged commit a9493c1 into apache:main Jan 5, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Jan 5, 2023

Awesome work, congrats on your first merged pull request!

@ShelRoman ShelRoman deleted the airflow-providers-aws-emr-addstep-waiter-config branch January 6, 2023 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants