New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix name format in the batch requests #32070
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
Also experiencing this problem. |
@HaeSe0ng For the interim, you might be able to switch to non-deferable as it doesn't do a get_batch request up front and instead waits on the initial job request completion. |
After the incident in our dags, I confirmed that the issue only occurs when FYI. Even if |
Yep, we are tracking that deficiency. This is just the only route for current production Airflow. We are in contact with GCP and have been told that the API behavior is being reverted. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for submitting this PR. It looks good!
Let's wait for the CI to pass before merging it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks it passed/ Merging.
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Thank you for the reviews. May I know what version would include this PR and when would it be released? |
For providers it is always it's 'next' and 'when ready' |
Srsly. Watch the announcements, and release notes if you want to know when it is released. In case of providers we determine the version and timing Just before releasing them - based on what kind of changes are included. It is impossible to tell when and what version it will be. You need to watch. But it will be included in the "next" version (after we release the versions that we were voting on while this one was merged). |
Oh, I see. I'll keep watching the announcements. Thanks for letting me know. |
Co-authored-by: Hussein Awala <hussein@awala.fr>
closes: #32069
The name argument in the hook code follows the format "projects/PROJECT_ID/regions/DATAPROC_REGION/batches/BATCH_ID". However, according to the Google Cloud DataProc API Reference, it should be in the format "projects/PROJECT_ID/locations/DATAPROC_REGION/batches/BATCH_ID .