Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YandexCloud provider: Support new Yandex SDK features: log_group_id, user-agent, maven packages #20103

Merged
merged 6 commits into from
Dec 14, 2021

Conversation

Piatachock
Copy link
Contributor

@Piatachock Piatachock commented Dec 7, 2021

  • Support log_group_id parameter passing in DataprocCreateClusterOperator.
  • Support User-Agent header in GRPC requests via SDK
  • Support Maven packages parameters for Spark and PySpark jobs

Functionality is tested on local AirFlow installation.

@boring-cyborg
Copy link

boring-cyborg bot commented Dec 7, 2021

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@Piatachock Piatachock changed the title YandexCloud provider: Support log_group_id parameter [WIP] YandexCloud provider: Support log_group_id parameter Dec 7, 2021
@Piatachock Piatachock force-pushed the yandex_log_group_id branch 2 times, most recently from 17b5640 to b7cf379 Compare December 9, 2021 10:26
@Piatachock Piatachock changed the title [WIP] YandexCloud provider: Support log_group_id parameter [WIP] YandexCloud provider: Support new Yandex SDK features: log_group_id, user-agent, maven pacakges Dec 9, 2021
@Piatachock Piatachock changed the title [WIP] YandexCloud provider: Support new Yandex SDK features: log_group_id, user-agent, maven pacakges [WIP] YandexCloud provider: Support new Yandex SDK features: log_group_id, user-agent, maven packages Dec 9, 2021
@Piatachock Piatachock changed the title [WIP] YandexCloud provider: Support new Yandex SDK features: log_group_id, user-agent, maven packages YandexCloud provider: Support new Yandex SDK features: log_group_id, user-agent, maven packages Dec 10, 2021
@Piatachock
Copy link
Contributor Author

Piatachock commented Dec 13, 2021

PR changes are fixed, reviewers are welcome :)

@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Dec 13, 2021
@Piatachock
Copy link
Contributor Author

Helm Chart Executor Upgrade task timed out. Not sure if this PR could be a cause for this.

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

Helm Chart Executor Upgrade task timed out. Not sure if this PR could be a cause for this.

Nope. I think it is just very slow now. We need to speed it up or better - parallelise

@Piatachock
Copy link
Contributor Author

Ok; anything I can do to make this PR happen?

@potiuk potiuk merged commit 41c49c7 into apache:main Dec 14, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 14, 2021

Awesome work, congrats on your first merged pull request!

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

You just did :). Merged. The fix to the too long upgrade check is running here #20289

@Piatachock Piatachock deleted the yandex_log_group_id branch December 16, 2021 11:53
@potiuk potiuk added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jan 22, 2022
@potiuk potiuk added this to the Airflow 2.2.4 milestone Jan 22, 2022
potiuk pushed a commit that referenced this pull request Jan 22, 2022
…user-agent, maven packages (#20103)

(cherry picked from commit 41c49c7)
jedcunningham pushed a commit that referenced this pull request Jan 27, 2022
…user-agent, maven packages (#20103)

(cherry picked from commit 41c49c7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants