Add BranchPythonVirtualenvOperator #3
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jscheffl
force-pushed
the
feature/add-branch-venv-operator
branch
2 times, most recently
from
9941e5d
to
4e66370
Compare
Put a quote to the entire curl command
Redis 5 relased last week breaks celery, celery is limiting it for now and will resolve it later, we should similarly limit redis on our side to limit redis for users who will not upgrade to celery that will be released shortly. Fixes: apache#33744
Only run it when the map may have changed (migration or core version change), and also only check migration files (ignore things like pycache).
There were still some left-overs of EAGER_UPGRADE in PROD image building. Howwever "eager upgrade" only makes sense for CI images. PROD images when being built should use eager upgrades as they are produced in the CI image step. This PR does the following: * removes eager upgrade parameters from PROD image * instead, prod image build has a new flag for installing the images: --use-constraints-for-context-packages which will automatically use constraints from "docker-context-files" if they are present there. * modifies the CI workflows to upload constraints as artifacts and download them for PROD image build when "eager upgrade" has been used and directs it to use "source" constraints * adds back support to "upgrade to newer dependencies" label that makes it easy to test "eager upgrade" As the result, when PROD image is build in CI: * when regular PR is run, it will use latest github "source" constraints * whwn "eager upgrade" PR is run, it will use the eager-upgrade constraints that were generated during CI build
…che#33790) The apache#33626 change missed an edge case where back reference new link was in the top-level folder and crashed with ValueError. Unfortunately back-reference command is one that we do not run (yet) in CI to verify if it works, because we need a checked-out airflow-site for it, so such errors might go unnoticed. However this error is an indication that we likely should add it to CI - in order to avoid situation when the command is expected to be working at release time.
) * Run back-reference breeze command as part of our CI builds The `back-reference` command post-processes our docuementation to add redirections for new pages in the old documentation, so that user browsing the page can change version without hittin 404 error. This command was not run in CI and some refactorings could break it (as it already happened in apache#33626 - fixed by apache#33790). Most other breeze commands are already run in CI, so this one should be no different. This PR adds back-reference generation as part of our regular documentation building step.
* Vastly speed up provider documentation publishing Publishing documentation is quite slow when preparing multiple provider documentation - and this is mostly because we are serializing copying of directories and checking if the directories are present. Hhowever we can easily speed it up by parallelising the publishing per-package. This will speed up both CI and release-manager's docs publishing step (vastly). Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
When we are building PROD image in CI for non main branch, we are installing providers from PyPI rather than building them locally from sources. Therefore we should use `PyPI` constraints for such builds not the "source" constraints (they might differ). This PR adds two steps: * In the CI build, when we do not build providers we generate PyPI constraints additionally to source constraints * In the PROD build we use the PyPI constraints in case we do not build providers locally
…che#33741) As the name suggests, this test should verify that a task is not adopted if it belongs to a DAG which is not in RUNNING state. Before this change: the test setup creates a DAG in RUNNING state, but the test passes because the task has queued_by_job_id set to a live scheduler (so is not adoptable for a different reason - it's not an orphan). It looks l ike it was a copy-paste from the previous test case (test_reset_orphaned_tasks_no_orphans) which is supposed to test this scenario. After this change: the test setup is aligned with the test name - the task is an orphan but its DAG is not running so the task cannot be adopted. Co-authored-by: daniel.dylag <danieldylag1990@gmail.com>
* Chart: Update postgres subchart * Move newsfragment
…hem into a type-checking block (apache#33781)
… type-checking block (apache#33783)
…pe-checking block (apache#33754)
Minor grammatical fix
* Custom fetch all handler for vertica to not miss errors * missing parameter * Fix test (set nextset to none) * fix static checks * fix static-check error * fix static-check error * rename variable * add docstring * fix docstring
The tests for Salescforce hooh were too strict - they were expecting a specific version and when new salesforce-simple library has been released (1.12.5), the version changed - the tests started to fail. This PR fixes the tests by accepting any version.
…pache#34119) The elasticsearch handler got all configuraiton parameters from the "elasticsearch_config" section but it means that in airflow versions pre 2.7 it could get old config keys which renders the new provider unusable. This PR filters out configuration parameter to only pass valid parameters for the new handler. Fixes: apache#34099
…eline_task (apache#32903) --------- Co-authored-by: Adam Best <abest@adams-mbp.lan> Co-authored-by: Adam Best <> Co-authored-by: Adam Best <unknown> Co-authored-by: Hussein Awala <hussein@awala.fr>
* allow empty access control on dags * Add test which demonstrates loss of information during ser/de * Keep empty value when serializing access_control dict
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
The docstring in the class UndeployModelOperator regarding the endpoint_id is not correct. This parameter accept only the id of the endpoint and not the path as written 'projects/{project}/locations/{location}/endpoints/{endpoint}'
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
The query param `tab` is set on the DAGs view when any of the tabs _within_ the details panel is selected, thus we should show the details panel, even if the user previously hid it. There are 2 main scenarios where a user would navigate to such a link: 1. User clicks on one of the pills on the DAG view (Graph, Gantt, Code). Imo, the user will expect to see the relevant tab in the details panel, even if they have it hidden. 2. User clicks on a link (e.g. from DAG runs, or someone sharing a link). Less clear cut than scenario 1, but still I think the most user friendly result is to show the requested tab (in the details panel).
…when exception is raised (apache#34165)
) We should only tag :latest image when default image is used because this step always tags `apache/airflow:X.Y.Z` (defaulti Python) to become the `apache/airflow:latest`. This step was previously run for all Python versions which could have two effects: * if non-default image completed before the default such tagging would fail as the `apache/airflow:X.Y.Z` was not ready yet to be used as target of `apache/airflow:latesti` tag * if non-default image completed later, it would be no-op because the `apache/airflow:latest` already pointed to `apache/airflow:X.Y.Z`
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a split of the PR apache#33017 to add BranchPythonVirtualenvOperator.
As it builds on top of the other branch it needs to be merged behind, else it will not work - but pytests are needed.
Therefore before merging, the target repo need to be switched to apache/airflow:main.
FYI @AutomationDev85 @clellmann @wolfdn