Selective checks for API did not match API test specification#28319
Merged
potiuk merged 1 commit intoapache:mainfrom Dec 12, 2022
Merged
Selective checks for API did not match API test specification#28319potiuk merged 1 commit intoapache:mainfrom
potiuk merged 1 commit intoapache:mainfrom
Conversation
The selective checks for the API had "^tests/api" as matching regexp, where we used "tests/api", "tests/api_connexion" to trigger the tests. This means that changes to "tests/api_internal" were not treated as "Other" as they were included in "^tests/api". This PR fixes it.
jedcunningham
approved these changes
Dec 12, 2022
potiuk
force-pushed
the
fix-api-selection-for-selective-tests
branch
from
197c1d3 to
2bce2d5
Compare
Contributor
|
Looks like the actual command is constructed in another place: https://github.com/apache/airflow/actions/runs/3680179020/jobs/6225576889#step:6:1901 Running tests tests/api tests/api_connexion
Starting the tests with those pytest arguments: --verbosity=0 --strict-markers --durations=100 --maxfail=50 --color=yes --junitxml=/files/test_result-API-sqlite.xml --timeouts-order moi --setup-timeout=60 --execution-timeout=60 --teardown-timeout=60 --output=/files/warnings-API-sqlite.txt --disable-warnings -rfEX --with-db-init tests/api tests/api_connexion |
Member
Author
Yep. This is exactly what we are synchronizing with. The test command specifically calls out the directories to use and the regexp in selective checks is used to determine if it should be run or not. So what the change brings is syncing these two:
The problem was that the regexp also matched When this PR is merged, if there is a change coming in |
Contributor
|
Yeah that might be some kind of misunderstanding with test categorisation. And one question about |
Member
Author
System tests are specifically excluded - this is entire different "category" of tests and they are not automatically run in CI. |
Contributor
|
If we exclude Running tests tests/triggers tests/timetables tests/testconfig tests/testconfig/conf tests/test_utils tests/test_utils/perf
tests/test_utils/perf/perf_kit tests/test_utils/perf/dags tests/test_utils/operators tests/task tests/task/task_runner
tests/system tests/system/utils tests/system/providers tests/system/providers/zendesk tests/system/providers/yandex
tests/system/providers/trino tests/system/providers/telegram tests/system/providers/tabular
tests/system/providers/tableau tests/system/providers/sqlite tests/system/providers/snowflake
tests/system/providers/slack tests/system/providers/singularity tests/system/providers/salesforce
tests/system/providers/qubole tests/system/providers/presto tests/system/providers/postgres
tests/system/providers/plexus tests/system/providers/papermill tests/system/providers/opsgenie
tests/system/providers/neo4j tests/system/providers/mysql tests/system/providers/microsoft
tests/system/providers/microsoft/winrm tests/system/providers/microsoft/mssql tests/system/providers/microsoft/azure
tests/system/providers/jenkins tests/system/providers/jdbc tests/system/providers/influxdb tests/system/providers/http
tests/system/providers/google tests/system/providers/google/workplace tests/system/providers/google/suite
tests/system/providers/google/suite/resources tests/system/providers/google/marketing_platform
tests/system/providers/google/leveldb tests/system/providers/google/firebase tests/system/providers/google/datacatalog
tests/system/providers/google/cloud tests/system/providers/google/cloud/workflows
tests/system/providers/google/cloud/vision tests/system/providers/google/cloud/video_intelligence
tests/system/providers/google/cloud/vertex_ai tests/system/providers/google/cloud/vertex_ai/resources
tests/system/providers/google/cloud/translate_speech tests/system/providers/google/cloud/translate
tests/system/providers/google/cloud/transfers tests/system/providers/google/cloud/transfers/resources
tests/system/providers/google/cloud/text_to_speech tests/system/providers/google/cloud/tasks
tests/system/providers/google/cloud/storage_transfer tests/system/providers/google/cloud/storage_transfer/resources
tests/system/providers/google/cloud/stackdriver tests/system/providers/google/cloud/sql_to_sheets
tests/system/providers/google/cloud/speech_to_text tests/system/providers/google/cloud/spanner
tests/system/providers/google/cloud/pubsub tests/system/providers/google/cloud/natural_language
tests/system/providers/google/cloud/ml_engine tests/system/providers/google/cloud/life_sciences
tests/system/providers/google/cloud/life_sciences/resources tests/system/providers/google/cloud/kubernetes_engine
tests/system/providers/google/cloud/gcs tests/system/providers/google/cloud/gcs/resources
tests/system/providers/google/cloud/datastore tests/system/providers/google/cloud/dataproc_metastore
tests/system/providers/google/cloud/dataproc tests/system/providers/google/cloud/dataproc/resources
tests/system/providers/google/cloud/dataprep tests/system/providers/google/cloud/dataplex
tests/system/providers/google/cloud/dataplex/resources tests/system/providers/google/cloud/dataform
tests/system/providers/google/cloud/dataflow tests/system/providers/google/cloud/dataflow/resources
tests/system/providers/google/cloud/data_loss_prevention
tests/system/providers/google/cloud/data_loss_prevention/resources tests/system/providers/google/cloud/compute
tests/system/providers/google/cloud/composer tests/system/providers/google/cloud/cloud_sql
tests/system/providers/google/cloud/cloud_memorystore tests/system/providers/google/cloud/cloud_functions
tests/system/providers/google/cloud/cloud_build tests/system/providers/google/cloud/cloud_build/resources
tests/system/providers/google/cloud/bigtable tests/system/providers/google/cloud/bigquery
tests/system/providers/google/cloud/bigquery/resources tests/system/providers/google/cloud/azure
tests/system/providers/google/cloud/automl tests/system/providers/google/ads tests/system/providers/github
tests/system/providers/ftp tests/system/providers/elasticsearch tests/system/providers/docker
tests/system/providers/dingding tests/system/providers/dbt tests/system/providers/dbt/cloud
tests/system/providers/databricks tests/system/providers/common tests/system/providers/common/sql
tests/system/providers/cncf tests/system/providers/cncf/kubernetes tests/system/providers/asana
tests/system/providers/apache tests/system/providers/apache/spark tests/system/providers/apache/pig
tests/system/providers/apache/livy tests/system/providers/apache/kylin tests/system/providers/apache/hive
tests/system/providers/apache/druid tests/system/providers/apache/drill tests/system/providers/apache/cassandra
tests/system/providers/apache/beam tests/system/providers/amazon tests/system/providers/amazon/aws
tests/system/providers/amazon/aws/utils tests/system/providers/alibaba tests/system/providers/airbyte tests/sensors
tests/security tests/secrets tests/plugins tests/operators tests/macros tests/listeners tests/lineage tests/kubernetes
tests/kubernetes/models tests/internal_api tests/hooks tests/decorators tests/datasets tests/dag_processing
tests/config_templates tests/cluster_policies tests/callbacks
Starting the tests with those pytest arguments: --verbosity=0 --strict-markers --durations=100 --maxfail=50 --color=yes --junitxml=/files/test_result-Other-sqlite.xml --timeouts-order moi --setup-timeout=60 --execution-timeout=60 --teardown-timeout=60 --output=/files/warnings-Other-sqlite.txt --disable-warnings -rfEX --with-db-init tests/triggers tests/timetables tests/testconfig tests/testconfig/conf tests/test_utils tests/test_utils/perf tests/test_utils/perf/perf_kit tests/test_utils/perf/dags tests/test_utils/operators tests/task tests/task/task_runner tests/system tests/system/utils tests/system/providers tests/system/providers/zendesk tests/system/providers/yandex tests/system/providers/trino tests/system/providers/telegram tests/system/providers/tabular tests/system/providers/tableau tests/system/providers/sqlite tests/system/providers/snowflake tests/system/providers/slack tests/system/providers/singularity tests/system/providers/salesforce tests/system/providers/qubole tests/system/providers/presto tests/system/providers/postgres tests/system/providers/plexus tests/system/providers/papermill tests/system/providers/opsgenie tests/system/providers/neo4j tests/system/providers/mysql tests/system/providers/microsoft tests/system/providers/microsoft/winrm tests/system/providers/microsoft/mssql tests/system/providers/microsoft/azure tests/system/providers/jenkins tests/system/providers/jdbc tests/system/providers/influxdb tests/system/providers/http tests/system/providers/google tests/system/providers/google/workplace tests/system/providers/google/suite tests/system/providers/google/suite/resources tests/system/providers/google/marketing_platform tests/system/providers/google/leveldb tests/system/providers/google/firebase tests/system/providers/google/datacatalog tests/system/providers/google/cloud tests/system/providers/google/cloud/workflows tests/system/providers/google/cloud/vision tests/system/providers/google/cloud/video_intelligence tests/system/providers/google/cloud/vertex_ai tests/system/providers/google/cloud/vertex_ai/resources tests/system/providers/google/cloud/translate_speech tests/system/providers/google/cloud/translate tests/system/providers/google/cloud/transfers tests/system/providers/google/cloud/transfers/resources tests/system/providers/google/cloud/text_to_speech tests/system/providers/google/cloud/tasks tests/system/providers/google/cloud/storage_transfer tests/system/providers/google/cloud/storage_transfer/resources tests/system/providers/google/cloud/stackdriver tests/system/providers/google/cloud/sql_to_sheets tests/system/providers/google/cloud/speech_to_text tests/system/providers/google/cloud/spanner tests/system/providers/google/cloud/pubsub tests/system/providers/google/cloud/natural_language tests/system/providers/google/cloud/ml_engine tests/system/providers/google/cloud/life_sciences tests/system/providers/google/cloud/life_sciences/resources tests/system/providers/google/cloud/kubernetes_engine tests/system/providers/google/cloud/gcs tests/system/providers/google/cloud/gcs/resources tests/system/providers/google/cloud/datastore tests/system/providers/google/cloud/dataproc_metastore tests/system/providers/google/cloud/dataproc tests/system/providers/google/cloud/dataproc/resources tests/system/providers/google/cloud/dataprep tests/system/providers/google/cloud/dataplex tests/system/providers/google/cloud/dataplex/resources tests/system/providers/google/cloud/dataform tests/system/providers/google/cloud/dataflow tests/system/providers/google/cloud/dataflow/resources tests/system/providers/google/cloud/data_loss_prevention tests/system/providers/google/cloud/data_loss_prevention/resources tests/system/providers/google/cloud/compute tests/system/providers/google/cloud/composer tests/system/providers/google/cloud/cloud_sql tests/system/providers/google/cloud/cloud_memorystore tests/system/providers/google/cloud/cloud_functions tests/system/providers/google/cloud/cloud_build tests/system/providers/google/cloud/cloud_build/resources tests/system/providers/google/cloud/bigtable tests/system/providers/google/cloud/bigquery tests/system/providers/google/cloud/bigquery/resources tests/system/providers/google/cloud/azure tests/system/providers/google/cloud/automl tests/system/providers/google/ads tests/system/providers/github tests/system/providers/ftp tests/system/providers/elasticsearch tests/system/providers/docker tests/system/providers/dingding tests/system/providers/dbt tests/system/providers/dbt/cloud tests/system/providers/databricks tests/system/providers/common tests/system/providers/common/sql tests/system/providers/cncf tests/system/providers/cncf/kubernetes tests/system/providers/asana tests/system/providers/apache tests/system/providers/apache/spark tests/system/providers/apache/pig tests/system/providers/apache/livy tests/system/providers/apache/kylin tests/system/providers/apache/hive tests/system/providers/apache/druid tests/system/providers/apache/drill tests/system/providers/apache/cassandra tests/system/providers/apache/beam tests/system/providers/amazon tests/system/providers/amazon/aws tests/system/providers/amazon/aws/utils tests/system/providers/alibaba tests/system/providers/airbyte tests/sensors tests/security tests/secrets tests/plugins tests/operators tests/macros tests/listeners tests/lineage tests/kubernetes tests/kubernetes/models tests/internal_api tests/hooks tests/decorators tests/datasets tests/dag_processing tests/config_templates tests/cluster_policies tests/callbacks |
Member
Author
|
We can exclude them from Other category when running tests actually. |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Dec 28, 2022
The "Other" test category automatically selects all tests that are not included in any of the regular categories. That is to make sure that we do not forget to add any directory that has been added. However this led to a long directory selection for "Other" category including system tests that have been automatically added there. However those tests are always skipped in regular tests and collecting those tests during "Other" execution is not needed and slows it down. Similarly System tests changes were treated as "Other change" for incoming PRs. This means that any change to system tests would trigger "all tests" rather than selective subset of those - the same as any core change. However System tests are also part of the documentation, so any change in system tests should trigger docs builds. This change improves it in a few ways: * Other tests now do not include "System Tests" - they are treated the same way as other test categories (but not included in test category selection for now - until we get a good way of breeze-integration for System Tests * They are also excluded from treating them as "other" change when considering which tests to run. Changes to system tests will not trigger "all" tests, just those that accompanying changes would trigger. * The changes to system tests only, however, triggers docs build because those are triggered by any source change.a * The __pycache__ directories are removed from the list of "Other" packages to run. * In order to make sure system tests are pytest-collectable, we perform pytest collection for all tests right after downloading the CI images and verifying them. This makes sure that the tests are collectible before we even attempt to run them - this way we avoid unnecessary machine spin-up and breze installation for the multiple jobs that run the tests. This slows down feedback time a litle, but should increase overall robustness of the test suite. * an old, unused nosetest collection script doing the same in the past has been removed (it was discovered during implementation) Noticed in: apache#28319
potiuk
added a commit
that referenced
this pull request
Dec 28, 2022
The "Other" test category automatically selects all tests that are not included in any of the regular categories. That is to make sure that we do not forget to add any directory that has been added. However this led to a long directory selection for "Other" category including system tests that have been automatically added there. However those tests are always skipped in regular tests and collecting those tests during "Other" execution is not needed and slows it down. Similarly System tests changes were treated as "Other change" for incoming PRs. This means that any change to system tests would trigger "all tests" rather than selective subset of those - the same as any core change. However System tests are also part of the documentation, so any change in system tests should trigger docs builds. This change improves it in a few ways: * Other tests now do not include "System Tests" - they are treated the same way as other test categories (but not included in test category selection for now - until we get a good way of breeze-integration for System Tests * They are also excluded from treating them as "other" change when considering which tests to run. Changes to system tests will not trigger "all" tests, just those that accompanying changes would trigger. * The changes to system tests only, however, triggers docs build because those are triggered by any source change.a * The __pycache__ directories are removed from the list of "Other" packages to run. * In order to make sure system tests are pytest-collectable, we perform pytest collection for all tests right after downloading the CI images and verifying them. This makes sure that the tests are collectible before we even attempt to run them - this way we avoid unnecessary machine spin-up and breze installation for the multiple jobs that run the tests. This slows down feedback time a litle, but should increase overall robustness of the test suite. * an old, unused nosetest collection script doing the same in the past has been removed (it was discovered during implementation) Noticed in: #28319
ephraimbuddy
pushed a commit
that referenced
this pull request
Jan 12, 2023
The selective checks for the API had "^tests/api" as matching regexp, where we used "tests/api", "tests/api_connexion" to trigger the tests. This means that changes to "tests/api_internal" were not treated as "Other" as they were included in "^tests/api". This PR fixes it. (cherry picked from commit 17131b3)
ephraimbuddy
added
the
changelog:skip
ephraimbuddy
pushed a commit
that referenced
this pull request
Mar 8, 2023
The "Other" test category automatically selects all tests that are not included in any of the regular categories. That is to make sure that we do not forget to add any directory that has been added. However this led to a long directory selection for "Other" category including system tests that have been automatically added there. However those tests are always skipped in regular tests and collecting those tests during "Other" execution is not needed and slows it down. Similarly System tests changes were treated as "Other change" for incoming PRs. This means that any change to system tests would trigger "all tests" rather than selective subset of those - the same as any core change. However System tests are also part of the documentation, so any change in system tests should trigger docs builds. This change improves it in a few ways: * Other tests now do not include "System Tests" - they are treated the same way as other test categories (but not included in test category selection for now - until we get a good way of breeze-integration for System Tests * They are also excluded from treating them as "other" change when considering which tests to run. Changes to system tests will not trigger "all" tests, just those that accompanying changes would trigger. * The changes to system tests only, however, triggers docs build because those are triggered by any source change.a * The __pycache__ directories are removed from the list of "Other" packages to run. * In order to make sure system tests are pytest-collectable, we perform pytest collection for all tests right after downloading the CI images and verifying them. This makes sure that the tests are collectible before we even attempt to run them - this way we avoid unnecessary machine spin-up and breze installation for the multiple jobs that run the tests. This slows down feedback time a litle, but should increase overall robustness of the test suite. * an old, unused nosetest collection script doing the same in the past has been removed (it was discovered during implementation) Noticed in: #28319 (cherry picked from commit e8657ce)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The selective checks for the API had "^tests/api" as matching regexp, where we used "tests/api_connexion" to trigger the tests. This means that changes to "tests/api_internal" were not treated as "Other", because they were included in the API regexp.
This PR fixes it.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.