Skip to content

Comments

AWS system tests: increase Redshift timeouts#31484

Merged
o-nikolas merged 4 commits intoapache:mainfrom
aws-mwaa:vincbeck/tests/example_redshift_s3_transfers
May 25, 2023
Merged

AWS system tests: increase Redshift timeouts#31484
o-nikolas merged 4 commits intoapache:mainfrom
aws-mwaa:vincbeck/tests/example_redshift_s3_transfers

Conversation

@vincbeck
Copy link
Contributor

@vincbeck vincbeck commented May 23, 2023

Similar to #31418. One of our system test executions failed because it timed out while waiting for the cluster to be created. Increasing the timeout should resolve the issue.

Edit: It happens to another of our system test that uses RedshiftClusterSensor. Somehow, Redshift takes more time now to spin clusters up. Thus, I updated all our system tests using RedshiftClusterSensor to set/increase the timeout.

INFO     airflow.task.operators:redshift_cluster.py:64 Poked cluster envfd53fef2-redshift-cluster for status 'available', found status 'creating'
--
ERROR    airflow.task:taskinstance.py:1903 Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/airflow/sensors/base.py", line 242, in execute
raise AirflowSensorTimeout(message)
airflow.exceptions.AirflowSensorTimeout: Sensor has timed out; run duration of 901.579888 seconds exceeds the specified timeout of 900.



^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@vincbeck vincbeck changed the title AWS system test example_redshift_s3_transfers: increase Redshift timeout AWS system tests: increase Redshift timeouts May 25, 2023
@vincbeck
Copy link
Contributor Author

It happened this morning to another of our system test that uses RedshiftClusterSensor. Somehow, Redshift takes more time now to spin clusters up. Thus, I updated all our system tests using RedshiftClusterSensor to set/increase the timeout.

@o-nikolas
Copy link
Contributor

It happened this morning to another of our system test that uses RedshiftClusterSensor. Somehow, Redshift takes more time now to spin clusters up. Thus, I updated all our system tests using RedshiftClusterSensor to set/increase the timeout.

Outside the scope of this PR: We should change the default value for that sensor, if the default is doomed to fail with a timeout then it's not a good user experience.

@o-nikolas o-nikolas merged commit f00c131 into apache:main May 25, 2023
@vincbeck vincbeck deleted the vincbeck/tests/example_redshift_s3_transfers branch May 25, 2023 16:53
@vincbeck
Copy link
Contributor Author

It happened this morning to another of our system test that uses RedshiftClusterSensor. Somehow, Redshift takes more time now to spin clusters up. Thus, I updated all our system tests using RedshiftClusterSensor to set/increase the timeout.

Outside the scope of this PR: We should change the default value for that sensor, if the default is doomed to fail with a timeout then it's not a good user experience.

The default value is actually the default value set by Airflow (1 week), see here. So no need to update the sensor, we were just setting too low timeouts in our system tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants