Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(smart_sensor): runtime and unbound var errors #14774

Merged
merged 1 commit into from Jun 23, 2021

Conversation

shivanshs9
Copy link
Contributor

Closes #14770


Fixes various runtime errors in smart sensor:

  1. UnboundLocalError
UnboundLocalError: local variable 'query_result' referenced before assignment
  File "airflow/sensors/smart_sensor.py", line 639, in _execute_sensor_work
    self._mark_multi_state(
  File "airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "airflow/sensors/smart_sensor.py", line 470, in _mark_multi_state
    self.log.info("Marked %s tasks out of %s to state %s", count_marked, len(query_result), state)
  1. RuntimeError
  File "airflow/sentry.py", line 159, in wrapper
    return func(task_instance, *args, session=session, **kwargs)
  File "airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "airflow/sensors/smart_sensor.py", line 736, in execute
    self.flush_cached_sensor_poke_results()
  File "airflow/sensors/smart_sensor.py", line 681, in flush_cached_sensor_poke_results
    for ti_key, sensor_exception in self.cached_sensor_exceptions.items():
  1. InvalidRequestError (should be related to 1)
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Timeout, PID: 24375 (Background on this error at: http://sqlalche.me/e/13/7s2a)
  File "airflow/sensors/smart_sensor.py", line 639, in _execute_sensor_work
    self._mark_multi_state(
  File "airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "contextlib.py", line 120, in __exit__
    next(self.gen)
  File "airflow/utils/session.py", line 32, in create_session
    session.commit()
  File "sqlalchemy/orm/session.py", line 1046, in commit
    self.transaction.commit()
  File "sqlalchemy/orm/session.py", line 502, in commit
    self._assert_active(prepared_ok=True)
  File "sqlalchemy/orm/session.py", line 289, in _assert_active
    raise sa_exc.InvalidRequestError(

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@kaxil kaxil added this to the Airflow 2.0.2 milestone Mar 15, 2021
Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashb ashb modified the milestones: Airflow 2.0.2, Airflow 2.0.3 Apr 22, 2021
@ashb ashb modified the milestones: Airflow 2.0.3, Airflow 2.1.1 May 7, 2021
@kaxil kaxil modified the milestones: Airflow 2.1.1, Airflow 2.1.2 Jun 22, 2021
@kaxil
Copy link
Member

kaxil commented Jun 22, 2021

@shivanshs9 Can you rebase your PR on Main please and fix the conflicts

Signed-off-by: Shivansh Saini <shivanshs9@gmail.com>
@shivanshs9
Copy link
Contributor Author

@kaxil I rebased the PR and apparently the RuntimeError is fixed in another PR - #14920, explained in this comment.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jun 23, 2021
@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@kaxil kaxil merged commit 4aec25a into apache:main Jun 23, 2021
Jorricks pushed a commit to Jorricks/airflow that referenced this pull request Jun 24, 2021
Signed-off-by: Shivansh Saini <shivanshs9@gmail.com>

Closes apache#14770
@ashb ashb modified the milestones: Airflow 2.1.2, Airflow 2.1.3 Jul 7, 2021
jhtimmins pushed a commit that referenced this pull request Aug 5, 2021
Signed-off-by: Shivansh Saini <shivanshs9@gmail.com>

Closes #14770

(cherry picked from commit 4aec25a)
jhtimmins pushed a commit that referenced this pull request Aug 13, 2021
Signed-off-by: Shivansh Saini <shivanshs9@gmail.com>

Closes #14770

(cherry picked from commit 4aec25a)
kaxil pushed a commit that referenced this pull request Aug 17, 2021
Signed-off-by: Shivansh Saini <shivanshs9@gmail.com>

Closes #14770

(cherry picked from commit 4aec25a)
jhtimmins pushed a commit that referenced this pull request Aug 17, 2021
Signed-off-by: Shivansh Saini <shivanshs9@gmail.com>

Closes #14770

(cherry picked from commit 4aec25a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Smart sensor] Runtime error: dictionary changed size during iteration
3 participants