[AIRFLOW-5862] Fix failing build on kubernetes by potiuk · Pull Request #6513 · apache/airflow

potiuk · 2019-11-06T21:14:15Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5862

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
- If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

potiuk · 2019-11-06T21:27:49Z

It worked in the PR but did not work after merging to master because of the optimisations of Travis build scripts - > common dir was missing in the new Docker and it was needed in the new Kubernetes scripts but the Kubernetes scripts have not changed so we have a weakness in the optimisation scripts.

potiuk · 2019-11-07T04:35:43Z

I made the tests a bit more "smart" - they will also run kubernetes/unit tests also whenever any of the scripts/ci files change. That should prevent this happening in the future.

Please take a look asap - for now we have master failing.

feluelle

LGTM

feluelle · 2019-11-07T10:22:08Z

But shellcheck hook is complaining:

Check Shell scripts syntax correctness...................................Failed

hookid: shellcheck

Unable to find image 'koalaman/shellcheck:stable' locally

stable: Pulling from koalaman/shellcheck

Status: Downloaded newer image for koalaman/shellcheck:stable

Unable to find image 'koalaman/shellcheck:stable' locally

stable: Pulling from koalaman/shellcheck

Status: Downloaded newer image for koalaman/shellcheck:stable

In scripts/ci/ci_run_airflow_testing.sh line 73:

if [[ ${START_KUBERNETES_CLUSTER} == "true" ]]; then

                                                ^-- SC1009: The mentioned syntax error was in this then clause.

In scripts/ci/ci_run_airflow_testing.sh line 80:

    # shellcheck: disable=SC2016

    ^-- SC1073: Couldn't parse this shellcheck directive. Fix to allow more checks.

                ^-- SC1072:  Fix any mentioned problems and try again.

For more information:

  https://www.shellcheck.net/wiki/SC1072 --  Fix any mentioned problems and t...

  https://www.shellcheck.net/wiki/SC1073 -- Couldn't parse this shellcheck di...

  https://www.shellcheck.net/wiki/SC1009 -- The mentioned syntax error was in...

feluelle · 2019-11-07T12:00:14Z

@potiuk

======================================================================

1) FAIL: test_integration_run_dag (tests.integration.kubernetes.test_kubernetes_executor.TestKubernetesExecutor)

----------------------------------------------------------------------

   Traceback (most recent call last):

    tests/integration/kubernetes/test_kubernetes_executor.py line 198 in test_integration_run_dag

      expected_final_state='success', timeout=100)

    tests/integration/kubernetes/test_kubernetes_executor.py line 105 in monitor_task

      self.assertEqual(state, expected_final_state)

   AssertionError: 'queued' != 'success'

   - queued

   + success

   

   -------------------- >> begin captured stdout << ---------------------

   Found the job with execution date 2019-11-07T11:47:40+00:00

   Attempt 0: Current state of operator is queued

   Attempt 1: Current state of operator is queued

   Attempt 2: Current state of operator is queued

   Attempt 3: Current state of operator is queued

   Attempt 4: Current state of operator is queued

   Attempt 5: Current state of operator is queued

   Attempt 6: Current state of operator is queued

   Attempt 7: Current state of operator is queued

   Attempt 8: Current state of operator is queued

   Attempt 9: Current state of operator is queued

   Attempt 10: Current state of operator is queued

   Attempt 11: Current state of operator is queued

   Attempt 12: Current state of operator is queued

   Attempt 13: Current state of operator is queued

   Attempt 14: Current state of operator is queued

   Attempt 15: Current state of operator is queued

   Attempt 16: Current state of operator is queued

   Attempt 17: Current state of operator is queued

   Attempt 18: Current state of operator is queued

   Attempt 19: Current state of operator is queued

   

   --------------------- >> end captured stdout << ----------------------

   -------------------- >> begin captured logging << --------------------

   airflow.utils.log.logging_mixin.LoggingMixin: INFO: Reading the config from /root/airflow/airflow.cfg

   airflow.settings: INFO: Configured default timezone <Timezone [UTC]>

   --------------------- >> end captured logging << ---------------------

======================================================================

2) FAIL: test_integration_run_dag_with_scheduler_failure (tests.integration.kubernetes.test_kubernetes_executor.TestKubernetesExecutor)

----------------------------------------------------------------------

   Traceback (most recent call last):

    tests/integration/kubernetes/test_kubernetes_executor.py line 225 in test_integration_run_dag_with_scheduler_failure

      expected_final_state='success', timeout=120)

    tests/integration/kubernetes/test_kubernetes_executor.py line 105 in monitor_task

      self.assertEqual(state, expected_final_state)

   AssertionError: 'queued' != 'success'

   - queued

   + success

   

   -------------------- >> begin captured stdout << ---------------------

   Found the job with execution date 2019-11-07T11:49:23+00:00

   Attempt 0: Current state of operator is queued

   Attempt 1: Current state of operator is queued

   Attempt 2: Current state of operator is queued

   Attempt 3: Current state of operator is queued

   Attempt 4: Current state of operator is queued

   Attempt 5: Current state of operator is queued

   Attempt 6: Current state of operator is queued

   Attempt 7: Current state of operator is queued

   Attempt 8: Current state of operator is queued

   Attempt 9: Current state of operator is queued

   Attempt 10: Current state of operator is queued

   Attempt 11: Current state of operator is queued

   Attempt 12: Current state of operator is queued

   Attempt 13: Current state of operator is queued

   Attempt 14: Current state of operator is queued

   Attempt 15: Current state of operator is queued

   Attempt 16: Current state of operator is queued

   Attempt 17: Current state of operator is queued

   Attempt 18: Current state of operator is queued

   Attempt 19: Current state of operator is queued

   Attempt 20: Current state of operator is queued

   Attempt 21: Current state of operator is queued

   Attempt 22: Current state of operator is queued

   Attempt 23: Current state of operator is queued

   

   --------------------- >> end captured stdout << ----------------------

potiuk · 2019-11-07T12:49:42Z

I reverted the original #6496 as i need to fix those problems first. So I close this one and will merge it into final version of kind improvement.

[AIRFLOW-5862] Fix failing build on kubernetes

2764d72

potiuk requested review from ashb, dimberman and mik-laj November 6, 2019 21:14

potiuk added 2 commits November 7, 2019 04:59

Force running tests when only CI scripts change

decde04

Fixup! Correct path

2d435ad

potiuk added 4 commits November 7, 2019 06:01

Added parameters passing

5eb51fa

Add extra $0 parameter for bash -c

b972d6c

And fixed passing parameters.

1cfcf6b

Disable shellcheck problem

4790440

feluelle approved these changes Nov 7, 2019

View reviewed changes

Fix the annoying :

dbabed9

potiuk closed this Nov 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-5862] Fix failing build on kubernetes#6513

[AIRFLOW-5862] Fix failing build on kubernetes#6513
potiuk wants to merge 8 commits into
apache:masterfrom
PolideaInternal:fix_failing_build_kubernetes

potiuk commented Nov 6, 2019

Uh oh!

potiuk commented Nov 6, 2019 •

edited

Loading

Uh oh!

potiuk commented Nov 7, 2019

Uh oh!

feluelle left a comment

Uh oh!

feluelle commented Nov 7, 2019

Uh oh!

feluelle commented Nov 7, 2019

Uh oh!

potiuk commented Nov 7, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

potiuk commented Nov 6, 2019

Jira

Description

Tests

Commits

Documentation

Uh oh!

potiuk commented Nov 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Nov 7, 2019

Uh oh!

feluelle left a comment

Choose a reason for hiding this comment

Uh oh!

feluelle commented Nov 7, 2019

Uh oh!

feluelle commented Nov 7, 2019

Uh oh!

potiuk commented Nov 7, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

potiuk commented Nov 6, 2019 •

edited

Loading