Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8057] [AIP-31] Add @task decorator #5

Closed
wants to merge 28 commits into from
Closed

Commits on May 21, 2020

  1. Configuration menu
    Copy the full SHA
    c6224e2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    12c22e0 View commit details
    Browse the repository at this point in the history
  3. Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work (apache#8938)

    Debian Buster only ships with a JDK11, and Hive/Hadoop fails in odd,
    hard to debug ways (complains about metastore not being initalized,
    possibly related to the class loader issues.)
    
    Until we rip Hive out from the CI (replacing it with Hadoop in a seprate
    integration, only on for some builds) we'll have to stick with JRE8
    
    Our previous approach of installing openjdk-8 from Sid/Unstable started
    failing as Debian Sid has a new (and conflicting) version of GCC/libc.
    The adoptopenjdk package archive is designed for Buster so should be
    more resilient
    ashb committed May 21, 2020
    Configuration menu
    Copy the full SHA
    8476c1e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f17b4bb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a9dfd7d View commit details
    Browse the repository at this point in the history
  6. Add TaskInstance state to TI Tooltip to be colour-blind friendlier (a…

    …pache#8910)
    
    Currently there is no way to determine the state of a TaskInstance in the graph view or tree view for people with colour blindness
    
    Approximately 4.5% of people experience some form of colour vision deficiency
    harrisjoseph committed May 21, 2020
    Configuration menu
    Copy the full SHA
    f3f74c7 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    8d3acd7 View commit details
    Browse the repository at this point in the history
  8. Remove singularity from CI images (apache#8945)

    The singularity operator tests _have always_ used mocking, so we were
    adding 700MB to our docker image for nothing.
    
    Fixes apache#8774
    ashb committed May 21, 2020
    Configuration menu
    Copy the full SHA
    47413d9 View commit details
    Browse the repository at this point in the history
  9. Update example webserver_config.py to show correct CSRF config (apach…

    …e#8944)
    
    CSRF_ENABLED does nothing.
    
    Thankfully, due to sensible defaults in flask-wtf, CSRF is on by
    default, but we should set this correctly.
    
    Fixes apache#8915
    ashb committed May 21, 2020
    Configuration menu
    Copy the full SHA
    16206cd View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    97b6cc7 View commit details
    Browse the repository at this point in the history
  11. Python base images are stored in cache (apache#8943)

    All PRs will used cached "latest good" version of the python
    base images from our GitHub registry. The python versions in
    the Github Registry will only get updated after a master
    build (which pulls latest Python image from DockerHub) builds
    and passes test correctly.
    
    This is to avoid problems that we had recently with Python
    patchlevel releases breaking our Docker builds.
    potiuk committed May 21, 2020
    Configuration menu
    Copy the full SHA
    41481bb View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    b26b3ca View commit details
    Browse the repository at this point in the history
  13. Make scheduler_dag_execution_timing grok dynamic start date of elasti…

    …c dag (apache#8952)
    
    The scheduler_dag_execution_timing script wants to run _n_ dag runs to
    completion. However since the start date of those dags is Dynamic (`now
    - delta`) we can't pre-compute the execution_dates like we were before.
    (This is because the execution_date of the very first dag run would be
    `now()` of the parser process, but if we try to pre-compute that in
    the benchmark process it would see a different value of now().)
    
    This PR changes it to instead watch for the first _n_ dag runs to be
    completed. This should make it work with more dags with less changes to
    them.
    ashb committed May 21, 2020
    Configuration menu
    Copy the full SHA
    113982b View commit details
    Browse the repository at this point in the history
  14. Cache 1 10 ci images (apache#8955)

    * Push CI images to Docker packcage cache for v1-10 branches
    
    This is done as a commit to master so that we can keep the two branches
    in sync
    
    Co-Authored-By: Ash Berlin-Taylor <ash_github@firemirror.com>
    
    * Run Github Actions against v1-10-stable too
    
    Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
    dimberman and ashb committed May 21, 2020
    Configuration menu
    Copy the full SHA
    90a07d8 View commit details
    Browse the repository at this point in the history
  15. Pin Version of Azure Cosmos to <4 (apache#8956)

    Old Repo: https://github.com/Azure/azure-cosmos-python
    New Repo: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/cosmos/azure-cosmos
    
    azure-cosmos==4.0.0 was released on 20 May 2020 that breaks Airflow
    kaxil committed May 21, 2020
    Configuration menu
    Copy the full SHA
    dd72040 View commit details
    Browse the repository at this point in the history
  16. Pin google-cloud-datacatalog to <0.8 (apache#8957)

    `field_path` was renamed to `tag_template_field_path` in >=0.8 and there might be other unknown errors
    kaxil committed May 21, 2020
    Configuration menu
    Copy the full SHA
    94a7673 View commit details
    Browse the repository at this point in the history
  17. add PythonFunctionalOperator

    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    2bdb60f View commit details
    Browse the repository at this point in the history
  18. add task decorator

    add return XComArg
    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    c0bbde8 View commit details
    Browse the repository at this point in the history
  19. handle kwargs, change super to baseoperator, assign dag if no dag on …

    …__call__
    
    handle empty dict/list in sequencial calls
    
    revert unnecessary dependency addition
    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    7b4dbf6 View commit details
    Browse the repository at this point in the history
  20. capture no xcom pushed settings

    s/Exception/AirflowException
    
    make pythonfunctionaloperator a private class
    
    missing documentation
    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    3758dbd View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    0e3fb99 View commit details
    Browse the repository at this point in the history
  22. add tests

    remove docs
    
    remove linter errors
    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    0f06caf View commit details
    Browse the repository at this point in the history
  23. add task to airflow and DAG

    migrate to pytest
    
    use DagRunType and fix small issue w XComArg
    
    some more pylint fixes
    
    pylint and isort issues
    
    add task into STATICA_HACK
    
    remove cyclic dependency
    
    disable pylint nomember for .key
    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    ba2baeb View commit details
    Browse the repository at this point in the history
  24. add documentation

    remove unneded line
    
    address mypy issues
    
    fix more pylint mypy issues
    
    fix docs
    
    add _called variable
    
    change task decorator name to operator
    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    d2ff60b View commit details
    Browse the repository at this point in the history
  25. add decorators module

    casassg committed May 21, 2020
    Configuration menu
    Copy the full SHA
    d6564a8 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2020

  1. Configuration menu
    Copy the full SHA
    846f80a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4f157fa View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8448bff View commit details
    Browse the repository at this point in the history