Skip to content

Commit

Permalink
Move providers docs to separate package + Spell-check in a common job…
Browse files Browse the repository at this point in the history
… with docs-build (#12527)
  • Loading branch information
mik-laj committed Nov 22, 2020
1 parent f2569de commit ef4af21
Show file tree
Hide file tree
Showing 28 changed files with 479 additions and 331 deletions.
23 changes: 2 additions & 21 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,7 @@ jobs:
VERBOSE: false

docs:
timeout-minutes: 30
timeout-minutes: 45
name: "Build docs"
runs-on: ubuntu-20.04
needs: [build-info, ci-images]
Expand All @@ -311,7 +311,7 @@ jobs:
- name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Build docs"
run: ./scripts/ci/docs/ci_docs.sh --docs-only
run: ./scripts/ci/docs/ci_docs.sh
- name: "Upload documentation"
uses: actions/upload-artifact@v2
if: always() && github.event_name == 'pull_request'
Expand All @@ -333,22 +333,6 @@ jobs:
github.event_name == 'push'
run: aws s3 sync ./files/documentation s3://apache-airflow-docs

docs-spell-check:
timeout-minutes: 30
name: "Spell check docs"
runs-on: ubuntu-20.04
needs: [build-info, ci-images]
env:
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
if: needs.build-info.outputs.docs-build == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Spell check docs"
run: ./scripts/ci/docs/ci_docs.sh --spellcheck-only

prepare-backport-provider-packages:
timeout-minutes: 30
name: "Backport packages"
Expand Down Expand Up @@ -839,7 +823,6 @@ jobs:
- prepare-provider-packages
- prod-images
- docs
- docs-spell-check
if: >
(github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test') &&
github.event_name != 'schedule'
Expand Down Expand Up @@ -878,7 +861,6 @@ jobs:
- prepare-backport-provider-packages
- ci-images
- docs
- docs-spell-check
if: >
(github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ) &&
github.event_name != 'schedule'
Expand Down Expand Up @@ -979,7 +961,6 @@ jobs:
runs-on: ubuntu-20.04
needs:
- docs
- docs-spell-check
- static-checks
- static-checks-pylint
- tests-sqlite
Expand Down
22 changes: 11 additions & 11 deletions CHANGELOG.txt
Original file line number Diff line number Diff line change
Expand Up @@ -946,7 +946,7 @@ Doc-only changes
- [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319)
- [AIRFLOW-XXX] Fix Documentation for adding extra Operator Links (#6301)
- [AIRFLOW-XXX] Add section on task lifecycle & correct casing in docs (#4681)
- [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md (#6240)
- [AIRFLOW-XXX] Make it clear that 1.10.5 was not accidentally omitted from UPDATING.md (#6240)
- [AIRFLOW-XXX] Improve format in code-block directives (#6242)
- [AIRFLOW-XXX] Format Sendgrid docs (#6245)
- [AIRFLOW-XXX] Update to new logo (#6066)
Expand Down Expand Up @@ -1090,7 +1090,7 @@ New Features
- [AIRFLOW-4521] Pause dag also pause its subdags (#5283)
- [AIRFLOW-4738] Enforce exampleinclude for example DAGs (#5375)
- [AIRFLOW-4326] Airflow AWS SQS Operator (#5110)
- [AIRFLOW-3729] Support "DownwardAPI" in env variables for KubernetesPodOperator (#4554)
- [AIRFLOW-3729] Support `DownwardAPI` in env variables for KubernetesPodOperator (#4554)
- [AIRFLOW-4585] Implement Kubernetes Pod Mutation Hook (#5359)
- [AIRFLOW-161] New redirect route and extra links (#5059)
- [AIRFLOW-4420] Backfill respects task_concurrency (#5221)
Expand Down Expand Up @@ -1272,7 +1272,7 @@ Bug fixes
- [AIRFLOW-4455] dag_details broken for subdags in RBAC UI (#5234)
- [AIRFLOW-2955] Fix kubernetes pod operator to set requests and limits on task pods (#4551)
- [AIRFLOW-4459] Fix wrong DAG count in /home page when DAG count is zero (#5235)
- [AIRFLOW-3876] AttributeError: module 'distutils' has no attribute 'util'
- [AIRFLOW-3876] AttributeError: module `distutils` has no attribute 'util'
- [AIRFLOW-4146] Fix CgroupTaskRunner errors (#5224)
- [AIRFLOW-4524] Fix bug with "Ignore \*" toggles in RBAC mode (#5378)
- [AIRFLOW-4765] Fix DataProcPigOperator execute method (#5426)
Expand Down Expand Up @@ -1544,7 +1544,7 @@ Improvement
Bug fixes
"""""""""

- [AIRFLOW-4248] Fix 'FileExistsError' makedirs race in file_processor_handler (#5047)
- [AIRFLOW-4248] Fix `FileExistsError` makedirs race in file_processor_handler (#5047)
- [AIRFLOW-4240] State-changing actions should be POST requests (#5039)
- [AIRFLOW-4246] Flask-Oauthlib needs downstream dependencies pinning due to breaking changes (#5045)
- [AIRFLOW-3887] Downgrade dagre-d3 to 0.4.18 (#4713)
Expand Down Expand Up @@ -2782,7 +2782,7 @@ Airflow 1.10.0, 2018-08-03
- [AIRFLOW-1799] Fix logging line which raises errors
- [AIRFLOW-1102] Upgrade Gunicorn >=19.4.0
- [AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook
- [AIRFLOW-1797] S3Hook.load_string didn't work on Python3
- [AIRFLOW-1797] S3Hook.load_string did not work on Python3
- [AIRFLOW-646] Add docutils to setup_requires
- [AIRFLOW-1792] Missing intervals DruidOperator
- [AIRFLOW-1789][AIRFLOW-1712] Log SSHOperator stderr to log.warning
Expand Down Expand Up @@ -2927,7 +2927,7 @@ Airflow 1.9.0, 2018-01-02
- [AIRFLOW-1799] Fix logging line which raises errors
- [AIRFLOW-1102] Upgrade Gunicorn >=19.4.0
- [AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook
- [AIRFLOW-1797] S3Hook.load_string didn't work on Python3
- [AIRFLOW-1797] S3Hook.load_string did not work on Python3
- [AIRFLOW-1792] Missing intervals DruidOperator
- [AIRFLOW-1789][AIRFLOW-1712] Log SSHOperator stderr to log.warning
- [AIRFLOW-1669] Fix Docker and pin Moto to 1.1.19
Expand Down Expand Up @@ -3262,7 +3262,7 @@ Airflow 1.9.0, 2018-01-02
- [AIRFLOW-1045] Make log level configurable via airflow.cfg
- [AIRFLOW-1047] Sanitize strings passed to Markup
- [AIRFLOW-1040] Fix some small typos in comments and docstrings
- [AIRFLOW-1017] get_task_instance shouldn't throw exception when no TI
- [AIRFLOW-1017] get_task_instance should not throw exception when no TI
- [AIRFLOW-1006] Add config_templates to MANIFEST
- [AIRFLOW-999] Add support for Redis database
- [AIRFLOW-1009] Remove SQLOperator from Concepts page
Expand Down Expand Up @@ -3307,7 +3307,7 @@ Airflow 1.9.0, 2018-01-02
- [AIRFLOW-937] Improve performance of task_stats
- [AIRFLOW-933] use ast.literal_eval rather eval because ast.literal_eval does not execute input.
- [AIRFLOW-925] Revert airflow.hooks change that cherry-pick picked
- [AIRFLOW-919] Running tasks with no start date shouldn't break a DAGs UI
- [AIRFLOW-919] Running tasks with no start date should not break a DAGs UI
- [AIRFLOW-802][AIRFLOW-1] Add spark-submit operator/hook
- [AIRFLOW-725] Use keyring to store credentials for JIRA
- [AIRFLOW-916] Remove deprecated readfp function
Expand Down Expand Up @@ -3415,7 +3415,7 @@ Airflow 1.8.1, 2017-05-09
- [AIRFLOW-1017] get_task_instance should return None instead of throw an exception for non-existent TIs
- [AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs
- [AIRFLOW-1004] `airflow webserver -D` runs in foreground
- [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'" on example_subdag_operator
- [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime' and `NoneType`" on example_subdag_operator
- [AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow
- [AIRFLOW-989] Clear Task Regression
- [AIRFLOW-974] airflow.util.file mkdir has a race condition
Expand Down Expand Up @@ -3456,7 +3456,7 @@ Airflow 1.8.0, 2017-03-12
- [AIRFLOW-937] Improve performance of task_stats
- [AIRFLOW-933] use ast.literal_eval rather eval because ast.literal_eval does not execute input.
- [AIRFLOW-925] Revert airflow.hooks change that cherry-pick picked
- [AIRFLOW-919] Running tasks with no start date shouldn't break a DAGs UI
- [AIRFLOW-919] Running tasks with no start date should not break a DAGs UI
- [AIRFLOW-802] Add spark-submit operator/hook
- [AIRFLOW-897] Prevent dagruns from failing with unfinished tasks
- [AIRFLOW-861] make pickle_info endpoint be login_required
Expand Down Expand Up @@ -3765,7 +3765,7 @@ Airflow 1.7.2
- [AIRFLOW-262] Simplify commands in MANIFEST.in
- [AIRFLOW-31] Add zope dependency
- [AIRFLOW-6] Remove dependency on Highcharts
- [AIRFLOW-234] make task that aren't `running` self-terminate
- [AIRFLOW-234] make task that are not `running` self-terminate
- [AIRFLOW-256] Fix test_scheduler_reschedule heartrate
- Add Python 3 compatibility fix
- [AIRFLOW-31] Use standard imports for hooks/operators
Expand Down
2 changes: 1 addition & 1 deletion airflow/providers/snowflake/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ integrations:
external-doc-url: https://snowflake.com/
how-to-guide:
- /docs/howto/operator/snowflake.rst
tags: [software]
tags: [service]

operators:
- integration-name: Snowflake
Expand Down
6 changes: 3 additions & 3 deletions airflow/providers/yandex/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,13 @@ versions:
integrations:
- integration-name: Yandex.Cloud
external-doc-url: https://cloud.yandex.com/
tags: [yandex]
tags: [service]

- integration-name: Yandex.Cloud Dataproc
external-doc-url: https://cloud.yandex.com/dataproc
how-to-guide:
- /docs/howto/operator/yandexcloud.rst
tags: [yandex]
tags: [service]

operators:
- integration-name: Yandex.Cloud Dataproc
Expand All @@ -44,6 +44,6 @@ hooks:
- integration-name: Yandex.Cloud
python-modules:
- airflow.providers.yandex.hooks.yandex
- integration-name: Yandex.Cloud
- integration-name: Yandex.Cloud Dataproc
python-modules:
- airflow.providers.yandex.hooks.yandexcloud_dataproc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
Expand Down Expand Up @@ -118,3 +119,13 @@ A. It depends on the scope of customization. There is no need to upgrade the pro
Generally speaking, with Airflow 2 we are following the `Semver <https://semver.org/>`_ approach where
we will introduce backwards-incompatible changes in Major releases, so all your modifications (as long
as you have not used internal Airflow classes) should work for All Airflow 2.* versions.


Content
-------

.. toctree::
:maxdepth: 1

Packages <packages-ref>
Operators and hooks <operators-and-hooks-ref/index>
42 changes: 42 additions & 0 deletions docs/apache-airflow-providers/operators-and-hooks-ref/apache.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
ASF: Apache Software Foundation
===============================

Airflow supports various software created by `Apache Software Foundation <https://www.apache.org/foundation/>`__.

Software operators and hooks
----------------------------

These integrations allow you to perform various operations within software developed by Apache Software
Foundation.

.. operators-hooks-ref::
:tags: apache
:header-separator: "


Transfer operators and hooks
----------------------------

These integrations allow you to copy data from/to software developed by Apache Software
Foundation.

.. transfers-ref::
:tags: apache
:header-separator: "
43 changes: 43 additions & 0 deletions docs/apache-airflow-providers/operators-and-hooks-ref/aws.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. _AWS:

AWS: Amazon Web Services
------------------------

Airflow has support for `Amazon Web Services <https://aws.amazon.com/>`__.

All hooks are based on :mod:`airflow.providers.amazon.aws.hooks.base_aws`.

Service operators and hooks
'''''''''''''''''''''''''''

These integrations allow you to perform various operations within the Amazon Web Services.

.. operators-hooks-ref::
:tags: aws
:header-separator: "

Transfer operators and hooks
''''''''''''''''''''''''''''

These integrations allow you to copy data from/to Amazon Web Services.

.. transfers-ref::
:tags: aws
:header-separator: "
42 changes: 42 additions & 0 deletions docs/apache-airflow-providers/operators-and-hooks-ref/azure.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Azure: Microsoft Azure
----------------------

Airflow has limited support for `Microsoft Azure <https://azure.microsoft.com/>`__.

Some hooks are based on :mod:`airflow.providers.microsoft.azure.hooks.base_azure`
which authenticate Azure's Python SDK Clients.

Service operators and hooks
'''''''''''''''''''''''''''

These integrations allow you to perform various operations within the Microsoft Azure.

.. operators-hooks-ref::
:tags: azure
:header-separator: "

Transfer operators and hooks
''''''''''''''''''''''''''''

These integrations allow you to copy data from/to Microsoft Azure.

.. transfers-ref::
:tags: azure
:header-separator: "
Loading

0 comments on commit ef4af21

Please sign in to comment.