Move providers docs to separate package + Spell-check in a common job with docs-build

[mik-laj]

I am extracting the provider's docs to a separate documentation package, so that we can update it independently of other documentation.
Unfortunately, it has now turned out that building documentation from multiple packages and checking spelling in a separate job does not work well with each other. The spell checker checks that all links are valid, but this is not possible if the documentation has not been built at least once.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

Do I get correctly thaat the spell check will happen inside the docs build utomatically? If so - we should also remove the flags in ci_docs.sh, I think and we should fix teh documentation in CONTRIBUTING and Breeze about it (just look for --docs-only or --spellcheck-only).

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[mik-laj]

As far as I know, this documentation building doesn't mean the spelling will be checked. These are two separate steps.

This change causes spelling and building to be done in one job on the CI, but if the user wants to iterate locally over the documentation in small steps, they can still do so. If they don't build all the documentation beforehand (a situation similar to spell check in a separate CI Job), the process may end with a message that not all was successful, but the spelling will be checked and the user can start working on other changes. If they want a full check, they can send the change to the CI.

All this will allow us to shorten the iteration time, because we will only perform selected steps and iterate over them.

$ time ./build_docs.py --spellcheck-only --package apache-airflow
real	1m10.113s
user	0m54.709s
sys	0m12.332s

$ time ./build_docs.py --docs-only --package apache-airflow
real	1m32.180s
user	1m24.968s
sys	0m5.662s

The documentation for building the documentation has been. moved to /docs/README.rst to make it easier to find. For now this file may not be very visible, but I will clean up this directory a bit soon and it will be the only .rst file in this directory. All other files will be in subdirectories following to the structure of the output directory.

[potiuk]

The documentation for building the documentation has been. moved to /docs/README.rst to make it easier to find. For now this file may not be very visible, but I will clean up this directory a bit soon and it will be the only .rst file in this directory. All other files will be in subdirectories following to the structure of the output directory.

So why do we neet do join it in one job then? What are the benefits?

If they are split into two separate jobs, they run in parallel and you can get the feedback much faster if you run a small doc-only PR and there are not many PRs in the queue.

In case we run those two jobs in parallel, rather than waiting 2:40 seconds user will have to wait 1:32. I think this is a good enough reason to keep those jobs as two separate jobs. Unless I am missing something.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[mik-laj]

@potiuk The problem is that the spell check will fail because sphinx will not have the inventory file it needs. This file is only created when building documentation in HTML format.

To understand the problem that I am trying to solve I need to tell you a little bit about how airflow_internsphinx works. It is an extension that works with sphinx.ext.intersphinx.. sphinx.ext.intersphinx allows us to create links between different documentation. For this purpose, the HTML documentation creates an objects.inv file in the output directory during build.
For example:

https://airflow.readthedocs.io/en/latest/objects.inv
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow-providers-apache-kylin/latest/objects.inv

This file lists all the objects that are in the document along with the relative path to this document.
For example:

:class:`airflow.providers.apache.kylin.hooks.kylin` => _api/airflow/providers/apache/kylin/hooks/kylin/index.html#airflow.providers.apache.kylin.hooks.kylin.KylinHook

If you want to see the full list of contents then you can run the script /docs/list-roles.py.

airflow_intersphiinx is an extension that sets up intersphinx for each documentation package we build. For each, two potential locations of the objects.inv file are checked:

• Local objects.inv - this file may not exist if documentation for the package has not been built.
• Remote cache on S3

If both sources are not found, we have a warning and the documentation building/spell checking fails. This can happen when we add a new package that has not yet been built, but another package that is being built earlier references it. And it surely will, as airflow_intersphinx creates a reference between all packages.

My proposed solution is to detect this situation and build problematic packages again. Then the local objects.inv files will be used and the whole operation will be successful.

This workaround cannot be used for spell checking. If we have a problem while checking the spelling, we have no other choice - we have to fail the build. Therefore, I would like to combine these steps into one job CI so that spell check has access to the local cache.

I realize that this is not a perfect solution, but I think it will not be too burdensome for us as we can build the documentation quickly locally. It was a big problem before, if building documentation always took 10+ minutes. Then running a task on CI allowed us to unblock our computer.

[potiuk]

I see - ok @mik-laj - many thanks for the detailed explanation. It's now perfectly clear. Great stuff!

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[mik-laj]

Now the Sphinx has started detecting some extra typos.

[mik-laj]

I wanted to avoid creating a new page with only one operator. Earlier I was hoping that Yandex would add other integrations as well and then a separate section would make sense, but they didn't.

[github-actions]

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!