More Pythonic way of package separation

[potiuk]

Currenlty - for convenience, we prepare packages using custom scripts and Breeze where we copy sources and prepare packages from the same sources.

We might want to make it in more pythonic way possibly where each package shares their own setup.py maybe, or at most symbolic link to the same setup.py.

We should make a POC to check if this is possible and what it gives us vs. the current solution.

[potiuk]

cc: @mik-laj - I believe you mentioned that this something possible and possibly you would like to take care of this when it gets to that. It might be implemented as late as before we release 2.0 so there is still time and I consider that as an optional task as the current solution is not the most beautiful, but it works.

[potiuk]

As part of the #11546 fixing I prepared a PR #11630 where I think I got it really sorted out very well:

1. we generate setup.py and config.py dynamically based on provider and information found in the Airflow's setup.py (dependencies etrc.). Those are automatically generated from JINJA templates for those files the same way we generate README.md and CHANGES files.

2. In the procedure of preparing such provider releases, those files are regenerated (using ./breeze prepare-provider-readme) and committed to the repository in the appropriate "airfllow/providers/" dir. We have (for now) two versions of those setup.py, setup.cfg, README, CHANGES files (one for backports, and one for regular providers). Whe the backports reach end-of-life we will be able to get rid of the BACKPORT)ORIVDERS_ ones.

3. when prepping the packages, I copy the sources to "provider_packages" - one, because it is needed to do refactor for backports and two - because the "setuptools" only works when setup.py, setup,cfg, README.md, CHANGELOG,txt are named as such. If we do it in main directory, we will be overwriting airflow ones. And we cannot do it easily in "providers" folder, because also setuptools require "airflow/providers" subdir for python files. I think this can stay like that permanently.

The nice thing about the final result (sdist packages) is that those .tar.gz packages when uncompressed, only contain sources of the provider they relate to + setup.py + setup.cfg + README.md + CHANGELOG.txt + LICENCE + NOTICE that makes sense for that provider, so we are now generating packages that "look like" and "behave like" packages prepared from separate repositories. Which is cool.

I think - if everyone else agrees with me - that this one can be closed once we merge #11630

WDYT?