Add Multi-Tenant architecture diagram

[potiuk]

This image is not yet connected anywhere, I just want to keep it in Airflow while we are discussing "AIP-67 - Multi-tenant Deployment of Airflow components" and while we are iterating it - it will be easy to collaborate on the diagrams.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vincbeck]

Statics are failing du to recently merged new rule I think

[vincbeck]

Also:

• Should we put the internal API component in each tenant box?
• I am a bit confused by the links between the DAG files and triggerers and workers. Do these link exist? I thought the only component using DAG files is DAG processor and then the other components just use parsed DAGs from DB. But I might be wrong

[potiuk]

Also:

• Should we put the internal API component in each tenant box?

They are really "on the edge" so it's difficult to make them both in and out the tenant box - but I will try to figure out something :)

• I am a bit confused by the links between the DAG files and triggerers and workers. Do these link exist? I thought the only component using DAG files is DAG processor and then the other components just use parsed DAGs from DB. But I might be wrong

The picture is correct.

1. Workers

Workers cannot execute the DAGs from DB because they have to execute the code (With the imports and all the other things that are in the DAG folder) - the DB contains two things there:

a) Serialized for of DAG - json (that UI uses to display the DAG and Scheduler to schedule tasks from)
b) Code of the DAG file where the DAG comes from - but this one does not contain any imported libraries, utils etc. - it's literally source code of the DAG file the DAG came from.

So not nearly enough to get workers to run tasks. You really need to have access to DAG folder and be able to execute any code there from the worker.

2. The Triggerers

Triggerers have to execute Triggers. And you can define Triggers not only in your operators but also you can define custom Triggers in your DAG code and Trigger shoud be able to execute them in the event loop. And for that Triggerer similarly to worker should have access to DAG folder and be able to parse the complete code (including imports of any shared code(,

[vincbeck]

Also:

• Should we put the internal API component in each tenant box?

They are really "on the edge" so it's difficult to make them both in and out the tenant box - but I will try to figure out something :)

• I am a bit confused by the links between the DAG files and triggerers and workers. Do these link exist? I thought the only component using DAG files is DAG processor and then the other components just use parsed DAGs from DB. But I might be wrong

The picture is correct.

1. Workers

Workers cannot execute the DAGs from DB because they have to execute the code (With the imports and all the other things that are in the DAG folder) - the DB contains two things there:

a) Serialized for of DAG - json (that UI uses to display the DAG and Scheduler to schedule tasks from) b) Code of the DAG file where the DAG comes from - but this one does not contain any imported libraries, utils etc. - it's literally source code of the DAG file the DAG came from.

So not nearly enough to get workers to run tasks. You really need to have access to DAG folder and be able to execute any code there from the worker.

2. The Triggerers

Triggerers have to execute Triggers. And you can define Triggers not only in your operators but also you can define custom Triggers in your DAG code and Trigger shoud be able to execute them in the event loop. And for that Triggerer similarly to worker should have access to DAG folder and be able to parse the complete code (including imports of any shared code(,

I see. Thanks for the explanations

[potiuk]

Should be much nicer now @vincbeck

[vincbeck]

Awesome!