specs: add architecture.md

[antoniivanov]

The goal of the architecture file is to provide a comprehensive overview of the VDK's architecture, components, and their relationships, serving as a reference document for developers, contributors, stakeholders, and users to understand the system's structure and make informed decisions.

The diagrams are based on C4 model https://c4model.com

The architecture is based on OnePage documents we use in VMware and we (Yoan (@yonitoo) and I) used the following architecture overviews as an example of good (data related) open source project arch documents: argo, dbt, airbyte, metaflow, dagster, flyte

Closes #858

Signed-off-by: Antoni Ivanov aivanov@vmware.com
Co-authored-by: Yoan Salambashev ysalambashev@vmware.com

[dakodakov]

I believe the following things are missing:

• "glossary" section at the top of the page - it might contain links to other pages, but I believe it would be beneficial.
• Out of this document, it's not clear, that the Control Service itself is running in a k8s cluster. I believe we should state that explicitly
• it's worth clarifying that when a user creates a data job, a data job container image is created for it and deployed as a cron job - this is roughly explained in the builder job/data job sections, however I don't think it's going to be clear to a new/user
• VDK SDK Component Diagram - how about explaining with a couple of sentences what each rectangle in this diagram does?

[mivanov1988]

I believe the following things are missing:

• Out of this document, it's not clear, that the Control Service itself is running in a k8s cluster. I believe we should state that explicitly

Including a deployment diagram or a visual representation illustrating a typical Control Service deployment could be advantageous. The inclusion of such a diagram or visual representation will enable the reader to gain a clear understanding of the location of all the components.

[mivanov1988]

To be honest, the style of the diagram is too unconventional for me.

[mivanov1988]

LGTM!

[antoniivanov]

I believe the following things are missing:

• "glossary" section at the top of the page - it might contain links to other pages, but I believe it would be beneficial.
• Out of this document, it's not clear, that the Control Service itself is running in a k8s cluster. I believe we should state that explicitly
• it's worth clarifying that when a user creates a data job, a data job container image is created for it and deployed as a cron job - this is roughly explained in the builder job/data job sections, however I don't think it's going to be clear to a new/user
• VDK SDK Component Diagram - how about explaining with a couple of sentences what each rectangle in this diagram does?

Addressed all parts. I did not add Glossary but I have added links to the Dictionary - https://github.com/vmware/versatile-data-kit/wiki/dictionary in a couple of places.

[antoniivanov]

To be honest, the style of the diagram is too unconventional for me.

https://c4model.com/ is getting more traction so I hope it's going to be familiar more and more people. ALso I find it simple to understand and learn. So I will keep it. I tried to change most diagram slightly to make them slightly more conventional-looking but not too much it is still following https://c4model.com/

[sabadzhiev]

#2: The levels explanation between the 'Control Service ' and 'VDK SDK' looks different:

• Control Service: System context + Container diagram
• VDK SDK: System context + Component diagram.
  Not sure if this was intentional, but brings me the feeling that the document is not complete.

In the c4 model, a container is a separately runnable/deployable unit (e.g. a separate process space) that executes code or stores data. The whole SDK is a single container so its components are the next level of detail.

[murphp15]

I'm surprised you don't go into the helm arch deployment.
This would address the comments above about saying the control service runs in k8s.

[antoniivanov]

I'm surprised you don't go into the helm arch deployment. This would address the comments above about saying the control service runs in k8s.

What do you mean ?