Seperate python script for armada v1 and v2 system diagrams

[Pradeep-Kurapati]

This is a documentation improvement pull request

What this PR does:

This pull request gives the seperate diagram for Armada V1 System.

Issue PR fixes:

Fixes #2738

Special notes for your reviewer:

1. Please check if the diagram is correct.
2. Should we remove generate.py as I'm thinking of generate_v1.py and generate_v2.py?
3. If this diagram is correct, I will be commiting generate_v2.py

Here is the resultant diagram:

┆Issue is synchronized with this Jira Task by Unito

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (3fa5ab2) 47.40% compared to head (23dd9ab) 47.40%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2758   +/-   ##
=======================================
  Coverage   47.40%   47.40%           
=======================================
  Files         395      395           
  Lines       44045    44045           
  Branches      487      487           
=======================================
  Hits        20879    20879           
  Misses      21582    21582           
  Partials     1584     1584           

Flag	Coverage Δ
unittests	47.40% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Sharpz7]

Great start! A few things:

Armada V1 actually uses lookout V2, but in a special way. The pulsar -> Lookout V2 Ingester -> Lookout V2 API loop exists, but instead, it also talks to the Lookout V1 UI. can you add that in?

Also, Pulsar does not talk to the executors, instead, the executors talk to Armada Server (This way, Armada Server is not responsible for keeping track of its executors, the executors need to send data to the server, and it responds with a decision). Also, if you are able to add in the Armada Clients talking to The server like in the V2 Doc that would be great.

Also, the "Redis Scheduler" should be an Armada Component, not the Redis Icon.

[Pradeep-Kurapati]

This looks very wrong. Please review this. Thank you:

[Sharpz7]

Yeah so this needs a few more things:

• The lookout V2 API talks to The Lookout V1 UI.
• Pulsar does not talk to the lookout V1 UI.
• The Executor API does not talk to the Redis Scheduler, in fact, remove the redis scheduler, as for V1 it is included inside the Armada Server. I.e make all connections going to the Redis scheduler go to the server instead
• https://drive.google.com/uc?id=1eeaX7EMdJfCXBgVKcNkiHV9yG1zqxBqO These components need to be duplicated for the V1 Lookout. There should, be two of these, with the Lookout V1 and V2 APIs being the only components which talk to the Lookout UI

I know this looks messy... but welcome to Armada :))

[Pradeep-Kurapati]

I did not clearly understand the last point. When you said V1 Lookout, were you referring to Lookout V1 UI or is it a component I didn't add? Some clarification about the last point would be great!

[Pradeep-Kurapati]

@Sharpz7, please review Armada V2 System diagram also:

[Pradeep-Kurapati]

@Sharpz7 kindly review the changes and provide feedback. Thank you.

[Sharpz7]

V2 is looking good now!

V1 Needs this change https://drive.google.com/uc?id=1cBMG1pR6c5KUGAwRqvLA03tbmQMHZyEM. You need to add a 2nd ingester and API in for lookout V1 and connect it as shown.

Also, now it is mostly ready, give some documentation a go! Try building armada yourself and experimenting, add the images into the PR as well please directly

[Pradeep-Kurapati]

@Sharpz7 Please review V1 diagram:

[Sharpz7]

This looks good! Lets get the documentation written including these diagrams and then we can ask more people to review :))

You probably want to try and get armada running so you can see what is going on. Go to the developer docs to see how that happens :))

[Pradeep-Kurapati]

Roger that!

[Pradeep-Kurapati]

I updated relationships_diagram.md to include our new diagrams. Please suggest necessary reviews. Thanks!

[Pradeep-Kurapati]

@Sharpz7 @kannon92 Hi there! I just wanted to check in on the status of this pull request. Would appreciate any feedback or suggestions. Thank you!

[Pradeep-Kurapati]

@Sharpz7 @kannon92 Hi there! I just wanted to check in on the status of this pull request. Would appreciate any feedback or suggestions. Thank you!

[Sharpz7]

Apolgies for not getting back to you, will look at this today.

[Sharpz7]

I am happy for no more time to be spent on this. I think the diagrams do a good enough job.