Create Networking/Data flow diagrams

[michellecbox]

Networking/data flow diagrams for the communications around how the cca operater/jenkins/rancher/kubernetes/nginx/letsencrypt/etc are all working together to complement the current documentation

Acceptance Criteria

John Jediny / Tom Wood / Andrew Bond understand to their satisfaction (?) the structure of the cluster

Tasks

• Draft the diagram
• Test out with relevant parties
• Finalize

Background

John Jediny (Data.gov / DCA / he/him)
I’d like us to start working on the networking/data flow diagrams for the communications around how the cca operater/jenkins/rancher/kubernetes/nginx/letsencrypt/etc are all working together to complement the current documentation. We’d like to do this as code and model https://cloud.gov/docs/compliance/diagrams/ using mermaid

John Jediny (Data.gov / DCA / he/him)
https://raw.githubusercontent.com/18F/cg-diagrams/master/source/diagrams/10-1-network.mmd

John Jediny (Data.gov / DCA / he/him)
c/p to https://mermaidjs.github.io/mermaid-live-editor/

[rufuspollock]

@akariv has this in progress and will have version to share by Friday or before.

[akariv]

This is a first draft of the flow diagram - any feedback would be most welcome:

https://hackmd.io/iRbcMO7BQ3-G7XaffqL-ag?both

@woodt @JJediny

[akariv]

/cc @rufuspollock

[adborden]

Embarrassed to ask, but I cannot figure how to export, download, or make the svg bigger from hackmd. Is it possible to generate the svg and include a download link so I can get a better look?

[akariv]

I also uploaded to mermaid live editor:

here

You can download the SVG from there

[adborden]

Unfortunately, when after downloading the SVG, I still cannot view it. Did it work for you?

I seem to be running into this half-resolved bug mermaid-js/mermaid#384 (comment)

I had to manually hack the SVG to work, annoying.

[adborden]

Why is the Database Proxy necessary? As a shared service, it seems like performance bottle neck and lack of isolation for security between CKAN instances.

[akariv]

There are two main reasons for the DB proxy:

The first is to create a connection pool to the main DB.
DBs (in general, and specifically) have a limit on the number of concurrent connections. When running multiple CKAN instances on the same cluster, backed by the same database, you can hit that limit fast.
(I think that these DBs are usually tuned for size and throughput and not for concurrent users).
The second reason is to have a single, unchanging, DB endpoint that instances access, instead of sharing the actual RDS address among the instances. This allows later to modify and administer the DB without having to propagate a configuration change throughout the cluster.

[BTW - At the current PoC configuration, demo instances access RDS directly, but in production this won't scale.]

As for a bottleneck - I agree, but not more than RDS itself (which is backed by an actual machine). From my experience, CKAN DB tend to be relatively small and queries are simple, so it's usually not very loaded. Either way, we could partition the system (i.e. create more than one combination of proxy & rds which instances will use) or have more complex configurations if we ever see a need for that.

[rufuspollock]

@adborden @JJediny is it clear now and can we now close as fixed :-) ?

[adborden]

Is the purpose of the diagram Tobe used for our SSP? If so I think there are some additional things we want to capture (I can provide more guidance on that).

Otherwise I understand this and we can call this done.

[adborden]

Clarified, this issue can be closed. We'll circle around again to do more formal network diagrams for the SSP in another issue.