[JUJU-553] Add tool to infer entity schemas and render a ER-like diagram using graphviz

[achilleasa]

This PR contributes a CLI tool that analyzes the set of structs defined in Juju's state package and attempts to generate an ER-like diagram to aid with the data modeling work which is part of the transition from mongo to sqlite.

The actual diagram looks more like a UML class diagram as the number of entities (tables) and attributes makes it more difficult for graphviz to figure out a suitable layout for the graph nodes.

The tool automatically normalizes both table and field names to the underscore-cased format that folks are more familiar with when working with SQL databases. In addition, the tool will attempt to cluster together tables that share a common name prefix.

Future work ideas/suggestions:

• Infer relations between tables by analyzing the names of fields that serve as foreign keys (this can also yield a more accurate clustering result). Alternatively, the tool could be seeded with an external (JSON/YAML etc.) file to provide hints as to which tables should be grouped together.
• The tool currently lists the Go type for each attribute. However, for attributes that are nested objects or collections of objects we could automatically generate auxiliary tables for storing them and also annotate the graph with edges that will help identify the relationships between fields.

QA steps

$ cd scripts/dqlite/cmd

# NOTE 1: use 'open' instead of xdg-open if running on OSX
# NOTE 2: you must use `fdp` as your layout engine to get nice, rectangular output for the graph!
$ go run infer_schema.go | fdp -o erd.svg -Tsvg && xdg-open erd.svg

Yields something like this:

[juanmanuel-tirado]

Nice
LGTM

[SimonRichardson]

Can we also expand nested types, for example:

graph {
charm [shape=record, label=<{<b>charm</b><br/>github.com/juju/juju/state/charm.go | doc_id (string) | url (*charm.URL) | charm_version (string) | life (Life) | pending_upload (bool) | placeholder (bool) | bundle_sha256 (string) | storage_path (string) | macaroon ([]byte) | meta (*charm.Meta) | config (*charm.Config) | manifest (*charm.Manifest) | actions (*charm.Actions) | metrics (*charm.Metrics) | lxdprofile (*LXDProfile)}>];
}

Can we expand "manifest (*charm.Manifest)"?

[achilleasa]

Can we also expand nested types, for example:

graph {
charm [shape=record, label=<{<b>charm</b><br/>github.com/juju/juju/state/charm.go | doc_id (string) | url (*charm.URL) | charm_version (string) | life (Life) | pending_upload (bool) | placeholder (bool) | bundle_sha256 (string) | storage_path (string) | macaroon ([]byte) | meta (*charm.Meta) | config (*charm.Config) | manifest (*charm.Manifest) | actions (*charm.Actions) | metrics (*charm.Metrics) | lxdprofile (*LXDProfile)}>];
}

Can we expand "manifest (*charm.Manifest)"?

See the future work/suggestion list. Ideally we would want to identify complex types and create aux tables from them. So in this example, we would have a manifests table and the charm would have a FK to the manifest ID. Not sure how much effort that would require though...

I will play around with this idea and see how far I get.

[achilleasa]

$$merge$$

[achilleasa]

$$merge$$ (CI was happy but branch merge policy prevented the merge)

[achilleasa]

$$merge$$

[achilleasa]

$$merge$$