Establish a docs framework - Resolves #60

[rgeyer]

Should have committed way earlier...
Added a makefile with embedmd which pulls in the usage info into the main docs file.
Created a docs directory and rules sub directory to establish heirarchy.
Created a script to replace rule names surrounded in backticks with links to their docs (if present).

[rgeyer]

@v-zhuravlev can you help provide some context here? I know you've raised concerns about enforcing these rules, so if we can have that discussion in the comments here, or flesh-out this document it would be appreciated.

[v-zhuravlev]

As it turns out, there is a problem with $job (and $instance as well) filter becomes visible when you start to compose dashboards that could have panels from multiple sources. For example, prometheus node_exporter and promtail logs from the same host. Or from node_exporter and process_exporter exporters that reside on the same host.

I think that at first the idea of this rule was to have a two-level grouping of instances:

• 1st level: instance
• 2nd level: job

This is convenient, but since job is built-in label, using it for grouping/filtering makes prometheus/grafana-agent configs complex (if you have multiple metrics/logs sources):

• It requires rewriting already present labels, so the match for metrics&logs , making scrape_configs complex due to additional relabelling rules.
• Since the label is present, relabelling seems to be the only way to overwrite it. If this label were vacant then we could just used external_labels or labels to set the common job in grafana-agent/prometheus.

Here are some examples of configs where such user confusing relabeling taking place:
https://grafana.com/docs/grafana-cloud/integrations/integrations/integration-docker/#post-install-configuration-for-the-docker-integration
https://grafana.com/docs/grafana-cloud/integrations/integrations/integration-mysql/

So, we should ask, what is the main reason we enforce this?
If we need a second level of grouping, perhaps we should use label like cluster instead? (and make it optional, where it is applicable)

[rgeyer]

I went ahead and documented this in it's "current state" to get this document framework committed.

Let's revisit the effectiveness of these rules, and we can address in a follow-up PR. I'll create an issue to track.

[mshahzeb]

LGTM! Couple of nits and comments.

[alyssawada]

Nice work, @rgeyer and thanks for your patience! Feel free to use whatever feels right and let me know if you have any questions on my suggests/comments :)