JSON artifact updates

[beckjake]

Describe the feature

There are a number of changes we should make to dbt's JSON artifact outputs to make some new features easier to develop:

• We should combine the various SQL fields in the manifest. I can see some value in the raw_sql field existing, but we should change injected_sql to overwrite compiled_sql instead of adding a new field.
• Include the dbt version that produced JSON artifacts.
• Include the invocation_id in produced JSON artifacts.
• Accept a run_id as an environment variable and include that in produced JSON artifacts.
• All output artifacts should have common metadata

On the behavior side of things, dbt should write compiled manifests at the end of the dbt run. Currently we only write a compiled manifest after running the internal compilation phase in dbt docs generate.

Describe alternatives you've considered

Status quo

Who will this benefit?

Consumers of dbt's JSON files.

[jtcohen6]

• Common metadata field across all artifacts, to include timestamps + invocation_id. Already exists in manifest.
• Instead of a single string value run_id, we check for something like DBT_ENV_CUSTOM_ENV_-prefixed env variables and nest into a dict in every artifact. (Under metadata field)

On the behavior side of things, dbt should write compiled manifests at the end of the dbt run.

If we rewrite the manifest at the end of the run, we can remove the node field from the run results

[beckjake]

I've split this into separate issues, except for the dbt version part, which is already captured in #2670

[jtcohen6]

Resolved!