feat(quickstart): Remove kafka-setup as a hard deployment requirement

[pedro93]

This PR does a few improvements to our docker cli behaviour, namely:

• Makes kafka-setup job optional behind the flag: --kafka-setup which works similar to standalone consumer containers.
• Makes docker-compose Quickstart generation logic able to handle duplicate env vars with last-wins heuristic. This was done because we have conflicting configs coming from docker env files & explicit docker-compose environment variables.
• Generates Neo4j M1-compatible docker-compose files which removes a long-standing limitation in our cli.
• M1 Quickstart was not being generated by our internal scripts. This makes it very easy for changes in our partial compose files to not get mirrored in the final Quickstart file. This has now been fixed.
• Makes standalone consumer configs for neo4j deployments consistent.
• Improves Quickstart comparison logic more flexible with the use of yq
• Bumps Confluent dependencies to 7.2.2

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[pedro93]

Note, this PR will fail if run locally to test the new flag because it assumes that the files already exist in Github.

You will need the following git diff to test this locally:

diff --git a/metadata-ingestion/src/datahub/utilities/sample_data.py b/metadata-ingestion/src/datahub/utilities/sample_data.py
index b908f7f89f..990e86fc97 100644
--- a/metadata-ingestion/src/datahub/utilities/sample_data.py
+++ b/metadata-ingestion/src/datahub/utilities/sample_data.py
@@ -6,7 +6,8 @@ import requests

 DOCKER_COMPOSE_BASE = os.getenv(
     "DOCKER_COMPOSE_BASE",
-    "https://raw.githubusercontent.com/datahub-project/datahub/master",
+    "https://raw.githubusercontent.com/acryldata/datahub/remove-kafka-setup-from-cli",
+    #"https://raw.githubusercontent.com/datahub-project/datahub/master",
 )
 BOOTSTRAP_MCES_FILE = "metadata-ingestion/examples/mce_files/bootstrap_mce.json"
 BOOTSTRAP_MCES_URL = f"{DOCKER_COMPOSE_BASE}/{BOOTSTRAP_MCES_FILE}"

[david-leifker]

Like the approach, probably you've found an issue with the mce consumer quickstart generation.

[david-leifker]

I am wondering if we even need a separate m1 flavor anymore? Seems like folks have moved to generate arm images consistently, however perhaps I am overlooking something here.

[pedro93]

I am wondering if we even need a separate m1 flavor anymore? Seems like folks have moved to generate arm images consistently, however perhaps I am overlooking something here.

There are a few images that we need M1 alternatives for still such as Postgres & Neo4j.