refactor(restli-mce-consumer)

[david-leifker]

Checklist

This PR contains a number of changes which first addresses the reliability and performance of the Restli Service endpoints for ingestion. Additional updates to allow debugging on M1 with 100% locally built images.

• Scale Restli Service thread pool based on CPU
• Add retry (exp backoff) to Restli Entity Client
• MCE no longer relies on GMS for Restli service
• Converted Restli Service from standalone servlet to Spring injectable
• Docker build externalized (significantly faster on m1, <7 minute build times, based on this)
• Frontend asset generation refactor (causing tests to fail intermittently)
• Added ./gradlew quickstart and ./gradlew quickstartNuke for building and running quickstart locally

The new gradle task needs documentation, however I'd like to get some folks to try it out besides myself before advertising it. This is actually building the docker images locally (datahub ones anyways).

$ ./gradlew clean quickstartNuke quickstart

...

✔ DataHub is now running
Ingest some demo data using `datahub docker ingest-sample-data`,
or head to http://localhost:9002 (username: datahub, password: datahub) to play around with the frontend.
Need support? Get in touch on Slack: https://slack.datahubproject.io/

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 5m 55s
334 actionable tasks: 310 executed, 24 up-to-date

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[david-leifker]

I am making this ready for review, however I am not 100% sure about the one test, if it is in fact a bug in these changes.

[jjoyce0510]

One obvious downside of this is that the standalone consumers have more entanglement, less loose coupling with the state stores.

The complexity of maintaining this is something to think about - are the gains worth it?

[david-leifker]

I would like to merge the 3 subcharts in helm one day to reduce configuration complexity. Really mce/mae/gms are tightly coupled and the same configuration should always be applied to all three. Fortunately even though the helm subcharts are separate the configuration is global so no additional helm chart values are required!

I believe this is the key to scaling ingestion horizontally and separating impact of ingestion on the GMS graphql read path for the UI.

[jjoyce0510]

We initially architected with datahub-gms serving as a front for elastic and MySQL, hiding the consumers from the complexity of knowing how to talk to each state store and encapsulating the complex configurations.. Really want to be sure this is worth splitting open

[david-leifker]

The concept is preserved in the sense that a common library still abstracts the storage layers. The exact same library components are used, we are merely changing the routing of the requests at runtime to bypass the GMS bottleneck. The datastore configuration parameters are shared in the helm chart so no duplication there. I believe the complexity of the configuration is no greater for the end user than it is today.

[jjoyce0510]

Are we spinning up an entire servlet in the mce consumer?

[RyanHolstien]

Yeah, this is probably worth talking about at like a lunch and learn/project updates space since it's a non-trivial architectural change in general. We have talked about the goal of this inside the platform team at length.

Basically this is enabling the offloading of ingestion onto just MCE Consumer in cases where we are using Async ingestion. I.e. if we're using GMS to just bounce back to MCE Consumer by generating a MCP, why not just directly hit MCE Consumer to take network load off of GMS?

[david-leifker]

Exactly, this is the core idea, remove GMS as the bottleneck for async ingestion and decouple the UI load from ingestion load. This was the original goal for the async consumers and its now fully realized.

[pedro93]

Curious: Why do we need the version bump?

[david-leifker]

Not required, however keeping minor updated versions in general is a good idea.