[HUDI-6399] Warn when datadog api key is wrong

[parisni]

Change Logs

Currently when the datadog api key is wrong the job will fail. We likely should not fail but log a warning to avoid the whole pipeline fails

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[parisni]

@xushiyan maybe ?

[hudi-bot]

CI report:

• 0763a87 UNKNOWN

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

Should we still fail the job if the metrics collector does not work?

[parisni]

We just should show a warning as proposed here

[parisni]

do you mean, catch any exeption ? this makes sense

[xushiyan]

Should we still fail the job if the metrics collector does not work?

+1

[yihua]

What I meant is, we should not change the behavior of throwing an exception here. If the metric collection does not work due to API key, it should fail the job so that the user knows and fixes it before proceeding.

[yihua]

I don't understand the rational here. Should user know the API key is not working and fix it before running the job again to properly generate the metrics? It's not a good idea to silently fail here.

[parisni]

To me, the metric provider is responsible to contact the user if metrics won't work (mailing alarm, oncall ...). But the ingestion jobs should not stop working. Not having metrics is a minor problem versus having all the company pipelines broken because of a token renewal issue.

Also users configure the metrics provider to alarm in case of no metrics.

At least I assume some user won't want their nightly jobs broken because of token, this would also be the case for an API or any metrics collection, outage is a minor problem versus stopping working.

Currently same apply for pushing metrics, if it does not work, it is only a warning see

hudi/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/Metrics.java

Lines 124 to 132 in 0763a87

	try {
	MetricRegistry registry = Metrics.getInstance().getRegistry();
	HoodieGauge guage = (HoodieGauge) registry.gauge(metricName, () -> new HoodieGauge<>(value));
	guage.setValue(value);
	} catch (Exception e) {
	// Here we catch all exception, so the major upsert pipeline will not be affected if the
	// metrics system has some issues.
	LOG.error("Failed to send metrics: ", e);
	}

[yihua]

Got it. I suggest having a feature flag on whether to fail the job if the metric provider does not work. By default, it's on, i.e., failing the job due to metric provider, the same behavior as before, while users can turn this off in the case metrics can be skipped.

[parisni]

Sure, will do that

[parisni]

https://hudi.apache.org/docs/configurations#hoodiemetricsdatadogapikeyskipvalidation
shame on me, the option to skip validation already exist. Then this PR is useless.