[GOBBLIN-1491] Add option to mark up/down D2 servers for gobblin service on leadership change

[jack-moseley]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-1491

Description

• Here are some details about my PR, including screenshots (if applicable):

Currently, D2 servers are started on all hosts at the time of service start. This change will allow using D2 delayed start, then marking up/down the servers when a host becomes the master/slave respectively. This means only the master will be announced to D2, so all requests will be routed to it.

The logic for markup is linkedin specific so must be implemented in internal code.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-commenter]

Codecov Report

Merging #3333 (b98d23f) into master (08db23e) will decrease coverage by 0.01%.
The diff coverage is 40.00%.

@@             Coverage Diff              @@
##             master    #3333      +/-   ##
============================================
- Coverage     46.55%   46.54%   -0.02%     
+ Complexity    10138    10136       -2     
============================================
  Files          2051     2052       +1     
  Lines         79547    79557      +10     
  Branches       8880     8882       +2     
============================================
- Hits          37036    37028       -8     
- Misses        39086    39096      +10     
- Partials       3425     3433       +8     

Impacted Files	Coverage Δ
.../org/apache/gobblin/service/ServiceConfigKeys.java	0.00% <ø> (ø)
...in/service/modules/core/GobblinServiceManager.java	50.20% <0.00%> (-0.84%)	⬇️
.../gobblin/service/modules/core/NoopD2Announcer.java	33.33% <33.33%> (ø)
...vice/modules/core/GobblinServiceConfiguration.java	93.33% <100.00%> (+0.31%)	⬆️
...ervice/modules/core/GobblinServiceGuiceModule.java	88.65% <100.00%> (+0.11%)	⬆️
...a/org/apache/gobblin/util/limiter/NoopLimiter.java	40.00% <0.00%> (-20.00%)	⬇️
...lin/util/filesystem/FileSystemInstrumentation.java	92.85% <0.00%> (-7.15%)	⬇️
.../management/conversion/hive/source/HiveSource.java	64.08% <0.00%> (-2.77%)	⬇️
...gobblin/service/modules/core/GitConfigMonitor.java	81.35% <0.00%> (-1.70%)	⬇️
...a/management/copy/publisher/CopyDataPublisher.java	74.17% <0.00%> (-1.33%)	⬇️
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08db23e...b98d23f. Read the comment docs.

[aplex]

LGTM. Can you fix the build issues? I'll merge it after.