[CELEBORN-1550] Add support of providing custom dynamic store backend implementation

[s0nskar]

What changes were proposed in this pull request?

Adding support of providing custom dynamic store backend implementation, users can now pass there own implementation for dynamic config store backend.

This change also keep the backwards compatibility of supporting short names for backend like "FS" and "DB"

Why are the changes needed?

Currently celeborn only supports File and DB based backend while there can be other ways of managing these configs.

Does this PR introduce any user-facing change?

NO, user facing behaviour will be same.

How was this patch tested?

Existing UTs verifies that this change is working for "FS" and "DB" implementation.

[SteNicholas]

@s0nskar, why is the implementation not based on SPI?

[s0nskar]

@SteNicholas We have an internal implementation of config management service which enforce authn/authz and constraints on specific configs. It also helps us manage the rollout better between different regions.

[AngersZhuuuu]

Should also update doc here https://celeborn.apache.org/docs/latest/developers/configuration/#dynamic-configuration

[s0nskar]

@AngersZhuuuu Updated the doc.

[SteNicholas]

@s0nskar, ConfigService should introduce getName() interface for short name configuration.

[s0nskar]

@SteNicholas Added. Q: do you want me to make use of it in this PR?

[SteNicholas]

@s0nskar, the implementation of ConfigService could use getName interface to load the implementation instance instead of hard code in factory. Or you could regard DynamicConfigServiceFactory as interface and the getName interface could be introduced in DynamicConfigServiceFactory.

public interface DynamicConfigServiceFactory {
   public String getName();

   public ConfigService getConfigService(CelebornConf celebornConf);
}

public class FSConfigServiceFactory implements DynamicConfigServiceFactory {
   public String getName() { return "FS";}
   public ConfigService getConfigService(CelebornConf celebornConf) { ...}
}

[SteNicholas]

User could not add short name of the custom ConfigService implementation in dynamicConfigStoreBackendShortNames.

[s0nskar]

i was sort of following what spark does for shuffle manager. It allows short names for sort and tungesten-sort but if user is passing custom shuffle manager then they have to pass fully qualified name.

[s0nskar]

the implementation of ConfigService could use getName interface to load the implementation instance instead of hard code in factory.

@SteNicholas Trying to understand this comment better – I think for this will have to make getName() static method but abstract static does not work.

The factory approach will also need a lot of abstraction IMO it is not necessary for this as we will only have couple of predefined names. Let me know if you think otherwise then i will try to make it work.

[waitinfuture]

LGTM, thanks!

[RexXiong]

LGTM

[waitinfuture]

Merging to main(v0.6.0)/branch-0.5(v0.5.2)