[MD] Support legacy client for data source

[zhongnansu]

Signed-off-by: Su szhongna@amazon.com

Description

Legacy clients are using elasticsearch.js
new clients are using openearch.js

Usage:
data_source.context.legacy.getClient(datasourceId).callAPI("ping", {params: something}, {options: something})

• Add support for data source legacy client
  • launch another pool within the data-source service setup stage, specifically for legacy clients
  • returns a function interface LegacyAPICaller as required by elasticsearch.js clients
• Refactor index pattern related logic to use data_source.legacy.getClient
• Added UT

Issues Resolved

#2133
#2386

Check List

• New functionality includes testing.
  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
    • yarn test:ftr
• New functionality has been documented.
• Commits are signed per the DCO using --signoff

[zengyan-amazon]

do we have to export it

[zhongnansu]

Have to export it, or otherwise copy the entire thing to datasource. Becuase we are trying to re-use it in the legacy client wrapper for datasource.

[zhongnansu]

Add more context. callAPI is not only the interface, but it wraps some validation, parsing logic for user input. To make sure the data_source.legacy.client has the exact same behavior as default legacy client, and doesn't break backwards compatibility for consumer plugins. We have to re-use it.

OpenSearch-Dashboards/src/core/server/opensearch/legacy/cluster_client.ts

Lines 68 to 83 in 65005be

	const callAPI = async (
	client: Client,
	endpoint: string,
	clientParams: Record<string, any> = {},
	options: LegacyCallAPIOptions = { wrap401Errors: true }
	) => {
	const clientPath = endpoint.split('.');
	const api: any = get(client, clientPath);
	if (!api) {
	throw new Error(`called with an invalid endpoint: ${endpoint}`);
	}
	const apiContext = clientPath.length === 1 ? client : get(client, clientPath.slice(0, -1));
	try {
	return await new Promise((resolve, reject) => {
	const request = api.call(apiContext, clientParams);

cc @noCharger

[zhongnansu]

With the new request from #2386, I think it worth to make a copy of callAPI and wire the wrap401Error there

[zengyan-amazon]

Agreed, we can create a new callAPI for now, while exploring other options.

[noCharger]

Overall looks good. Also curious if export callAPI is the best practice here.

[kristenTian]

index pattern related migration lgtm

[zhongnansu]

@zengyan-amazon Yan can you help take another look？I updated the PR based on our recent refactor and code changes

[seanneumann]

Can I get an update? @zengyan-amazon ??

[zhongnansu]

re-posted this PR to main, and include the changes requested in #2386
@zengyan-amazon @kristenTian @noCharger Please take another look

The base branch was changed.

[zengyan-amazon]

we have this pool includes a LRU of <endpoint, root_client> . since now we support Legacy client, shall we have one pool for legacy client, and another pool for new client? otherwise, it will cause runtime error when when a use case need a legacy client while getting a new client root client from the pool, and verse versa.

[zhongnansu]

There are in fact 2 pool instances being initiated in the constructor of DataSourceService here. One only stores new client, the other one for legacy client.

Since the only different of pool impl for legacy or new client is the "value" type of the LRU, I didn't make a copy of client_pool to avoid having too much duplicate code. Instead I defined the LRU type as multiple type.

[zengyan-amazon]

I'd prefer to have 2 cache in the pool to avoid misuse the pool. but the current implementation makes sense to me.

[codecov-commenter]

Codecov Report

Merging #2204 (5435d9d) into main (0279588) will increase coverage by 0.00%.
The diff coverage is 68.42%.

@@           Coverage Diff           @@
##             main    #2204   +/-   ##
=======================================
  Coverage   66.74%   66.75%           
=======================================
  Files        3197     3200    +3     
  Lines       60828    60886   +58     
  Branches     9243     9250    +7     
=======================================
+ Hits        40601    40642   +41     
- Misses      18019    18032   +13     
- Partials     2208     2212    +4     

Impacted Files	Coverage Δ
src/core/server/http/router/router.ts	74.54% <ø> (ø)
...r/index_patterns/fetcher/index_patterns_fetcher.ts	0.00% <ø> (ø)
...tcher/lib/field_capabilities/field_capabilities.ts	84.61% <ø> (ø)
...erver/index_patterns/fetcher/lib/opensearch_api.ts	100.00% <ø> (+36.36%)	⬆️
...index_patterns/fetcher/lib/resolve_time_pattern.ts	100.00% <ø> (ø)
src/plugins/data_source/server/lib/error.ts	100.00% <ø> (ø)
src/plugins/data/server/index_patterns/routes.ts	2.56% <50.00%> (+0.12%)	⬆️
...gins/data_source/server/client/configure_client.ts	80.00% <57.14%> (-4.85%)	⬇️
...ta_source/server/legacy/configure_legacy_client.ts	64.58% <64.58%> (ø)
.../plugins/data_source/server/data_source_service.ts	75.00% <70.00%> (-8.34%)	⬇️
... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-2204-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 746b9df179f9f095befe5a2ec9df841fc287baf7
# Push it to GitHub
git push --set-upstream origin backport/backport-2204-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2204-to-2.x.