Create HTTP Agent manager #137748

gsoldevila · 2022-08-01T16:17:24Z

The PR defines a class to better control which instances of HTTP(s) Agent are created.

gsoldevila · 2022-09-05T15:53:37Z

packages/core/elasticsearch/core-elasticsearch-server-internal/src/elasticsearch_service.ts

@@ -172,13 +174,16 @@ export class ElasticsearchService
    clientConfig: Partial<ElasticsearchClientConfig> = {}
  ) {
    const config = mergeConfig(baseConfig, clientConfig);
+
+    // tweak config to inject agent


Self-review: remove this obsolete comment

elasticmachine · 2022-09-05T15:53:50Z

Pinging @elastic/kibana-core (Team:Core)

pgayvallet

Kinda hard to say without concrete usages being added in the PR, but overall the implementation looks good to me.

A bunch of questions and NITs before a second pass:

pgayvallet · 2022-09-07T06:49:41Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/cluster_client.ts

  }: {
    config: ElasticsearchClientConfig;
    logger: Logger;
    type: string;
    authHeaders?: IAuthHeadersStorage;
    getExecutionContext?: () => string | undefined;
    getUnauthorizedErrorHandler?: () => UnauthorizedErrorHandler | undefined;
+    agentManager?: AgentManager;


May we want to make this mandatory instead of optional?

Having it optional is an easy way to fallback to the previous behavior (as per my comment above).

when an agentManager is not supplied I'd say our clients no longer behave as expected e.g. the meaning of maxSockets will change. It's only core using this constructor so it would be easy to change if we switch to undici.

Fair enough, I'll make it mandatory.

pgayvallet · 2022-09-07T06:50:21Z

...ages/core/elasticsearch/core-elasticsearch-client-server-internal/src/cluster_client.test.ts

@@ -83,11 +83,13 @@ describe('ClusterClient', () => {
    expect(configureClientMock).toHaveBeenCalledTimes(2);
    expect(configureClientMock).toHaveBeenCalledWith(config, {
      logger,
+      agentManager: undefined,


NIT: add test to ensure that the agent manager is passed down to the configureClient calls when present.

pgayvallet · 2022-09-07T06:55:23Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.ts

+   *   'https:': [agentInstance5]
+   * }
+   */
+  private agentStore: Record<string, Record<Protocol, Array<NetworkAgent | undefined>>>;


I think I would use a Map here instead of a Record, at least for the top-level one.

I don't have a strong opinion for or against either of the two, I'll switch to Map.

pgayvallet · 2022-09-07T06:56:09Z

packages/core/elasticsearch/core-elasticsearch-server-internal/src/elasticsearch_service.ts

@@ -172,13 +174,16 @@ export class ElasticsearchService
    clientConfig: Partial<ElasticsearchClientConfig> = {}
  ) {
    const config = mergeConfig(baseConfig, clientConfig);
+
+    // tweak config to inject agent


pgayvallet · 2022-09-07T06:58:31Z

packages/core/elasticsearch/core-elasticsearch-server-mocks/src/elasticsearch_service.mock.ts

  >;
 }

 const createPrebootContractMock = () => {
  const prebootContract: MockedElasticSearchServicePreboot = {
    config: { hosts: [], credentialsSpecified: false },
-    createClient: jest.fn(),
+    createClient: jest.fn((type: string) => elasticsearchClientMock.createCustomClusterClient()),


NIT: no need to specify the type: string parameter if unused to match the signature (same L70)

TS complains with the following message if I don't:

Type '[]' is not assignable to type '[type: string, clientConfig?: Partial<ElasticsearchClientConfig> | undefined]'. Source has 0 element(s) but target requires 1.ts(2322) index.d.ts(226, 14): The expected type comes from property 'createClient' which is declared here on type 'MockedElasticSearchServicePreboot'

pgayvallet · 2022-09-07T07:17:39Z

x-pack/plugins/ml/server/saved_objects/sync_task.ts

@@ -126,7 +126,7 @@ export class SavedObjectsSyncService {

      return taskInstance;
    } catch (e) {
-      this.log.error(`Error running task: ${SAVED_OBJECTS_SYNC_TASK_ID}, `, e?.message() ?? e);
+      this.log.error(`Error running task: ${SAVED_OBJECTS_SYNC_TASK_ID}, `, e?.message ?? e);


Why was this change necessary?

Nope, not really.
It's a bug in the code that prevents from logging the caught error though, and causes a new Exception to be thrown.
I uncovered it during some tests.

I can create a separate PR if needed. @elastic/ml-ui WDYT?

This looks like a typo.
I'm happy for this to be fixed in this PR, no need to create a new one.
Thanks for spotting and fixing it.

pgayvallet · 2022-09-07T07:19:06Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.test.ts

+    const HttpAgentMock = HttpAgent as jest.Mock<HttpAgent>;
+    const HttpsAgentMock = HttpsAgent as jest.Mock<HttpsAgent>;


NIT: I think these two lines can be extracted to the top level describeblock, or even to depth 0 of the file.

pgayvallet · 2022-09-07T07:23:24Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.test.ts

+      agentFactory({ url: new URL('http://elastic-node-1:9200') });
+      expect(HttpAgent).toBeCalledTimes(1);
+      expect(HttpAgent).toBeCalledWith({


NIT: I would mock the return of HttpAgent / HttpsAgent and assert the return value of agentFactory(...) accordingly.

pgayvallet · 2022-09-07T07:25:46Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.test.ts

+      const agentManager = new AgentManager(logger);
+      expect(() => {
+        agentManager.getAgentFactory('anotherTest', { keepAliveTimeout: 2000 });
+      }).toThrowError();


NIT: toThrowErrorMatchingInlineSnapshot()

pgayvallet · 2022-09-07T07:31:32Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.test.ts

+        const agent1 = agentFactory1({ url: new URL('http://elastic-node-1:9200') });
+        const agent2 = agentFactory2({ url: new URL('http://elastic-node-1:9200') });
+        expect(agent1).not.toEqual(agent2);


TIL: that's true in a test environment with

jest.mock('http'); jest.mock('https');

I would expect both to be undefined given the constructor has been mocked without specifying return values.

Jest seems to be mocking the class automatically, and returning an instance that mocks all its methods too.

jgowdyelastic

ML changes LGTM

rudolf · 2022-09-08T09:59:49Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.ts

+   *   'https:': [agentInstance5]
+   * }
+   */
+  private agentStore: Record<string, Record<Protocol, Array<NetworkAgent | undefined>>>;


It's not clear to me why there could be several agents per type.protocol.

Like in your example why does data.http have agentInstance1 AND agentInstance2 shouldn't the factory always return the same agent?

It seems like this is because the AgentManager can create multiple factories, one factory per type. But if we're already creating a new agent per type why do we also need a new factory per type?

In my head I was kinda imagining that the agentStore would be something like a WeakSet with type+protocol keys and NetworkAgent values.

See #137748 (comment), I think that's the same question

Pierre's right, please refer to the comment above.

At the moment I am logging a warning if multiple factories of the same type are requested.
IMO if we want to ensure there's only one Agent instance per type we can work at a higher level and have a ClusterClient store instead.

FWIW, if I'm not mistaken the different data requests go through the 'data' instance managed internally by the elasticsearch service, so at the end of the day we will not have lots of instances of each type:

One data instance, one data-scoped.

A bunch of instances of other types (enroll, ping, authenticate, ...) that are used in the interactive mode only.

A monitoring instance (used by the monitoring plugin).

Also, note that if we use WeakSets or WeakMaps we can't really iterate through them to aggregate metrics for monitoring on the next step.

Now that I think about it, perhaps we could throw an Exception if we try to fetch more than one factory for the same type. That would greatly simplify the store structure and would encourage users to reuse the existing ES Client instance.

~~It would cause some tests to fail, as they systematically create instances using the same type, but we might be able to workaround it for tests. I'll give it a try.~~

UPDATE: I'm afraid we can't really do that without impacting the interactive-setup plugin.

rudolf

I really like the direction this is going, it's already a lot simpler 🎉

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.ts

pgayvallet

Revisited implementation looks way better to me. LGTM on my side, but we should wait for @rudolf's approval too.

A few nits:

pgayvallet · 2022-09-09T09:21:36Z

packages/core/elasticsearch/core-elasticsearch-server-internal/src/elasticsearch_service.ts

    return new ClusterClient({
      config,
      logger: this.coreContext.logger.get('elasticsearch'),
      type,
      authHeaders: this.authHeaders,
      getExecutionContext: () => this.executionContextClient?.getAsHeader(),
      getUnauthorizedErrorHandler: () => this.unauthorizedErrorHandler,
+      agentManager: this.agentManager,


NIT: update

kibana/packages/core/elasticsearch/core-elasticsearch-server-internal/src/elasticsearch_service.test.ts

Line 124 in 3508350

describe('#createClient', () => {

accordingly to assert the agentManager is passed down to the ClusterClient constructor

pgayvallet · 2022-09-09T09:25:11Z

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/client_config.ts

+): ClientOptions & { agent: HttpAgentOptions } {
+  const clientOptions: ClientOptions & { agent: HttpAgentOptions } = {


NIT: I would introduce an explicit type instead of an inline def

export type ParsedClientOptions = Omit<ClientOptions, 'agent'> & { agent: HttpAgentOptions }

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.ts

kibana-ci · 2022-09-14T08:49:14Z

💚 Build Succeeded

Buildkite Build
Commit: 9437590

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/core-elasticsearch-client-server-internal`	9	10	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/core-elasticsearch-client-server-internal`	0	1	+1

Unknown metric groups

API count

id	before	after	diff
`@kbn/core-elasticsearch-client-server-internal`	11	12	+1

ESLint disabled line counts

id	before	after	diff
`@kbn/core-elasticsearch-client-server-internal`	0	2	+2
`@kbn/core-elasticsearch-server-internal`	0	1	+1
total			+3

Total ESLint disabled count

id	before	after	diff
`@kbn/core-elasticsearch-client-server-internal`	0	2	+2
`@kbn/core-elasticsearch-server-internal`	0	1	+1
total			+3

History

💚 Build #70851 succeeded 6331e35
💔 Build #70754 failed 86b14b9
💚 Build #70477 succeeded 72d8388
💔 Build #70464 failed 74a4886
💚 Build #69440 succeeded 53a25f1
💚 Build #68297 succeeded 0ff81aa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Create HTTP Agent factory

4265fe3

gsoldevila added 24 commits August 2, 2022 10:58

Merge branch 'main' into kbn-137734-http-agent-factory

b3a2d1c

Properly extract agent options

2f0550b

Use independent Agent for preboot

24e731d

Merge branch 'main' into kbn-137734-http-agent-factory

bf19f70

Create AgentManager to obtain factories

941f5db

Merge branch 'main' into kbn-137734-http-agent-factory

ca8d434

Make client type mandatory, fix outdated mocks

93d033c

Temporarily force new Agent creation

0e8591c

Revert changes in utils

582e4e7

Merge branch 'main' into kbn-137734-http-agent-factory

11431a3

Add correct defaults for Agent Options, support proxy agent.

a3b153d

Merge branch 'main' into kbn-137734-http-agent-factory

46232e7

Forgot to push package.json

f891b0a

Add hpagent dependency in BUILD.bazel

8270d13

Get rid of hpagent (proxy param is not exposed in kibana.yml)

a8b7915

Merge branch 'main' into kbn-137734-http-agent-factory

0954407

Remove hpagent from BUILD.bazel

210a326

Use different agents for normal Vs scoped client

df464cc

Merge branch 'main' into kbn-137734-http-agent-factory

9acb1dc

Fix Agent constructor params

41fe166

Merge branch 'main' into kbn-137734-http-agent-factory

1aa1a4c

Fix incorrect access to err.message

8149c53

Use separate Agent for scoped client

e2b53d6

Create different agents for std vs scoped

fca3af2

gsoldevila added 6 commits August 29, 2022 18:45

Provide different Agent instances if config differs

0d0ba54

Merge branch 'main' into kbn-137734-http-agent-factory

5dcb57a

Create a new Agent for each ES Client

5038277

Merge branch 'main' into kbn-137734-http-agent-factory

0ff81aa

Restructure agent store. Add UTs

01a4d17

Merge branch 'main' into kbn-137734-http-agent-factory

53a25f1

gsoldevila changed the title ~~Create HTTP Agent factory~~ Create HTTP Agent manager Sep 5, 2022

Remove obsolete comment

afe9f35

gsoldevila commented Sep 5, 2022

View reviewed changes

gsoldevila marked this pull request as ready for review September 5, 2022 15:53

gsoldevila requested review from a team as code owners September 5, 2022 15:53

pgayvallet reviewed Sep 7, 2022

View reviewed changes

jgowdyelastic approved these changes Sep 7, 2022

View reviewed changes

rudolf reviewed Sep 8, 2022

View reviewed changes

gsoldevila added 3 commits September 8, 2022 16:10

Simplify AgentManager store structure (no type needed)

5aae94b

Merge branch 'main' into kbn-137734-http-agent-factory

74a4886

Fine tune client_config return type

72d8388

rudolf reviewed Sep 8, 2022

View reviewed changes

packages/core/elasticsearch/core-elasticsearch-client-server-internal/src/agent_manager.ts Show resolved Hide resolved

pgayvallet approved these changes Sep 9, 2022

View reviewed changes

gsoldevila added 2 commits September 9, 2022 12:26

Misc enhancements following PR comments

290bcff

Merge branch 'main' into kbn-137734-http-agent-factory

86b14b9

rudolf approved these changes Sep 9, 2022

View reviewed changes

gsoldevila added 2 commits September 9, 2022 15:56

Fix missing param in cli_setup/utils

6331e35

Merge branch 'main' into kbn-137734-http-agent-factory

9437590

gsoldevila merged commit a77f8ff into elastic:main Sep 14, 2022

gsoldevila mentioned this pull request Sep 22, 2022

Collect metrics about the active/idle connections to ES nodes #141434

Merged

		const HttpAgentMock = HttpAgent as jest.Mock<HttpAgent>;
		const HttpsAgentMock = HttpsAgent as jest.Mock<HttpsAgent>;

		): ClientOptions & { agent: HttpAgentOptions } {
		const clientOptions: ClientOptions & { agent: HttpAgentOptions } = {

Create HTTP Agent manager #137748

Create HTTP Agent manager #137748

Conversation

gsoldevila commented Aug 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 5, 2022

pgayvallet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsoldevila Sep 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgayvallet Sep 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsoldevila Sep 8, 2022 • edited

Choose a reason for hiding this comment

gsoldevila Sep 8, 2022 • edited

Choose a reason for hiding this comment

rudolf left a comment

Choose a reason for hiding this comment

pgayvallet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Sep 14, 2022

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

ESLint disabled line counts

Total ESLint disabled count

History

gsoldevila commented Aug 1, 2022 •

edited

gsoldevila Sep 7, 2022 •

edited

pgayvallet Sep 7, 2022 •

edited

gsoldevila Sep 8, 2022 •

edited

gsoldevila Sep 8, 2022 •

edited