Request Timeout when trying to add new data source

[kaiyan-sheng]

Kibana version:
master branch

Elasticsearch version:
master branch

Server OS version:
mac 10.15.4

Browser version:
Chrome Version 81.0.4044.138 (Official Build) (64-bit)

Steps to reproduce:

1. Pull the latest elastic/package-registry and run mage build
2. Under elastic/package-registry directory, run docker build --rm -t docker.elastic.co/package-registry/package-registry:master .
3. cd testing/environments
4. Run this command to start the test environment: docker-compose -f snapshot.yml -f local.yml up --force-recreate
5. In Kibana, enroll new agent:

6. Click default config:

7. Click create data source:

8. Choose `aws` in the integration list and click `Save data source`:

9. This should lead to the next screenshot after the timeout. 10. I also tried to change Kibana URL under global output from `http://0:5601`(default) to `http:localhost:5601` and same thing happend.

Screenshots:

Provide logs and/or server output (if relevant):

2020-05-12T12:07:49-06:00 DEBUG	fleet_gateway.go:166	FleetGateway is sleeping, next update in 30s
2020-05-12T12:08:24-06:00 DEBUG	fleet_gateway.go:146	FleetGateway calling Checkin API
2020-05-12T12:08:24-06:00 DEBUG	client.go:170	Request method: POST, path: /api/ingest_manager/fleet/agents/c2cc3aeb-6d15-4088-856c-73cb6e1a606d/checkin
2020-05-12T12:09:54-06:00 ERROR	fleet_gateway.go:180	Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post http://0:5601/api/ingest_manager/fleet/agents/c2cc3aeb-6d15-4088-856c-73cb6e1a606d/checkin?: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
	fail to checkin to fleet: Post http://0:5601/api/ingest_manager/fleet/agents/c2cc3aeb-6d15-4088-856c-73cb6e1a606d/checkin?: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2020-05-12T12:09:55-06:00 DEBUG	client.go:170	Request method: POST, path: /api/ingest_manager/fleet/agents/c2cc3aeb-6d15-4088-856c-73cb6e1a606d/checkin
#^C2020-05-12T12:10:46-06:00 INFO	managed_mode.go:211	Agent is stopped

I also tried to ping Kibana and it works seem like:

kaiyansheng@KaiyanMacBookPro:~$ curl -XGET localhost:9200 -u $USER:$PASSWORD{
  "name" : "4eb38faf6e2e",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "NeQPWzjPRFSc_S5kdvMdPA",
  "version" : {
    "number" : "8.0.0-SNAPSHOT",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "c9943698939ed733556193aa52d1099bb1f6cfd5",
    "build_date" : "2020-05-11T11:04:36.268539Z",
    "build_snapshot" : true,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "7.9.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@nchaulet Could you take a look? @kaiyan-sheng tried with the default datasource and it work, this seems related to the AWS configuration.

[nchaulet]

Looking right now

[kaiyan-sheng]

I was able to add a mysql data source on top of default system config. So maybe it's something wrong with the aws data source itself?

More log messages that are related:

kibana_1                 | {"type":"log","@timestamp":"2020-05-19T20:56:39+00:00","tags":["warning","elasticsearch","data"],"pid":8,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1                 | {"type":"log","@timestamp":"2020-05-19T20:56:39+00:00","tags":["warning","plugins","licensing"],"pid":8,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
kibana_1                 | {"type":"log","@timestamp":"2020-05-19T20:56:39+00:00","tags":["warning","elasticsearch","admin"],"pid":8,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1                 | {"type":"log","@timestamp":"2020-05-19T20:56:39+00:00","tags":["error","plugins","taskManager","taskManager"],"pid":8,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}

[nchaulet]

Locally it was working, but it took 21s, I am investigating which steps are really long and if we can do more things concurently.

[jfsiii]

@nchaulet I have been chasing slow startup / install time for #66125 I was just working on a branch to add APM debugging #67069

I installed aws locally and it took about 15s. I'm short on time to leave more details but here's the breakdown

It looks just like the issues with the system package in that draft PR, so the quick fixes I had in mind (ETA ~1 day) should apply here, too.

You're welcome to make your own improvements, of course or ping me on Slack / Zoom and we can chat.

[nchaulet]

Pefect @jfsiii I will to see if the fix for the system package fix this too.

[kaiyan-sheng]

Thank you for your help! Is there a workaround for this problem? For example, is there a setting to change the request timeout?

[jfsiii]

@kaiyan-sheng
I have added some tracing for insight into what's taking time. Here's the AWS installation

There can be long times but it always works. I cannot get to 30s once, let alone reliably. I am using yarn es snapshot -E xpack.security.authc.api_key.enabled=true for ES and yarn start --no-base-path --xpack.ingestManager.enabled=true for Kibana, I'll try the Docker steps you show in the original, but I don't know of a quick way to speed up the ES install times. I'm "hoping" there's some other network/JS/etc issue going on in your case.

[ph]

@nchaulet did you have time to look at this?

[nchaulet]

@kaiyan-sheng can you tell me what ressource did you allocated to docker for mac? Looks like giving more ressource allow to not have a timeout (6CPU on my end and 6GB of memory)

[kaiyan-sheng]

@kaiyan-sheng can you tell me what ressource did you allocated to docker for mac? Looks like giving more ressource allow to not have a timeout (6CPU on my end and 6GB of memory)

Thank you for your help @nchaulet! I have 6 CPU and 2GB memory. Let me increase the memory to 6GB and give it a try!

[kaiyan-sheng]

@nchaulet 6GB memory works! Thanks!

[nchaulet]

If anybody need a a repro you can run ES in a docker environment with limited ressources.

I think we can close this issue for now.