New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load ML using Kibana API #6604

Merged
merged 10 commits into from Mar 25, 2018

Conversation

Projects
None yet
5 participants
@kvch
Contributor

kvch commented Mar 20, 2018

Since Kibana 6.1 new ML API is available to load jobs, datafeeds, etc. Thus, it's not necessary to store ML configs in Beats and load them using setup.
From now on if Filebeat connects to a Kibana 6.1 or newer the Kibana API is called. If Kibana is older than 6.1, Filebeat loads the ML config from the repo, as it's done in previous versions.

Setting up ML for nginx:

./filebeat setup --machine-learning -modules=nginx
Show outdated Hide outdated libbeat/ml-importer/importer.go Outdated
Show outdated Hide outdated libbeat/ml-importer/importer.go Outdated
Show outdated Hide outdated libbeat/ml-importer/importer.go Outdated
Show outdated Hide outdated filebeat/fileset/modules.go Outdated
@kvch

This comment has been minimized.

Show comment
Hide comment
@kvch

kvch Mar 21, 2018

Contributor

The filebeat test failure is due to the regression I mentioned above.

Contributor

kvch commented Mar 21, 2018

The filebeat test failure is due to the regression I mentioned above.

Show outdated Hide outdated libbeat/ml-importer/importer.go Outdated
Show outdated Hide outdated filebeat/fileset/modules.go Outdated
Show outdated Hide outdated libbeat/ml-importer/importer.go Outdated
Show outdated Hide outdated filebeat/fileset/modules.go Outdated
@kvch

This comment has been minimized.

Show comment
Hide comment
@kvch

kvch Mar 21, 2018

Contributor

I added a workaround to make it possible to run ./filebeat setup without specifying modules for ML.

Also, found a minor mistake in estest, which is fixed now.

Contributor

kvch commented Mar 21, 2018

I added a workaround to make it possible to run ./filebeat setup without specifying modules for ML.

Also, found a minor mistake in estest, which is fixed now.

Show outdated Hide outdated filebeat/fileset/modules.go Outdated
Show outdated Hide outdated filebeat/fileset/modules.go Outdated

kvch added some commits Mar 21, 2018

@kvch

This comment has been minimized.

Show comment
Hide comment
@kvch

kvch Mar 22, 2018

Contributor

The failing tests are unrelated to the change.

Contributor

kvch commented Mar 22, 2018

The failing tests are unrelated to the change.

@kvch kvch removed the in progress label Mar 22, 2018

}
func isElasticsearchLoads(kibanaVersion *common.Version) bool {
if kibanaVersion.Major < 6 || kibanaVersion.Major == 6 && kibanaVersion.Minor < 1 {

This comment has been minimized.

@tsg

tsg Mar 22, 2018

Collaborator

Is it 6.1 the demarcation line, I seem to remember it was 6.2, but could be wrong. @jgowdyelastic, can you confirm from which version is the new way of loading ML jobs available?

@tsg

tsg Mar 22, 2018

Collaborator

Is it 6.1 the demarcation line, I seem to remember it was 6.2, but could be wrong. @jgowdyelastic, can you confirm from which version is the new way of loading ML jobs available?

This comment has been minimized.

@jgowdyelastic

jgowdyelastic Mar 22, 2018

Member

The ML endpoints were added in 6.1.

@jgowdyelastic

jgowdyelastic Mar 22, 2018

Member

The ML endpoints were added in 6.1.

This comment has been minimized.

@tsg

tsg Mar 22, 2018

Collaborator

Thank you!

@tsg

tsg Mar 22, 2018

Collaborator

Thank you!

@tsg

This comment has been minimized.

Show comment
Hide comment
@tsg

tsg Mar 22, 2018

Collaborator

I seem to be getting some errors when testing against ES/KB 6.2.1. It looks to me like adding the jobs works, but some associated dashboards are not created.

$ ./filebeat setup -E "setup.dashboards.directory=_meta/kibana"                             1 ↵
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Exiting: 2 errors: Error setting up ML for nginx: 9 errors: error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart; Error setting up ML for nginx: 19 errors: [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-visitor_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-response_code] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-low_request_rate] already exists; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-visitor_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-response_code'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_url_count'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_request_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-low_request_rate'. The Id is already used.; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart

The full log in debug mode follows, in case it's helpful:

$ ./filebeat setup  -e -d "*" -E "setup.dashboards.directory=_meta/kibana"                  1 ↵
2018-03-22T12:49:36.126+0100	INFO	instance/beat.go:479	Home path: [/Users/tsg/src/github.com/elastic/beats/filebeat] Config path: [/Users/tsg/src/github.com/elastic/beats/filebeat] Data path: [/Users/tsg/src/github.com/elastic/beats/filebeat/data] Logs path: [/Users/tsg/src/github.com/elastic/beats/filebeat/logs]
2018-03-22T12:49:36.127+0100	DEBUG	[beat]	instance/beat.go:506	Beat metadata path: /Users/tsg/src/github.com/elastic/beats/filebeat/data/meta.json
2018-03-22T12:49:36.127+0100	INFO	instance/beat.go:486	Beat UUID: 5bd96265-e2cb-4694-bea2-e35a1897f023
2018-03-22T12:49:36.127+0100	INFO	instance/beat.go:222	Setup Beat: filebeat; Version: 7.0.0-alpha1
2018-03-22T12:49:36.127+0100	DEBUG	[beat]	instance/beat.go:239	Initializing output plugins
2018-03-22T12:49:36.127+0100	DEBUG	[processors]	processors/processor.go:49	Processors:
2018-03-22T12:49:36.128+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:49:36.129+0100	INFO	pipeline/module.go:76	Beat name: where-is-my-esc-key.local
2018-03-22T12:49:36.130+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:49:36.131+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:666	ES Ping(url=http://localhost:9200)
2018-03-22T12:49:36.137+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:689	Ping status code: 200
2018-03-22T12:49:36.137+0100	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 6.2.1
2018-03-22T12:49:36.137+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	HEAD http://localhost:9200/_template/filebeat-7.0.0-alpha1  <nil>
2018-03-22T12:49:36.139+0100	INFO	template/load.go:55	Loading template for Elasticsearch version: 6.2.1
2018-03-22T12:49:36.151+0100	DEBUG	[template]	template/load.go:83	Try loading template with name: filebeat-7.0.0-alpha1
2018-03-22T12:49:36.152+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	PUT http://localhost:9200/_template/filebeat-7.0.0-alpha1  map[settings:{"index":{"mapping":{"total_fields":{"limit":10000}},"number_of_routing_shards":30,"number_of_shards":3,"refresh_interval":"5s"}} index_patterns:[filebeat-7.0.0-alpha1-*] mappings:{"doc":{"_meta":{"version":"7.0.0-alpha1"},"date_detection":false,"dynamic_templates":[{"fields":{"mapping":{"type":"keyword"},"match_mapping_type":"string","path_match":"fields.*"}},{"docker.container.labels":{"mapping":{"type":"keyword"},"match_mapping_type":"string","path_match":"docker.container.labels.*"}},{"strings_as_keyword":{"mapping":{"ignore_above":1024,"type":"keyword"},"match_mapping_type":"string"}}],"properties":{"@timestamp":{"type":"date"},"apache2":{"properties":{"access":{"properties":{"agent":{"norms":false,"type":"text"},"body_sent":{"properties":{"bytes":{"type":"long"}}},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"http_version":{"ignore_above":1024,"type":"keyword"},"method":{"ignore_above":1024,"type":"keyword"},"referrer":{"ignore_above":1024,"type":"keyword"},"remote_ip":{"ignore_above":1024,"type":"keyword"},"response_code":{"type":"long"},"url":{"ignore_above":1024,"type":"keyword"},"user_agent":{"properties":{"device":{"ignore_above":1024,"type":"keyword"},"major":{"type":"long"},"minor":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"},"os":{"ignore_above":1024,"type":"keyword"},"os_major":{"type":"long"},"os_minor":{"type":"long"},"os_name":{"ignore_above":1024,"type":"keyword"},"patch":{"ignore_above":1024,"type":"keyword"}}},"user_name":{"ignore_above":1024,"type":"keyword"}}},"error":{"properties":{"client":{"ignore_above":1024,"type":"keyword"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"module":{"ignore_above":1024,"type":"keyword"},"pid":{"type":"long"},"tid":{"type":"long"}}}}},"auditd":{"properties":{"log":{"properties":{"a0":{"ignore_above":1024,"type":"keyword"},"acct":{"ignore_above":1024,"type":"keyword"},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"item":{"ignore_above":1024,"type":"keyword"},"items":{"ignore_above":1024,"type":"keyword"},"new_auid":{"ignore_above":1024,"type":"keyword"},"new_ses":{"ignore_above":1024,"type":"keyword"},"old_auid":{"ignore_above":1024,"type":"keyword"},"old_ses":{"ignore_above":1024,"type":"keyword"},"pid":{"ignore_above":1024,"type":"keyword"},"ppid":{"ignore_above":1024,"type":"keyword"},"record_type":{"ignore_above":1024,"type":"keyword"},"res":{"ignore_above":1024,"type":"keyword"},"sequence":{"type":"long"}}}}},"beat":{"properties":{"hostname":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"},"timezone":{"ignore_above":1024,"type":"keyword"},"version":{"ignore_above":1024,"type":"keyword"}}},"docker":{"properties":{"container":{"properties":{"id":{"ignore_above":1024,"type":"keyword"},"image":{"ignore_above":1024,"type":"keyword"},"labels":{"type":"object"},"name":{"ignore_above":1024,"type":"keyword"}}}}},"error":{"properties":{"code":{"type":"long"},"message":{"norms":false,"type":"text"},"type":{"ignore_above":1024,"type":"keyword"}}},"fields":{"type":"object"},"fileset":{"properties":{"module":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"}}},"icinga":{"properties":{"debug":{"properties":{"facility":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"severity":{"ignore_above":1024,"type":"keyword"}}},"main":{"properties":{"facility":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"severity":{"ignore_above":1024,"type":"keyword"}}},"startup":{"properties":{"facility":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"severity":{"ignore_above":1024,"type":"keyword"}}}}},"kafka":{"properties":{"log":{"properties":{"class":{"norms":false,"type":"text"},"component":{"ignore_above":1024,"type":"keyword"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"timestamp":{"ignore_above":1024,"type":"keyword"},"trace":{"properties":{"class":{"ignore_above":1024,"type":"keyword"},"full":{"norms":false,"type":"text"},"message":{"norms":false,"type":"text"}}}}}}},"kubernetes":{"properties":{"annotations":{"type":"object"},"container":{"properties":{"image":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"}}},"labels":{"type":"object"},"namespace":{"ignore_above":1024,"type":"keyword"},"node":{"properties":{"name":{"ignore_above":1024,"type":"keyword"}}},"pod":{"properties":{"name":{"ignore_above":1024,"type":"keyword"}}}}},"logstash":{"properties":{"log":{"properties":{"level":{"ignore_above":1024,"type":"keyword"},"log_event":{"type":"object"},"message":{"norms":false,"type":"text"},"module":{"ignore_above":1024,"type":"keyword"},"thread":{"norms":false,"type":"text"}}},"slowlog":{"properties":{"event":{"norms":false,"type":"text"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"module":{"ignore_above":1024,"type":"keyword"},"plugin_name":{"ignore_above":1024,"type":"keyword"},"plugin_params":{"norms":false,"type":"text"},"plugin_params_object":{"type":"object"},"plugin_type":{"ignore_above":1024,"type":"keyword"},"thread":{"norms":false,"type":"text"},"took_in_millis":{"type":"long"},"took_in_nanos":{"type":"long"}}}}},"message":{"norms":false,"type":"text"},"meta":{"properties":{"cloud":{"properties":{"availability_zone":{"ignore_above":1024,"type":"keyword"},"instance_id":{"ignore_above":1024,"type":"keyword"},"instance_name":{"ignore_above":1024,"type":"keyword"},"machine_type":{"ignore_above":1024,"type":"keyword"},"project_id":{"ignore_above":1024,"type":"keyword"},"provider":{"ignore_above":1024,"type":"keyword"},"region":{"ignore_above":1024,"type":"keyword"}}}}},"mysql":{"properties":{"error":{"properties":{"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"thread_id":{"type":"long"},"timestamp":{"ignore_above":1024,"type":"keyword"}}},"slowlog":{"properties":{"host":{"ignore_above":1024,"type":"keyword"},"id":{"type":"long"},"ip":{"ignore_above":1024,"type":"keyword"},"lock_time":{"properties":{"sec":{"type":"float"}}},"query":{"ignore_above":1024,"type":"keyword"},"query_time":{"properties":{"sec":{"type":"float"}}},"rows_examined":{"type":"long"},"rows_sent":{"type":"long"},"timestamp":{"type":"long"},"user":{"ignore_above":1024,"type":"keyword"}}}}},"nginx":{"properties":{"access":{"properties":{"agent":{"norms":false,"type":"text"},"body_sent":{"properties":{"bytes":{"type":"long"}}},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"http_version":{"ignore_above":1024,"type":"keyword"},"method":{"ignore_above":1024,"type":"keyword"},"referrer":{"ignore_above":1024,"type":"keyword"},"remote_ip":{"ignore_above":1024,"type":"keyword"},"response_code":{"type":"long"},"url":{"ignore_above":1024,"type":"keyword"},"user_agent":{"properties":{"device":{"ignore_above":1024,"type":"keyword"},"major":{"type":"long"},"minor":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"},"os":{"ignore_above":1024,"type":"keyword"},"os_major":{"type":"long"},"os_minor":{"type":"long"},"os_name":{"ignore_above":1024,"type":"keyword"},"patch":{"ignore_above":1024,"type":"keyword"}}},"user_name":{"ignore_above":1024,"type":"keyword"}}},"error":{"properties":{"connection_id":{"type":"long"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"pid":{"type":"long"},"tid":{"type":"long"}}}}},"offset":{"type":"long"},"osquery":{"properties":{"result":{"properties":{"action":{"ignore_above":1024,"type":"keyword"},"calendar_time":{"ignore_above":1024,"type":"keyword"},"host_identifier":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"},"unix_time":{"type":"long"}}}}},"postgresql":{"properties":{"log":{"properties":{"database":{"ignore_above":1024,"type":"keyword"},"duration":{"type":"float"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"query":{"ignore_above":1024,"type":"keyword"},"thread_id":{"type":"long"},"timestamp":{"ignore_above":1024,"type":"keyword"},"timezone":{"ignore_above":1024,"type":"keyword"},"user":{"ignore_above":1024,"type":"keyword"}}}}},"prospector":{"properties":{"type":{"ignore_above":1024,"type":"keyword"}}},"read_timestamp":{"ignore_above":1024,"type":"keyword"},"redis":{"properties":{"log":{"properties":{"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"pid":{"type":"long"},"role":{"ignore_above":1024,"type":"keyword"}}},"slowlog":{"properties":{"args":{"ignore_above":1024,"type":"keyword"},"cmd":{"ignore_above":1024,"type":"keyword"},"duration":{"properties":{"us":{"type":"long"}}},"id":{"type":"long"},"key":{"ignore_above":1024,"type":"keyword"}}}}},"source":{"ignore_above":1024,"type":"keyword"},"stream":{"ignore_above":1024,"type":"keyword"},"system":{"properties":{"auth":{"properties":{"groupadd":{"properties":{"gid":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"}}},"hostname":{"ignore_above":1024,"type":"keyword"},"message":{"ignore_above":1024,"type":"keyword"},"pid":{"type":"long"},"program":{"ignore_above":1024,"type":"keyword"},"ssh":{"properties":{"dropped_ip":{"type":"ip"},"event":{"ignore_above":1024,"type":"keyword"},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"ip":{"type":"ip"},"method":{"ignore_above":1024,"type":"keyword"},"port":{"type":"long"},"signature":{"ignore_above":1024,"type":"keyword"}}},"sudo":{"properties":{"command":{"ignore_above":1024,"type":"keyword"},"error":{"ignore_above":1024,"type":"keyword"},"pwd":{"ignore_above":1024,"type":"keyword"},"tty":{"ignore_above":1024,"type":"keyword"},"user":{"ignore_above":1024,"type":"keyword"}}},"timestamp":{"ignore_above":1024,"type":"keyword"},"user":{"ignore_above":1024,"type":"keyword"},"useradd":{"properties":{"gid":{"type":"long"},"home":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"},"shell":{"ignore_above":1024,"type":"keyword"},"uid":{"type":"long"}}}}},"syslog":{"properties":{"hostname":{"ignore_above":1024,"type":"keyword"},"message":{"ignore_above":1024,"type":"keyword"},"pid":{"ignore_above":1024,"type":"keyword"},"program":{"ignore_above":1024,"type":"keyword"},"timestamp":{"ignore_above":1024,"type":"keyword"}}}}},"tags":{"ignore_above":1024,"type":"keyword"},"traefik":{"properties":{"access":{"properties":{"agent":{"norms":false,"type":"text"},"backend_url":{"norms":false,"type":"text"},"body_sent":{"properties":{"bytes":{"type":"long"}}},"frontend_name":{"norms":false,"type":"text"},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"http_version":{"ignore_above":1024,"type":"keyword"},"method":{"ignore_above":1024,"type":"keyword"},"referrer":{"ignore_above":1024,"type":"keyword"},"remote_ip":{"ignore_above":1024,"type":"keyword"},"request_count":{"type":"long"},"response_code":{"type":"long"},"url":{"ignore_above":1024,"type":"keyword"},"user_agent":{"properties":{"device":{"ignore_above":1024,"type":"keyword"},"major":{"type":"long"},"minor":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"},"os":{"ignore_above":1024,"type":"keyword"},"os_major":{"type":"long"},"os_minor":{"type":"long"},"os_name":{"ignore_above":1024,"type":"keyword"},"patch":{"ignore_above":1024,"type":"keyword"}}},"user_name":{"ignore_above":1024,"type":"keyword"}}}}}}}} order:1]
2018-03-22T12:49:36.175+0100	INFO	template/load.go:89	Elasticsearch template with name 'filebeat-7.0.0-alpha1' loaded
Loaded index template
Loading dashboards (Kibana must be running and reachable)
2018-03-22T12:49:36.175+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:49:36.175+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:666	ES Ping(url=http://localhost:9200)
2018-03-22T12:49:36.179+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:689	Ping status code: 200
2018-03-22T12:49:36.179+0100	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 6.2.1
2018-03-22T12:49:36.179+0100	DEBUG	[dashboards]	dashboards/es_loader.go:309	Initialize the Elasticsearch 6.2.1 loader
2018-03-22T12:49:36.179+0100	DEBUG	[dashboards]	dashboards/es_loader.go:309	Elasticsearch URL http://localhost:9200
2018-03-22T12:49:36.179+0100	INFO	kibana/client.go:90	Kibana url: http://localhost:5601
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Initialize the Kibana 6.2.1 loader
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Kibana URL http://localhost:5601
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Importing directory _meta/kibana/6
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import directory _meta/kibana/6
2018-03-22T12:49:36.191+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import index-pattern from _meta/kibana/6/index-pattern/filebeat.json

2018-03-22T12:49:37.361+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import directory _meta/kibana/6
2018-03-22T12:49:37.364+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-Kafka-overview.json

2018-03-22T12:49:38.377+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-Postgresql-overview.json

2018-03-22T12:49:39.397+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-Postgresql-slowlogs.json

2018-03-22T12:49:40.417+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-apache2.json

2018-03-22T12:49:41.442+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-auditd.json

2018-03-22T12:49:42.471+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-auth-sudo-commands.json

2018-03-22T12:49:43.480+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-icinga-debug-log.json

2018-03-22T12:49:44.511+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-icinga-main-log.json

2018-03-22T12:49:45.521+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-icinga-startup-errors.json

2018-03-22T12:49:46.547+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-logstash-log.json

2018-03-22T12:49:47.576+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-logstash-slowlog.json

2018-03-22T12:49:48.605+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-mysql.json

2018-03-22T12:49:49.614+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-new-users-and-groups.json

2018-03-22T12:49:50.632+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-nginx-logs.json

2018-03-22T12:49:51.662+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-nginx-overview.json

2018-03-22T12:49:52.681+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-redis.json

2018-03-22T12:49:53.699+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-ssh-login-attempts.json

2018-03-22T12:49:54.727+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-syslog.json

2018-03-22T12:49:55.736+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-traefik-overview.json

2018-03-22T12:49:56.757+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-nginx-access-remote-ip-count-explorer.json

2018-03-22T12:49:57.776+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-nginx-remote-ip-url-explorer.json

2018-03-22T12:49:58.803+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-traefik-access-remote-ip-count-explorer.json

2018-03-22T12:49:59.813+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-traefik-remote-ip-url-explorer.json

2018-03-22T12:50:00.842+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/osquery-compliance.json

2018-03-22T12:50:01.849+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/osquery-rootkit.json

2018-03-22T12:50:02.867+0100	INFO	instance/beat.go:594	Kibana dashboards successfully loaded.
Loaded dashboards
2018-03-22T12:50:02.867+0100	DEBUG	[machine-learning]	beater/filebeat.go:151	Setting up ML jobs for modules
2018-03-22T12:50:02.868+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:50:02.868+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:666	ES Ping(url=http://localhost:9200)
2018-03-22T12:50:02.872+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:689	Ping status code: 200
2018-03-22T12:50:02.872+0100	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 6.2.1
2018-03-22T12:50:02.872+0100	INFO	kibana/client.go:90	Kibana url: http://localhost:5601
2018-03-22T12:50:02.893+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	GET http://localhost:9200/_xpack  <nil>
2018-03-22T12:50:05.296+0100	DEBUG	[cfgfile]	cfgfile/cfgfile.go:143	Load config from file: /Users/tsg/src/github.com/elastic/beats/filebeat/modules.d/nginx.yml
2018-03-22T12:50:05.298+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	GET http://localhost:9200/_xpack  <nil>
2018-03-22T12:50:05.476+0100	ERROR	instance/beat.go:678	Exiting: 2 errors: Error setting up ML for nginx: 9 errors: error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart; Error setting up ML for nginx: 19 errors: [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-visitor_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-low_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-response_code] already exists; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-visitor_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-response_code'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_url_count'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_request_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-low_request_rate'. The Id is already used.; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart
Exiting: 2 errors: Error setting up ML for nginx: 9 errors: error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart; Error setting up ML for nginx: 19 errors: [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-visitor_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-low_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-response_code] already exists; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-visitor_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-response_code'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_url_count'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_request_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-low_request_rate'. The Id is already used.; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart

I should add that this happens on a completely fresh installation of ES/KB.

Collaborator

tsg commented Mar 22, 2018

I seem to be getting some errors when testing against ES/KB 6.2.1. It looks to me like adding the jobs works, but some associated dashboards are not created.

$ ./filebeat setup -E "setup.dashboards.directory=_meta/kibana"                             1 ↵
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Exiting: 2 errors: Error setting up ML for nginx: 9 errors: error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart; Error setting up ML for nginx: 19 errors: [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-visitor_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-response_code] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-low_request_rate] already exists; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-visitor_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-response_code'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_url_count'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_request_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-low_request_rate'. The Id is already used.; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart

The full log in debug mode follows, in case it's helpful:

$ ./filebeat setup  -e -d "*" -E "setup.dashboards.directory=_meta/kibana"                  1 ↵
2018-03-22T12:49:36.126+0100	INFO	instance/beat.go:479	Home path: [/Users/tsg/src/github.com/elastic/beats/filebeat] Config path: [/Users/tsg/src/github.com/elastic/beats/filebeat] Data path: [/Users/tsg/src/github.com/elastic/beats/filebeat/data] Logs path: [/Users/tsg/src/github.com/elastic/beats/filebeat/logs]
2018-03-22T12:49:36.127+0100	DEBUG	[beat]	instance/beat.go:506	Beat metadata path: /Users/tsg/src/github.com/elastic/beats/filebeat/data/meta.json
2018-03-22T12:49:36.127+0100	INFO	instance/beat.go:486	Beat UUID: 5bd96265-e2cb-4694-bea2-e35a1897f023
2018-03-22T12:49:36.127+0100	INFO	instance/beat.go:222	Setup Beat: filebeat; Version: 7.0.0-alpha1
2018-03-22T12:49:36.127+0100	DEBUG	[beat]	instance/beat.go:239	Initializing output plugins
2018-03-22T12:49:36.127+0100	DEBUG	[processors]	processors/processor.go:49	Processors:
2018-03-22T12:49:36.128+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:49:36.129+0100	INFO	pipeline/module.go:76	Beat name: where-is-my-esc-key.local
2018-03-22T12:49:36.130+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:49:36.131+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:666	ES Ping(url=http://localhost:9200)
2018-03-22T12:49:36.137+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:689	Ping status code: 200
2018-03-22T12:49:36.137+0100	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 6.2.1
2018-03-22T12:49:36.137+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	HEAD http://localhost:9200/_template/filebeat-7.0.0-alpha1  <nil>
2018-03-22T12:49:36.139+0100	INFO	template/load.go:55	Loading template for Elasticsearch version: 6.2.1
2018-03-22T12:49:36.151+0100	DEBUG	[template]	template/load.go:83	Try loading template with name: filebeat-7.0.0-alpha1
2018-03-22T12:49:36.152+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	PUT http://localhost:9200/_template/filebeat-7.0.0-alpha1  map[settings:{"index":{"mapping":{"total_fields":{"limit":10000}},"number_of_routing_shards":30,"number_of_shards":3,"refresh_interval":"5s"}} index_patterns:[filebeat-7.0.0-alpha1-*] mappings:{"doc":{"_meta":{"version":"7.0.0-alpha1"},"date_detection":false,"dynamic_templates":[{"fields":{"mapping":{"type":"keyword"},"match_mapping_type":"string","path_match":"fields.*"}},{"docker.container.labels":{"mapping":{"type":"keyword"},"match_mapping_type":"string","path_match":"docker.container.labels.*"}},{"strings_as_keyword":{"mapping":{"ignore_above":1024,"type":"keyword"},"match_mapping_type":"string"}}],"properties":{"@timestamp":{"type":"date"},"apache2":{"properties":{"access":{"properties":{"agent":{"norms":false,"type":"text"},"body_sent":{"properties":{"bytes":{"type":"long"}}},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"http_version":{"ignore_above":1024,"type":"keyword"},"method":{"ignore_above":1024,"type":"keyword"},"referrer":{"ignore_above":1024,"type":"keyword"},"remote_ip":{"ignore_above":1024,"type":"keyword"},"response_code":{"type":"long"},"url":{"ignore_above":1024,"type":"keyword"},"user_agent":{"properties":{"device":{"ignore_above":1024,"type":"keyword"},"major":{"type":"long"},"minor":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"},"os":{"ignore_above":1024,"type":"keyword"},"os_major":{"type":"long"},"os_minor":{"type":"long"},"os_name":{"ignore_above":1024,"type":"keyword"},"patch":{"ignore_above":1024,"type":"keyword"}}},"user_name":{"ignore_above":1024,"type":"keyword"}}},"error":{"properties":{"client":{"ignore_above":1024,"type":"keyword"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"module":{"ignore_above":1024,"type":"keyword"},"pid":{"type":"long"},"tid":{"type":"long"}}}}},"auditd":{"properties":{"log":{"properties":{"a0":{"ignore_above":1024,"type":"keyword"},"acct":{"ignore_above":1024,"type":"keyword"},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"item":{"ignore_above":1024,"type":"keyword"},"items":{"ignore_above":1024,"type":"keyword"},"new_auid":{"ignore_above":1024,"type":"keyword"},"new_ses":{"ignore_above":1024,"type":"keyword"},"old_auid":{"ignore_above":1024,"type":"keyword"},"old_ses":{"ignore_above":1024,"type":"keyword"},"pid":{"ignore_above":1024,"type":"keyword"},"ppid":{"ignore_above":1024,"type":"keyword"},"record_type":{"ignore_above":1024,"type":"keyword"},"res":{"ignore_above":1024,"type":"keyword"},"sequence":{"type":"long"}}}}},"beat":{"properties":{"hostname":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"},"timezone":{"ignore_above":1024,"type":"keyword"},"version":{"ignore_above":1024,"type":"keyword"}}},"docker":{"properties":{"container":{"properties":{"id":{"ignore_above":1024,"type":"keyword"},"image":{"ignore_above":1024,"type":"keyword"},"labels":{"type":"object"},"name":{"ignore_above":1024,"type":"keyword"}}}}},"error":{"properties":{"code":{"type":"long"},"message":{"norms":false,"type":"text"},"type":{"ignore_above":1024,"type":"keyword"}}},"fields":{"type":"object"},"fileset":{"properties":{"module":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"}}},"icinga":{"properties":{"debug":{"properties":{"facility":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"severity":{"ignore_above":1024,"type":"keyword"}}},"main":{"properties":{"facility":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"severity":{"ignore_above":1024,"type":"keyword"}}},"startup":{"properties":{"facility":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"severity":{"ignore_above":1024,"type":"keyword"}}}}},"kafka":{"properties":{"log":{"properties":{"class":{"norms":false,"type":"text"},"component":{"ignore_above":1024,"type":"keyword"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"timestamp":{"ignore_above":1024,"type":"keyword"},"trace":{"properties":{"class":{"ignore_above":1024,"type":"keyword"},"full":{"norms":false,"type":"text"},"message":{"norms":false,"type":"text"}}}}}}},"kubernetes":{"properties":{"annotations":{"type":"object"},"container":{"properties":{"image":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"}}},"labels":{"type":"object"},"namespace":{"ignore_above":1024,"type":"keyword"},"node":{"properties":{"name":{"ignore_above":1024,"type":"keyword"}}},"pod":{"properties":{"name":{"ignore_above":1024,"type":"keyword"}}}}},"logstash":{"properties":{"log":{"properties":{"level":{"ignore_above":1024,"type":"keyword"},"log_event":{"type":"object"},"message":{"norms":false,"type":"text"},"module":{"ignore_above":1024,"type":"keyword"},"thread":{"norms":false,"type":"text"}}},"slowlog":{"properties":{"event":{"norms":false,"type":"text"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"module":{"ignore_above":1024,"type":"keyword"},"plugin_name":{"ignore_above":1024,"type":"keyword"},"plugin_params":{"norms":false,"type":"text"},"plugin_params_object":{"type":"object"},"plugin_type":{"ignore_above":1024,"type":"keyword"},"thread":{"norms":false,"type":"text"},"took_in_millis":{"type":"long"},"took_in_nanos":{"type":"long"}}}}},"message":{"norms":false,"type":"text"},"meta":{"properties":{"cloud":{"properties":{"availability_zone":{"ignore_above":1024,"type":"keyword"},"instance_id":{"ignore_above":1024,"type":"keyword"},"instance_name":{"ignore_above":1024,"type":"keyword"},"machine_type":{"ignore_above":1024,"type":"keyword"},"project_id":{"ignore_above":1024,"type":"keyword"},"provider":{"ignore_above":1024,"type":"keyword"},"region":{"ignore_above":1024,"type":"keyword"}}}}},"mysql":{"properties":{"error":{"properties":{"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"thread_id":{"type":"long"},"timestamp":{"ignore_above":1024,"type":"keyword"}}},"slowlog":{"properties":{"host":{"ignore_above":1024,"type":"keyword"},"id":{"type":"long"},"ip":{"ignore_above":1024,"type":"keyword"},"lock_time":{"properties":{"sec":{"type":"float"}}},"query":{"ignore_above":1024,"type":"keyword"},"query_time":{"properties":{"sec":{"type":"float"}}},"rows_examined":{"type":"long"},"rows_sent":{"type":"long"},"timestamp":{"type":"long"},"user":{"ignore_above":1024,"type":"keyword"}}}}},"nginx":{"properties":{"access":{"properties":{"agent":{"norms":false,"type":"text"},"body_sent":{"properties":{"bytes":{"type":"long"}}},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"http_version":{"ignore_above":1024,"type":"keyword"},"method":{"ignore_above":1024,"type":"keyword"},"referrer":{"ignore_above":1024,"type":"keyword"},"remote_ip":{"ignore_above":1024,"type":"keyword"},"response_code":{"type":"long"},"url":{"ignore_above":1024,"type":"keyword"},"user_agent":{"properties":{"device":{"ignore_above":1024,"type":"keyword"},"major":{"type":"long"},"minor":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"},"os":{"ignore_above":1024,"type":"keyword"},"os_major":{"type":"long"},"os_minor":{"type":"long"},"os_name":{"ignore_above":1024,"type":"keyword"},"patch":{"ignore_above":1024,"type":"keyword"}}},"user_name":{"ignore_above":1024,"type":"keyword"}}},"error":{"properties":{"connection_id":{"type":"long"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"pid":{"type":"long"},"tid":{"type":"long"}}}}},"offset":{"type":"long"},"osquery":{"properties":{"result":{"properties":{"action":{"ignore_above":1024,"type":"keyword"},"calendar_time":{"ignore_above":1024,"type":"keyword"},"host_identifier":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"},"unix_time":{"type":"long"}}}}},"postgresql":{"properties":{"log":{"properties":{"database":{"ignore_above":1024,"type":"keyword"},"duration":{"type":"float"},"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"query":{"ignore_above":1024,"type":"keyword"},"thread_id":{"type":"long"},"timestamp":{"ignore_above":1024,"type":"keyword"},"timezone":{"ignore_above":1024,"type":"keyword"},"user":{"ignore_above":1024,"type":"keyword"}}}}},"prospector":{"properties":{"type":{"ignore_above":1024,"type":"keyword"}}},"read_timestamp":{"ignore_above":1024,"type":"keyword"},"redis":{"properties":{"log":{"properties":{"level":{"ignore_above":1024,"type":"keyword"},"message":{"norms":false,"type":"text"},"pid":{"type":"long"},"role":{"ignore_above":1024,"type":"keyword"}}},"slowlog":{"properties":{"args":{"ignore_above":1024,"type":"keyword"},"cmd":{"ignore_above":1024,"type":"keyword"},"duration":{"properties":{"us":{"type":"long"}}},"id":{"type":"long"},"key":{"ignore_above":1024,"type":"keyword"}}}}},"source":{"ignore_above":1024,"type":"keyword"},"stream":{"ignore_above":1024,"type":"keyword"},"system":{"properties":{"auth":{"properties":{"groupadd":{"properties":{"gid":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"}}},"hostname":{"ignore_above":1024,"type":"keyword"},"message":{"ignore_above":1024,"type":"keyword"},"pid":{"type":"long"},"program":{"ignore_above":1024,"type":"keyword"},"ssh":{"properties":{"dropped_ip":{"type":"ip"},"event":{"ignore_above":1024,"type":"keyword"},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"ip":{"type":"ip"},"method":{"ignore_above":1024,"type":"keyword"},"port":{"type":"long"},"signature":{"ignore_above":1024,"type":"keyword"}}},"sudo":{"properties":{"command":{"ignore_above":1024,"type":"keyword"},"error":{"ignore_above":1024,"type":"keyword"},"pwd":{"ignore_above":1024,"type":"keyword"},"tty":{"ignore_above":1024,"type":"keyword"},"user":{"ignore_above":1024,"type":"keyword"}}},"timestamp":{"ignore_above":1024,"type":"keyword"},"user":{"ignore_above":1024,"type":"keyword"},"useradd":{"properties":{"gid":{"type":"long"},"home":{"ignore_above":1024,"type":"keyword"},"name":{"ignore_above":1024,"type":"keyword"},"shell":{"ignore_above":1024,"type":"keyword"},"uid":{"type":"long"}}}}},"syslog":{"properties":{"hostname":{"ignore_above":1024,"type":"keyword"},"message":{"ignore_above":1024,"type":"keyword"},"pid":{"ignore_above":1024,"type":"keyword"},"program":{"ignore_above":1024,"type":"keyword"},"timestamp":{"ignore_above":1024,"type":"keyword"}}}}},"tags":{"ignore_above":1024,"type":"keyword"},"traefik":{"properties":{"access":{"properties":{"agent":{"norms":false,"type":"text"},"backend_url":{"norms":false,"type":"text"},"body_sent":{"properties":{"bytes":{"type":"long"}}},"frontend_name":{"norms":false,"type":"text"},"geoip":{"properties":{"city_name":{"ignore_above":1024,"type":"keyword"},"continent_name":{"ignore_above":1024,"type":"keyword"},"country_iso_code":{"ignore_above":1024,"type":"keyword"},"location":{"type":"geo_point"},"region_name":{"ignore_above":1024,"type":"keyword"}}},"http_version":{"ignore_above":1024,"type":"keyword"},"method":{"ignore_above":1024,"type":"keyword"},"referrer":{"ignore_above":1024,"type":"keyword"},"remote_ip":{"ignore_above":1024,"type":"keyword"},"request_count":{"type":"long"},"response_code":{"type":"long"},"url":{"ignore_above":1024,"type":"keyword"},"user_agent":{"properties":{"device":{"ignore_above":1024,"type":"keyword"},"major":{"type":"long"},"minor":{"type":"long"},"name":{"ignore_above":1024,"type":"keyword"},"os":{"ignore_above":1024,"type":"keyword"},"os_major":{"type":"long"},"os_minor":{"type":"long"},"os_name":{"ignore_above":1024,"type":"keyword"},"patch":{"ignore_above":1024,"type":"keyword"}}},"user_name":{"ignore_above":1024,"type":"keyword"}}}}}}}} order:1]
2018-03-22T12:49:36.175+0100	INFO	template/load.go:89	Elasticsearch template with name 'filebeat-7.0.0-alpha1' loaded
Loaded index template
Loading dashboards (Kibana must be running and reachable)
2018-03-22T12:49:36.175+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:49:36.175+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:666	ES Ping(url=http://localhost:9200)
2018-03-22T12:49:36.179+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:689	Ping status code: 200
2018-03-22T12:49:36.179+0100	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 6.2.1
2018-03-22T12:49:36.179+0100	DEBUG	[dashboards]	dashboards/es_loader.go:309	Initialize the Elasticsearch 6.2.1 loader
2018-03-22T12:49:36.179+0100	DEBUG	[dashboards]	dashboards/es_loader.go:309	Elasticsearch URL http://localhost:9200
2018-03-22T12:49:36.179+0100	INFO	kibana/client.go:90	Kibana url: http://localhost:5601
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Initialize the Kibana 6.2.1 loader
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Kibana URL http://localhost:5601
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Importing directory _meta/kibana/6
2018-03-22T12:49:36.190+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import directory _meta/kibana/6
2018-03-22T12:49:36.191+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import index-pattern from _meta/kibana/6/index-pattern/filebeat.json

2018-03-22T12:49:37.361+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import directory _meta/kibana/6
2018-03-22T12:49:37.364+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-Kafka-overview.json

2018-03-22T12:49:38.377+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-Postgresql-overview.json

2018-03-22T12:49:39.397+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-Postgresql-slowlogs.json

2018-03-22T12:49:40.417+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-apache2.json

2018-03-22T12:49:41.442+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-auditd.json

2018-03-22T12:49:42.471+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-auth-sudo-commands.json

2018-03-22T12:49:43.480+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-icinga-debug-log.json

2018-03-22T12:49:44.511+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-icinga-main-log.json

2018-03-22T12:49:45.521+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-icinga-startup-errors.json

2018-03-22T12:49:46.547+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-logstash-log.json

2018-03-22T12:49:47.576+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-logstash-slowlog.json

2018-03-22T12:49:48.605+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-mysql.json

2018-03-22T12:49:49.614+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-new-users-and-groups.json

2018-03-22T12:49:50.632+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-nginx-logs.json

2018-03-22T12:49:51.662+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-nginx-overview.json

2018-03-22T12:49:52.681+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-redis.json

2018-03-22T12:49:53.699+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-ssh-login-attempts.json

2018-03-22T12:49:54.727+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-syslog.json

2018-03-22T12:49:55.736+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/Filebeat-traefik-overview.json

2018-03-22T12:49:56.757+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-nginx-access-remote-ip-count-explorer.json

2018-03-22T12:49:57.776+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-nginx-remote-ip-url-explorer.json

2018-03-22T12:49:58.803+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-traefik-access-remote-ip-count-explorer.json

2018-03-22T12:49:59.813+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/ml-traefik-remote-ip-url-explorer.json

2018-03-22T12:50:00.842+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/osquery-compliance.json

2018-03-22T12:50:01.849+0100	DEBUG	[dashboards]	dashboards/kibana_loader.go:121	Import dashboard from _meta/kibana/6/dashboard/osquery-rootkit.json

2018-03-22T12:50:02.867+0100	INFO	instance/beat.go:594	Kibana dashboards successfully loaded.
Loaded dashboards
2018-03-22T12:50:02.867+0100	DEBUG	[machine-learning]	beater/filebeat.go:151	Setting up ML jobs for modules
2018-03-22T12:50:02.868+0100	INFO	elasticsearch/client.go:145	Elasticsearch url: http://localhost:9200
2018-03-22T12:50:02.868+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:666	ES Ping(url=http://localhost:9200)
2018-03-22T12:50:02.872+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:689	Ping status code: 200
2018-03-22T12:50:02.872+0100	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 6.2.1
2018-03-22T12:50:02.872+0100	INFO	kibana/client.go:90	Kibana url: http://localhost:5601
2018-03-22T12:50:02.893+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	GET http://localhost:9200/_xpack  <nil>
2018-03-22T12:50:05.296+0100	DEBUG	[cfgfile]	cfgfile/cfgfile.go:143	Load config from file: /Users/tsg/src/github.com/elastic/beats/filebeat/modules.d/nginx.yml
2018-03-22T12:50:05.298+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:708	GET http://localhost:9200/_xpack  <nil>
2018-03-22T12:50:05.476+0100	ERROR	instance/beat.go:678	Exiting: 2 errors: Error setting up ML for nginx: 9 errors: error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart; Error setting up ML for nginx: 19 errors: [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-visitor_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-low_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-response_code] already exists; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-visitor_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-response_code'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_url_count'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_request_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-low_request_rate'. The Id is already used.; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart
Exiting: 2 errors: Error setting up ML for nginx: 9 errors: error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart; Error setting up ML for nginx: 19 errors: [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-visitor_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-low_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists; [resource_already_exists_exception] A datafeed with id [datafeed-filebeat-nginx-access-response_code] already exists; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-visitor_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-response_code'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_url_count'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-remote_ip_request_rate'. The Id is already used.; [resource_already_exists_exception] The job cannot be created with the Id 'filebeat-nginx-access-low_request_rate'. The Id is already used.; error while setting up dashboard: ML-Nginx-Access-Remote-IP-Count-Explorer; error while setting up dashboard: ML-Nginx-Remote-IP-URL-Explorer; error while setting up search: ML-Filebeat-Nginx-Access; error while setting up visualization: ML-Nginx-Access-Map; error while setting up visualization: ML-Nginx-Access-Remote-IP-Timechart; error while setting up visualization: ML-Nginx-Access-Response-Code-Timechart; error while setting up visualization: ML-Nginx-Access-Top-Remote-IPs-Table; error while setting up visualization: ML-Nginx-Access-Top-URLs-Table; error while setting up visualization: ML-Nginx-Access-Unique-Count-URL-Timechart

I should add that this happens on a completely fresh installation of ES/KB.

@kvch

This comment has been minimized.

Show comment
Hide comment
@kvch

kvch Mar 22, 2018

Contributor

@tsg The problems you encountered must be fixed now.

Contributor

kvch commented Mar 22, 2018

@tsg The problems you encountered must be fixed now.

@tsg

This comment has been minimized.

Show comment
Hide comment
@tsg

tsg Mar 22, 2018

Collaborator

@kvch I tested again and looks good!

Collaborator

tsg commented Mar 22, 2018

@kvch I tested again and looks good!

for _, feed := range resp.Datafeeds {
if !feed.Success {
if strings.HasPrefix(feed.Error.Msg, "[resource_already_exists_exception]") {

This comment has been minimized.

@andrewkroh

andrewkroh Mar 23, 2018

Member

My intuition here is that these types of checks are prone to breakages. I don't know how well defined the API spec is for this interface, but my guess is that error message is not documented to begin with this string so that could change.

I think it's OK to do these types of checks when there are no other options, but we should have tests in place to detect upstream changes in the error messages. Are there already tests for these cases? I'm not seeing any coverage in the report: https://codecov.io/gh/elastic/beats/pull/6604/diff#D1-219

@andrewkroh

andrewkroh Mar 23, 2018

Member

My intuition here is that these types of checks are prone to breakages. I don't know how well defined the API spec is for this interface, but my guess is that error message is not documented to begin with this string so that could change.

I think it's OK to do these types of checks when there are no other options, but we should have tests in place to detect upstream changes in the error messages. Are there already tests for these cases? I'm not seeing any coverage in the report: https://codecov.io/gh/elastic/beats/pull/6604/diff#D1-219

This comment has been minimized.

@kvch

kvch Mar 23, 2018

Contributor

I agree, this part is quite dirty. Unfortunately, I haven't been able to find anything else which would tell the reason of the failure.

I will add tests to avoid breaking anything.

@kvch

kvch Mar 23, 2018

Contributor

I agree, this part is quite dirty. Unfortunately, I haven't been able to find anything else which would tell the reason of the failure.

I will add tests to avoid breaking anything.

Show outdated Hide outdated libbeat/ml-importer/importer.go Outdated
@andrewkroh

We don’t have to hold this up for tests. If you want they can be added in another PR.

@tsg tsg merged commit 6e27e4c into elastic:master Mar 25, 2018

2 of 3 checks passed

continuous-integration/travis-ci/pr The Travis CI build failed
Details
beats-ci Build finished.
Details
hound No violations found. Woof!
@tsg

This comment has been minimized.

Show comment
Hide comment
@tsg

tsg Mar 25, 2018

Collaborator

I merged the PR, @kvch please remember to do a follow up on tests.

Collaborator

tsg commented Mar 25, 2018

I merged the PR, @kvch please remember to do a follow up on tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment