Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add System Package #390

Merged
merged 4 commits into from May 5, 2020
Merged

Conversation

fearful-symmetry
Copy link
Contributor

I'm going over the guide here: https://github.com/elastic/package-registry/blob/master/CONTRIBUTING.md#run-the-whole-setup

This is just the result of env PACKAGES=system mage ImportBeats. I can't really get any farther, as when I try to register and start the agent, filebeat crashes and I can't get any farther.

Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest you best add 0.0.2 of the package. And lets get your FB issues fixed @michalpristas

@mtojek
Copy link
Contributor

mtojek commented Apr 28, 2020

I apologize if it's not clear in the instruction (let me know if it is), but in general package data in dev/packages/beats are volatile and can be changed with every run of mage ImportBeats. Once you consider that you're ready for manual modifications, please copy whole system package to the dev/packages/alpha and continue operating on this one.

Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please post any errors (include agent's), so we can jump on the root cause analysis.

@fearful-symmetry
Copy link
Contributor Author

Here's the errors I'm seeing in the agent:

2020-04-28T09:45:06-07:00 DEBUG	operator.go:230	running operation 'operation-config' for filebeat.8.0.0
2020-04-28T09:45:43-07:00 ERROR	reporter.go:47	2020-04-28T09:45:43-07:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: filebeat[9c1da1fc-baa2-4756-8c07-372143a7b74c]: application 'filebeat--8.0.0' crashed: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/plugin/app/app.go[206]: unknown error
2020-04-28T09:45:44-07:00 INFO	process.go:73	address assigned to the process '/home/alexk/go/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-linux-x86_64/data/install/filebeat-8.0.0-linux-x86_64/filebeat': '127.0.0.1:15182'

@ruflin
Copy link
Member

ruflin commented Apr 29, 2020

@michalpristas ^

@mtojek
Copy link
Contributor

mtojek commented Apr 29, 2020

This is the suggestion I received from @michalpristas and it worked:

i usually clean x-pack/filebeat/build, x-pack/metricbeat/build and x-pack/elastic-agent/build
i also do docker system prune -a occasionally

@michalpristas
Copy link

what marcin said. if this does not help check logs. as PR which makes it easier to find process logs is not in yet my advice would be to allow monitoring monitoring.enabled: true monitoring.logs: true and then logs should appear in /var/log/elastic-agent/default/filebeat

if monitoring is not enabled they are usually at the install location of a beat under logs subdir if you stick with default package then it is where the agent is located + data/install/filebeat-xxx-x86_64/logs

we might learn more about what's happening there

@fearful-symmetry
Copy link
Contributor Author

Okay, now I'm running into different issues. I can enroll the agent, same as normal, but when I try and view the agent, I get a JS error:

ko@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:52958
No/<@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:57752
qa@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:66589
xs@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:106361
ml@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:90020
hl@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:89943
ol@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:87291
Gi/<@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45733
__kbnSharedDeps__</t.unstable_runWithPriority@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:350:3462
Hi@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45442
Gi@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45680
Yi@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45613
el@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:84080
Aa@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:62991
_callee2$@http://buzzard.nest:5601/32729/bundles/plugin/esUiShared/esUiShared.plugin.js:1:668088
l@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:969133
s/o._invoke</<@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:968887
_/</e[t]@http://buzzard.nest:5601/32729/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:969490
asyncGeneratorStep@http://buzzard.nest:5601/32729/bundles/plugin/esUiShared/esUiShared.plugin.js:1:664439
_next@http://buzzard.nest:5601/32729/bundles/plugin/esUiShared/esUiShared.plugin.js:1:664768

No backend errors on the developer console.

The agent just reports this:

2020-04-30T14:03:12-07:00 DEBUG	action_dispatcher.go:77	No action to dispatch
2020-04-30T14:03:12-07:00 DEBUG	fleet_gateway.go:163	FleetGateway is sleeping, next update in 30s
2020-04-30T14:03:46-07:00 DEBUG	fleet_gateway.go:143	FleetGateway calling Checkin API
2020-04-30T14:03:46-07:00 DEBUG	client.go:170	Request method: POST, path: /api/ingest_manager/fleet/agents/bd41bbad-3f2e-4fa5-a393-fd0822c412fa/checkin
{"action":"checkin","success":true,"actions":[]}

@fearful-symmetry
Copy link
Contributor Author

Also, trying to SIGIT produces a panic:

panic: close of closed channel

goroutine 41 [running]:
github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/application.(*fleetGateway).Stop(0xc0000eabb0)
	/go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/fleet_gateway.go:234 +0xd2
github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/application.(*fleetGateway).worker(0xc0000eabb0)
	/go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/fleet_gateway.go:167 +0x4cb
github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/application.(*fleetGateway).Start.func1(0xc0000eabb0, 0xc0000eac38)
	/go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/fleet_gateway.go:227 +0xfe
created by github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/application.(*fleetGateway).Start
	/go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/fleet_gateway.go:223 +0x6e

@fearful-symmetry
Copy link
Contributor Author

Okay. Finally got this working.

@fearful-symmetry
Copy link
Contributor Author

Checking over everything now, but I hope this is good.

- add_locale: ~
- add_fields:
target: ''
fields:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting. I wonder if we should add this still on the Beats side in the future. @ph

@@ -0,0 +1 @@
metricsets: ["diskio"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we miss period here as a config option? And also some other datasets?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

period is at the module level, if that's what you mean?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, there are two things here. First I assume this stream template should also contain an entry for the period like few other datasets? The second part is if it is a global option on the package level or per dataset. I think it should be per dataset and you have it specified in some of the datasets already. Important is that it can only be either on the package or the dataset level, not both.

The available bits of entropy
- name: pct
type: scaled_float
format: percent
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skh @neptunian Do we already support formatters in our kibana index patterns?
@fearful-symmetry Nothing to change here.

"filter": [],
"query": {
"language": "kuery",
"query": ""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this PR, but I'm wondering if we could query here for the stream.type and stream.dataset and then this would query much less data.

@ruflin
Copy link
Member

ruflin commented May 5, 2020

@fearful-symmetry If you would call this 0.0.3 it would be much easier to test as it would be newer then https://github.com/elastic/package-registry/tree/master/dev/packages/example/system/0.0.2 As soon as this PR is merged, I plan to remove my 0.0.2

@fearful-symmetry
Copy link
Contributor Author

@ruflin Sure. Still trying to figure out the readme.

@fearful-symmetry fearful-symmetry marked this pull request as ready for review May 5, 2020 21:06
@fearful-symmetry fearful-symmetry merged commit 0663885 into elastic:master May 5, 2020
@mtojek
Copy link
Contributor

mtojek commented May 6, 2020

@ruflin Sure. Still trying to figure out the readme.

README files (template and the rendered one) look good. Comparing to other packages I'm missing sample events, but I assume you gonna add real ones later once tested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants