Agent Configuration examples

[ph]

This is a work in progress configuration:

• Show how datasources works
• Show inputs and streams
• Show possible usage of multiples outputs
• Show definition of endpoint
• Show custom source and source generate from a package.

[elasticmachine]

Pinging @elastic/ingest (Project:fleet)

[ph]

@ruflin the link between the datasource and the "managing" app was mentioned by @jen-huang, I think we wanted to make the link hidden using the ids but maybe making it explicit in the configuration is a good idea.

[ph]

@jen-huang Had a chat with @ruflin and we do not need to have a managed_by defined in the datasource, actually that information should be in the package. This make sense if lets say we start making custom UI for specific datasources. Maybe a better UI for AWS services.

[jen-huang]

@ph So datasources created by Uptime, APM, Endpoint, etc will all specify a package? What if they don't use a package?

[ruflin]

Yes. They will have to have a package even if it is only to define this linking. Otherwise we could hardcode it into our UI as these links should not change too often.

[ph]

cc @michalpristas and @jen-huang since you are impacted by this, still consider this as a work in progress.

[kevinlog]

I know this is an example, but some of these could change. For instance we've talked about dropping threshold, is this still the case @ferullo @stevewritescode

[ruflin]

From an agent / fleet prespective, everything under input except type we don't know about and just forward. Any changes on this will not affect the agent itself.

[kevinlog]

at first glance, I prefer suggestion 1 as I'm not sure we would maintain different configurations for different platforms. We'll make the entire thing atomic, so we'll send down some extra config and the Endpoint will know what to read.

thoughts @ferullo ?

EDIT: I think I misunderstood at first, it looks like they're just different layouts and all platforms are contained in the same config, sorry for the confusion.

[kevinlog]

We've discussed adding versions into the configuration that need to match up with Endpoint, otherwise, it will reject the configuration. Is that what this version implies? Will we be able to manage individual versions for each package? (i.e. endpoint, metric beat, etc)?

A similar conversation on versions here:
https://github.com/elastic/endpoint-app-team/issues/129#issuecomment-579130066

[ruflin]

This version here is to know from which package and package version a config was created. This should help when we try to upgrade configs.

The part about limiting it to a specific agent version can be found here: https://github.com/elastic/beats/pull/15940/files#diff-043e80bbcc20fded11325ef31433396bR35

[kevinlog]

do the individual ids in here imply that we'll edit individual sections on the Agent configuration? Is there an overall Agent configuration ID that we'll use with Ingest API's edit the entire configuration?

I'm concerned about editing the configuration in pieces, it's preferable to send the entire configuration down to the Agent/Endpoint at once.

[ruflin]

There will be an overall unique ID for a configuration in the Fleet API. These "local" ids can be generated and become useful especially on error reporting, so we can tell the user which part of the config did not work.

It is still the plan to send down the full agent config down to the agent at once.

[kevinlog]

FYI @elastic/endpoint-management @scunningham

[michalpristas]

this should be ecs compliant?

[ruflin]

YES!

[ph]

Yes it should, I should say, that the constraints look is not final. but lets make sure the definition of them both work in the Agent and in the UI. It would be good to leverage that information to display more useful data to the user when they visualize and agent or a configuration.

[ruflin]

@ph Lets keep an eye on the id and as soon as we start using it in the code somewhere have a discussion on how we use it.

[ph]

Okay to not make it mandatory for now.

[roncohen]

should we just call this "sources"?

[ruflin]

I would stick with datasources. Sources is more generic and would be correct too. We have already source in ECS. Also I heard datasource term used by people less familiar with the project to describe things like the "nginx datasource" so I think it fits well.

[roncohen]

"description" rather than "title"?

[ph]

I was leaning toward "title" because I would that using description could be multiple lines where a title does not. but I do not have strong feeling to change it. Any opinion from the others?

[roncohen]

this, unfortunately, needs to be simpler.
Could we set a default "dataset" in the input so users don't have to specify it? And then let them specify metricsets as an array?

- id: system  # make optional if possible
  inputs:
    - type: metrics/system
      streams:
       - metricset: ["cpu", "memory", "diskio", "load", "process", "uptime", "filesystem"]
          # default dataset, period etc.

or a short form that gets converted automatically:

- inputs:
    - type: metrics/system
      metricsets: ["cpu", "memory", "diskio", "load", "process", "uptime", "filesystem"]

[ph]

First, I think we trying to optimize for the UI experience vs the YAML edition experience.

If you look at the metricbeat module definition in beats they are do have settings, they are not just opt-in thing. I've used the system module because it's one of the bigger one and if you look at the metricset they do have settings. I do not find that obvious to change these options using the key/name approache.

But @ruflin do you have a strong opinion? I think there were also a reason concerning dataset for using the list.

  cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.
  #filesystem.ignore_types: []
  #process.include_top_n:
    #enabled: true
    #by_cpu: 0
    #by_memory: 0
  #process.cmdline.cache.enabled: true
  #process.cgroups.enabled: true
  #process.env.whitelist: []
  #process.include_cpu_ticks: false
  #raid.mount_point: '/'

  #socket.reverse_lookup.enabled: false
  #socket.reverse_lookup.success_ttl: 60s
  #socket.reverse_lookup.failure_ttl: 60s
  #diskio.include_devices: []

[ph]

@roncohen What do you think about metrics module options as defined above? Where they fit in the shortened version?

[ruflin]

My current thinking:

• There is the long form which always works (as proposed)
• There is a short form with one of the proposals above which is needs a bit of magic on the agent side to make it work with metricbeat.

The magical part can always be added later so I don't see it as a blocker. Talking about magic: I would also like to see the following working:

- id: system  # make optional if possible
  inputs:
    - type: metrics/system

It is like enabling the system module today, it just picks the default metricsets which are defined in Metricbeat.

[roncohen]

OK, we can add more defaults and a rewrite step later 👍

[roncohen]

@kevinlog do you think it makes sense to have a separate "protections" section or similar? I imagine that there are a number of config settings that aren't really "inputs" or "streams". From my perspective, we don't need to over-generalize this Agent config in the sense that everything must fit as a stream.

There's no reason why we need to make it all generic and the Agent oblivious to endgame - if that makes sense.

[ph]

Maybe input is stretching the concept for the endpoint, but they still fit out definition they generate data, our requirements and reserved keywords are minimal everything else at the input level would be free form and passed directly to the endpoint. So they could have protections or anything they want at that level.

- id: myendpoint-1
  title: Endpoint configuration
   package:
     name: epm/endpoint # This establish the link with the package and will allow to link it to endpoint app.
     version: xxx
   inputs:
     - type: endpoint # Reserved key word

[ferullo]

Thanks for an example!

[ferullo]

Is there down supposed to still be under streams or is what's above not supposed to be under streams? Half looks indented one space more than the other half.

[ph]

Yes its supposed to be my vim-foo failed on the yaml format will fix it in the next round of changes.

[ferullo]

Suggestion 2 seems cleaner to me for Endpoint. I like nesting the entire config for an OS under an OS section rather than intermingling them. This will make it easier for Endpoint to read the config for it's OS and also feels more in line with the fact that there are many features that are not available on all OSes,

[ph]

I am OK with that.

[ferullo]

What do default, long_term_storage and monitoring mean? Am I correct in assuming that when writing data to Elasticsearch that each datatype Beats/Agent/Endpoint write will be hardcoded to go to particular destination? Is there room for more outputs if Endpoint ends up needing another Endpoint-specific output? Is there any way to control what index is written to for an output?

[ph]

I've discussed with @ferullo on zoom about this. but for clarity I will stil add part of the discussion here. The default, long_term_storage and monitoring are destination, in the initial version we would only support default and long_term_storage. When you define a datasource you can use the use_output option to defines to which named output to send the data.

So yes, if the endpoint need a special output destination it could be added.

Concerning the index, the index is not something that a user would be able to configure directly we do this to hide the complexity inherent to manage indices (templates mapping, ilm policies and dashboard) instead the agent is using the type of the input, the dataset and the defined namespace to generate the appropriate index destination. The only user editable part is the namespace, which is used as a grouping mechanism like "production" data.

I presume that endpoint should try to align with that indexing strategy and allow the user to set up the namespace maybe its something to discuss with @ruflin

[ph]

Add a namespace for endpoint and sync with @kevinlog

[michalpristas]

namespace is moved from stream to datasource, do you think it would be useful to have stream override of the namespace specified here?

[ph]

We try as much as possible to not support overrides, the way would be to duplicate a datasource and set a another namespace.

[ph]

Also, streams cannot override input settings.

[ruflin]

+1 on keeping it on the top level. I think having a namespace per stream is not a common case. If a user needs it, he can specify the datasource twice.

[urso]

Although we should restrict the usage of global processors, I wonder how processors like add_x_metadata fit in. These are often global, as these do enhance events based on the environment the Beat is run in. The add_host_metadata and add_cloud_metadata processors do query the host system on init time only. The add_docker_metadata and similar processors to enhance events based on events contents.

Do we actually want users to add any of the add_x_metadata processors to their configs? What if a datasource configured package is to be executed in different environment types (pure host and docker containers). Maybe the agent would add these envionrmental processors or even replace them with constants add_fields (processor) based on information the agent did query? In this case maybe the functionality might be 'hidden' behind an add_environment: <bool> setting, expecting the agent to do the right thing?

[mostlyjason]

Another setting to consider is adding the agent version for this configuration. This will allow future agent versions to make schema changes, and for us to provide an error if an agent is incompatible with the configuration version.

[ph]

Another setting to consider is adding the agent version for this configuration. This will allow future agent versions to make schema changes, and for us to provide an error if an agent is incompatible with the configuration version.

I think having a schema version (of lack of version means version is 1) is a good idea, it also depends how we do handle migration, if we assume that configuration is keep in fleet we can also assume that the following flow is true: update agent-> migrate configuration -> deploy configuration to agent. Which reduce agent to understand multiple version of a schema.

[ph]

@urso

Do we actually want users to add any of the add_x_metadata processors to their configs? What if a datasource configured package is to be executed in different environment types (pure host and docker containers). Maybe the agent would add these envionrmental processors or even replace them with constants add_fields (processor) based on information the agent did query? In this case maybe the functionality might be 'hidden' behind an add_environment: setting, expecting the agent to do the right thing?

When processors are globals or auto-added you are always at risk of having a conflict of fields (hey hostname case or source?) So I think if the solutions should be behind the package definition where we could either add the processors to the required processors by a streams or even have it at the datasource level.

add_environment: <bool> I think its a good suggestion if let's say that collecting the information is dependent on the running environment (like configured by a k8s streams of events?)

@ruflin seems like our talk concerning the processors should allow them to be defined at the datasource and maybe with a insert scope like processors.after or processor.before to control how the merging logic is applied.

Also is there anything preventing us to merge all the environment based processors to be merged into a single one?

[michalpristas]

are we assuming everything is elasticsearch or do we support type still

[ph]

Lets keep the type support that we have, we are so use to target elasticsearch I think this felt into a crack. so type: elasticsearch

[ph]

I just added the type to this PR.

[michalpristas]

we say paths but there is only one. is this an array or should this be a path?

[ph]

Well no its should still be plural since we are using go-ucfg, a single string "/var/log" and a slice of strings are equivalent, if you use the following type to unpack. This is true for a lot of things in beats.

type C struct {
 Paths []string `config:"path"`
}

[michalpristas]

in ucfg yes but in tests we're producing map[string]interface{} tree so i was wondering what's the intention here.

[ph]

This is correct, but they will be reparsed after so that should not be a problem, or I am missing something?

[michalpristas]

no no you're not, i just wanted to have these trees in a form which reflects our intention. so double checking whether this is supposed to be multiple paths or single value as with metricset

[ph]

@ruflin I have added examples for Elasticsearc/Kubernetes/Docker and AWS.

Looking at Kubernetes, I wonder if they should be two different datasources.

[ruflin]

About Kubernetes: When @exekias created the K8s module we had many discussions should it be one or 2 modules. We ended up with 1 module because of the config if I remember correctly. It was always a bit odd as it was 1 module with 2 shared configs. But now with datasources and multiple inputs I think it makes much more sense:

• 1 Datasource: Kubernetes
• 3 inputs: logs, node metrics, state metrics

I would keep it the way you specified it with 3 inputs. I expect the user to configure it in one go.

@exekias I see at the end of the file we separate out the event metricset: https://github.com/elastic/beats/blob/master/metricbeat/module/kubernetes/_meta/config.yml#L46 I assume this is because it does not have a collection period. Should this go under the node metrics input or is this an additional input?

[ruflin]

Can you add a few more common configs here like username, password, period, ssl.cerficate_authorities to make it more obvious the common configs are here? https://github.com/elastic/beats/blob/master/metricbeat/module/elasticsearch/_meta/config.reference.yml#L12

[ph]

@ruflin I thought period was at the stream level and not on the common part of the input?

[ruflin]

It is, skip it for now

We will probably need to come back to the question: Are there configs that can be specified on both levels like period as default and have a local period that overwrites it. Lets have this discussion later.

[ruflin]

If possible, I think we should make this 1 input with multiple streams. Why do you think it should be multiple inputs?

[ph]

@ruflin From our previous discussion, inputs would contain hosts/ssl configuration. In that context looking at the filebeat's AWS module, the events are read from files inside an s3 bucket.

To fix that problem either:

1. We do what I've done, where multiple inputs are defined where each input have access to a different S3 bucket.
2. Or we allow the S3 input to ignore keys or consider keys that match a certain pattern. (This would require a bit of change in the s3 input, looking at the configuration its not supported)

We do have to think about the use-case, what it's more common? Use a single bucket for all the logs from AWS service or different buckets. I think because of policy/permission the different bucket route might be more popular.

I am no expert in AWS @kaiyan-sheng might know more here about the usage.

[ruflin]

Is the queue url defining from where the events are fetched in the end? If yes, shouldn't this be on the stream level instead?

[kaiyan-sheng]

For AWS logs, I think better setup is to have different S3 buckets for different logs or different services. For example, one bucket for s3access log, one bucket for elb logs. Or one for vpc flow logs from VPC1 and one for vpc flow logs from VPC2. But all these buckets can have the same notification setup with the same SQS queue(if they are in the same region).

The queue url is not where the logs are fetched, it only points to the S3 bucket where the actual log needs to be read.

[ruflin]

How does the input then now, which logs to pick and where to send it to?

[kaiyan-sheng]

@ruflin The input then uses the message from SQS queue to locate where the log is(in which s3 bucket and what's the name of the log) and read from S3.

[ruflin]

I got that part. But it means, each dataset needs to have a different SQS queue as otherwise one dataset would pick up the logs from the others too.

[ph]

@kaiyan-sheng Yes, it know where (bucket) and which (the file) but there is nothing I can see in the code that says: OK this file is actually a VPCflow.

So the way I am understand it is the AWS modules assumes that all the events coming from a specific SQS queue can only be the same type. I can be wrong but I don't see how disambiguation could happen.

[kaiyan-sheng]

Ahh sorry you are right! With different kinds of logs, they should be separated to different queues if we want to ingest them at the same time.

[ruflin]

Nit: I keep stumbling over the type. I would expect it to be docker/metrics. The reason is the order I think of collecting: Let me get some data from Docker and select only the metrics. I understand that from an agent perspective it is the other way around, select metricbeat and then run docker.

[ph]

I don't have a strong feeling about metric before of after docker, I think your reasoning makes sense and might be more future proof if we do want to collect other thing. The mnemonic might be better.

[ph]

I've made the change on the config.

[exekias]

@exekias I see at the end of the file we separate out the event metricset: https://github.com/elastic/beats/blob/master/metricbeat/module/kubernetes/_meta/config.yml#L46 I assume this is because it does not have a collection period. Should this go under the node metrics input or is this an additional input?

This is node specific, but cluster wise. I would say it should be a new input

[ph]

@ruflin I think I've addressed all the concerns above?

[ruflin]

@ph I'm good with getting it in as is and have follow up conversation with smaller PRs.

[ruflin]

Is the queue url defining from where the events are fetched in the end? If yes, shouldn't this be on the stream level instead?

[ph]

@ruflin Good, I've made the change for the AWS inputs vs stream and also added more credential for the aws/metrics.

[ph]

@ruflin I am merging this and lets make iteration on top of it.