Add additional data to the documents

[cooperspencer]

I would like to add the hostname to the documents sent to Elasticsearch. Is that possible?

[dadoonet]

Hi @cooperspencer

Could you share what your fscrawler settings files are looking like? I mean the fs.url setting specifically or if you are using FTP or SSH or anything else.

But anyway, you might want to use the .meta.yml file, which can be added in all the dirs you are scanning:

external:
  hostname: "server001"

That should be added as external.hostname field. Which you can rename using an ingest pipeline.

See more at https://fscrawler.readthedocs.io/en/latest/admin/fs/tags.html

[cooperspencer]

Thanks a lot, that was exactly what I was looking for.

[dadoonet]

But I'm wondering if we should not implement another solution, like adding static values for each documents for a given configuration. In the fscrawler settings for example, like:

fs:
  url: "/path/to/docs"
tags:
  static:
    external:
      hostname: "server001"

Which will be applied to every document indexed without needing to add a file in every single dir?

[cooperspencer]

IMHO, that solution would be way more elegant. It would also save a lot of configuration/copy and paste.

[dadoonet]

Thanks for confirming. I added #2120 to track it.

[dadoonet]

@cooperspencer I added a new way to support global metadata. Have a look at: https://fscrawler.readthedocs.io/en/latest/admin/fs/tags.html#static-metadata

HTH