Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Usage of generated Go types #603

Closed
hamishforbes opened this issue Nov 5, 2019 · 6 comments
Closed

Usage of generated Go types #603

hamishforbes opened this issue Nov 5, 2019 · 6 comments
Assignees
Labels
wontfix This will not be worked on

Comments

@hamishforbes
Copy link

Are the generated types in this repo actually intended to be usable or are they a WIP / experiment that's not ready yet?

I'm working on a tool to filter and re-structure our logging data at the edge (before shipping from hosts to save network bandwidth and also Logstash resource usage), being able to use these generated types to validate outgoing messages are correctly formed would be very useful.

However the current generated code doesn't seem quite ready. It's got these custom ecs tags and dot-format field names which the default Go JSON marshaller doesn't understand.

Is there intended to be a custom marshaller to go along with these?

Related to #343 but the refactor PR mentioned in that issue is already merged without fixing any of these issue that I can see?

@mytototo
Copy link

I'm also trying to use generated Go types for ECS. I tried to find how it works but didn't find anything relevant. For now I just have a parent struct containing all other struct, but it seems a bit silly and doesn't support JSON.

Where can I find docs or examples about it ?

@webmat
Copy link
Contributor

webmat commented Jan 21, 2020

I'm not sure this library is in use nor how well it works. I've been wanting to remove this from the ECS repo for a while.

You're welcome to check out this repo https://github.com/elastic/ecs-logging, which links to concerted efforts to building ECS-compatible logging libraries.

Golang isn't there yet, but it's one of the languages we want to cover eventually as well.

@mytototo
Copy link

Thank you for the details @webmat.

So in your opinion, what is the best way to respect the ECS in a Go application-- in a very short term ? Do you think replicating the schema as Go types with JSON support is the best way to handle this ?

@hamishforbes
Copy link
Author

In the end I hacked up a python script (loosely based on some of the scripts in this repo) to generate proper nested and JSON annotated Go types from the yaml schemas in https://github.com/elastic/ecs/tree/master/generated/ecs

@webmat
Copy link
Contributor

webmat commented Jan 30, 2020

I think if the goal is solely to emit correct events, you can get away with generating JSON documents that would be ingested properly by Elasticsearch in an ECS index. By this I mean:

  • numerics and booleans would be unquoted
  • everything else (including IP and date/times) would be quoted
  • respecting which fields should be arrays (to be formalized soon via Mark which fields should be arrays. #727 )

But I'm a Rubyist, and that's just my preference :-)

Folks from statically typed languages (e.g. friends working on https://github.com/elastic/ecs-dotnet) tend to prefer to map all of the datatypes native to the programming language and ensure type correctness. This approach enables reading back events and having good native objects to work with. However this approach requires a ton more work to get right. I think there will still be some impedance mismatches between ECS -- a standardization of an Elasticsearch mapping -- and your favourite language.

So it really depends on what you're trying to accomplish.

@webmat webmat added the wontfix This will not be worked on label Aug 25, 2020
@ebeahan
Copy link
Member

ebeahan commented Dec 20, 2021

This Go-type generated is being removed in ECS 8.0 (see #1567), and this issue is no longer relevant.

@ebeahan ebeahan closed this as completed Dec 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

5 participants