Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue: Add Log Namespace support to sinks #15453

Closed
24 tasks done
fuchsnj opened this issue Dec 5, 2022 · 0 comments
Closed
24 tasks done

Tracking issue: Add Log Namespace support to sinks #15453

fuchsnj opened this issue Dec 5, 2022 · 0 comments
Labels
type: feature A value-adding code addition that introduce new functionality.

Comments

@fuchsnj
Copy link
Member

fuchsnj commented Dec 5, 2022

Sinks need updated to work correctly when the vector log namespace is enabled. The only change required is related to the "global log schema". The vector namespace no longer uses the "global log schema", so sinks that use this to fetch data need to switch to getting this data from somewhere else. Usually either from "semantic meaning", or from vector metadata.

Global Log Schema Field Migrate To
message semantic meaning ("message")
host semantic meaning ("host")
timestamp semantic meaning ("timestamp"), defaulting to the current time if it doesn't exist and is appropriate
source_type %vector.source_type should always exist for messages using the vector namespace

Since these sinks will rely on certain semantic meaning fields, it would useful to also have sink requirements which can enforce that the required meanings are set, and that they are the correct type. This involves creating a schema::Requirement object that just lists the semantic meaning name, the type, and whether it is required / optional, and adding it to the input in the SinkConfig::input function. These should be fairly straightforward to add to sinks.

Since sink requirements (at least initially) will only be required (for most sinks) when the Vector namespace is used, we should either check the LogNamespace when creating the requirements, or potentially only enable sink requirements with the Vector namespace.

For each sink:

  1. Migrate all uses of the "Global Log Schema" (except for tests / default config options) to use helper methods that fetch data from either the "Global Log Schema" (Legacy namespace) or from semantic meaning / Vector metadata (Vector namespace).
  2. Add sink requirements

Sinks

Other Work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Development

No branches or pull requests

1 participant