You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sinks need updated to work correctly when the vector log namespace is enabled. The only change required is related to the "global log schema". The vector namespace no longer uses the "global log schema", so sinks that use this to fetch data need to switch to getting this data from somewhere else. Usually either from "semantic meaning", or from vector metadata.
Global Log Schema Field
Migrate To
message
semantic meaning ("message")
host
semantic meaning ("host")
timestamp
semantic meaning ("timestamp"), defaulting to the current time if it doesn't exist and is appropriate
source_type
%vector.source_type should always exist for messages using the vector namespace
Since these sinks will rely on certain semantic meaning fields, it would useful to also have sink requirements which can enforce that the required meanings are set, and that they are the correct type. This involves creating a schema::Requirement object that just lists the semantic meaning name, the type, and whether it is required / optional, and adding it to the input in the SinkConfig::input function. These should be fairly straightforward to add to sinks.
Since sink requirements (at least initially) will only be required (for most sinks) when the Vector namespace is used, we should either check the LogNamespace when creating the requirements, or potentially only enable sink requirements with the Vector namespace.
For each sink:
Migrate all uses of the "Global Log Schema" (except for tests / default config options) to use helper methods that fetch data from either the "Global Log Schema" (Legacy namespace) or from semantic meaning / Vector metadata (Vector namespace).
Audit sources / transforms to ensure that semantic meaning for "global log schema" values are correct. These are message, host, timestamp (source_type is sometimes used, but can be fetched directly from vector metadata)
Sinks need updated to work correctly when the
vector
log namespace is enabled. The only change required is related to the "global log schema". Thevector
namespace no longer uses the "global log schema", so sinks that use this to fetch data need to switch to getting this data from somewhere else. Usually either from "semantic meaning", or from vector metadata.%vector.source_type
should always exist for messages using thevector
namespaceSince these sinks will rely on certain semantic meaning fields, it would useful to also have sink requirements which can enforce that the required meanings are set, and that they are the correct type. This involves creating a
schema::Requirement
object that just lists the semantic meaning name, the type, and whether it is required / optional, and adding it to the input in theSinkConfig::input
function. These should be fairly straightforward to add to sinks.Since sink requirements (at least initially) will only be required (for most sinks) when the Vector namespace is used, we should either check the
LogNamespace
when creating the requirements, or potentially only enable sink requirements with the Vector namespace.For each sink:
Sinks
datadog_logs
sink #15473Humiowraps splunk_hec_logs and only useslog_schema
in config defaultsinfluxdb_logs
sink #16539Other Work
message
,host
,timestamp
(source_type
is sometimes used, but can be fetched directly from vector metadata)schema.validation
) without schema definitions (schema.enabled
) #16534TransformConfig::outputs
should accept a list of schema definitions #16730The text was updated successfully, but these errors were encountered: