Confusion about required changes described in 0.57 upgrade guide

[leandrojmp]

Question

Hello,

We use vector in a couple of ways in an ingestion data flow, our configuration files use both env variable interpolation and templates to dynamically send data into a sink.

For environment variable interpolation it seems that I would need to set VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true in the environment used by the systemd vector service.

But what would be the alternative? Using secrets? It is not clear from the upgrade guide.

Having to set a variable named VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION will probably generate issues from our security team and scanning tools.

Regarding the template confinement for sinks, it is correct to assume that it will only affect templates that does not have a fixed prefix ?

For example, on our kafka sink we dynamically create the topic using something like this:

topic: "logs-{{ log_topic }}"

The field log_topic is extracted from the message, in this case vector will consider that it is already confined because of the prefix and will start normally, right?

I'm sharing an example of our config as well:

Vector Config

# vector
# ingestion layer
#

log_schema:
  timestamp_key: "vector.ingestion_timestamp"
  message_key: "vector.ingestion_message"

sources:
  logs:
    type: http_server
    address: 0.0.0.0:5045
    encoding: json
    method: POST
    auth:
      username: "${VECTOR_HTTP_USER}"
      password: "${VECTOR_HTTP_PASS}"
    path: /logs
    path_key: vector_request_path
    response_code: 200
    strict_path: false

  status:
    type: http_server
    address: 0.0.0.0:5040
    encoding: json
    method: GET
    path: /status
    path_key: vector_request_path
    response_code: 200
    strict_path: true

# transform

transforms:
  topic:
    inputs: [ "logs" ]
    type: remap
    drop_on_error: true
    source: |
      . |= parse_groks!(
                .vector_request_path,
                patterns: [
                    "^/logs/%{DATA:log_topic}"
                ]
            )
      del(.source_type)
      del(.vector_request_path)
  event:
    inputs: ["*"]
    type: filter
    condition:
      type: "vrl"
      source: ".log_topic != null"

# sinks

sinks:
  kafka:
    type: kafka
    inputs: ["event"]
    bootstrap_servers: "${VECTOR_KAFKA_BROKERS}"
    topic: "logs-{{log_topic}}"
    encoding:
      codec: json
    librdkafka_options:
      message.max.bytes: "10000000"

Vector Logs

No response

[leandrojmp]

Just saw that the question related to template confinement is answered by the description of this PR.

[pront]

Hi @leandrojmp, thank you for taking the time to create a discussion. Don't hesitate to ask more questions if the following response isn't clear.

1. Environment variable interpolation

The flag name is intentionally alarming. Its purpose is to prompt operators to consciously decide whether env var interpolation is safe in their environment, not to imply you must avoid it entirely. For example, in a locked-down deployment opting in is valid but in shared or not fully controlled environments you probably want to disable it.

You can also maintain you current configs and add a pre-processing step, e.g. using envsubst. This is not mentioned in the upgrade guide but it is mentioned here: https://vector.dev/docs/reference/environment_variables/#security-restrictions

2. Template confinement

topic: "logs-{{ log_topic }}"

No changes needed. Your template already has a literal prefix (logs-), which is exactly what the new confinement rule requires. The breaking change only affects templates without a a literal prefix, like {{ log_topic }} by itself. You can confirm this with vector validate. For example:

$ vector validate /tmp/vector_test_unconfined.yaml
√ Loaded ["/tmp/vector_test_unconfined.yaml"]
√ Transforms configuration

Component errors
----------------
x Sink "kafka_out": template references event fields (["log_topic"]) but has no literal string prefix to derive a confinement base from. Add a static prefix to your template, or set `dangerously_allow_unconfined_template_resolution: true` to opt out.

$ vector validate /tmp/vector_test_confined.yaml
√ Loaded ["/tmp/vector_test_confined.yaml"]
√ Transforms configuration
√ Component configuration

Does this fully answer your questions?

[leandrojmp]

Thanks @pront!

Regarding the envsubst, how would something like this work when vector is executed as a service?

We use systemd to run vector, the environment variables are used to avoid commiting configurations and secrets in the file as this is managed in git.

The environemnt are already sourced from a file defined in the systemd service, so probably we should switch to use secrets with a file backend?

[pront]

Yes, the file secrets backend is a clean fit for your setup. Instead of sourcing env vars, you point Vector at a JSON file:

secret:
  mysecrets:
    type: file
    path: /etc/vector/secrets.json

{ "api_key": "...", "password": "..." }

Then reference them in config as SECRET[mysecrets.api_key].

One thing to be aware of: secrets backends are designed for sensitive values (credentials, tokens). If some of your env vars are non-sensitive config values (topic prefixes, hostnames, etc.), those are better handled by hardcoding them directly in the config or using a pre-processing step like envsubst before Vector starts.

[leandrojmp]

But will they work for all settings or just the ones marked as sensitive?

For example, bootstrap_servers: "${VECTOR_KAFKA_BROKERS}" may not be sensitive, but it will be different depending on the environment vector is running and we want to avoid having hardcoded values in github.

How envsubst would work when running vector as a service?

[pront]

will they work for all settings or just the ones marked as sensitive?

Secrets backends work for any string config value, there's no restriction to "sensitive" fields. You can always give it a try and see for yourself.

How envsubst would work when running vector as a service?

That would be a step that is completely outside of Vector and up to the deployment admin. You would just need to point Vector to the resolved config file.