Skip to content

fix(expand): use transform.Bytes to fix interpolation of large environment variables#137

Merged
sywhang merged 1 commit into
uber-go:masterfrom
aetheryx:zain.ali/fix-expand-short-buffer
May 12, 2026
Merged

fix(expand): use transform.Bytes to fix interpolation of large environment variables#137
sywhang merged 1 commit into
uber-go:masterfrom
aetheryx:zain.ali/fix-expand-short-buffer

Conversation

@aetheryx
Copy link
Copy Markdown
Contributor

@aetheryx aetheryx commented May 6, 2026

Originally reported in #115.

Problem

When trying to use environment variable interpolation with large environment variable values, expand fails with couldn't expand environment: transform: short destination buffer. For example:

test:
  value: ${TEST_VALUE}
type TestConfig struct {
  cfgparse.Struct `config:"test"`
  Value           string `yaml:"value"`
}

func main(config TestConfig) {
  fmt.Printf("value: %d\n", len(config.Value))
}
% export TEST_VALUE=$(printf 'a%.0s' {1..4096})
% bazel run 
[...]
value: 4096

% export TEST_VALUE=$(printf 'a%.0s' {1..4097}) # 4096 + 1
[...]
couldn't expand environment: transform: short destination buffer

Changes

Updated expand.go to use transform.Bytes(transformer, bytes) instead of ioutil.ReadAll(transform.NewReader(bytes, transformer)).

The reason for this is that transform.NewReader transforms chunks at fixed sizes, and when an iteration of the transformer returns ErrShortDst without making progress (nDst == 0 && nSrc == 0), NewReader bubbles up ErrShortDst by design and won't grow dst. transform.Bytes does grow dst in this case, as it treats the entire buffer as a whole instead of chunking.

Test plan

  1. Unit tests pass
  2. Minimum reproducible example from above passes on this branch:
% export TEST_VALUE=$(printf 'a%.0s' {1..100000})
% bazel run 
[...]
value: 100000

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 6, 2026

CLA assistant check
All committers have signed the CLA.

@aetheryx aetheryx changed the title fix(expand): use transform.Bytes to fix large environment variables fix(expand): use transform.Bytes to fix interpolation of large environment variables May 6, 2026
@sywhang sywhang merged commit 1f375e0 into uber-go:master May 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants