Setting Property in Array via Environment

[Benehiko]

Hi There!

I have trouble with setting a specific property of an array via its environment variable.
Here's an example, let's say I have a configuration file like this:

providers:
    - id: google
      provider: google
      client_id: ''
      client_secret: ''
      ...
    - id: github
      ...

One could add many providers and each provider has its own client_id and client_secret.

So now let's say I want to add the client_secret to the google provider.
How would the environment variable look? Also would it override the rest of the providers or the other provider properties?
I want it to look something like this SELFSERVICE_METHODS_OIDC_CONFIG_PROVIDERS_GOOGLE_CLIENT_SECRET?

The issue is tracked from another project here:
ory/kratos#1186 (reply in thread)

[knadh]

Some context on reading and merging env variables: https://github.com/knadh/koanf/#reading-environment-variables

Unfortunately, in the example you've shared, there's no automatic way for env variables to be merged into arrays. If it wasn't an array but a map, where "google" was the key, and provided that the _ in the key names were escaped with __ like in the above documentation, the variables would've looked like:

PROVIDERS_GOOGLE_CLIENT__ID              # translates to:  providers.google.client_id
PROVIDERS_GOOGLE_CLIENT__SECRET

The implementer could use the env key transformation functions to internally parse something like this to arrays:

PROVIDERS_0_GOOGLE_CLIENT__ID
PROVIDERS_0_GOOGLE_CLIENT__SECRET

PROVIDERS_1_GOOGLE_CLIENT__ID
PROVIDERS_1_GOOGLE_CLIENT__SECRET

[Benehiko]

Hi @knadh

Thank you for your fast reply! :)

I will see what I can figure out in the ory/kratos code base, really appreciate your help!

[knadh]

https://github.com/knadh/koanf#reading-environment-variables

The new env.ProviderWithValue() supports a callback that can take a key and a value and then transform the value into any type like an array.

[tina-junold]

FYI: I have/had the same problem. I've solved this with by creating a provider which reads the env but returns json. this conversion and setting to json is done by tidwall/sjson

package envtojson

import (
	"errors"
	"github.com/tidwall/sjson"
	"os"
	"strings"
)

type Env struct {
	prefix string
	delim  string
	cb     func(key string, value string) (string, interface{})
	out    string
}

func Provider(prefix, delim string, cb func(s string) string) *Env {
	e := &Env{
		prefix: prefix,
		delim:  delim,
		out:    "{}",
	}
	if cb != nil {
		e.cb = func(key string, value string) (string, interface{}) {
			return cb(key), value
		}
	}
	return e
}

// ProviderWithValue works exactly the same as Provider except the callback
// takes a (key, value) with the variable name and value and allows you
// to modify both. This is useful for cases where you may want to return
// other types like a string slice instead of just a string.
func ProviderWithValue(prefix, delim string, cb func(key string, value string) (string, interface{})) *Env {
	return &Env{
		prefix: prefix,
		delim:  delim,
		cb:     cb,
	}
}

// ReadBytes reads the contents of a file on disk and returns the bytes.
func (e *Env) ReadBytes() ([]byte, error) {
	// Collect the environment variable keys.
	var keys []string
	for _, k := range os.Environ() {
		if e.prefix != "" {
			if strings.HasPrefix(k, e.prefix) {
				keys = append(keys, k)
			}
		} else {
			keys = append(keys, k)
		}
	}

	for _, k := range keys {
		parts := strings.SplitN(k, "=", 2)

		var (
			key   string
			value interface{}
		)

		// If there's a transformation callback,
		// run it through every key/value.
		if e.cb != nil {
			key, value = e.cb(parts[0], parts[1])
			// If the callback blanked the key, it should be omitted
			if key == "" {
				continue
			}
		} else {
			key = parts[0]
			value = parts[1]
		}

		if err := e.set(key, value); err != nil {
			return []byte{}, err
		}
	}

	return []byte(e.out), nil
}

func (e *Env) set(key string, value interface{}) error {
	out, err := sjson.Set(e.out, strings.Replace(key, e.delim, ".", -1), value)
	if err != nil {
		return err
	}

	e.out = out

	return nil
}

// Read is not supported by the file provider.
func (e *Env) Read() (map[string]interface{}, error) {
	return nil, errors.New("envextended provider does not support this method")
}

Here is a small example how to use it. it works quite well with keys like:

PREFIX__HTTP__0__NAME=test
PREFIX__HTTP__0__ENABLED=true
PREFIX__PORTS__0=1234
PREFIX__PORTS__2=5678

koanfEnv := koanf.New(".")
_ = koanfEnv.Load(envtojson.Provider(configEnvPrefix, configEnvDelim, func(s string) string {
return strings.ToLower(strings.TrimPrefix(s, cfg.Source.EnvPrefix+configEnvDelim))
}), json.Parser())
_ = koanfEnv.Unmarshal("", cfg)