Strict Environment Handling #4449

nathanhammond · 2023-04-04T15:21:18Z

To provide a way to reduce information leakage into the task execution environment, add a strict mode for environment variable handling. This mode does not propagate non-enumerated variables to the environment of the task being executed.

Things that changed to make sure to review which are harder to do in code, pop into the deploy:

turbo.json schema.
Docs for CLI and Configuration.

Open Design Option

This partitions the global cache key based solely upon existence of --env=strict vs. --env=loose. It could instead do that in tandem with existence of globalPassthroughEnv if and only if --env=strict information is moved into the task hashes which have passthroughEnv specified.

That will increase possible granularity, but I suspect that the use case for this feature is going to be people wanting to lock it down.

vercel · 2023-04-04T15:21:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
examples-designsystem-docs	🔄 Building (Inspect)	Visit Preview		Apr 6, 2023 6:52pm
examples-native-web	🔄 Building (Inspect)	Visit Preview		Apr 6, 2023 6:52pm
examples-tailwind-web	🔄 Building (Inspect)	Visit Preview		Apr 6, 2023 6:52pm
turbo-site	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 6, 2023 6:52pm

7 Ignored Deployments

Name	Status	Preview	Updated (UTC)
examples-basic-web	⬜️ Ignored (Inspect)	Visit Preview	Apr 6, 2023 6:52pm
examples-cra-web	⬜️ Ignored (Inspect)	Visit Preview	Apr 6, 2023 6:52pm
examples-gatsby-web	⬜️ Ignored (Inspect)	Visit Preview	Apr 6, 2023 6:52pm
examples-kitchensink-blog	⬜️ Ignored (Inspect)	Visit Preview	Apr 6, 2023 6:52pm
examples-nonmonorepo	⬜️ Ignored (Inspect)		Apr 6, 2023 6:52pm
examples-svelte-web	⬜️ Ignored (Inspect)		Apr 6, 2023 6:52pm
examples-vite-web	⬜️ Ignored (Inspect)	Visit Preview	Apr 6, 2023 6:52pm

vercel · 2023-04-04T15:21:30Z

@nathanhammond is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

nathanhammond

Reviewer's guide.

cli/internal/env/env.go

cli/internal/fs/turbo_json.go

nathanhammond · 2023-04-04T15:58:10Z

cli/internal/fs/turbo_json.go

 	globalFileDependencies := make(util.Set)

 	for _, value := range raw.GlobalEnv {
 		if strings.HasPrefix(value, envPipelineDelimiter) {
 			// Hard error to help people specify this correctly during migration.
 			// TODO: Remove this error after we have run summary.
-			return fmt.Errorf("You specified \"%s\" in the \"env\" key. You should not prefix your environment variables with \"%s\"", value, envPipelineDelimiter)
+			return fmt.Errorf("You specified \"%s\" in the \"globalEnv\" key. You should not prefix your environment variables with \"%s\"", value, envPipelineDelimiter)


Minor textual correction!

nathanhammond · 2023-04-04T15:58:38Z

cli/internal/fs/turbo_json.go

 		}

 		envVarDependencies.Add(value)
 	}

+	for _, value := range raw.GlobalPassthroughEnv {


I like the hard error, so I kept this consistent.

nathanhammond · 2023-04-04T15:59:40Z

cli/internal/run/real_run.go

This is really the heart of the PR. It's ... a small diff?

NicholasLYang

Mostly needs some tests but otherwise LGTM

crates/turborepo-lib/src/cli.rs

nathanhammond

Some decisions about what causes hash changes.

nathanhammond · 2023-04-04T19:06:45Z

cli/integration_tests/strict_env_vars/global_passthrough_env.t

+  [-0-9:.TWZ+]+ |[DEBUG] turbo: local cache folder: path="" (re)
+  No tasks were executed as part of this run.
+
+Loose + ["BONUS_VAR"]


This has the same hash as the above one, since loose explicitly means "don't pay attention to it."

nathanhammond · 2023-04-04T19:12:56Z

cli/integration_tests/strict_env_vars/strict_env_vars.t

+  $ . ${TESTDIR}/../setup.sh
+  $ . ${TESTDIR}/setup.sh $(pwd) monorepo
+
+Strict with no passthroughEnv opts you into the new hashing behavior.


This is an optional decision. We could make it use the old hashing behavior here. ("You specified a nonsense combination of options.")

If we do that, though, we need to move the EnvMode flag into each individual task hash.

cli/internal/run/run.go

gsoltis · 2023-04-04T23:03:20Z

cli/internal/run/real_run.go

+	passthroughEnv := env.EnvironmentVariableMap{}
+
+	if strictEnv {
+		defaultPassthrough := []string{


I think Windows will also require SYSTEMROOT. Go adds it by default if it's missing, but let's be explicit that we're including it.

cli/integration_tests/prune/produces_valid_turbo_json.t

gsoltis

Unless I missed it, I don't think we define what "strict" or "loose" modes are in the docs anywhere. We discuss how to control which one is picked, but not what they are or what they do.

Let's also make sure that we are including SYSTEMROOT in the default list of passthrough vars, it's required on Windows. Go will do it by default, but there should be no harm in us being explicit and that will ensure it doesn't get lost in the porting effort.

gsoltis · 2023-04-06T17:27:39Z

docs/pages/repo/docs/reference/command-line-reference.mdx

+
+`type: string`
+
+Controls the available variables to your tasks based upon the values specified in the `globalEnvPassthrough` and `envPassthrough` fields in `turbo.json`.


Here and elsewhere on this page globalEnvPassthrough -> experimentalGlobalEnvPassthrough, etc.

gsoltis · 2023-04-06T17:29:05Z

docs/pages/repo/docs/reference/configuration.mdx

+
+`type: string[]`
+
+An allowlist of environment variables that should be made available to this task but should not contribute to the task's cache key. Using this key opts this task into strict environment variable mode.


Maybe add a note that the names contribute to the hash key, but the values don't. So updating this list will bust the cache, but changing the values of vars in the list won't

mehulkar · 2023-04-06T18:07:54Z

@gsoltis added SYSTEMROOT (with tests) and moved docs to #4490, since we want to wait till release on those anyway.

nathanhammond added 3 commits April 4, 2023 20:09

First draft.

e928702

incremental

45fb7da

Source data from turbo.json instead.

abc01a2

nathanhammond requested a review from a team as a code owner April 4, 2023 15:21

nathanhammond requested review from tknickman and NicholasLYang April 4, 2023 15:21

turbo-orchestrator bot added area: docs Improvements or additions to documentation team: turborepo labels Apr 4, 2023

vercel bot deployed to Preview – turbo-site April 4, 2023 15:33 View deployment

nathanhammond added 2 commits April 4, 2023 23:54

Tests for the Go parsing.

5958ed3

Test for globalPassthroughEnv

6fb240b

nathanhammond commented Apr 4, 2023

View reviewed changes

Consider the infer variable in the hash.

da5ac59

NicholasLYang approved these changes Apr 4, 2023

View reviewed changes

crates/turborepo-lib/src/cli.rs Outdated Show resolved Hide resolved

nathanhammond added 2 commits April 5, 2023 01:52

add integration test

f45227e

more testing

1dde429

nathanhammond commented Apr 4, 2023

View reviewed changes

gsoltis reviewed Apr 4, 2023

View reviewed changes

vercel bot deployed to Preview – turbo-site April 5, 2023 22:23 View deployment

vercel bot deployed to Preview – examples-cra-web April 5, 2023 23:15 View deployment

vercel bot deployed to Preview – examples-gatsby-web April 5, 2023 23:17 View deployment

vercel bot deployed to Preview – examples-vite-web April 5, 2023 23:18 View deployment

vercel bot deployed to Preview – examples-basic-web April 5, 2023 23:19 View deployment

vercel bot deployed to Preview – examples-native-web April 5, 2023 23:20 View deployment

vercel bot deployed to Preview – examples-designsystem-docs April 5, 2023 23:20 View deployment

vercel bot deployed to Preview – examples-kitchensink-blog April 5, 2023 23:22 View deployment

vercel bot had a problem deploying to Preview – examples-svelte-web April 5, 2023 23:25 Failure

mehulkar reviewed Apr 6, 2023

View reviewed changes

cli/integration_tests/prune/produces_valid_turbo_json.t Outdated Show resolved Hide resolved

vercel bot deployed to Preview – turbo-site April 6, 2023 06:49 View deployment

mehulkar requested review from gsoltis and NicholasLYang April 6, 2023 16:48

Merge branch 'main' into opt-in-env

e7cfe6a

vercel bot deployed to Preview – examples-cra-web April 6, 2023 16:49 View deployment

vercel bot deployed to Preview – examples-designsystem-docs April 6, 2023 16:50 View deployment

gsoltis suggested changes Apr 6, 2023

View reviewed changes

mehulkar added 3 commits April 6, 2023 11:03

Match casing in key name

2357746

Ensure systemroot is made available in all modes

c946929

Remove docs for this PR

92e9fd3

turbo-orchestrator bot removed the area: docs Improvements or additions to documentation label Apr 6, 2023

revert schema docs

61e3f1b

mehulkar requested a review from gsoltis April 6, 2023 18:08

mehulkar approved these changes Apr 6, 2023

View reviewed changes

gsoltis approved these changes Apr 6, 2023

View reviewed changes

mehulkar enabled auto-merge (squash) April 6, 2023 18:15

make test work in ci

65f35e9

vercel bot deployed to Preview – turbo-site April 6, 2023 18:26 View deployment

rm test for shell being set in task process, not working in ci

571f291

vercel bot deployed to Preview – turbo-site April 6, 2023 18:52 View deployment

mehulkar merged commit 46bba2e into vercel:main Apr 6, 2023

mehulkar deleted the opt-in-env branch April 6, 2023 19:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strict Environment Handling #4449

Strict Environment Handling #4449

nathanhammond commented Apr 4, 2023 •

edited

Loading

vercel bot commented Apr 4, 2023 •

edited

Loading

vercel bot commented Apr 4, 2023

nathanhammond left a comment

nathanhammond Apr 4, 2023

nathanhammond Apr 4, 2023

nathanhammond Apr 4, 2023

NicholasLYang left a comment

nathanhammond left a comment

nathanhammond Apr 4, 2023

nathanhammond Apr 4, 2023

gsoltis Apr 4, 2023

gsoltis left a comment

gsoltis Apr 6, 2023

gsoltis Apr 6, 2023

mehulkar commented Apr 6, 2023


		`type: string`

		Controls the available variables to your tasks based upon the values specified in the `globalEnvPassthrough` and `envPassthrough` fields in `turbo.json`.


		`type: string[]`

		An allowlist of environment variables that should be made available to this task but should not contribute to the task's cache key. Using this key opts this task into strict environment variable mode.

Strict Environment Handling #4449

Strict Environment Handling #4449

Conversation

nathanhammond commented Apr 4, 2023 • edited Loading

Open Design Option

vercel bot commented Apr 4, 2023 • edited Loading

vercel bot commented Apr 4, 2023

nathanhammond left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicholasLYang left a comment

Choose a reason for hiding this comment

nathanhammond left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsoltis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mehulkar commented Apr 6, 2023

nathanhammond commented Apr 4, 2023 •

edited

Loading

vercel bot commented Apr 4, 2023 •

edited

Loading