Scrubbing env vars from logs #1315

helsaawy · 2022-03-07T17:57:30Z

Scrubbing environment variables from logs of container configuration to prevent leaking sensitive information.

Added config parameter, ScrubLogs, to enable scrubbing.

Will have to be re-vendored into CRI fork to be deployed properly.

Signed-off-by: Hamza El-Saawy hamzaelsaawy@microsoft.com

Added code to remove environment variables from code path. Need to add toggle mechanism Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

jterry75 · 2022-03-07T18:16:15Z

While this works, it doesnt prevent containerd from logging the env's right? ttrpc trace exporter right? have we considered using tagging on the task proto and trying to work with containerd to support this? It seems like something along the lines of

type test struct {
   a string `scrub:always`
   b string `scrub:when-disabled`
   c string `scrub:never`
}

And then the policy of the scrub can do something more generic when writing out the struct to text in a trace message?

helsaawy · 2022-03-07T18:27:34Z

I have another PR on our fork of containerd to scrub the environment variables from its logs.
I hadnt considered tagging the structs, but I'll look into that.
Since most of the logs here used already json-serialized strings that I had to unmarshal, scrub, and then re-marshal, I was hesitant to create a generic solution since that would involve repeated json parsing, which seemed excessive:

v = &A{}
if err := json.Unmarshal(b, &pp); err == nil {
    //scrub
    // ...
    return json.Marshal(b, v)
}
v = &B{}
if err := json.Unmarshal(b, &pp); err == nil {
    // ...
}

jterry75 · 2022-03-07T18:32:28Z

I have another PR on our fork of containerd to scrub the environment variables from its logs. I hadnt considered tagging the structs, but I'll look into that. Since most of the logs here used already json-serialized strings that I had to unmarshal, scrub, and then re-marshal, I was hesitant to create a generic solution since that would involve repeated json parsing, which seemed excessive:
v = &A{}
if err := json.Unmarshal(b, &pp); err == nil {
    //scrub
    // ...
    return json.Marshal(b, v)
}
v = &B{}
if err := json.Unmarshal(b, &pp); err == nil {
    // ...
}

Ugh.. :). Logging right...

helsaawy · 2022-03-07T18:37:41Z

Ugh.. :). Logging right...

Yeah ... unfortunately the function we log receive serialized strings (or structs with serialized strings as fields) rather than structs as parameters, so the whole pars-scrub-reparse song and dance is unavoidable, and I am really hesitant to incur a serious performance hit :(

cmd/containerd-shim-runhcs-v1/options/runhcs.proto

dcantah · 2022-03-08T15:45:59Z

internal/scrub/scrub.go

@@ -0,0 +1,174 @@
+// This package scrubs objects of customer information to pass to logging


I'd almost say maybe this makes sense in /internal/log instead to keep them close together (in a scrub.go file)

Although I guess the scenario I'm picturing in my mind is you'd set an option on the logger itself and it would handle scrubbing internally.

I was debating adding a flag in the context, but that, along with setting a flag on the logger, wouldnt work because a lot of places create a new logger from scratch rather than inherit it from parent functions.
I do like the idea of moving it to internal/log though

internal/scrub/scrub.go

dcantah · 2022-03-08T15:52:21Z

While this works, it doesnt prevent containerd from logging the env's right? ttrpc trace exporter right? have we considered using tagging on the task proto and trying to work with containerd to support this? It seems like something along the lines of
type test struct {
   a string `scrub:always`
   b string `scrub:when-disabled`
   c string `scrub:never`
}
And then the policy of the scrub can do something more generic when writing out the struct to text in a trace message?

This seems like a great idea, I'd tried something similar for marking fields as deprecated (for a different project, not here).

anmaxvl · 2022-03-08T06:42:42Z

internal/scrub/scrub.go

+
+// scrubbing utilities to remove user information from arbitrary objects
+
+type genMap = map[string]interface{}


is it the same as type genMap map[string]interface{}?

nope, this is a type alias, so genMap is equivalent to a map[string]interface{} and I dont have to convert between the two.
This is just to save keystrokes and make functions look cleaner.

internal/scrub/scrub.go

helsaawy · 2022-03-08T17:12:15Z

And then the policy of the scrub can do something more generic when writing out the struct to text in a trace message?

This seems like a great idea, I'd tried something similar for marking fields as deprecated (for a different project, not here).

So I am trying to implement this: hcsschema.ProcessParameters is auto-generated (via Swagger), and I'm nervous that if it is regenerated and the scrub:always tag isn't manually added back in, then scrubbing comes crashing down.

I do think that we should revisit this with a more robust solution after closing up these leaks.

dcantah · 2022-03-08T17:15:17Z

And then the policy of the scrub can do something more generic when writing out the struct to text in a trace message?

This seems like a great idea, I'd tried something similar for marking fields as deprecated (for a different project, not here).

So I am trying to implement this: hcsschema.ProcessParameters is auto-generated (via Swagger), and I'm nervous that if it is regenerated and the scrub:always tag isn't manually added back in, then scrubbing comes crashing down.

I do think that we should revisit this with a more robust solution after closing up these leaks.

Ahh, I was replying about on the containerd end for the protos. Yea, not sure how we'd get away with our swagger files..

Moved scrubbing under `internal/log`. Added proper godocs to exported functions Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

katiewasnothere · 2022-03-09T23:33:30Z

internal/gcs/bridge.go

@@ -365,6 +367,7 @@ func (brdg *bridge) recvLoop() error {
 func (brdg *bridge) sendLoop() {
 	var buf bytes.Buffer
 	enc := json.NewEncoder(&buf)
+	enc.SetEscapeHTML(false)


Why did we add this?

default behavior is to escape characters for HTML, and since the replacement is <scrubbed>, that gets encoded as \u003cscrubbed\u0033, which i felt would cause issues for pattern matching down the line

katiewasnothere · 2022-03-09T23:39:10Z

internal/log/scrub.go

+type genMap = map[string]interface{}
+type scrubberFunc func(genMap) error
+
+const ScrubbedReplacement = "<scrubbed>"


This doesn't need to be exported right?

no, ill also unexport it

katiewasnothere · 2022-03-09T23:48:48Z

internal/log/scrub.go

+func ScrubProcessParameters(s string) (string, error) {
+	// todo: deal with v1 ProcessConfig
+	b := []byte(s)
+	if !IsScrubbingEnabled() || !hasKeywords(b) || !json.Valid(b) {


Should we be returning an error if the string isn't a valid json?

it could be a go-formatted string, or some other parameter that is passed in erroneously, in which case I felt it best to just return it as is

katiewasnothere · 2022-03-09T23:52:13Z

internal/log/scrub.go

+	}
+	pp.Environment = map[string]string{ScrubbedReplacement: ScrubbedReplacement}
+
+	buf := bytes.NewBuffer(b[:0])


Why did we need to use a NewBuffer type here but in scrubBytes we don't?

since we control the byte slice here (from converting the string to a byte string), we can reuse the storage
for scrubBytes, the byte slice is passed in from the caller and we dont own it, so using it here would overwrite the contents, which, for the bridge, is the payload being sent.

katiewasnothere · 2022-03-09T23:56:31Z

internal/log/scrub.go

+func isRequestBase(m genMap) bool {
+	// neither of these are (currently) `omitempty`
+	_, a := m["ActivityId"]
+	_, c := m["ContainerId"]
+	return a && c
+}


I'm not a huge fan of parsing all of the things in this file in string format, this seems like an unreliable way to determine fields. The alternative was marshalling and unmarshalling right?

yes, but that required exporting the internal bridge protocol structs, and I didnt know if that was an issue

Related work items: #1240, #1241, #1258, #1260, #1270, #1271, #1272, #1279, #1286, #1288, #1289, #1292, #1293, #1294, #1295, #1297, #1298, #1299, #1301, #1305, #1307, #1308, #1309, #1310, #1313, #1314, #1315, #1318, #1320, #1321, #1322, #1324, #1326, #1328, #1332, #1333, #1334, #1335, #1337

Added code to remove environment variables from code path. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Scrubbing env vars from logs

1574549

Added code to remove environment variables from code path. Need to add toggle mechanism Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

helsaawy requested a review from a team as a code owner March 7, 2022 17:57

dcantah reviewed Mar 8, 2022

View reviewed changes

cmd/containerd-shim-runhcs-v1/options/runhcs.proto Outdated Show resolved Hide resolved

dcantah reviewed Mar 8, 2022

View reviewed changes

internal/scrub/scrub.go Outdated Show resolved Hide resolved

dcantah reviewed Mar 8, 2022

View reviewed changes

internal/scrub/scrub.go Outdated Show resolved Hide resolved

dcantah reviewed Mar 8, 2022

View reviewed changes

internal/scrub/scrub.go Outdated Show resolved Hide resolved

anmaxvl approved these changes Mar 8, 2022

View reviewed changes

helsaawy force-pushed the he/env branch from 41a3083 to f7c9865 Compare March 8, 2022 17:05

PR: documentation and organization

7eaa713

Moved scrubbing under `internal/log`. Added proper godocs to exported functions Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

helsaawy force-pushed the he/env branch from f7c9865 to 7eaa713 Compare March 8, 2022 18:02

dcantah approved these changes Mar 8, 2022

View reviewed changes

helsaawy merged commit d512c70 into microsoft:master Mar 8, 2022

katiewasnothere mentioned this pull request Mar 9, 2022

Scrubbing env from logs kevpar/cri#24

Merged

katiewasnothere reviewed Mar 9, 2022

View reviewed changes

princepereira pushed a commit to princepereira/hcsshim that referenced this pull request Aug 29, 2024

Scrubbing env vars from logs (microsoft#1315)

d605133

Added code to remove environment variables from code path. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scrubbing env vars from logs #1315

Scrubbing env vars from logs #1315

helsaawy commented Mar 7, 2022 •

edited

Loading

jterry75 commented Mar 7, 2022

helsaawy commented Mar 7, 2022

jterry75 commented Mar 7, 2022

helsaawy commented Mar 7, 2022 •

edited

Loading

dcantah Mar 8, 2022

dcantah Mar 8, 2022

helsaawy Mar 8, 2022 •

edited

Loading

dcantah commented Mar 8, 2022 •

edited

Loading

anmaxvl Mar 8, 2022

helsaawy Mar 8, 2022

helsaawy commented Mar 8, 2022 •

edited

Loading

dcantah commented Mar 8, 2022

katiewasnothere Mar 9, 2022

helsaawy Mar 10, 2022

katiewasnothere Mar 9, 2022

helsaawy Mar 10, 2022

katiewasnothere Mar 9, 2022

helsaawy Mar 10, 2022

katiewasnothere Mar 9, 2022

helsaawy Mar 10, 2022

katiewasnothere Mar 9, 2022

helsaawy Mar 10, 2022

		@@ -0,0 +1,174 @@
		// This package scrubs objects of customer information to pass to logging


		// scrubbing utilities to remove user information from arbitrary objects

		type genMap = map[string]interface{}

Scrubbing env vars from logs #1315

Scrubbing env vars from logs #1315

Conversation

helsaawy commented Mar 7, 2022 • edited Loading

jterry75 commented Mar 7, 2022

helsaawy commented Mar 7, 2022

jterry75 commented Mar 7, 2022

helsaawy commented Mar 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helsaawy Mar 8, 2022 • edited Loading

Choose a reason for hiding this comment

dcantah commented Mar 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helsaawy commented Mar 8, 2022 • edited Loading

dcantah commented Mar 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helsaawy commented Mar 7, 2022 •

edited

Loading

helsaawy commented Mar 7, 2022 •

edited

Loading

helsaawy Mar 8, 2022 •

edited

Loading

dcantah commented Mar 8, 2022 •

edited

Loading

helsaawy commented Mar 8, 2022 •

edited

Loading