fix(detect-secrets): improve how secret detection works #6109

pvinis · 2022-01-26T17:17:07Z

The type of this PR is: TYPE

This PR resolves []

Description

Secret detection has been added, but it had some small issues with certain filepaths and how we handle these.

There's some context info here https://artsy.slack.com/archives/CP9P4KR35/p1643215901266300?thread_ts=1643198172.246900&cid=CP9P4KR35 but the gist of it is:

eigen has a bunch of lockfiles, image files, and generated files, that we can skip when detecting secrets. i added this filtering.
eigen has some filepaths with spaces, ', (, ) in them. the original bash script did not handle them, and to account for them we just skipped them. i fixed that by making xargs do better delimiting so the command can run.
added --verbose so we can see progress and any wrong running of the command.
we had some excluded lines and secrets that were for the relay generated files. i removed that since we are filtering these files out with --exclude-files.
removed the python version of detect-secrets and used the brew version. (more info: PLATFORM-3565: add detect-secrets formula homebrew-formulas#13)
added some basic troubleshooting docs for secrets.

so the idea is:

we run the generate script at the start. that is done already. (we might need to run it again in the future, but it should be fine for a while, if we dont change many things).
we run the check scripts on precommit hook and on ci, to make sure we dont commit secrets.

todo:

make sure CI works.

PR Checklist (tick all before merging)

I have included screenshots or videos to illustrate my changes, or I have not changed anything that impacts the UI.
I have tested my changes on iOS and Android.
I have added tests/stories for my changes, or my changes don't require testing/stories, or I have included a link to a separate Jira ticket covering the tests.
I have added a feature flag, or my changes don't require a feature flag. (How do I add one?)
I have documented any follow-up work that this PR will require, or it does not require any.
I have added an app state migration, or my changes do not require one. (What are migrations?)
I have added a changelog entry below or my changes do not require one.

To the reviewers 👀

I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

better secret detection - pavlos

ArtsyOpenSource · 2022-01-26T17:20:52Z

This PR contains the following changes:

Dev changes (better secret detection - pavlos)

Generated by 🚫 dangerJS against cbe270d

ovasdi

LGTM.

I have not run this PR on the simulator or device. Happy to do tomorrow. I haven't run eigen since summer so I suspect im out of date as far as deps/tooling goes. Is there something specific I should look out for?

ovasdi · 2022-01-26T19:53:59Z

docs/troubleshooting.md

+
+This happens when you try to commit some code that looks like a secret, a key, a token, etc.
+Make sure what you are committing has no sensitive data in it.
+If you are sure is it _not_ sensitive data, then you can add an inline comment containing `pragma: allowlist secret`, to signify it is ok to commit. Then try to commit again, and it will work this time.


Nice. I will update the playbook to include this suggestion as well.
One thing I recalled when looking at this, there is an open issue when using the next line variant of the inline allowlist (_pragma: allowlist nextline secret _), it does not work in .env files

i think we would only need to use the nextline for typescript, since prettier might format things to fit for width. for .env files, i think we are ok with the inline one, and the line can go as long as we want. good to know though.

ovasdi · 2022-01-26T20:01:46Z

scripts/secrets-add-all

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash


This is really nice. Moving these commands with exclusions into dedicated scripts keeps things clean and contained. 🙌

ovasdi · 2022-01-26T20:06:24Z

.circleci/config.yml

@@ -221,7 +221,7 @@ jobs:
    working_directory: /usr/src/app
    steps:
      - checkout
-      - run: detect-secrets-hook --baseline .secrets.baseline --exclude-secrets '[a-fA-F0-9]{24}' --exclude-lines 'W/"[!#-\x7E]*"' $(git ls-files | grep -v stickerpack)
+      - run: ./scripts/secrets-check-all


Has eigen had any funkiness in CI with this tool? We had to implement a work around (in some projects) for what seems to be a random mutation of the baseline generated_at field. I have never heard eigen having this issue tho.

i think we did have this at some point at the beginning, but it was only while people didnt have their stations set up, so the ci would be the first time this would run. we havent had an issue since.

i dont want to complicate things for now, so if we do end up having this again often, ill definitely try to fix it then.

pvinis · 2022-01-27T09:45:31Z

no need to run eigen for this, especially if you havent touched it in a while. i will show this on next KS and do a demo, but if you wanna try it out, i think the minimum thing is:

checkout this pr
make a branch from that
add a line in a tsx file like const SECRET_KEY = "ghp_wowsecret5347gANkA3ADQTKpkQS544eoyvU"
maybe another line console.log(SECRET_KEY) so the compiler stops crying
try to commit. you should get an error about the secret, and you cant commit
then try to add the pragma line and commit again. that should now commit fine.
then try to remove the pragma line, commit again. it should fail again.
that should be all

extra:

one extra thing could be if we want to commit this without pragma, and in that special case you can run yarn secrets-add-all and it will scan all files and add the new one on the baseline file. verify that happens in your git, and we are good to go, should be able to commit just fine.

dblandin

Nice!!

In addition to line comments I have couple higher level questions:

Do we want to ignore files via --exclude-files for some commands and not others? I'm wondering if we should be consistent with which files we're excluding.
Is there an opportunity to DRY up some of the scripts? Seems like most common change is what we're piping in (git ls-files vs git diff --staged --name-only) - we might be able to have fewer scripts and inline that input into the package.json commands
Do we intend to run the "add/generate" commands regularly? If not, would it make sense to document that process / commands rather than defining these package.json commands?

dblandin · 2022-01-27T14:08:08Z

package.json

+    "secrets-check-all": "scripts/secrets-check-all",
+    "secrets-add-staged": "scripts/secrets-add-staged",
+    "secrets-add-all": "scripts/secrets-add-all",
+    "secrets-generate-once": "scripts/secrets-generate-baseline",


Looks like these commands might benefit from some namespacing. What do you think of a secrets: command namespace and a ./scripts/secrets/ folder?

"secrets:audit": "detect-secrets audit .secrets.baseline", "secrets:check:staged": "scripts/secrets/check-staged", "secrets:check:all": "scripts/secrets/check-all", "secrets:generate": "scripts/secrets/generate",

thats a nice idea

dblandin · 2022-01-27T14:09:17Z

requirements.txt

-detect-secrets >= 1.1
+# commented out for now, and replaced with the brew version
+# read more here: https://github.com/artsy/homebrew-formulas/pull/13
+# detect-secrets==1.1


suggestion: I feel like we might as well git remove this file if we're commenting all of the lines. We can always bring the file back if we need to but chances are that we'll stick with the brew-based installation and this file will remain unused and cluttering the root directory.

i guess i needed just a tiny extra push haha. doing it.

ovasdi · 2022-01-27T14:28:45Z

@pvinis

➜  eigen git:(verify-secrets-improvement) git diff
modified: src/lib/AndroidApp.tsx
@ src/lib/AndroidApp.tsx:37 @ addTrackingProvider("console", ConsoleTrackingProvider)
if (UIManager.setLayoutAnimationEnabledExperimental) {
  UIManager.setLayoutAnimationEnabledExperimental(true)
}

const SECRET_KEY = "ghp_wowsecret5347gANkA3ADQTKpkQS544eoyvU"
console.log(SECRET_KEY)
const Main: React.FC<{}> = track()(({}) => {
  useEffect(() => {
    GoogleSignin.configure({

➜  eigen git:(verify-secrets-improvement) ✗ git commit -m "add secret"
yarn run v1.22.17
$ /Users/ozzievasdi/code/eigen/node_modules/.bin/lint-staged
✔ Preparing...
✔ Running tasks...
✔ Applying modifications...
✔ Cleaning up...
✨  Done in 10.15s.
yarn run v1.22.17
$ scripts/secrets-check-staged
+ git diff --staged --name-only
+ tr '\n' '\0'
+ xargs -0 detect-secrets-hook --baseline .secrets.baseline --verbose
[scan]	INFO	Checking file: src/lib/AndroidApp.tsx
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
ERROR: Potential secrets about to be committed to git repo!

Secret Type: Base64 High Entropy String
Location:    src/lib/AndroidApp.tsx:37

Secret Type: Secret Keyword
Location:    src/lib/AndroidApp.tsx:37

Possible mitigations:
  - For information about putting your secrets in a safer place, please ask in
    #security
  - Mark false positives with an inline `pragma: allowlist secret`
    comment

If a secret has already been committed, visit
https://help.github.com/articles/removing-sensitive-data-from-a-repository
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
husky - pre-commit hook exited with code 1 (error)

➜  eigen git:(verify-secrets-improvement) ✗ git diff
modified: src/lib/AndroidApp.tsx
@ src/lib/AndroidApp.tsx:37 @ addTrackingProvider("console", ConsoleTrackingProvider)
if (UIManager.setLayoutAnimationEnabledExperimental) {
  UIManager.setLayoutAnimationEnabledExperimental(true)
}

const SECRET_KEY = "ghp_wowsecret5347gANkA3ADQTKpkQS544eoyvU"
const SECRET_KEY = "ghp_wowsecret5347gANkA3ADQTKpkQS544eoyvU" // pragma: allowlist secret
console.log(SECRET_KEY)
const Main: React.FC<{}> = track()(({}) => {
  useEffect(() => {

➜  eigen git:(verify-secrets-improvement) ✗ git commit -m "add secret"
yarn run v1.22.17
$ /Users/ozzievasdi/code/eigen/node_modules/.bin/lint-staged
✔ Preparing...
✔ Hiding unstaged changes to partially staged files...
✔ Running tasks...
✔ Applying modifications...
✔ Restoring unstaged changes to partially staged files...
✔ Cleaning up...
✨  Done in 10.04s.
yarn run v1.22.17
$ scripts/secrets-check-staged
+ git diff --staged --name-only
+ tr '\n' '\0'
+ xargs -0 detect-secrets-hook --baseline .secrets.baseline --verbose
[scan]	INFO	Checking file: src/lib/AndroidApp.tsx
[scan]	INFO	Skipping secret due to `detect_secrets.filters.allowlist.is_line_allowlisted`.
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
[scan]	INFO	Skipping secret due to `detect_secrets.filters.heuristic.is_indirect_reference`.
✨  Done in 0.94s.
[verify-secrets-improvement e15146ca8] add secret
 1 file changed, 2 insertions(+), 1 deletion(-)

removed inline allowlist

➜  eigen git:(verify-secrets-improvement) yarn secrets-add-all
yarn run v1.22.17
$ scripts/secrets-add-all
+ git ls-files
+ tr '\n' '\0'
+ xargs -0 detect-secrets scan --exclude-files '/.lock$/' --exclude-files '/.png$/' --exclude-files '/.webp$/' --exclude-files '/.jpg$/' --exclude-files '/.jar$/' --exclude-files /__generated__/ --baseline .secrets.baseline
✨  Done in 13.03s.

➜  eigen git:(verify-secrets-improvement) ✗ git diff
modified: .secrets.baseline
@ .secrets.baseline:412 @
        "line_number": 151
      }
    ],
    "src/lib/AndroidApp.tsx": [
      {
        "type": "Base64 High Entropy String",
        "filename": "src/lib/AndroidApp.tsx",
        "hashed_secret": "7d13f5838b78a45234b6e3c3ee7676a6ce6ae875",
        "is_verified": false,
        "line_number": 37
      },
      {
        "type": "Secret Keyword",
        "filename": "src/lib/AndroidApp.tsx",
        "hashed_secret": "7d13f5838b78a45234b6e3c3ee7676a6ce6ae875",
        "is_verified": false,
        "line_number": 37
      }
    ],
    "src/lib/Scenes/Artwork/Components/CommercialInformation.tests.tsx": [
      {
        "type": "Hex High Entropy String",
@ .secrets.baseline:578 @
      }
    ]
  },
  "generated_at": "2022-01-26T19:00:43Z"
  "generated_at": "2022-01-27T14:19:05Z"
}

ovasdi · 2022-01-27T14:56:08Z

Do we intend to run the "add/generate" commands regularly? If not, would it make sense to document that process / commands rather than defining these package.json commands?

@dblandin I've been involved with this tool for some time and I have not had to add secrets and I don't immediately recall having to inline exclude any (it may have happened) however I've had to re-generate the baseline on a number of occasions, thought not in a while. This was something that would happen more frequently after we started to incorporate this tool to projects. Just wanted to share my experience.

package.json

.husky/pre-commit

scripts/install

brainbicycle

this looks great to me, might be worth sharing out in mobile-practice before or after it is merged, good to merge from my perspective

pvinis · 2022-02-03T08:43:30Z

this should be good to go. ill mark it as #squashongreen and we can show it on the next free KS. the last two were taken over by other stuff.

changes

64ee202

pvinis requested review from olerichter00, brainbicycle, joeyAghion and ovasdi January 26, 2022 17:19

pvinis assigned brainbicycle Jan 26, 2022

pvinis added 5 commits January 26, 2022 19:51

more

8562c5b

some more

b3a9cc6

back to bash

08dcc1a

rename

9bf4cba

fix

cabdd95

pvinis marked this pull request as ready for review January 26, 2022 19:30

ovasdi previously approved these changes Jan 26, 2022

View reviewed changes

dblandin reviewed Jan 27, 2022

View reviewed changes

pvinis commented Jan 28, 2022

View reviewed changes

package.json Outdated Show resolved Hide resolved

Delete requirements.txt

121471b

pvinis dismissed ovasdi’s stale review via 121471b January 28, 2022 15:59

pvinis commented Jan 28, 2022

View reviewed changes

.husky/pre-commit Outdated Show resolved Hide resolved

pvinis commented Jan 28, 2022

View reviewed changes

scripts/install Outdated Show resolved Hide resolved

Update scripts/install

6364f70

brainbicycle previously approved these changes Feb 1, 2022

View reviewed changes

Update .husky/pre-commit

cdf66b2

pvinis dismissed brainbicycle’s stale review via cdf66b2 February 3, 2022 08:27

pvinis added 2 commits February 3, 2022 10:38

Update package.json

234babe

Merge branch 'main' into pvinis/secrets-improvement

c23bc2f

artsy-peril bot added the Squash On Green A label to indicate that Peril should squash-merge this PR when all statuses are green label Feb 3, 2022

small update

cbe270d

artsy-peril bot merged commit 4b4f347 into main Feb 3, 2022

artsy-peril bot deleted the pvinis/secrets-improvement branch February 3, 2022 10:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(detect-secrets): improve how secret detection works #6109

fix(detect-secrets): improve how secret detection works #6109

pvinis commented Jan 26, 2022 •

edited

ArtsyOpenSource commented Jan 26, 2022 •

edited

ovasdi left a comment •

edited

ovasdi Jan 26, 2022

pvinis Jan 27, 2022

ovasdi Jan 26, 2022

ovasdi Jan 26, 2022

pvinis Jan 27, 2022

pvinis commented Jan 27, 2022

dblandin left a comment

dblandin Jan 27, 2022

pvinis Jan 28, 2022

dblandin Jan 27, 2022

pvinis Jan 28, 2022

ovasdi commented Jan 27, 2022

ovasdi commented Jan 27, 2022 •

edited

brainbicycle left a comment •

edited

pvinis commented Feb 3, 2022

fix(detect-secrets): improve how secret detection works #6109

fix(detect-secrets): improve how secret detection works #6109

Conversation

pvinis commented Jan 26, 2022 • edited

Description

PR Checklist (tick all before merging)

To the reviewers 👀

Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

ArtsyOpenSource commented Jan 26, 2022 • edited

This PR contains the following changes:

ovasdi left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvinis commented Jan 27, 2022

dblandin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ovasdi commented Jan 27, 2022

ovasdi commented Jan 27, 2022 • edited

brainbicycle left a comment • edited

Choose a reason for hiding this comment

pvinis commented Feb 3, 2022

pvinis commented Jan 26, 2022 •

edited

ArtsyOpenSource commented Jan 26, 2022 •

edited

ovasdi left a comment •

edited

ovasdi commented Jan 27, 2022 •

edited

brainbicycle left a comment •

edited