ValueError: substring not found

[smuu]

I get the error below when running the audit on my baseline.

Command:
detect-secrets audit --json .secrets.baseline

Error Message:

Traceback (most recent call last):
  File "/home/codespace/.local/bin/detect-secrets", line 8, in <module>
    sys.exit(main())
  File "/home/codespace/.local/lib/python3.8/site-packages/detect_secrets/main.py", line 32, in main
    handle_audit_action(args)
  File "/home/codespace/.local/lib/python3.8/site-packages/detect_secrets/main.py", line 152, in handle_audit_action
    audit.audit_baseline(args.filename[0])
  File "/home/codespace/.local/lib/python3.8/site-packages/detect_secrets/audit/audit.py", line 25, in audit_baseline
    if _classify_secrets(get_secret_iterator(secrets)):
  File "/home/codespace/.local/lib/python3.8/site-packages/detect_secrets/audit/audit.py", line 42, in _classify_secrets
    io.print_context(
  File "/home/codespace/.local/lib/python3.8/site-packages/detect_secrets/audit/io.py", line 37, in print_context
    context.snippet.highlight_line(context.secret.secret_value)
  File "/home/codespace/.local/lib/python3.8/site-packages/detect_secrets/util/code_snippet.py", line 75, in highlight_line
    index_of_payload = self.target_line.lower().index(payload.lower())
ValueError: substring not found

Additional Information:

        "type": "Hex High Entropy String",
        "filename": "<removed string>",
        "hashed_secret": "<removed string>",
        "is_verified": false,
        "line_number": 678

Corresponding part of the file:

678:        example: >-
679:          <removed string>

It seems that the tool can't handle such syntax in yaml files.

[uklft]

Same here for a huge nested single-line json file (valid, though):

Secret:      3 of 3
Filename:    filename.json
Secret Type: Base64 High Entropy String
----------
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/secret_env/bin/detect-secrets", line 8, in <module>
    sys.exit(main())
  File "/home/user/anaconda3/envs/secret_env/lib/python3.9/site-packages/detect_secrets/main.py", line 32, in main
    handle_audit_action(args)
  File "/home/user/anaconda3/envs/secret_env/lib/python3.9/site-packages/detect_secrets/main.py", line 152, in handle_audit_action
    audit.audit_baseline(args.filename[0])
  File "/home/user/anaconda3/envs/secret_env/lib/python3.9/site-packages/detect_secrets/audit/audit.py", line 25, in audit_baseline
    if _classify_secrets(get_secret_iterator(secrets)):
  File "/home/user/anaconda3/envs/secret_env/lib/python3.9/site-packages/detect_secrets/audit/audit.py", line 42, in _classify_secrets
    io.print_context(
  File "/home/user/anaconda3/envs/secret_env/lib/python3.9/site-packages/detect_secrets/audit/io.py", line 37, in print_context
    context.snippet.highlight_line(context.secret.secret_value)
  File "/home/user/anaconda3/envs/secret_env/lib/python3.9/site-packages/detect_secrets/util/code_snippet.py", line 75, in highlight_line
    index_of_payload = self.target_line.lower().index(payload.lower())
ValueError: substring not found

[JuanJTorres11]

I have the same issue, it seems that the tool has a problem with the | character in a yml. It seems that it detects a secret in the line below, but states that the line is the one with the | for some reason, so when the audit searches for this secret to show it, it fails because it doesn't exist in that line. While that is fixed, a quick workaround is to cath that Error to avoid the tool falling and not completing the rest of the audit.

[jpdakran]

Yes, this looks like to be an issue with multi-line secrets. As stated in the README, multi-line secrets are not supported. I do agree that failures caused by this caveat should be caught and fail gracefully.

[jpdakran]

I investigated this further and it does look like this is a deeper rooted issue with the YAML file parser - specifically block vs literal style. For the time being, I will have a fix so we catch the error and allow the tool to continue processing. I will also add a work item for myself to further investigate the YAML file parser.

[ghilainm]

Is there any way to install some kind of snapshot/nightly of detect-secrets which include this fix?

[jpdakran]

@ghilainm Not at the present time. We will have to wait until a new release of detect-secrets. This should be happening very soon. We will keep this point as a take-away to determine how we can ship fixes faster in the future.

[hwine]

Is there any way to install some kind of snapshot/nightly of detect-secrets which include this fix?

@ghilainm @ArisBee If you're desperate, and willing to take a hack from a stranger, see this note.