Initial set of Azure secrets for #539

[dvasdekis]

Description:

Please find in this PR an almost-complete set of Azure secrets as sourced from here, plus some new ones from my own experience.

Checklist:

• Does your PR pass tests?
• Have you written new tests for your changes?
• Have you lint your code locally prior to submission?

[jit-ci]

Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset.

All security workflows are defined in a centralized repository named .jit.
In case there are security findings, they will be communicated to you as a comment inside the PR.

Hope you’ll enjoy using Jit.

Questions? Comments? Want to learn more? Get in touch with us.

[dvasdekis]

I've commented out the secret types I couldn't get to work (mostly the XML ones) but the rest pass go run main.go now. Hopefully you can approve and pull in 👍 and some future genius can fix my missing searches.

[dvasdekis]

Sure, I've accepted the changes

[dvasdekis]

Hi, can I get these changed merged please?

[JoostVoskuil]

Hi @zricethezav , can this be merged? This is a big addition for the Microsoft techstack. Or is there an additional step required?

[rgmz]

Just curious, what's the reason for changing some of the patterns?

-<XstoreAccountInfo[ -~"\s\S\n\r\t]+accountSharedKey\s*=\s*"[^"]{30}[ -~"\s\S\n\r\t]+/>
+[<XstoreAccountInfo].*accountSharedKey\s*=\s*['"].*['"]

In particular, this won't work as expected because [<XstoreAccountInfo] will only match a single character from inside the square brackets, not <XstoreAccountInfo.

[dvasdekis]

Probably because there were like 100 rules and I blew through them with all the enthusiasm that I had, now exhausted. When I get some free time I'll try it again, with the above change

[tgrnwd]

Definitely interested in seeing these rules included. Anything I can do to help @dvasdekis ?

[dvasdekis]

These rules are complete. Not sure why they're not merged

[JoostVoskuil]

@zricethezav how can we complete this?

[zricethezav]

Actually, jk, there are some issues here. Most of these rules do not have keywords, which are necessary if gitleaks is to maintain reasonable performance. Opening up a follow up commit.

FWIW, after adding this PR the scan time more than doubled. Gotta have them keywords

[dvasdekis]

Okay, I can amend the original code with some keywords, when they're available from Azure. But for those in the spec that don't have keywords, are you planning to never merge then?

[nikolamalesevic]

@zricethezav This is not a way to handle this. Many of us have been waiting for this PR for a long time, with inexplicable months of silence from your side, and now jokes and fun and let's throw everything in the fire.

There must be a better way to communicate a way forward.

[JoostVoskuil]

@nikolamalesevic be kind please

[zricethezav]

@nikolamalesevic lol I'm really really sorry for not providing good enough free labor. get real dude

Anyways, this PR adds rules to the default configuration. If you want to use these rules for your own scans then you can, you just need to specify it using --config or one of the other methods.

I admit, I was trigger happy and merged this PR too quickly. Something that should be added is a check that makes sure all new rules have at least one keyword and that the delta in scan time does not exceed some %.

[zricethezav]

@dvasdekis

Okay, I can amend the original code with some keywords, when they're available from Azure. But for those in the spec that don't have keywords, are you planning to never merge then?

I'd be happy to merge if the rules can be updated with keywords. The reason why keywords are so important is because gitleaks does a string compare on a chunk of text before attempting to match a regex. So if no keyword is specified, then that rule would be running regex on every chunk of text, which slows things down dramatically. You can see how this is implemented here:

gitleaks/detect/detect.go

Lines 578 to 596 in 7804d65

	for _, rule := range d.Config.Rules {
	if len(rule.Keywords) == 0 {
	// if not keywords are associated with the rule always scan the
	// fragment using the rule
	findings = append(findings, d.detectRule(fragment, rule)...)
	continue
	}
	fragmentContainsKeyword := false
	// check if keywords are in the fragment
	for _, k := range rule.Keywords {
	if _, ok := fragment.keywords[strings.ToLower(k)]; ok {
	fragmentContainsKeyword = true
	}
	}
	if fragmentContainsKeyword {
	findings = append(findings, d.detectRule(fragment, rule)...)
	}
	}
	return filter(findings, d.Redact)

Feel free to reopen a PR when keywords have been added. Again, sorry for the merging then reverting.

[nikolamalesevic]

@nikolamalesevic be kind please

@nikolamalesevic lol I'm really really sorry for not providing good enough free labor. get real dude

The comment was meant to be an honest feedback specific to handling of this PR (however negative), and never did I say that I do not appreciate a hard work put into this. I do apologize if I came out as being rude.

[zricethezav]

The comment was meant to be an honest feedback specific to handling of this PR (however negative), and never did I say that I do not appreciate a hard work put into this. I do apologize if I came out as being rude.

It's all good, I need a performance check to make sure new rules don't increase the scan time by some %. Again, I'm happy to review new rules if someone is up to the task.