Support full wildcard syntax in robots.txt directives

[anjackson]

We only support trailing * wildcards at present. Ideally we should support wildcards as defined in https://developers.google.com/search/reference/robots_txt

The code to modify would be:

heritrix3/modules/src/main/java/org/archive/modules/net/RobotsDirectives.java

Lines 40 to 42 in 0581170

	public boolean allows(String path) {
	return !(longestPrefixLength(disallows, path) > longestPrefixLength(allows, path));
	}

The actual wildcards are not that difficult, but getting the precedence right is harder. Perhaps we can use a standard library e.g. the crawler commons code?

[anjackson]

see https://github.com/crawler-commons/crawler-commons/blob/bef1b8437e63930bcbab82a3f754bf835cda5cca/src/main/java/crawlercommons/robots/SimpleRobotRules.java#L153

[jr-ewing]

Maybe interesting https://github.com/google/robotstxt

[ato]

There's a Java port of Google's parser too https://github.com/google/robotstxt-java/ but unfortunately it doesn't seem to be in Maven Central.