Keep the fetcher off private address space by default by abhinav-phi · Pull Request #2127 · apache/stormcrawler

abhinav-phi · 2026-09-05T23:29:39Z

Fixes #2080.

The connection-time IP filter was the only check that looked at the address the fetcher actually connects to, and both of its keys shipped commented out in crawler-default.yaml, so at defaults the interceptor was never installed. The regex exclusion list in the archetypes matched literal localhost and four dotted-quad ranges and nothing else — link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 forms the JVM resolver maps to 127.0.0.1 all passed it, and InetAddress.getByName maps 127.1 and 2130706433 to loopback without any DNS control.

crawler-default.yaml ships http.filter.ipaddress.exclude enabled: localhost,sitelocal,linklocal,100.64.0.0/10,fd00::/8. A fetched page decides which hosts the fetcher connects to, and these ranges host unauthenticated services (e.g. cloud instance metadata at 169.254.169.254, reachable whatever DNS name a link used, since the check runs on the resolved socket address) which a public index must not leak into
crawling an intranet or a loopback service needs an explicit opt-out — the comment next to the key documents how
the archetype default-regex-filters.txt (and its opensearch/solr copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8, IPv6 fc00::/7 and fe80::/10, abbreviated loopback (127.1) and integer IPv4 hosts (2130706433), and states that a regex list judges the bytes of the URL only — it can never enforce where a host name resolves, so the IP filter stays the authoritative check

Release note needed: enabling the exclude list by default changes behaviour for anyone crawling an intranet or a loopback service; the way to opt out is documented next to the key.

The IP address filter was the only check that looked at the address the fetcher actually connects to, and both of its keys shipped commented out, so the interceptor was never installed at library defaults. The regex exclusion list in the archetypes matched literal localhost and four dotted-quad ranges and nothing else: link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 forms the resolver maps to 127.0.0.1 all passed it. - crawler-default.yaml ships http.filter.ipaddress.exclude enabled (loopback, RFC1918, link-local, CGNAT, IPv6 unique-local); a fetched page decides which hosts the fetcher connects to, and these ranges host unauthenticated services a public index must not leak into. Crawling an intranet needs an explicit opt-out, documented in the comment next to the key - the archetype default-regex-filters.txt (and its opensearch/solr copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8, IPv6 fc00::/7 and fe80::/10, the abbreviated loopback forms (127.1) and integer IPv4 hosts (2130706433), and says in its comment that a regex list judges the bytes of the URL only, so the IP filter stays the authoritative check

rzo1

Currently I am short on time, so first round AI-based review)

Enabling http.filter.ipaddress.exclude by default is correct, and the new NOTE saying the post-DNS filter is the authoritative check belongs in this file. One regex is wrong; see line 45.

Unrelated change. This PR also swaps the skip-suffix regex in core/src/test/resources/default-regex-filters.txt for a much longer list and rewrites a comment. RegexFilterTest and URLFilterBoltTest both load that file. Aligning it with the archetype copy is fine, but it does not belong to #2080. Please split it out, or explain in the description why it has to be here.

Test style. DefaultRegexFiltersPrivateRangeTest.java has imports out of order (java.nio.charset before java.net) and several inline fully-qualified names producing over-long lines. CI runs the format check (-Dskip.format.code=false), so please import Utils, ConfUtils, IPFilterRules and InetAddress properly.

Worth putting in the description so nobody chases it: the JVM resolver rejects hex (0x7f000001 gives UnknownHostException) and does not treat leading zeros as octal (0177.0.0.1 gives 177.0.0.1), so the \d{1,10} rule closes the only numeric form that actually works.

Note this overlaps with #2119, which also touches the http.filter.ipaddress.exclude example. Worth agreeing which PR owns the shipped value.

rzo1 · 2026-09-06T14:02:09Z

+#     169.254.0.0/16 (link-local, e.g. cloud instance metadata services)
+-^https?://(?:169\.254(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+#     100.64.0.0/10 (carrier-grade NAT)
+-^https?://(?:100\.(?:6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)


100.64.0.0/10 is 100.64.0.0 to 100.127.255.255, i.e. second octet 64 to 127.

The alternation is 6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7], which blocks 60 to 64 (60 to 63 are public space) and misses 65 to 69 entirely. The test only checks 100.64.0.1 and 100.127.255.254, so neither end of the mistake is covered.

Suggested change

-^https?://(?:100\.(?:6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)

-^https?://(?:100\.(?:6[4-9]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)

Please apply to all four copies of the file and add 100.65.0.1 and 100.63.0.1 to otherNonRoutableRangesAreRejected.

rzo1 · 2026-09-06T14:02:09Z

+#     0.0.0.0/8 ("this network")
+-^https?://(?:0(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){1,3})(?::\d+)?(?:/|$)
+#     IPv6 unique-local (fc00::/7) and link-local (fe80::/10) addresses
+-^https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)


Two gaps here.

Case. RegexURLFilter does not compile with CASE_INSENSITIVE (the suffix rule at the top of the file carries its own (?i)), so [FD00::1] and [FE80::1] pass. Same for LOCALHOST on line 29.

Loopback spellings. Both of these reach loopback through the JVM resolver while matching neither rule:

InetAddress.getByName("::ffff:127.0.0.1") -> /127.0.0.1 InetAddress.getByName("0:0:0:0:0:0:0:1") -> /0:0:0:0:0:0:0:1 (== ::1)

Suggested change

-^https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)

-^(?i)https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)

# loopback reached through an IPv4-mapped or fully expanded IPv6 literal

-^(?i)https?://\[(?:::ffff:127\.|(?:0*:){7}0*1\])

isLoopbackAddress() catches all of these after DNS now that the IP filter is on by default, so this is defence in depth, but catching them earlier is what this file is for.

rzo1 · 2026-09-06T14:02:09Z

+#     http://2130706433/
 #     http://[::1]/
-^https?://(?:localhost|127(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){3}|\[::1\])(?::\d+)?(?:/|$)
+-^https?://(?:localhost|127(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){0,3}|\[::1\])(?::\d+)?(?:/|$)


Case-sensitive, so LOCALHOST passes. Worth (?i) here too, same as the IPv6 rule on line 49.

rzo1 · 2026-09-06T14:02:09Z

+#     169.254.0.0/16 (link-local, e.g. cloud instance metadata services)
+-^https?://(?:169\.254(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)
+#     100.64.0.0/10 (carrier-grade NAT)
+-^https?://(?:100\.(?:6[0-4]|[7-9][0-9]|1[01][0-9]|12[0-7])(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/|$)


Same CGNAT bug as in the archetype copy; see the comment there.

rzo1 · 2026-09-06T14:02:10Z

+#     0.0.0.0/8 ("this network")
+-^https?://(?:0(?:\.(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))){1,3})(?::\d+)?(?:/|$)
+#     IPv6 unique-local (fc00::/7) and link-local (fe80::/10) addresses
+-^https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)


Same IPv6 gaps as in the archetype copy; see the comment there.

rzo1 · 2026-09-06T14:02:10Z

+  # The exclude list is enabled by default: a fetched page decides which hosts
+  # the fetcher connects to, and loopback, private and link-local ranges host
+  # unauthenticated services (e.g. cloud instance metadata) which a public
+  # index must not leak into. Set http.filter.ipaddress.include to any value


This advice does not work as written.

Set http.filter.ipaddress.include to any value (e.g. "0.0.0.0/0") and leave the exclude list empty

Exclude wins over include in IPFilterRules.accept, so setting only include changes nothing: the shipped exclude list on line 169 still applies. The operator has to write http.filter.ipaddress.exclude: "" explicitly. Please say that.

Also, fd00::/8 here only covers the locally-assigned half of ULA; fc00::/7 is the full range. Same point as in #2119.

rzo1 · 2026-09-06T14:15:17Z

@abhinav-phi CI is red here, could you have a look please?

Two Checkstyle violations, both the import order I flagged in the review:

core/src/test/java/org/apache/stormcrawler/filtering/DefaultRegexFiltersPrivateRangeTest.java:[26,1]
  CustomImportOrder: 'java.net.MalformedURLException' should be before 'java.nio.charset.StandardCharsets'
core/src/test/java/org/apache/stormcrawler/filtering/DefaultRegexFiltersPrivateRangeTest.java:[27,1]
  CustomImportOrder: 'java.net.URL' should be before 'java.nio.charset.StandardCharsets'

Reproduce with mvn -B -Prat -DskipTests verify -Dskip.format.code=false, fix with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false.

Run 33998813190

rzo1 reviewed Sep 6, 2026

View reviewed changes

rzo1 assigned abhinav-phi Sep 6, 2026

rzo1 requested review from dpol1, jnioche, mvolikas and sigee September 6, 2026 14:07

rzo1 added this to the 4.0.0 milestone Sep 6, 2026

Navigation Menu

Uh oh!

Keep the fetcher off private address space by default - #2127

Keep the fetcher off private address space by default#2127
abhinav-phi wants to merge 1 commit into
apache:mainfrom
abhinav-phi:issue-2080-private-address-default

abhinav-phi commented Sep 5, 2026

Uh oh!

rzo1 left a comment

Uh oh!

rzo1 Sep 6, 2026

Uh oh!

rzo1 Sep 6, 2026

Uh oh!

rzo1 Sep 6, 2026

Uh oh!

rzo1 Sep 6, 2026

Uh oh!

rzo1 Sep 6, 2026

Uh oh!

rzo1 Sep 6, 2026

Uh oh!

rzo1 commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	-^https?://(?:100\.(?:6[0-4]\|[7-9][0-9]\|1[01][0-9]\|12[0-7])(?:\.(?:(?:25[0-5]\|2[0-4][0-9]\|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/\|$)
	-^https?://(?:100\.(?:6[4-9]\|[7-9][0-9]\|1[01][0-9]\|12[0-7])(?:\.(?:(?:25[0-5]\|2[0-4][0-9]\|[01]?[0-9][0-9]?))){2})(?::\d+)?(?:/\|$)

--^https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)
+-^(?i)https?://\[(?:f[cd][0-9a-f]{2}|fe[89ab][0-9a-f]):[0-9a-f:]+\](?::\d+)?(?:/|$)
+#     loopback reached through an IPv4-mapped or fully expanded IPv6 literal
+-^(?i)https?://\[(?:::ffff:127\.|(?:0*:){7}0*1\])

Uh oh!

Conversation

abhinav-phi commented Sep 5, 2026

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants