Skip to content

fix: check all security.txt paths before returning isPresent false #292

Merged
Lissy93 merged 2 commits intoLissy93:masterfrom
jorgearma:fix/security-txt-fake-negatives
Apr 20, 2026
Merged

fix: check all security.txt paths before returning isPresent false #292
Lissy93 merged 2 commits intoLissy93:masterfrom
jorgearma:fix/security-txt-fake-negatives

Conversation

@jorgearma
Copy link
Copy Markdown
Contributor

@jorgearma jorgearma commented Apr 16, 2026

Bug

When /security.txt returns HTML (e.g. a homepage instead of a 404),
the checker immediately returns isPresent: false without testing the fallback path
/.well-known/security.txt (RFC 9116 standard location).

Expected Behavior

The checker should attempt all known security.txt locations before concluding that it is not present.

According to RFC 9116, /.well-known/security.txt is the canonical location and must always be checked, even if /security.txt returns an unexpected response.

Root Cause

The condition on line 56:

if (result && result.includes('<html')) return { isPresent: false };

exits the function early, preventing the fallback path from being evaluated.

Fix

- if (result && result.includes('<html')) return { isPresent: false };
+ if (result && result.includes('<html')) continue;

Instead of returning early, the loop continues to evaluate the remaining paths.

Impact

This causes false negatives in real-world scenarios where:

  • /security.txt returns HTML (e.g. homepage or redirect)
  • but /.well-known/security.txt exists and is valid

As a result, valid security.txt files are incorrectly reported as missing.

Evidence

The screenshot below shows a real case:

  • /security.txt returns HTML (homepage)
  • /.well-known/security.txt exists and is valid
  • the tool incorrectly reports it as not present
fake-negative

Related

Relates to #189, but addresses the issue more precisely.

Removing the HTML check entirely (as proposed in #189) may introduce false positives by allowing HTML responses to be treated as valid.

This approach:

  • skips invalid HTML responses
  • preserves fallback logic
  • avoids both false negatives and false positives

@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 16, 2026 •
edited
Loading

✅ Deploy Preview for web-check ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit 795d41f
🔍 Latest deploy log https://app.netlify.com/projects/web-check/deploys/69e6a9b986f0eb000830a236
😎 Deploy Preview https://deploy-preview-292--web-check.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@Lissy93
Copy link
Copy Markdown
Owner

Lissy93 commented Apr 20, 2026

LGTM, thanks!

@Lissy93 Lissy93 merged commit 55d8657 into Lissy93:master Apr 20, 2026
5 of 6 checks passed
@jorgearma
Copy link
Copy Markdown
Contributor Author

jorgearma commented Apr 21, 2026

I didn't get the chance to explain this second commit before it was merged, so I wanted to add context here.

About the second commit — User-Agent header

While working on the first fix, I identified a second issue: Node.js does not send a User-Agent header by default. Some servers (like Reddit) detect this and return a blocked/HTML page
instead of the actual security.txt. This triggered the HTML detection logic and caused a false negative even after the continue fix was applied.

The second commit adds User-Agent: curl/8.0.0 to the HTTPS request to prevent this.

Reproduction:

# No User-Agent (Node.js default) → server blocks, returns HTML
curl -s -A "" https://www.reddit.com/.well-known/security.txt | head -3
# → <!doctype html>
                                                                                                                                                                                            
# With User-Agent: curl/8.0.0 → returns valid security.txt
curl -sL -A "curl/8.0.0" https://www.reddit.com/.well-known/security.txt | head -3                                                                                                         
# → -----BEGIN PGP SIGNED MESSAGE-----

Note on production: Despite both fixes being live, reddit.com still returns {"isPresent":false} on web-check.as93.net. The fix works correctly from a residential IP, but the production
Netlify function gets a 403 from Reddit. This suggests Reddit may also be blocking requests from Netlify/AWS datacenter IP ranges at the infrastructure level, though this hasn't been fully
confirmed.

# From residential IP → works
curl -sL "https://reddit.com/.well-known/security.txt" | head -1
# → -----BEGIN PGP SIGNED MESSAGE-----                                                                                                                                                      
 
# Netlify → github.com works fine                                                                                                                                                           
curl -sL "https://web-check.as93.net/.netlify/functions/status?url=https://github.com"
# → {"isUp":true,"responseCode":200}                                                                                                                                                        
 
# Netlify → Reddit gets 403 regardless                                                                                                                                                      
curl -sL "https://web-check.as93.net/.netlify/functions/status?url=https://reddit.com"
# → {"error":"Received non-success response code: 403"}
                                                                                                                                                                                            
I will continue investigating a more robust solution to reduce false negatives caused by server-side blocking of automated requests from datacenter IPs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lissy93 Lissy93 Lissy93 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jorgearma @Lissy93