docs: add Common Recipes section for well-known files#2527
Merged
Conversation
bump version to v1.8.1 (projectdiscovery#2381)
release httpx v1.9.0
Per maintainer feedback on PR projectdiscovery#2472, add a 'Common Recipes' section demonstrating how to detect security.txt, robots.txt, sitemap.xml, and other well-known URIs using httpx's composable primitives. Includes: - security.txt detection with RFC 9116 compliance checks - robots.txt and sitemap.xml probes - IANA well-known URIs registry reference - One-liner examples using -path, -mc, -mct, -mr flags Reference: projectdiscovery#2468 (comment)
WalkthroughAdds exported well-known probing recipes, fixture-backed validation for positive and negative matches, and README examples for common files and standardized URIs. ChangesWell-known recipes
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Member
|
@coderabbitai create a tracking issue for this PR, do not assign it to me |
3 tasks
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
repo="projectdiscovery/httpx"
gh issue create \
--repo "$repo" \
--title "Track: docs - Common Recipes section for well-known files (`#2527`)" \
--body "$(cat <<'EOF'
## Summary
Tracking issue for PR `#2527`, which adds a **Common Recipes** section to the README demonstrating how to detect `security.txt`, and other well-known URIs using httpx's composable primitives (`-path`, `-mc`, `-mct`, `-mr`).
## Background
Per maintainer feedback in `#2472` and the follow-up discussion in `#2468` (comment: https://github.com/projectdiscovery/httpx/issues/2468#issuecomment-4298328468), this PR turns the previously closed PR `#2472` (which proposed adding per-file flags) into documentation showing recipes for well-known files, aligning with httpx's philosophy of composability over specialized flags.
## Scope / Affected areas
- `README.md`: new "Common Recipes" section inserted before the "# Notes" section, with one-liner examples and links to RFC 9116 (security.txt) and the IANA well-known URIs registry.
## Acceptance criteria
- [ ] Common Recipes section merged into README.md
- [ ] Wording/format reviewed and iterated on as needed
- [ ] Examples verified to work against `httpx`
## Links
- PR: https://github.com/projectdiscovery/httpx/pull/2527
- Requested by: `@Mzack9999`
EOF
)"Length of output: 1515
|
3 tasks
Mzack9999
approved these changes
Jul 16, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Line 288: Change the “Common Recipes” heading from h1 to h2 so the document’s
heading hierarchy does not skip levels and the markdownlint warning is resolved.
- Line 292: Update the README guidance for -mr/-ms to state that
OutputMatchRegex and OutputMatchString match the full raw response via resp.Raw,
rather than response headers only. Keep the -mdc description as the option for
structured field matching with DSL helpers such as contains.
In `@runner/wellknown_recipes_test.go`:
- Around line 18-41: Update TestMain to capture m.Run’s exit code, explicitly
call os.RemoveAll(tmp) and handle any cleanup error before invoking os.Exit, and
remove the ineffective deferred cleanup. Preserve the existing setup-error exits
while ensuring cleanup runs after the test suite completes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 519c1948-87c2-4f22-b9a2-b306c783940b
📒 Files selected for processing (3)
README.mdrunner/wellknown_recipes.gorunner/wellknown_recipes_test.go
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
README.md (1)
296-300: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAvoid claiming this matcher proves RFC 9116 validity.
The command only checks
content_type, the presence ofContact:, andmailto:/https://anywhere in the body; it does not validate RFC 9116 field syntax or associate the URI with theContactfield. Describe this as a heuristic/security.txt-like match, or strengthen the matcher before calling it “valid.”Proposed wording
- Probe for a valid [RFC 9116](https://www.rfc-editor.org/rfc/rfc9116.html) security.txt file at the standard paths (`/.well-known/security.txt`, `/security.txt`): + Probe for a security.txt-like response at the standard paths (`/.well-known/security.txt`, `/security.txt`):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 296 - 300, Update the security.txt section heading and description to avoid claiming RFC 9116 validity; describe the httpx matcher as a heuristic or security.txt-like detection. Keep the existing standard paths and command unchanged unless needed to align the wording with that limitation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@README.md`:
- Around line 296-300: Update the security.txt section heading and description
to avoid claiming RFC 9116 validity; describe the httpx matcher as a heuristic
or security.txt-like detection. Keep the existing standard paths and command
unchanged unless needed to align the wording with that limitation.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: bf6c165b-befb-4f36-ad31-0140fc89cd68
📒 Files selected for processing (2)
README.mdrunner/wellknown_recipes_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- runner/wellknown_recipes_test.go
There was a problem hiding this comment.
🧹 Nitpick comments (1)
runner/wellknown_recipes_test.go (1)
34-34: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid hardcoded array indices for recipe lookups.
Retrieving recipes by their hardcoded index makes the test suite brittle. If recipes are added, removed, or reordered in
WellKnownRecipes(), these tests will either break or silently evaluate the wrong recipe. Consider extracting a test helper to find a recipe by itsNameinstead of relying on its position in the array.
runner/wellknown_recipes_test.go#L34-L34: replaceWellKnownRecipes()[0]withfindRecipe(t, "security.txt").runner/wellknown_recipes_test.go#L40-L40: replaceWellKnownRecipes()[0]withfindRecipe(t, "security.txt").runner/wellknown_recipes_test.go#L50-L50: replaceWellKnownRecipes()[4]withfindRecipe(t, "ads.txt").runner/wellknown_recipes_test.go#L63-L63: replaceWellKnownRecipes()[len(WellKnownRecipes())-1]withfindRecipe(t, "well-known-uri-batch").♻️ Example helper function
func findRecipe(t *testing.T, name string) WellKnownRecipe { t.Helper() for _, r := range WellKnownRecipes() { if r.Name == name { return r } } t.Fatalf("recipe %q not found", name) return WellKnownRecipe{} }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@runner/wellknown_recipes_test.go` at line 34, Replace positional recipe lookups in runner/wellknown_recipes_test.go at lines 34, 40, 50, and 63 with findRecipe calls using names "security.txt", "security.txt", "ads.txt", and "well-known-uri-batch" respectively. Add the findRecipe testing helper to search WellKnownRecipes() by Name, call t.Helper(), and fail the test if no matching recipe is found.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@runner/wellknown_recipes_test.go`:
- Line 34: Replace positional recipe lookups in runner/wellknown_recipes_test.go
at lines 34, 40, 50, and 63 with findRecipe calls using names "security.txt",
"security.txt", "ads.txt", and "well-known-uri-batch" respectively. Add the
findRecipe testing helper to search WellKnownRecipes() by Name, call t.Helper(),
and fail the test if no matching recipe is found.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 88350c9e-b5e2-4327-9386-9700e68577ab
📒 Files selected for processing (2)
runner/wellknown_recipes.gorunner/wellknown_recipes_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- runner/wellknown_recipes.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per maintainer feedback on #2472, this PR adds a Common Recipes section to the README demonstrating how to detect , , , and other well-known URIs using httpx's composable primitives.
Changes
-path,-mc,-mct,-mrflagsContext
@Mzack9999 suggested in this comment turning the closed PR #2472 into docs showing well-known file recipes as an alternative to adding per-file flags.
This aligns with httpx's philosophy of composability over specialized flags.
Testing
Verified all one-liners locally:
cc @Mzack9999 — happy to iterate on wording/format if needed.
Summary by CodeRabbit
security.txt,robots.txt,sitemap.xml, OIDC discovery, Apple/Android app association assets,crossdomain.xml, plus a batch variant).-mdcfor combined checks.