CiteCue AI Auto-Fix WordPress plugin: AI-crawler middleware, llms.txt, content ingest#1
Conversation
… ingest WordPress middleware for the CiteCue delivery API: - Serve CiteCue-optimized pages to AI bots/crawlers via the authenticated v2 delivery channel (Bearer ck_live_ key, X-Citecue-Channel: wordpress), with ETag/304 revalidation, negative caching of the 404 miss sentinel, a failure circuit breaker, and stale-on-error fallback. Human traffic is never touched; any failure passes through to the normal page. - Publish the project's llms.txt at the site root, stamping the x-citecue headers CiteCue's install verification probes for. - Bundled AI-crawler UA registry (18 tokens) refreshed daily from the public /api/delivery/v1/crawlers feed via WP-Cron. - HMAC-signed REST endpoint (POST /wp-json/citecue/v1/content) through which CiteCue can push brand-building content into WordPress as draft posts: idempotent by external_id, status capped by a site-owner setting, local-edit and trash guards, per-hour rate limit. - Admin screen: connection test with project auto-selection from /api/delivery/v2/config, delivery toggles, ingest configuration and secret rotation, recent crawler activity view. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFudBFwAhjC7vhXH1ZBKun
📝 WalkthroughWalkthroughThe PR introduces the CiteCue WordPress plugin with AI-crawler page delivery, cached ChangesCiteCue WordPress plugin
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)Crawler page deliverysequenceDiagram
participant Crawler
participant Citecue_Proxy
participant Citecue_Crawlers
participant Citecue_Api_Client
participant Citecue_Cache
Crawler->>Citecue_Proxy: request page with User-Agent
Citecue_Proxy->>Citecue_Crawlers: match crawler token
Citecue_Proxy->>Citecue_Cache: check page cache and circuit
Citecue_Proxy->>Citecue_Api_Client: fetch optimized content
Citecue_Api_Client-->>Citecue_Proxy: return page response
Citecue_Proxy->>Citecue_Cache: cache body and ETag
Citecue_Proxy-->>Crawler: serve optimized HTML
Signed content pushsequenceDiagram
participant CiteCue
participant WordPress_REST
participant Citecue_Ingest
participant WordPress_Posts
CiteCue->>WordPress_REST: POST signed content payload
WordPress_REST->>Citecue_Ingest: verify timestamp, signature, and rate limit
Citecue_Ingest->>WordPress_Posts: insert or update content and metadata
WordPress_Posts-->>Citecue_Ingest: return persisted content
Citecue_Ingest-->>CiteCue: return status and links
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 447c59e036
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| $cache->touch_page( $url ); | ||
| $this->plugin->activity->record( $crawler, $path, 'served' ); |
There was a problem hiding this comment.
When a URL that was previously cached later returns the 404 miss sentinel (for example, the optimized page was removed or disabled), this only records a short miss and leaves the old page transient intact. After that 60-second miss expires, any later open circuit will still load $cached = $cache->get_page( $url ) and serve the stale optimized body, so crawlers can keep seeing content CiteCue explicitly told us to pass through. Please clear the page cache for this URL when handling the 404.
Useful? React with 👍 / 👎.
|
|
||
| <h2><?php esc_html_e( 'Tools', 'citecue' ); ?></h2> | ||
| <p> | ||
| <?php $this->action_button( 'citecue_test_connection', __( 'Test connection', 'citecue' ) ); ?> |
There was a problem hiding this comment.
For a first-time admin following the documented flow of pasting an API key and clicking “Test connection,” this button submits only its own action/nonce because it is rendered outside the settings form, so handle_test_connection() tests the previously saved option value (usually empty) and reports an auth failure instead of testing the key the admin just entered. Either include the connection fields in this action or make the UI require saving before testing.
Useful? React with 👍 / 👎.
Middleware: cart, checkout (incl. order-pay/order-received), account pages and every other WooCommerce endpoint are never intercepted, and cart-mutating ?add-to-cart= GETs and wc-ajax calls are skipped. Product and shop-archive pages remain served — the highest-value crawler pages. Ingest: type "product" creates a draft simple product through WooCommerce's CRUD API (name/description/short description, slug, SKU, regular_price, product_cat/product_tag terms), with the same status cap as posts. A push whose SKU matches an existing product not created by the plugin is refused with 409 unless force=true, so enriching an existing catalog item is an explicit opt-in. Cross-type external_id reuse is rejected, product pushes without WooCommerce return a clear 400, and /health now reports a woocommerce flag. Admin: product option in the default-type select and a note about the store-page exclusions, both shown only when WooCommerce is active. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFudBFwAhjC7vhXH1ZBKun
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
README.md (1)
11-25: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSpecify a language for the fenced code block.
The ASCII diagram block has no language specifier, which triggers markdownlint MD040. Adding a language hint (e.g.,
text) improves rendering in some markdown viewers.♻️ Proposed fix
-``` +```text AI crawler (GPTBot, ClaudeBot, …) Human visitor🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 11 - 25, Specify the fenced ASCII diagram as a text code block by adding the text language identifier to its opening fence, while leaving the diagram content unchanged.Source: Linters/SAST tools
uninstall.php (1)
14-26: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider cleaning up salt-keyed transients from the database.
The comment on line 24 notes that page/llms.txt transients are salt-keyed and expire on their own within a day. Without a persistent object cache these live in
wp_options, so they become orphaned rows until expiry. A targeted query for_transient_citecue_*keys would provide more thorough cleanup.♻️ Proposed addition
delete_transient( 'citecue_circuit' ); delete_transient( 'citecue_ingest_rate' ); // Page/llms.txt transients are salt-keyed and expire on their own within a day. + +// Clean up any orphaned salt-keyed transients from wp_options. +global $wpdb; +$wpdb->query( + $wpdb->prepare( + "DELETE FROM {$wpdb->options} WHERE option_name LIKE %s", + '\_transient\_citecue\_%' + ) +);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@uninstall.php` around lines 14 - 26, Extend the uninstall cleanup after the existing citecue transient deletions to remove salt-keyed page/llms.txt transients from wp_options. Use a targeted database query matching only _transient_citecue_* option keys, and also remove their corresponding timeout entries, while preserving the existing scheduled-hook cleanup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@includes/class-citecue-crawlers.php`:
- Around line 139-152: The refresh() method must reject registry responses whose
version is older than the bundled or currently stored version, preserving the
existing option on downgrade or incomplete data. For accepted responses,
sanitize the remote tokens, merge them with bundled_tokens(), and remove
case-insensitive duplicates before update_option(), while retaining the
validated version and fetched_at metadata.
In `@includes/class-citecue-ingest.php`:
- Around line 133-150: Update the authenticated request flow around the
signature validation and within_rate_limit() call to make signed requests
single-use within TIMESTAMP_WINDOW. Derive a deterministic request digest or
nonce from the authenticated request, store it for the timestamp window, and
detect duplicates before performing writes; either return the cached original
result or reject the duplicate with an appropriate WP_Error. Ensure replay
detection occurs only after authentication so unsigned requests cannot consume
replay state.
In `@includes/class-citecue-proxy.php`:
- Around line 64-106: Update the request flow around current_url(), cache
lookups, and api->get_page() to canonicalize URLs with CiteCue’s existing
query-parameter normalization rules before using them as cache or miss keys. Add
a shared global request/concurrency budget that is acquired before outbound
delivery requests and always released afterward, including failures, so unique
URLs cannot bypass protection. Preserve pass-through behavior when the budget is
exhausted.
---
Nitpick comments:
In `@README.md`:
- Around line 11-25: Specify the fenced ASCII diagram as a text code block by
adding the text language identifier to its opening fence, while leaving the
diagram content unchanged.
In `@uninstall.php`:
- Around line 14-26: Extend the uninstall cleanup after the existing citecue
transient deletions to remove salt-keyed page/llms.txt transients from
wp_options. Use a targeted database query matching only _transient_citecue_*
option keys, and also remove their corresponding timeout entries, while
preserving the existing scheduled-hook cleanup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 30072676-528f-4feb-9ab9-67946277e79e
📒 Files selected for processing (16)
.gitignoreREADME.mdcitecue.phpincludes/class-citecue-activity-log.phpincludes/class-citecue-admin.phpincludes/class-citecue-api-client.phpincludes/class-citecue-cache.phpincludes/class-citecue-crawlers.phpincludes/class-citecue-ingest.phpincludes/class-citecue-llms-txt.phpincludes/class-citecue-plugin.phpincludes/class-citecue-proxy.phpincludes/class-citecue-settings.phpincludes/index.phpreadme.txtuninstall.php
| $user_agent = isset( $_SERVER['HTTP_USER_AGENT'] ) ? sanitize_text_field( wp_unslash( $_SERVER['HTTP_USER_AGENT'] ) ) : ''; | ||
| $crawler = $this->plugin->crawlers->match( $user_agent ); | ||
| if ( null === $crawler ) { | ||
| return; | ||
| } | ||
|
|
||
| $url = $this->current_url(); | ||
| if ( '' === $url ) { | ||
| return; | ||
| } | ||
|
|
||
| /** | ||
| * Filters whether to serve optimized content for this crawler request. | ||
| * | ||
| * @param bool $should_serve Default true. | ||
| * @param string $crawler Matched UA token. | ||
| * @param string $url Absolute request URL. | ||
| */ | ||
| if ( ! apply_filters( 'citecue_should_serve', true, $crawler, $url ) ) { | ||
| return; | ||
| } | ||
|
|
||
| $path = (string) wp_parse_url( $url, PHP_URL_PATH ); | ||
| $cache = $this->plugin->cache; | ||
|
|
||
| // Recent miss for this URL: skip the API for a minute (mirrors the | ||
| // API's own max-age=60 on the miss sentinel). | ||
| if ( $cache->is_recent_miss( $url ) ) { | ||
| return; | ||
| } | ||
|
|
||
| $cached = $cache->get_page( $url ); | ||
|
|
||
| // Circuit open (recent timeout/auth failure): no API calls. Serve the | ||
| // stale cached copy when we have one, otherwise pass through. | ||
| if ( $cache->is_circuit_open() ) { | ||
| if ( $cached ) { | ||
| $this->serve( $cached['body'], $cached['mode'], true ); | ||
| } | ||
| return; | ||
| } | ||
|
|
||
| $response = $this->plugin->api->get_page( $url, $crawler, $cached ? $cached['etag'] : '' ); |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Denial of Service (CWE-400): Uncontrolled Resource Consumption
Reachability: External
● Entry
includes/class-citecue-admin.php:215
handle_refresh_crawlers
│
▼
● Hop
includes/class-citecue-crawlers.php:139
refresh
│
▼
● Hop
includes/class-citecue-api-client.php:40
__construct
│
▼
● Sink
includes/class-citecue-proxy.php
Bound externally triggerable delivery requests.
Any anonymous caller can spoof a crawler User-Agent and send unlimited unique query strings. Because the full URL is the cache key, every variation bypasses the page/miss caches and can hold a PHP worker on a three-second outbound call; successful 404 responses never open the circuit.
Normalize URLs locally using the same query-parameter rules as CiteCue and add a global request/concurrency budget so unique URLs cannot bypass protection.
Also applies to: 208-215
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@includes/class-citecue-proxy.php` around lines 64 - 106, Update the request
flow around current_url(), cache lookups, and api->get_page() to canonicalize
URLs with CiteCue’s existing query-parameter normalization rules before using
them as cache or miss keys. Add a shared global request/concurrency budget that
is acquired before outbound delivery requests and always released afterward,
including failures, so unique URLs cannot bypass protection. Preserve
pass-through behavior when the budget is exhausted.
… save-and-test - Ingest signatures are single-use: a replayed request within the timestamp window is rejected with 401 citecue_replayed (checked after authentication, before the rate limit, so unsigned traffic can consume neither replay state nor budget). Retries must re-sign with a fresh timestamp. - Delivery lookups are bounded by a per-minute budget (default 120, citecue_lookup_budget filter); beyond it crawler requests degrade like an open circuit. Cache/miss keys now use CiteCue-compatible URL normalization (scheme/www/trailing-slash/tracking-param dedupe) so URL variants share one entry. - A 404 miss for a previously optimized URL evicts the cached body immediately (and likewise for llms.txt), so content removed in CiteCue can no longer resurface through the stale-on-error path. - Crawler-registry refresh rejects version downgrades and always merges the bundled token floor back in, case-insensitively deduped — a truncated feed can never drop a bundled crawler. - The connection test is now a "Save & test connection" submit inside the settings form (via formaction), so a freshly pasted API key is saved and tested in one step instead of testing the stale stored key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFudBFwAhjC7vhXH1ZBKun
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (2)
includes/class-citecue-proxy.php (1)
106-114: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy liftDenial of Service (CWE-400): Uncontrolled Resource Consumption
Reachability: External
● Entry includes/class-citecue-llms-txt.php:55 maybe_serve │ ▼ ● Sink includes/class-citecue-proxy.phpMake the delivery budget atomic and concurrency-safe.
Parallel requests can all read the same count and pass before any
set_transient()executes, so a spoofed crawler burst can still fan out into unlimited simultaneous API calls. Use an atomic shared counter/lease and release concurrency slots in afinallypath.Also applies to: 227-240
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@includes/class-citecue-proxy.php` around lines 106 - 114, Update consume_lookup_budget() and the associated lookup flow to use an atomic shared counter or lease so concurrent requests cannot all pass the budget check before persistence. Treat unavailable leases as exhausted using the existing cached-response behavior, and ensure every acquired concurrency slot is released in a finally path around the outbound lookup, including success and failure cases.includes/class-citecue-ingest.php (1)
151-158: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy liftBroken Authentication (CWE-367): Time-of-check Time-of-use (TOCTOU) Race Condition
Reachability: External
Make replay recording atomic.
Concurrent copies of one valid signed request can both pass
get_transient()before either writes the marker, bypassing the single-use guarantee and potentially creating duplicate content. Use an atomic add-if-absent replay/idempotency record rather than a separate read then write.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@includes/class-citecue-ingest.php` around lines 151 - 158, The replay check in the signature validation flow must be atomic: replace the separate get_transient/set_transient calls around $replay_key with an add-if-absent transient operation, and reject the request when that operation indicates the key already exists. Preserve the existing replay error response and expiration of 2 * self::TIMESTAMP_WINDOW.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@includes/class-citecue-cache.php`:
- Around line 61-64: Update normalize_url’s authority construction to retain the
parsed $parts['port'] when present, appending it to the lowercased, www-stripped
host before generating the cache key. Keep URLs without a port normalized as
before, so distinct origins such as example.com and example.com:8443 remain
separate.
- Around line 71-82: Update normalizePageUrl() so query-string normalization
preserves repeated non-tracking parameters instead of using parse_str(), which
collapses duplicates; ensure distinct ordered pairs such as tag=a&tag=b and
tag=b remain distinct cache keys while continuing to remove the existing
tracking parameters. If repeated parameters are intentionally not meaningful,
add a test documenting that normalizePageUrl() treats them as equivalent
instead.
---
Duplicate comments:
In `@includes/class-citecue-ingest.php`:
- Around line 151-158: The replay check in the signature validation flow must be
atomic: replace the separate get_transient/set_transient calls around
$replay_key with an add-if-absent transient operation, and reject the request
when that operation indicates the key already exists. Preserve the existing
replay error response and expiration of 2 * self::TIMESTAMP_WINDOW.
In `@includes/class-citecue-proxy.php`:
- Around line 106-114: Update consume_lookup_budget() and the associated lookup
flow to use an atomic shared counter or lease so concurrent requests cannot all
pass the budget check before persistence. Treat unavailable leases as exhausted
using the existing cached-response behavior, and ensure every acquired
concurrency slot is released in a finally path around the outbound lookup,
including success and failure cases.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 04d9d4c1-71dd-4903-bae9-7b0b4952f9ea
📒 Files selected for processing (8)
README.mdincludes/class-citecue-admin.phpincludes/class-citecue-cache.phpincludes/class-citecue-crawlers.phpincludes/class-citecue-ingest.phpincludes/class-citecue-llms-txt.phpincludes/class-citecue-proxy.phpreadme.txt
🚧 Files skipped from review as they are similar to previous changes (5)
- readme.txt
- includes/class-citecue-crawlers.php
- README.md
- includes/class-citecue-admin.php
- includes/class-citecue-llms-txt.php
| $host = strtolower( $parts['host'] ); | ||
| if ( 0 === strpos( $host, 'www.' ) ) { | ||
| $host = substr( $host, 4 ); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the file and nearby lines with numbers.
sed -n '1,220p' includes/class-citecue-cache.php | cat -n
printf '\n---\n'
# Find related normalization and cache-key helpers.
rg -n "parse_url|wp_parse_url|parse_str|www\\.|port|query|transient|cache key|normalize" includes/class-citecue-cache.phpRepository: henry-mosh/citecue-wordpress-plugin
Length of output: 9042
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Search for any other normalization contract or mentions of ports.
rg -n "normalizePageUrl|non-default port|port|authority|www\\.|tracking params|utm_|fbclid|gclid" .
printf '\n--- files ---\n'
git ls-files | rg '(^README|docs/|includes/|src/|\.md$|\.php$)' | head -n 200Repository: henry-mosh/citecue-wordpress-plugin
Length of output: 2195
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '250,290p' includes/class-citecue-proxy.php | cat -n
printf '\n---\n'
sed -n '20,36p' README.md | cat -nRepository: henry-mosh/citecue-wordpress-plugin
Length of output: 3967
Preserve the port in cache-key normalization. normalize_url() drops $parts['port'], so https://example.com/foo and https://example.com:8443/foo collapse to the same transient key. If CiteCue doesn’t intentionally treat those origins as equivalent, include the parsed port in the normalized authority.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@includes/class-citecue-cache.php` around lines 61 - 64, Update
normalize_url’s authority construction to retain the parsed $parts['port'] when
present, appending it to the lowercased, www-stripped host before generating the
cache key. Keep URLs without a port normalized as before, so distinct origins
such as example.com and example.com:8443 remain separate.
| $query = ''; | ||
| if ( isset( $parts['query'] ) && '' !== $parts['query'] ) { | ||
| $pairs = array(); | ||
| parse_str( $parts['query'], $pairs ); | ||
| foreach ( array_keys( $pairs ) as $param ) { | ||
| if ( preg_match( '/^(utm_\w+|ref|fbclid|gclid)$/i', (string) $param ) ) { | ||
| unset( $pairs[ $param ] ); | ||
| } | ||
| } | ||
| if ( $pairs ) { | ||
| $query = '?' . http_build_query( $pairs ); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file list =="
git ls-files 'includes/class-citecue-cache.php' 'includes/*' 'tests/*' | sed -n '1,120p'
echo
echo "== citecue cache outline =="
ast-grep outline includes/class-citecue-cache.php --view expanded || true
echo
echo "== normalizePageUrl search =="
rg -n "normalizePageUrl|parse_str\(|http_build_query|utm_|fbclid|gclid|ref" includes tests -S || true
echo
echo "== relevant slice =="
nl -ba includes/class-citecue-cache.php | sed -n '1,180p'Repository: henry-mosh/citecue-wordpress-plugin
Length of output: 4400
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '40,110p' includes/class-citecue-cache.php | cat -nRepository: henry-mosh/citecue-wordpress-plugin
Length of output: 2652
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== search whole repo for normalization contract =="
rg -n "normalizePageUrl|normalize_url|tracking params|utm_\w+|fbclid|gclid|parse_str\(|http_build_query\(" . -S || true
echo
echo "== file list around docs/readme =="
git ls-files | rg '(^README|docs/|\.md$)' || trueRepository: henry-mosh/citecue-wordpress-plugin
Length of output: 1775
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '240,320p' includes/class-citecue-proxy.php | cat -nRepository: henry-mosh/citecue-wordpress-plugin
Length of output: 3317
🌐 Web query:
CiteCue normalizePageUrl duplicate query parameters parse_str duplicate parameters tracking params
💡 Result:
URL normalization practices regarding query parameters aim to produce a canonical representation of a URL to improve cache hit rates and ensure semantic equivalence [1][2][3]. Common approaches for handling duplicate query parameters and tracking parameters in normalizePageUrl functions (or similar utilities) are summarized below. Handling Duplicate Query Parameters Standard URL normalization usually preserves duplicate query parameters to maintain semantic integrity, as many applications use repeating keys to represent arrays (e.g., ?id=1&id=2) [1][3][4]. - Preservation: Most robust implementations, including those following the WHATWG URL Standard, preserve the original relative order of duplicate keys [5]. - Stable Sorting: When sorting parameters for canonicalization, a stable sort is used to ensure that even if the parameter order is adjusted, the relative order of identical keys remains unchanged [1][3][4]. - Avoiding Data Loss: Removing duplicate parameters is generally avoided because it can alter application behavior [4]. Handling Tracking Parameters Tracking parameters (e.g., utm_source, fbclid, gclid) are often identified as "noise" because they are typically used for marketing attribution and do not affect the rendered content of a page [1][2]. - Stripping: To prevent cache fragmentation, these parameters are often explicitly stripped from the URL before it is hashed for a cache key [1][2][6]. - Opt-in Configuration: Advanced implementations utilize an allow-list or block-list configuration for strip_query_params rather than dropping all unknown parameters, which prevents accidental removal of parameters required by the origin server [2]. Parsing and Normalization Utilities like parse_str (often found in PHP environments) or URLSearchParams (in JavaScript) are used to process these strings. When building custom normalizePageUrl logic: - Canonical Form: Logic typically involves parsing the query string, optionally stripping defined tracking keys, sorting the remaining keys alphabetically (or by key then value), and then rebuilding the query string [1][2][7][4]. - Implementation Risk: A common pitfall is using a naive implementation that treats query parameters as a simple object/dictionary. This approach often drops duplicate keys or fails to maintain the correct array structure expected by backend frameworks [3][5]. For high-reliability systems, it is recommended to use established libraries that support stable sorting and explicit handling of repeating parameters, rather than implementing manual regex-based stripping which may introduce bugs [2][3][8].
Citations:
- 1: https://edgecomet.com/docs/edge-gateway/url-normalization.html
- 2: https://modpagespeed.com/blog/cache-key-url-normalization-tracking-params/
- 3: https://wikitech-static.wikimedia.org/wiki/Query_string_normalization.html
- 4: https://jsonparser.ai/url-normalizer/
- 5: https://stackoverflow.com/questions/1746507/authoritative-position-of-duplicate-http-get-query-keys
- 6: perf: normalize loadPage URL to stop tracking-param cache fragmentation Weaverse/weaverse#458
- 7: browserbase/stagehand@8d7d414
- 8: https://github.com/sindresorhus/query-string/blob/main/readme.md
🌐 Web query:
CiteCue normalizePageUrl duplicate query parameters site:github.com CiteCue normalizePageUrl
💡 Result:
The term "normalizePageUrl" in the context of CiteCue and similar software development projects typically refers to a utility function designed to ensure consistent URL representation, often to improve cache hit rates or ensure reliability in data tracking [1][2][3]. There is no singular, universal "CiteCue" normalizePageUrl implementation that enforces specific behavior for duplicate query parameters. Instead, such utilities are custom-implemented in various projects to meet specific requirements [1][4][5]. When developers implement these functions, their handling of duplicate query parameters usually follows one of two common approaches: 1. Preservation or Collapse: Some implementations prioritize standardizing the URL key by sorting query parameters or stripping unnecessary tracking params (like utm_source), while keeping existing duplicates if they are significant [2][3]. 2. Normalization to Unique Keys: Other implementations use logic that may overwrite or collapse duplicate keys (e.g., keeping only the last occurrence) if the intended use case, such as generating a cache key or a clean sitemap reference, requires simplified, unique parameter sets [6][7][8]. If you are encountering unexpected behavior with duplicate query parameters in a project using a normalizePageUrl function, it is likely due to the specific logic chosen by the developers of that codebase rather than a standardized library behavior [3][5]. Common issues in similar custom URL normalization utilities often stem from: - Using URLSearchParams methods which may handle duplicates differently depending on the environment [2][8]. - Using manual string parsing that fails to correctly account for multiple instances of the same key [9][8]. - Intentionally stripping parameters to improve cache performance, which can inadvertently remove duplicate parameters that were intended to be preserved [2][3]. To resolve or debug this, examine the specific implementation of normalizePageUrl within your project's codebase to determine if it explicitly handles or collapses duplicate parameters [3][5]. If the current implementation is insufficient for your needs, you may need to modify the parsing logic to explicitly preserve or process multiple values for the same key [10].
Citations:
- 1: Weaverse/weaverse@fdd5f53
- 2: browserbase/stagehand@8d7d414
- 3: perf: normalize loadPage URL to stop tracking-param cache fragmentation Weaverse/weaverse#458
- 4: refactor: extract duplicated code into shared utilities ghostwright/phantom#157
- 5: setube/stackprism@388a2f4
- 6: https://github.com/Manu-sh/http_normalizer_parts
- 7: Rest API: duplicate values for query parameters Budibase/budibase#4844
- 8: Defect in duplicate query parameter handling in ExtAuthZ when using query_parameters_to_remove and query_parameters_to_add on a CheckResponse envoyproxy/envoy#26345
- 9: getWorkerBootstrapUrl duplicates query parameters and mishandles fragments when scriptPath contains an existing query string or fragment microsoft/vscode#303196
- 10: [BUG] URL Parser does not support duplicate query parameters CorentinTh/it-tools#800
Preserve repeated query pairs in the cache key. parse_str() collapses duplicate parameters, so ?tag=a&tag=b normalizes to the same key as ?tag=b. If repeated params are meaningful for CiteCue pages, this can collide distinct requests; otherwise add a test that locks the intended normalizePageUrl() behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@includes/class-citecue-cache.php` around lines 71 - 82, Update
normalizePageUrl() so query-string normalization preserves repeated non-tracking
parameters instead of using parse_str(), which collapses duplicates; ensure
distinct ordered pairs such as tag=a&tag=b and tag=b remain distinct cache keys
while continuing to remove the existing tracking parameters. If repeated
parameters are intentionally not meaningful, add a test documenting that
normalizePageUrl() treats them as equivalent instead.
2 participants
What this is
The WordPress plugin consuming CiteCue's authenticated v2 delivery channel — the CMS-plugin API shipped in citecue_app's "AI Auto-Fix Phase 2" (
/api/delivery/v2/{config,page,llms.txt},Authorization: Bearer ck_live_…,X-Citecue-Channel: wordpress). Repo root is the plugin root; plain PHP ≥ 7.4, no build step.Features
1. AI-crawler middleware (proxy) —
includes/class-citecue-proxy.phptemplate_redirect(priority 0); when the User-Agent matches an AI crawler (GPTBot, ClaudeBot, ChatGPT-User, PerplexityBot, …), fetches the optimized page fromGET /api/delivery/v2/page?k&u&band serves it with theX-Citecue: servedheader CiteCue's install verifier probes for.servedon 200/304,passthroughon 404), so Agent Traffic analytics need no extra beacon.If-None-Matchrevalidation with a locally cached body (304 → serve cached copy); miss sentinel negative-cached 60 s (mirrors the API'smax-age=60); timeouts/5xx open a 60 s circuit breaker (10 min on 401) with stale-on-error fallback. Human visitors are never touched; every failure mode passes through to the normal page.DONOTCACHEPAGE+Cache-Control: private, no-storeso page caches never store bot-only content.2. llms.txt —
includes/class-citecue-llms-txt.phpX-Citecue: llms-txt(what "Verify installation" checks). Falls through to WordPress when CiteCue has it disabled; a physical file in the web root always wins.3. Crawler registry —
includes/class-citecue-crawlers.phpmatchDeliveryCrawler(); refreshed daily from publicGET /api/delivery/v1/crawlersvia WP-Cron, so new crawlers are served without a plugin update.4. Content push (create posts) —
includes/class-citecue-ingest.phpPOST /wp-json/citecue/v1/content, authenticated by HMAC-SHA256 (X-Citecue-Signature: sha256=hex(hmac("{ts}.{body}", secret)), ±300 s timestamp window, constant-time compare) — the seam through which CiteCue pushes brand-building content (content briefs, FAQ packs, gap-filling pages) into WordPress.external_id; draft by default with a site-owner-controlled status cap; local-edit guard (409 unlessforce), trashed pushes stay trashed (410);wp_kses_postsanitation; categories/tags/meta-description support; post-auth rate limit (120/h, filterable). PlusGET /wp-json/citecue/v1/healthas a public handshake.5. WooCommerce support — proxy + ingest
?add-to-cart=GETs andwc-ajaxcalls. Product and shop-archive pages remain served — the highest-value crawler pages.type: "product"creates a draft simple product through WooCommerce's CRUD API (name/description/short description, slug, SKU,regular_price,product_cat/product_tagterms) under the same status cap. A push whose SKU matches an existing product not created by the plugin is refused with409 citecue_sku_existsunlessforce: true— enriching an existing catalog item is an explicit opt-in. Cross-typeexternal_idreuse → 409; product push without WooCommerce → clear 400;/healthreports awoocommerceflag.6. Admin screen —
includes/class-citecue-admin.php/v2/config) with project auto-selection by domain, delivery toggles, ingest configuration + secret rotation, cache flush, crawler-registry refresh, and a recent-crawler-activity table for on-site verification. WooCommerce-aware labels (product type option, store-exclusion note) appear only when WooCommerce is active.Verification
php -lclean on all 13 PHP files (PHP 8.4).sanitize_option_*re-entry trap), llms.txt path matching, cache/circuit-breaker behavior, and the WooCommerce matrix (store-page exclusions incl.add-to-cart/wc-ajax, product-type gating with/without WooCommerce, flow to the WC CRUD boundary) — all passing.x-citecueheadersprobeWorkerLiveness()/verify.post.tsrequire.🤖 Generated with Claude Code
https://claude.ai/code/session_01JFudBFwAhjC7vhXH1ZBKun
Summary by CodeRabbit
llms.txtdelivery at the site root with caching and conditional revalidation.POST /wp-json/citecue/v1/content) to create/update content with overwrite/conflict rules, plusGET /wp-json/citecue/v1/health..gitignoreto exclude common non-source artifacts.