fix: update search_timeline to use POST to resolve 404 error#412
Open
deltathecreatorr wants to merge 5 commits intod60:mainfrom
Open
fix: update search_timeline to use POST to resolve 404 error#412deltathecreatorr wants to merge 5 commits intod60:mainfrom
deltathecreatorr wants to merge 5 commits intod60:mainfrom
Conversation
…saction. This fix was from issue 408 by AkaraiRP
Reviewer's guide (collapsed on small PRs)Reviewer's GuideSwitches GraphQL search_timeline from GET to POST to fix 404s and updates the X client transaction logic for resolving the ondemand.s JavaScript asset by using new response patterns for the file index and hash. Sequence diagram for search_timeline using POST instead of GETsequenceDiagram
actor User
participant TwikitClient
participant TwitterGraphQLAPI
User ->> TwikitClient: search_timeline(query, cursor, product)
TwikitClient ->> TwikitClient: build variables and features
TwikitClient ->> TwitterGraphQLAPI: HTTP POST /graphql SEARCH_TIMELINE
TwitterGraphQLAPI -->> TwikitClient: 200 OK (timeline data)
TwikitClient -->> User: search results
Updated class diagram for gql client and transaction index resolutionclassDiagram
class TwikitClient {
+search_timeline(query, cursor, product)
+gql_get(endpoint, variables, features)
+gql_post(endpoint, variables, features)
}
class XClientTransaction {
+ON_DEMAND_FILE_REGEX: re.Pattern
+ON_DEMAND_HASH_PATTERN: str
+get_indices(home_page_response, session, headers)
}
TwikitClient <.. XClientTransaction: uses async HTTP session
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
📝 WalkthroughWalkthroughUpdated Changes
Sequence Diagram(s)mermaid mermaid Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The new
ON_DEMAND_FILE_REGEX.search(...).group(1)and subsequentregex.search(...).group(1)calls assume the matches always exist; consider restoring/addingif match is Nonehandling to avoid unexpectedAttributeErrorin cases where the response format changes. - You now call
str(response)andstr(on_demand_file_response.text)multiple times for regex searches; capturing these once into local variables (e.g.,html = str(response)) will make the code clearer and slightly more efficient. - The commented-out legacy code in
get_indices(#on_demand_file = ..., etc.) appears obsolete after the new approach; consider removing it to keep the function focused and easier to read.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `ON_DEMAND_FILE_REGEX.search(...).group(1)` and subsequent `regex.search(...).group(1)` calls assume the matches always exist; consider restoring/adding `if match is None` handling to avoid unexpected `AttributeError` in cases where the response format changes.
- You now call `str(response)` and `str(on_demand_file_response.text)` multiple times for regex searches; capturing these once into local variables (e.g., `html = str(response)`) will make the code clearer and slightly more efficient.
- The commented-out legacy code in `get_indices` (`#on_demand_file = ...`, etc.) appears obsolete after the new approach; consider removing it to keep the function focused and easier to read.
## Individual Comments
### Comment 1
<location path="twikit/x_client_transaction/transaction.py" line_range="45-50" />
<code_context>
- str(on_demand_file_response.text))
- for item in key_byte_indices_match:
- key_byte_indices.append(item.group(2))
+ on_demand_file_index = ON_DEMAND_FILE_REGEX.search(str(response)).group(1)
+ regex = re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index))
+ #on_demand_file = ON_DEMAND_FILE_REGEX.search(str(response))
+ #if on_demand_file:
+ #on_demand_file_url = f"https://abs.twimg.com/responsive-web/client-web/ondemand.s.{on_demand_file.group(1)}a.js"
+ filename = regex.search(str(response)).group(1)
+ on_demand_file_url = f"https://abs.twimg.com/responsive-web/client-web/ondemand.s.{filename}a.js"
+ on_demand_file_response = await session.request(method="GET", url=on_demand_file_url, headers=headers)
</code_context>
<issue_to_address>
**issue (bug_risk):** Regular expression `.group(1)` calls can raise if no match is found, changing the previous error-handling behavior.
The previous code only used the match when `on_demand_file` was truthy and raised a clear error when indices couldn’t be extracted. The new code calls `ON_DEMAND_FILE_REGEX.search(...).group(1)` and `regex.search(...).group(1)` without checking the result, so a non‑match will raise `AttributeError` instead of a controlled, descriptive exception. Please restore an explicit check (or catch the error) and raise a clear message when the response doesn’t match the expected pattern.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
twikit/x_client_transaction/transaction.py (1)
47-49: Remove commented-out code.The commented-out lines are remnants of the old implementation and should be removed for cleanliness. Git history preserves the old code if needed.
♻️ Proposed fix
on_demand_file_index = ON_DEMAND_FILE_REGEX.search(str(response)).group(1) regex = re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index)) - `#on_demand_file` = ON_DEMAND_FILE_REGEX.search(str(response)) - `#if` on_demand_file: - `#on_demand_file_url` = f"https://abs.twimg.com/responsive-web/client-web/ondemand.s.{on_demand_file.group(1)}a.js" filename = regex.search(str(response)).group(1)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@twikit/x_client_transaction/transaction.py` around lines 47 - 49, Remove the three commented-out lines that reference ON_DEMAND_FILE_REGEX and on_demand_file_url in transaction.py (the block containing "#on_demand_file = ON_DEMAND_FILE_REGEX.search(str(response))", "#if on_demand_file:", and the "#on_demand_file_url = ..." line); simply delete that commented legacy code so the file contains no leftover commented implementation—rely on git history if the old logic needs to be restored.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@twikit/x_client_transaction/transaction.py`:
- Around line 45-50: The code calls ON_DEMAND_FILE_REGEX.search(...) and
regex.search(...) then immediately .group(1) which can raise AttributeError if
no match is found; update the get_indices logic to null-check the search results
before calling .group(1): check the result of
ON_DEMAND_FILE_REGEX.search(str(response)) and if None gracefully handle (e.g.,
log and return/raise a descriptive error) and only compile/execute
ON_DEMAND_HASH_PATTERN when you have a valid on_demand_file_index, then
similarly check regex.search(str(response)) for None before extracting filename;
reference the existing symbols ON_DEMAND_FILE_REGEX, ON_DEMAND_HASH_PATTERN and
the function that contains this code (get_indices) to locate where to add the
guards and error handling.
---
Nitpick comments:
In `@twikit/x_client_transaction/transaction.py`:
- Around line 47-49: Remove the three commented-out lines that reference
ON_DEMAND_FILE_REGEX and on_demand_file_url in transaction.py (the block
containing "#on_demand_file = ON_DEMAND_FILE_REGEX.search(str(response))", "#if
on_demand_file:", and the "#on_demand_file_url = ..." line); simply delete that
commented legacy code so the file contains no leftover commented
implementation—rely on git history if the old logic needs to be restored.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 39d328fe-fb82-4ffe-bd06-97e6953c9125
📒 Files selected for processing (3)
.gitignoretwikit/client/gql.pytwikit/x_client_transaction/transaction.py
…ching in get_indices method
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
twikit/x_client_transaction/transaction.py (1)
45-51:⚠️ Potential issue | 🔴 CriticalNull checks are placed after
.group(1)calls, making them ineffective.The checks on lines 46-47 and 50-51 cannot prevent
AttributeErrorbecause.group(1)is called directly on the.search()result. If the regex doesn't match,.search()returnsNone, and calling.group(1)onNoneraisesAttributeErrorbefore the null check is ever reached.🐛 Proposed fix to add proper null-checks
- on_demand_file_index = ON_DEMAND_FILE_REGEX.search(str(response)).group(1) - if not on_demand_file_index: - raise Exception("Failed to match ON_DEMAND_FILE_REGEX in the home page response") - regex = re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index)) - filename = regex.search(str(response)).group(1) - if not filename: - raise Exception(f"Failed to match the hash pattern for on_demand_file index {on_demand_file_index}") + on_demand_match = ON_DEMAND_FILE_REGEX.search(str(response)) + if not on_demand_match: + raise Exception("Failed to match ON_DEMAND_FILE_REGEX in the home page response") + on_demand_file_index = on_demand_match.group(1) + regex = re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index)) + hash_match = regex.search(str(response)) + if not hash_match: + raise Exception(f"Failed to match the hash pattern for on_demand_file index {on_demand_file_index}") + filename = hash_match.group(1)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@twikit/x_client_transaction/transaction.py` around lines 45 - 51, The code calls .group(1) directly on regex.search(...) which can return None; change this to capture the Match objects first and null-check them before calling .group(1): assign m1 = ON_DEMAND_FILE_REGEX.search(str(response)), if m1 is None raise the existing exception, then set on_demand_file_index = m1.group(1); likewise build regex = re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index)), then assign m2 = regex.search(str(response)) and if m2 is None raise the filename-matching exception, otherwise set filename = m2.group(1); reference ON_DEMAND_FILE_REGEX, ON_DEMAND_HASH_PATTERN, response, on_demand_file_index, filename.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@twikit/x_client_transaction/transaction.py`:
- Around line 45-51: The code calls .group(1) directly on regex.search(...)
which can return None; change this to capture the Match objects first and
null-check them before calling .group(1): assign m1 =
ON_DEMAND_FILE_REGEX.search(str(response)), if m1 is None raise the existing
exception, then set on_demand_file_index = m1.group(1); likewise build regex =
re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index)), then assign m2
= regex.search(str(response)) and if m2 is None raise the filename-matching
exception, otherwise set filename = m2.group(1); reference ON_DEMAND_FILE_REGEX,
ON_DEMAND_HASH_PATTERN, response, on_demand_file_index, filename.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 40cb3df4-67ae-48db-8d50-03c67d21e67f
📒 Files selected for processing (1)
twikit/x_client_transaction/transaction.py
…REGEX and hash pattern matching
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
twikit/x_client_transaction/transaction.py (1)
15-16: Quote handling inconsistency between regex patterns.
ON_DEMAND_FILE_REGEXmatches both single and double quotes (["']), butON_DEMAND_HASH_PATTERNonly matches double quotes (\"). If Twitter's bundler output uses single quotes for hash values, the extraction will fail.Consider aligning the quote handling for consistency:
♻️ Suggested fix for consistent quote handling
-ON_DEMAND_HASH_PATTERN: str = r',{}:\"([0-9a-f]+)\"' +ON_DEMAND_HASH_PATTERN: str = r",{}:[\"']([0-9a-f]+)[\"']"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@twikit/x_client_transaction/transaction.py` around lines 15 - 16, ON_DEMAND_FILE_REGEX accepts both single and double quotes but ON_DEMAND_HASH_PATTERN only matches double quotes, so hash extraction can fail for single-quoted output; update ON_DEMAND_HASH_PATTERN to allow both quote types (mirror the ["'] usage from ON_DEMAND_FILE_REGEX) so the pattern consistently matches single or double quoted hash values when used by the code that extracts on-demand hashes.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@twikit/x_client_transaction/transaction.py`:
- Around line 15-16: ON_DEMAND_FILE_REGEX accepts both single and double quotes
but ON_DEMAND_HASH_PATTERN only matches double quotes, so hash extraction can
fail for single-quoted output; update ON_DEMAND_HASH_PATTERN to allow both quote
types (mirror the ["'] usage from ON_DEMAND_FILE_REGEX) so the pattern
consistently matches single or double quoted hash values when used by the code
that extracts on-demand hashes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 612f6261-add4-4216-8f0c-cd3c2a513940
📒 Files selected for processing (1)
twikit/x_client_transaction/transaction.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SsearchTimeline GET requests are being blocked, changed to a POST request so that the query variables are in the request body now.
In gql.py, I changed the gql_get on line 159 to gql_post and now the ability to search for tweets is operational again.
Summary by Sourcery
Update Twitter client internals to restore search functionality and align ondemand script parsing with current response format.
Bug Fixes:
Summary by CodeRabbit
Chores
Refactor