Skip to content

fix: Cache requests in RQ implementations by id #633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 20, 2025
Merged

fix: Cache requests in RQ implementations by id #633

merged 12 commits into from
Oct 20, 2025
+153 −115

Conversation

Pijukatel
Copy link
Contributor

@Pijukatel Pijukatel commented Oct 16, 2025 •
edited
Loading

Description

  • Cache requests in RQ implementations by id instead of unique_key due to the Apify platform truncating long unique_keys.

Issues

Testing

  • Added unit test

@github-actions github-actions bot added this to the 125th sprint - Tooling team milestone Oct 16, 2025
@github-actions github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Oct 16, 2025
@Pijukatel Pijukatel force-pushed the cache-by-request-id branch from 288e0b3 to d3dab06 Compare October 17, 2025 08:13
@Pijukatel Pijukatel force-pushed the cache-by-request-id branch from d3dab06 to 922a4c8 Compare October 17, 2025 08:32
@Pijukatel Pijukatel changed the title Draft fix: Cache requests in RQ implementations by id Oct 17, 2025
@Pijukatel Pijukatel force-pushed the cache-by-request-id branch from 59a33b1 to 4209d75 Compare October 17, 2025 12:36
@Pijukatel Pijukatel marked this pull request as ready for review October 17, 2025 12:37
@Pijukatel Pijukatel requested a review from vdusek October 17, 2025 12:37
vdusek
vdusek reviewed Oct 19, 2025
View reviewed changes
Copy link
Contributor

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few comments

src/apify/storage_clients/_apify/_request_queue_shared_client.py
response = await self._api_client.get_request(unique_key_to_request_id(unique_key))
return await self._get_request_by_id(unique_key_to_request_id(unique_key))

async def _get_request_by_id(self, request_id: str) -> Request | None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Methods order please (https://pypi.org/project/flake8-class-attributes-order/)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am reluctant to follow a formatting rule that is not enforced automatically by a tool. We do not even have a coding style defined, so how does one know what rule to follow? The code base is not compliant with the rule anyway. Cherry picking it randomly in some PR's is just annoying.

I will be happy to follow the rule as long as it is enforced by a tool.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw. it is common for IDEs to offer customizable method sorting to your liking, for example, in PyCharm
image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, not gonna force you. I'm not familiar with this functionality in VS Code, but anyway, you can't do it in GitHub - the point is that with proper ordering, the public interface is clearer and the whole class more readable. And what bothers me the most is the chaotic alternation between private and public methods.

src/apify/storage_clients/_apify/_request_queue_single_client.py
return self._requests_cache[unique_key]
return await self._get_request(id=unique_key_to_request_id(unique_key))

async def _get_request(self, id: str) -> Request | None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Methods order please (https://pypi.org/project/flake8-class-attributes-order/)

@Pijukatel Pijukatel requested a review from vdusek October 20, 2025 08:26
vdusek
vdusek requested changes Oct 20, 2025
View reviewed changes
Copy link
Contributor

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you noticed the failing integration tests?

vdusek
vdusek approved these changes Oct 20, 2025
View reviewed changes
Copy link
Contributor

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hate the chaotic alternation between private and public methods; otherwise LGTM.

@Pijukatel Pijukatel merged commit 76886ce into master Oct 20, 2025
91 of 97 checks passed
@Pijukatel Pijukatel deleted the cache-by-request-id branch October 20, 2025 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdusek vdusek vdusek approved these changes

Assignees

@Pijukatel Pijukatel

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Milestone

125th sprint - Tooling team

Development

Successfully merging this pull request may close these issues.

Apify RQ client list_head method returns truncated unique keys and URLs for long URLs

2 participants

@Pijukatel @vdusek