-
Notifications
You must be signed in to change notification settings - Fork 21
Fix: Related Content should be fetched during the fetch pipeline #873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
odesenfans
requested changes
Oct 22, 2025
Collaborator
odesenfans
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Remove the changes inherited by #871, they're out of focus
- Check permissions in the fetch pipeline
- To discuss: IMO we should keep on pinning files in the processing pipeline as well, I see the fetch pipeline as a way to speed things up but the processing pipeline should have the full logic to be 100% sure that files get stored properly.
856f6f4 to
6fbc36e
Compare
6fbc36e to
8d1fedc
Compare
… fetch related content & verify permissions
odesenfans
requested changes
Oct 23, 2025
Collaborator
odesenfans
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking pretty good, just one more comment.
odesenfans
approved these changes
Oct 24, 2025
odesenfans
pushed a commit
that referenced
this pull request
Nov 26, 2025
* Fix: messages & pre check balance should be done on fetch pipeline * Fix: unit test missing file in db due to fetching adding the pin in db instead of processing * Refactor: `verify_message` renamed to `verify_and_fetch_message`. now fetch related content & verify permissions * fix: processing pipeline should still fetch the file if the fetch pipeline didn't * fix: remove useless change from test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In case of store message, the related content (file) isn't fetched during the fetch pipeline but is direcly fetched while processing it, making the processing block for no reason.
Related Clickup or Jira tickets : ALEPH-XXX
Self proofreading checklist
Changes
This pull request refactors the message verification and content fetching flow to ensure that database sessions are consistently passed and that related content is fetched at the appropriate stage. It also updates tests to reflect these changes and ensures required files are present in the database before processing messages. The changes improve reliability and maintainability by enforcing a clearer separation of concerns between message verification, balance checking, and content fetching.
Core handler refactor:
verify_messagemethod inMessageHandlernow requires asessionargument and performs a balance pre-check within the method, streamlining the verification and pre-check process. (src/aleph/handlers/message_handler.py)processmethod no longer performs a separate balance pre-check or fetches related content; these responsibilities have been moved to the verification stage, simplifying the processing pipeline. (src/aleph/handlers/message_handler.py)Content fetching pipeline update:
src/aleph/jobs/fetch_pending_messages.py)Test suite updates:
tests/message_processing/test_process_forgets.py,tests/message_processing/test_process_stores.py,tests/test_network.py) [1] [2] [3] [4] [5] [6] [7] [8] [9]Imports and setup:
tests/message_processing/test_process_forgets.py,tests/message_processing/test_process_stores.py) [1] [2] [3]Notes
There is still an issue with the pinning of message blocking the pipeline, but better to be block during the fetch than the processing