Skip to content

[SYNPY-1800] Added new sync_to_synapse method#1353

Merged
andrewelamb merged 47 commits intodevelopfrom
SYNPY-1800
Apr 16, 2026
Merged

[SYNPY-1800] Added new sync_to_synapse method#1353
andrewelamb merged 47 commits intodevelopfrom
SYNPY-1800

Conversation

@andrewelamb
Copy link
Copy Markdown
Contributor

@andrewelamb andrewelamb commented Apr 6, 2026

Problem:

The syncToSynapse function doesn't fit the new OOP model paradigm.

Solution:

  • A new sync_to_synapse method was added to the StorableContainer class
  • syncToSynapse was deprecated

Testing:

  • Unit tests for the new helper functions
  • Integration tests for the new sync_to_synapse method

@andrewelamb andrewelamb requested a review from a team as a code owner April 6, 2026 15:17
@andrewelamb andrewelamb marked this pull request as draft April 6, 2026 15:17
@andrewelamb andrewelamb marked this pull request as ready for review April 6, 2026 16:43
Copy link
Copy Markdown
Contributor

@linglp linglp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Old syncToSynapse defaults to sendMessages=True, and new sync_to_synapse defaults to send_messages=False. Users migrating by following the deprecation example will silently stop receiving upload notifications with no warning. Is this behavior intentional?

Also, _create_local_test_file leaves cleanup to the caller. I think it will be better if clean up can happen within _create_local_test_file so that we could avoid all the scattered schedule_for_cleanup calls.

Comment thread docs/tutorials/python/tutorial_scripts/upload_data_in_bulk.py Outdated
Comment thread synapseutils/sync.py Outdated
Comment thread synapseclient/models/mixins/storable_container.py Outdated
Comment thread tests/integration/synapseclient/models/async/test_storable_container_async.py Outdated
@andrewelamb andrewelamb requested a review from linglp April 7, 2026 17:17
@andrewelamb andrewelamb changed the title added new sync_to_synapse method [SYNPY-1800] Added new sync_to_synapse method Apr 8, 2026
Copy link
Copy Markdown
Contributor

@linglp linglp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes! Just some small comments. Also, could you make sure that the documentation format is correct?

For example, the format here is slightly off. Also, the links in reference are not working (See here)

Comment thread tests/integration/synapseclient/models/async/test_storable_container_async.py Outdated
Comment thread synapseclient/models/protocols/storable_container_protocol.py Outdated
Comment thread docs/explanations/manifest_csv.md Outdated
@andrewelamb andrewelamb requested a review from linglp April 10, 2026 18:13
Copy link
Copy Markdown
Contributor

@linglp linglp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @andrewelamb ! I found more problems with the documentation. Please see comments below

Comment thread docs/tutorials/python/upload_data_in_bulk.md Outdated
Comment thread docs/explanations/manifest_csv.md Outdated
Comment thread docs/explanations/manifest_csv.md
Comment thread docs/tutorials/python/upload_data_in_bulk.md Outdated
Comment thread docs/tutorials/python/upload_data_in_bulk.md Outdated
Comment thread docs/tutorials/python/upload_data_in_bulk.md Outdated
Comment thread docs/tutorials/python/upload_data_in_bulk.md
Copy link
Copy Markdown
Contributor

@linglp linglp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewelamb Thanks for the last round of fixes. there are still some problems with tests and documentation. Could you check again?

Comment thread docs/explanations/manifest_csv.md Outdated
Comment thread docs/explanations/manifest_csv.md Outdated
Using multiple values for a single annotation should be used sparingly as it makes it more
difficult for you to manage the data. However, it is supported.

**Annotations can be comma `,` separated lists surrounded by brackets `[]`.**
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at the old documentation here: https://python-docs.synapse.org/en/stable/explanations/manifest_tsv/?h=manifest#multiple-values-of-annotations-per-key, the examples here are much shorter. Is this epxected?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I wasn't sure all the examples needed to be included. What do you think?

Comment thread docs/explanations/manifest_csv.md Outdated
Comment thread docs/explanations/manifest_csv.md
is_synapse_id_str,
is_url,
test_import_pandas,
topolgical_sort,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably this needs to be fixed in a different PR. Is there a typo in the function name here? should this be topological_sort instead of topolgical_sort?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. We would need to decide if this is a public function, and if so it would need to be deprecated first.

Comment thread tests/unit/synapseclient/models/async/unit_test_manifest_async.py Outdated
dep_file = _make_file_mock("/dep.txt", "syn_dep")

# Create a real future that resolves to dep_file
dep_future = asyncio.get_event_loop().create_future()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we be using asyncio.get_event_loop() here? Based on the documentation, for Python 3.10+, we should consider using asyncio.run

Copy link
Copy Markdown
Contributor Author

@andrewelamb andrewelamb Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So Claude says on this :

In this context — an async test function — neither asyncio.get_event_loop() nor asyncio.run() is the right choice. 
Since the test is already running inside an async event loop (managed by pytest-asyncio), the correct approach is:                                      
                                                                                                                                                                                                                                        
  dep_future = asyncio.get_running_loop().create_future()                                                                                                                                                                               
                                                                                                                                                                                                                                        
  - get_event_loop() is deprecated for this use case in 3.10+ and emits a deprecation warning when no running loop exists.                                                                                                              
  - asyncio.run() would try to create a new event loop, which conflicts with the already-running one.
  - get_running_loop() is the correct call from within an async function — it returns the loop that's already executing the test.  

Comment thread tests/unit/synapseclient/models/async/unit_test_manifest_async.py Outdated
@andrewelamb andrewelamb requested a review from linglp April 14, 2026 19:35
Comment thread synapseclient/models/mixins/storable_container.py
Comment thread tests/unit/synapseclient/models/async/unit_test_manifest_async.py Outdated
Comment thread tests/unit/synapseclient/models/async/unit_test_manifest_async.py Outdated
Copy link
Copy Markdown
Contributor

@linglp linglp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still some problems with tests and documentation. One concern that I have is that a lot of tests are relying on sync_from_synapse_async for clean up which means if the assertion failed after sync_from_synapse_async or the function itself failed, the test files won't be cleaned up at all. But overall it looks good!

Comment thread tests/integration/synapseclient/models/async/test_storable_container_async.py Outdated
Comment thread tests/unit/synapseclient/models/async/unit_test_storable_container_async.py Outdated
Comment thread docs/explanations/manifest_csv.md Outdated
Comment thread docs/explanations/manifest_csv.md Outdated
Comment thread docs/explanations/manifest_csv.md Outdated
@andrewelamb andrewelamb requested a review from linglp April 16, 2026 21:44
@andrewelamb andrewelamb merged commit 875c8f5 into develop Apr 16, 2026
22 of 23 checks passed
@andrewelamb andrewelamb deleted the SYNPY-1800 branch April 16, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants