v4.3.0

Highlights

• New tutorial:
  • Uploading data in bulk
    is our newest tutorial. It covers the basics of working with manifest files to manage synapse projects.
• Updates to syncToSynapse:
  • The syncToSynapse function has been refactored to improve performance and
    reliability.
  • Minor behavior change: File entities will no longer have it's version
    incremented during no-op changes. Only when file content, or fields on the file
    has been updated will a version number be incremented.
  • Optional booleans merge_existing_annotations and
    associate_activity_to_new_version have been added. Both are used to give more
    fine tuned control when working with this interface.
  • Check out the changes in the reference docs.

What's Changed

• [SYNPY-1474] Correct order of credentials in comment by @BryanFauble in #1091
• [SYNPY-1456] Flaky integration tests due to connection issues by @BryanFauble in #1085
• [SYNPY-1456] Revert to old ubuntu by @BryanFauble in #1092
• [SYNPY-1466] update user agent for command line by @danlu1 in #1090
• Update README.md by @allaway in #1093
• [SYNPY-1470] Remove SonarCloud's dependency on test job by @danlu1 in #1095
• [SYNPY-1470] Always run SonarCloud as long as previous jobs are not cancelled by @danlu1 in #1096
• [SYNPY-1356] Refactor sync to synapse by @BryanFauble in #1083
• [SYNPY-1356] Logic around disassociating Activity from File Entity by @BryanFauble in #1098
• [SYNPY-1341] Add CI step to containerize and upload synpy to the GHCR by @jaymedina in #1100
• [SYNPY-1384] Create uploading data in bulk tutorial by @BryanFauble in #1101

Full Changelog: v4.2.0...v4.3.0-rc

v4.3.0-rc

Highlights

• New tutorial:
  • Uploading data in bulk
    is our newest tutorial. It covers the basics of working with manifest files to manage synapse projects.
• Updates to syncToSynapse:
  • The syncToSynapse function has been refactored to improve performance and
    reliability.
  • Minor behavior change: File entities will no longer have it's version
    incremented during no-op changes. Only when file content, or fields on the file
    has been updated will a version number be incremented.
  • Optional booleans merge_existing_annotations and
    associate_activity_to_new_version have been added. Both are used to give more
    fine tuned control when working with this interface.
  • Check out the changes in the reference docs.

What's Changed

• [SYNPY-1474] Correct order of credentials in comment by @BryanFauble in #1091
• [SYNPY-1456] Flaky integration tests due to connection issues by @BryanFauble in #1085
• [SYNPY-1456] Revert to old ubuntu by @BryanFauble in #1092
• [SYNPY-1466] update user agent for command line by @danlu1 in #1090
• Update README.md by @allaway in #1093
• [SYNPY-1470] Remove SonarCloud's dependency on test job by @danlu1 in #1095
• [SYNPY-1470] Always run SonarCloud as long as previous jobs are not cancelled by @danlu1 in #1096
• [SYNPY-1356] Refactor sync to synapse by @BryanFauble in #1083
• [SYNPY-1356] Logic around disassociating Activity from File Entity by @BryanFauble in #1098
• [SYNPY-1341] Add CI step to containerize and upload synpy to the GHCR by @jaymedina in #1100
• [SYNPY-1384] Create uploading data in bulk tutorial by @BryanFauble in #1101

Full Changelog: v4.2.0...v4.3.0-rc

v4.2.0

Highlights

• Continued Async Support:
  • Multi-threaded Uploads Are Now Async: This change optimizes the upload process to handle multiple tasks simultaneously: uploading file parts concurrently, performing MD5 checksum calculations in separate processes, and making HTTP calls asynchronously. Further, memory management techniques were implemented to prevent crashes on low-resource EC2 instances. Benchmark results can be found here.
  • HTTPX Support & Async Client Instances: In this update, support for the HTTPX library has been introduced, enhancing the functionality of Synapse with two new async client instances for seamless interaction with Synapse and Storage Providers. Additionally, the retry and back-off mechanisms have been revamped to improve reliability and performance, ensuring smoother operation even under challenging network conditions.

What's Changed

• [SYNPY-1419] Httpx async client by @BryanFauble in #1076
• [SYNPY-1453] Guard around modified time item in cache by @BryanFauble in #1080
• [SYNPY-1420] Re-write uploads to mix async, multi-threading, and multi-processing by @BryanFauble in #1078
• [SYNPY-1417] Updates for the annotation model by @BryanFauble in #1081
• [SYNPY-1456] Flaky integration tests by @BryanFauble in #1082
• [SYNPY-1465] Adding Annotations to sync and async docs by @jaymedina in #1087

Full Changelog: v4.1.1...v4.2.0

v4.2.0-rc

Highlights

• Continued Async Support:
  • Multi-threaded Uploads Are Now Async: This change optimizes the upload process to handle multiple tasks simultaneously: uploading file parts concurrently, performing MD5 checksum calculations in separate processes, and making HTTP calls asynchronously. Further, memory management techniques were implemented to prevent crashes on low-resource EC2 instances. Benchmark results can be found here.
  • HTTPX Support & Async Client Instances: In this update, support for the HTTPX library has been introduced, enhancing the functionality of Synapse with two new async client instances for seamless interaction with Synapse and Storage Providers. Additionally, the retry and back-off mechanisms have been revamped to improve reliability and performance, ensuring smoother operation even under challenging network conditions.

What's Changed

• [SYNPY-1419] Httpx async client by @BryanFauble in #1076
• [SYNPY-1453] Guard around modified time item in cache by @BryanFauble in #1080
• [SYNPY-1420] Re-write uploads to mix async, multi-threading, and multi-processing by @BryanFauble in #1078
• [SYNPY-1417] Updates for the annotation model by @BryanFauble in #1081
• [SYNPY-1456] Flaky integration tests by @BryanFauble in #1082
• [SYNPY-1465] Adding Annotations to sync and async docs by @jaymedina in #1087

Full Changelog: v4.1.1...v4.2.0-rc-dev

v4.1.1

Highlights

• New Downloading Data in Bulk Tutorial
  • See Downloading data in bulk tutorial for more details on downloading data in bulk from Synapse.
• Downloading Files Troubleshooting Improvement
  • Improved error logging for when users are downloading files using commands like get-download-list from Synapse.

What's Changed

• Regen pipfile.lock for dependabot alert by @BryanFauble in #1069
• [SYNPY-1383] Downloading data in bulk tutorial by @BryanFauble in #1070
• [SYNPY-1443] Correct which fields are merged if there is a change by @BryanFauble in #1072
• [SYNPY-1442] Log update to capture exception when download-list fails by @BryanFauble in #1071
• [SYNPY-1445] Raise exception on HTTP error when retrieving file from storage location by @BryanFauble in #1074

Full Changelog: v4.1.0...v4.1.1

v4.1.0

Highlights

• New Interfaces:

  • Combines data and behavior into a single class, simplifying the understanding and usage of the system's models.
  • New Interface: Introduced a revamped interface in the Synapse Python Client, shifting from a functional programming approach to an object-oriented one.
  • Enhanced Developer Experience: This change highlights much needed quality of life updates for developers. Improved autocomplete, hoverdocs, and examples in docstrings provide a significantly better coding experience.
  • Asyncio Support: Introduced support for asyncio, enabling more efficient use of system resources and enhancing performance for IO-bound tasks.
  • Extensibility: Laying the foundation for an extensible platform, facilitating easier addition of new features, and improvements to the Synapse Python Client.

• synapseutils.walk Improvement:

  • Improved performance for synapseutils.walk.

• Pandas Range Expansion:

  • Expanded pandas range to >=1.5, <3.0.

• Version Notation Support:

  • Using syn123.version notation is now supported with syn.get, synapseutils.syncFromSynapse, and syn.setProvenance. This enhances consistency in version management across various activities.

What's Changed

• [SYNPY-1349] Detect annotation type by @BryanFauble in #1022
• [SYNPY-1345] Migrate to mkdocstrings by @BryanFauble in #1023
• [SYNPY-1412] Fixing typing issue in copy_functions by @jaymedina in #1048
• [SYNPY-1322] Object Orientated Programming Interfaces by @BryanFauble in #1013
• [SYNPY-1344] Creating a decorator to be used in async class methods by @BryanFauble in #1049
• [SYNPY-1344] Remove return type hinting on decorator by @BryanFauble in #1050
• [SYNPY-1344] Moving to pytest-socket for blocking socket connections in unit tests by @BryanFauble in #1051
• [SYNPY-1344] Set client in conftest since login is not used in unit tests by @BryanFauble in #1052
• [SYNPY-1344] Allows Socket Connections for Windows Systems in Unit Tests by @BWMac in #1054
• [SYNPY-1398] Support .[version] syntax for input SynIDs by @jaymedina in #1047
• [SYNPY-1344] Adding activity model for OOP by @BryanFauble in #1055
• [SYNPY-1398] Correct broken docstring by @BryanFauble in #1057
• [SYNPY-1344] Remove need to manually propogate otel context by @BryanFauble in #1056
• [SYNPY-1348] Add User profile and Permission models by @BryanFauble in #1058
• [SYNPY-1411] Prevent Repeated syn.get calls in _help_walk by @BWMac in #1059
• [SYNPY-1347] Adds OOP Team Model by @BWMac in #1053
• [SYNPY-1416] File model finishing touches for OOP by @BryanFauble in #1060
• [SYNPY-1416] Adding to the file.store docstring by @BryanFauble in #1062
• [SYNPY-1253] Check MD5 before upload to verify change in content by @BryanFauble in #1063
• [SYNPY-1415] Publish sequence diagram for walk function by @BryanFauble in #1064
• [SYNPY-448] allow changing Synapse name in changeFileMetaData by @danlu1 in #1065
• [SYNPY-1415] Folder model finishing touches by @BryanFauble in #1061
• [SYNPY-1326] Expand range for pandas dependency to <3.0 by @BryanFauble in #1066
• [SYNPY-1414] Project model finishing touches by @BryanFauble in #1067

Full Changelog: v4.0.0...v4.1.0-rc

v4.0.0

Highlights

• Only authentication through Personal Access Token
  (aka: Authentication bearer token) is supported. Review the Authentication document for information on setting up your usage of a Personal Access Token to authenticate with Synapse.
• Date type Annotations on Synapse entities are now timezone aware. Review our reference documentation for Annotations. The pytz package is reccomended if you regularly work with data across time zones.
  • If you do not set the tzinfo field on a date or datetime instance we will use the timezone of the machine where the code is executing.
  • If you are using the Manifest TSV for bulk actions on your projects you'll now see that [synapseutils.sync.syncFromSynapse][] will store dates as YYYY-MM-DDTHH:MM:SSZ. Review our documentation for an example manifest file. Additionally, if you'd like to upload an annotation in a specific timezone please make sure that it is in ISO 8601 format. If you do not specify a timezone it is assumed to use the timezone of the machine where the code is executing.
• Support for annotations with multiple values through the Manifest TSV with the usage of a comma delimited bracket wrapped list. Any manifest files wishing to take advantage of multi-value annotations need to match this format. Examples:
  • ["Annotation, with a comma", another annotation]
  • [1,2,3]
  • [2023-12-04T07:00:00Z,2000-01-01T07:00:00Z]
• Migration and expansion of the docs site! You'll see that the look, feel, and flow of all of the information on this site has been touched. As we move forward we hope that you'll provide the Data Processing and Engineering team feedback on areas we can improve.
• Expansion of the available Python Tutorials can be found starting here.

What's Changed

• Adding a label to the dockerfile to automatically label it for this repo by @BryanFauble in #1018
• Updates Dockerfile to Correctly Install Dependencies by @BWMac in #1019
• [SYNPY-1358] Correction of timestamp in annotations from manifest file by @BryanFauble in #1020
• [SYNPY-1336] Benchmarking upload with annotations by @BryanFauble in #1021
• [SYNPY-1321] Download benchmark results by @BryanFauble in #1024
• [SYNPY-1360] Migrating to mkdocstrings by @BryanFauble in #1025
• [SYNPY-1366] Add code coverage by @BryanFauble in #1029
• [SYNPY-1362] High level best practices for project structure by @thomasyu888 in #1028
• [SYNPY-1371] Migrate to Google Style by @BWMac in #1033
• [SYNPY-1302] Replace getPermission with get_acl and add new get_permissions by @danlu1 in #1037
• [SYNPY-1334] Revamp getting started docs by @BryanFauble in #1036
• [SYNPY-1332] Pypi deployment strategy by @BryanFauble in #1038
• [SYNPY-1370] Documentation Upgrade by @jaymedina in #1032
• [SYNPY-1370] Minor formatting fixes by @BryanFauble in #1039
• [SYNPY-1371] Doc fixes by @BryanFauble in #1040
• [SYNPY-1225] Support authToken only by @BryanFauble in #1041
• [SYNPY-1392] Remove some deprecated pieces by @BryanFauble in #1043
• [Synpy 1369] Migrate to Google style by @danlu1 in #1042
• [SYNPY-1387] Update Structure Project doc by @danlu1 in #1044
• [SYNPY-1357] Allow multiple values in manifest TSV by @BryanFauble in #1030

New Contributors

• @jaymedina made their first contribution in #1032

Full Changelog: v3.2.0...v4.0.0-rc

v3.2.0

What's Changed

• [SYNPY-1274] pre commit in gh action by @BryanFauble in #1001
• [SYNPY-1316] Make cryptography an optional install by @BryanFauble in #1003
• [SYNPY-1304] Introduction of OpenTelemetry by @BryanFauble in #1007
• [SYNPY-1316] Updating to caching logic to take in MD5 key by @BryanFauble in #1008
• [SYNPY-1305] Collect trace data from integration tests by @BryanFauble in #1009
• [SYNPY-1319] Lower binary file size by @BryanFauble in #1010
• [SYNPY-816] Enable test_teams again by @BryanFauble in #1011
• [SYNPY-1320] Upload benchmark + Documentation by @BryanFauble in #1012
• [SYNPY-1186] When a username is a number, getUserProfile cannot retrieve the user by @BWMac in #1014
• [SYNPY-1337] Adding in a comprehensive example for migration to another S3 bucket by @BryanFauble in #1015
• [SYNPY-1337] Adding note about unique .db file by @BryanFauble in #1016
• [SYNPY-1333] Print transfer progress for FTP by @BryanFauble in #1017

Full Changelog: v3.1.1...v3.2.0

v3.1.1

What's Changed

• [SYNPY-1298] Update annotation logic by @BryanFauble in #995
• [SYNPY-1299] Clean up contributing.md by @BryanFauble in #996
• [SYNPY-1299] Update contributing.md to point to service desk by @BryanFauble in #997
• [SYNPY-1316] Fix for cache item names matching by @BryanFauble in #998
• [SYNPY-1298] flakey integration tests by @BryanFauble in #999

Full Changelog: v3.1.0...v3.1.1-rc

v3.1.0

What's Changed

• [SYNPY-49] Aggregate acl based on groups by @BryanFauble in #979
• [SYNPY-967] deprecated memoize and added @lru_cache by @BryanFauble, @linglp in #983
• SYNPY-1285: Create pipfile by @BryanFauble in #984
• [SYNPY-1282] Adds Type Hinting to client.py by @BWMac in #987
• [SYNPY-1293] Update urllib3 version dependency by @BryanFauble in #988
• [SYNPY-1283] Replace Broken Link URL by @BWMac in #989
• [SYNPY-1296] Config client error with api key or PAT by @BryanFauble in #990* * [SYNPY-1283] Adds Missing Trailing Space (Broken Link Fix) by @BWMac in #991
• [SYNPY-1295] Adding to the credentials.rst doc by @BryanFauble in #992

New Contributors

• @BryanFauble made their first contribution in #979
• @BWMac made their first contribution in #987

Full Changelog: v3.0.0...v3.1.0-rc