Extend si.load capabilities - #3651
Conversation
| "pytest", | ||
| "pytest-dependency", | ||
| "psutil", | ||
| "s3fs", # for streaming tests |
There was a problem hiding this comment.
This is a big one for core tests no ?
There was a problem hiding this comment.
We decided to not have core tests that do downloading no ?
Because the matrix of version X OS is big and failure probability is bigger when need external URL.
I would move the S3 stuff in test_extractor somewhere.
|
Good idea but I am not sure to like the design with try except. |
|
We could also add the Motion object. |
|
And also Template |
|
@alejoe91 : I refactor everything to have more general case. |
for more information, see https://pre-commit.ci
| # This case shoudl deprecated soon because the read_zarr is ultra ambiguous | ||
| # just testing if the zarr contains unit_ids or channel_ids but many object also contains it (see template)!!!! | ||
| from .zarrextractors import read_zarr |
There was a problem hiding this comment.
@alejoe91 : any idea how to avoid using this read_zarr stuff ?
If think that read_zarr should deprecated because this is too ambiguous.
| storage_options = backend_options.get("storage_options", {}) | ||
|
|
||
| zarr_root = zarr.open_consolidated(str(folder), mode="r", storage_options=storage_options) | ||
| zarr_root = super_zarr_open(str(folder), mode="r", storage_options=storage_options) |
There was a problem hiding this comment.
@alejoe91 : Here, is it a problem that the zarr_root could be eventually not consolidated ?
There was a problem hiding this comment.
It could for some older assets. Plus in append mode we can't use consolidated
|
@alejoe91 : really cool. |
I'll add it to the steaming extractors tests |
2 participants
This PR extends the
si.loadmagic function to load alsoSortingAnalyzers.In addition, a better logic is implemented to load zarr files, both locally and remotely, including trying to open the folder with different options:
zarr.open/zarr.open_consolidatedandanon False/Truefor remote assets. This simplifies handling of zarr opening everywhere