v.1.11.0

[pgunn]

This release makes two disruptive changes, among others.

• If you use CLI demos, you'll find them to be totally reworked in this release to be more applike and driven by jsonfiles. This is intended to make them potentially usable without being edited.
• If you build other software that imports caiman, this release has more functions in the code that generate output files do so in the temp/ directory rather than leave them in a variety of random places. Not everything was changed; future releases will likely continue to increase tidiness on this front, so if you import caiman and need to know where files land, please reach out to pgunn@flatironinstitute.org if you want a heads-up or to make requests/suggestions. Eventually the codebase will likely have per-run directories and have standard file outputs that most functions produce, hopefully making it much easier to wrap caiman (either as a library or as an external call)

The codebase is moving towards the standard that if the user requests something be put in a file with a particular name:

• If that name contains a path, it will be put exactly in that path, meeting the request
• If that name does not contain a path, the codebase will find a suitable place to put it and return or store a value somewhere indicating where it was put. Right now this is always caiman_data/temp/ but in the future after we have per-run directories, there may be further distinction between logs, temporary files, and output files (we have not committed to specifics yet, but intend to move in this direction)

The full changelist:

• Significant SBX format reworking, tiff format fixes from ethanbb
• Bugfix in detecting N5 and ZARR files from marberi
• Removed support for the oldest (pre-hdf5) format of MAT files, better messages for that code in general
• Full refactoring of most CLI demos to be applications, driven by JSON, to be usable without code modification. Added a README
• CLI demos have some long-overdue fixes to display their results correctly. Note that demo_behaviour requires particular builds of OpenCV (that are built against Qt6 rather than standalone).
• Changes across the codebase to have it less often produce files all over the place during a run
• Removed some dead code and entries in data structures throughout the codebase (including some that used the now nonsupported spams library as a dynamic import). caiman.behavior.extract_components() no longer supports the method_factorization value of dict_learn
• Movie.save() now always returns the filename regardless of filetype; this is now preferred because if one doesn’t include a path it will save in temp/ and the caller can use the return value to know how to reopen it later
• environment.yml better handles alternate distributions of conda (like miniforge, which we still don’t recommended but it now more likely to work)
• Ensure that caimanmanager will not copy pycache directories into caiman_data
• Miscellaneous small bugfixes

---

This discussion was created from the release v.1.11.0.

[ethanbb]

Hi Pat, I appreciate the work you're doing to clean things up and implement sensible defaults! It's true that the change to where motion correction saves its results was a breaking change - not too disruptive for me, I just have to move the file after it's finished. But I did want to highlight one issue which is that there's currently no way to override the behavior and save in a different folder, because the base_name is always computed in motion_correction_batch_rigid or motion_correction_batch_pwrigid from a filename that is stripped of its directory. Would appreciate if this could be changed in the future, so I could avoid writing the file to one drive and then moving it to another after the function returns.

[pgunn]

Hi,
There actually is a documented way to set where files are saved - if you set the CAIMAN_DATA environment variable (ideally before you do a caimanmanager install) this will set where the path to caiman_data is (defaults to being under the user's home directory). I'm hoping this is sufficient for your needs; if not, maybe we can do a quick VC to talk about how you're managing storage because I'm hoping that the CAIMAN_DATA env var can be a one-size-fits-all solution and if that's not true it may need a redesign.

[kushalkolar]

You also have to set CAIMAN_NEW_TEMPFILE as True, example usage from mescore:

https://github.com/nel-lab/mesmerize-core/blob/22eedcac379c0747c46961d8dc290274337cc76a/mesmerize_core/algorithms/mcorr.py#L35-L41

[ethanbb]

Ah interesting, there seem to be two different strategies here. The example from @kushalkolar is helpful and shows how I can recover the behavior of saving in a directory of my choice by setting CAIMAN_TEMP each time (by the way, it looks like in the most recent release, CAIMAN_NEW_TEMPFILE has no effect). On the other hand, @pgunn you seem to be saying that setting the data dir once during installation and just using that for everything is the intended design.

For something like the F-order motion correction results, which I immediately re-save in C order, the file is truly temporary and thus maybe it would make sense to always just dump it in the temp folder. And it's good to know I can choose to put that on, e.g., the same shared drive where the rest of my results live and skip copying it afterwards. Even so, I worry somewhat about the slight change of a name collision when multiple analyses are running simultaneously (the recording names aren't necessary unique when removed from the context of the directory they're in), so it seems safer to just keep everything for each analysis session in one directory.

I think I will use the CAIMAN_TEMP method for now, but if you want to chat more about it, happy to get on a call.

[kushalkolar]

Use a randomly generated uuid for each dir to avoid collisions, if you're doing hyperparameter search by running the same vid multiple times you might want to use mesmerize-core.

[ethanbb]

Thanks I might just do that, it looks really neat!

[pgunn]

Most users probably don't have multiple separate storage devices they need to manage this way so they're probably all right. It's possible to use symlinks or CAIMAN_DATA for people with more opinions.

The most recent release is chasing an intermediate notion of cleanliness, but there were a few parts of the code that I didn't change yet (the breakage would've been more severe) - in some circumstances temporary files will still be dropped in weird places.

in the future we'll have per-run directories, which will finally make things pretty clean and well-behaved; with that we'll be able to entirely eliminate the worry of collisions. I'm looking forward to it, but the way I'm imagining doing it will introduce a new concept to the codebase and will be a fair amount of work; the immediate big thing I'm working on is something else.

[ethanbb]

OK, I agree that per-run directories will be ideal - looking forward to that, and until then the way it currently works is fine.