Add support for libraw snapshots?

[harshithdwivedi]

I was looking at libraw snapshots and it seems like they added support for reading Canon CR3 files and a few more camera supports in their 0.20 snapshot build.

While I understand that you don't want to add support for non-stable releases of libraw, but quoting from the release cycle of libraw: LibRaw/LibRaw#191 (comment)

We publish snapshots every 5-7 months.
We publish releases (0.18, 0.19, etc) every 1-1.5 years.
We publish hotfixes for releases (0.19.1, 0.19.2) ASAP, but this hotfixes are 'security fixes only'.

Waiting 1-1.5 years to get the changes in the snapshot build seems a bit too much imo; and as quoted by other sources (microsoft/vcpkg#6759), it seems like their snapshots are stable enough for normal use.
The only reason it's not released is that libraw doesn't publish a release unless all the features planned for the release have been finished: LibRaw/LibRaw#205 (comment)

My suggestion:

Have the current rawpy release (0.14) as it is and add a new snapshot release incorporating changes from the snapshots released by libraw.
As it would be a beta release, those who don't want to use it can get the 0.14 release via pip install.

Also since libraw snapshots are released once every 4-5 months, keeping up with their snapshot releases won't be much of a hassle anyways.

Let me know if this sounds good to you, it'll really save a lot of time for most of the people using rawpy and dealing with issues like #105.

Thanks again!

[letmaik]

What you say makes sense in principle. In practical terms, it means a few things:

1. Change the CI to produce and publish weekly (?) dev releases from the latest libraw master. I don't think there's much value in sticking to arbitrary "snapshots", given that there is no documentation which commit corresponds to which libraw "snapshot". But even if there was a mapping it also would mean more manual work to bump to the next snapshot each time.
2. Someone has to look after the CI and fix/analyse any errors that may occur due to the frequent releases off of master.
3. Docs have to be extended with instructions on how to install a dev release of rawpy.

I think the critical bit is maintenance effort that comes with frequent (dev) releases, especially based on master. If you like to help out with this and become a maintainer then I'm happy to go forward with your plan.

[harshithdwivedi]

Sounds like a plan!
I would love to help out with maintaining the beta releases. Since I'm depending heavily on rawpy in my current project, it only seems reasonable :)

However, maybe weekly builds would be a bit too much.
I'll look into their forums to see which commit hash represents which snapshot and i believe that they also have blog posts announcing releases so that should make things easier.

How do you suggest that we take this forward?

[letmaik]

OK, that makes sense. Not having automated weeklies/nightlies makes it easier.

I would suggest to create a new branch prerelease which is used exclusively for releasing new rawpy pre-releases based on libraw pre-releases (snapshots, but also alphas/betas/rcs for example). Any new rawpy features should always happen in master and then merged into prerelease. I don't think we need to consider the case of doing rawpy pre-releases based on libraw stable releases, which could create conflicts in versioning, e.g. rawpy 0.15.0a1 may then be either a pre-release of libraw stable or pre-release, based on master or prerelease... this would get too confusing. So let's just say any rawpy pre-release by definition must be based on a libraw pre-release and originate from the prerelease branch.

Currently, releases of rawpy are triggered by pushing a git tag starting with v, which means the same can be used for pre-releases, e.g. by tagging a commit from prerelease as v0.15.0a1.

So, to get started, I created the prerelease branch. How about you create a PR targeting that branch which bumps the libraw submodule to the latest snapshot and also bumps the rawpy version to 0.15.0a1 (see rawpy/_version.py). If the CI passes (which I doubt, given the issue you reported on macOS) then I can merge it and push the git tag to publish the pre-release. Unfortunately, it's not possible to have PRs for adding new git tags, so this is something I will have to do manually for now, but I guess it's ok since there won't be many pre-releases anyway.

[harshithdwivedi]

Sounds good, let me work on this.

[letmaik]

I tagged v0.15.0a1 and the new pre-release is up on PyPI. I also updated the README to say what a pre-release is and how to install it.