Bundled bitarray appears to not be Python 3.11 compatible

[djhoese]

The conda-forge community has started building packages for Python 3.11. It seems that pylibtiff is having trouble building under Python 3.11 with errors like:

  libtiff/bitarray-a1646c0/bitarray/_bitarray.c:148:23: error: lvalue required as left operand of assignment
    148 |         Py_SIZE(self) = newsize;
        |                       ^
  libtiff/bitarray-a1646c0/bitarray/_bitarray.c:175:19: error: lvalue required as left operand of assignment
    175 |     Py_SIZE(self) = newsize;
        |                   ^
  libtiff/bitarray-a1646c0/bitarray/_bitarray.c: In function 'newbitarrayobject':
  libtiff/bitarray-a1646c0/bitarray/_bitarray.c:196:18: error: lvalue required as left operand of assignment
    196 |     Py_SIZE(obj) = nbytes;
        |                  ^

I'm guessing this is due to how old the bundled bitarray is in pylibtiff. Any chance of updating it and making a new release?

Here is the conda-forge PR for Python 3.11: conda-forge/pylibtiff-feedstock#14

[djhoese]

@gcbrown (as mentioned on #135), my main problem with this bitarray thing is I don't know the history of why it is here or why it is at the version it is. Put another way:

• why is it embedded?
• is this specific version (a1646c0) the last version that worked with pylibtiff and later versions don't?

Perhaps @pearu or another maintainer can try to remember what happened here. If the answer to the second question is "yes, that's exactly the problem" then we have some major work ahead.

Theoretically we should be able to "just" update the version of bitarray embedded in pylibtiff (or remove it so it imports the installed bitarray) and then any tests should reveal what doesn't work. One problem with that is that the tests are still using TravisCI which is no longer free. They need to be migrated to GitHub Actions.

[gcbrown]

@djhoese Just to note (may be helpful), the oldest version of bitarray that installs without error is 1.6.3. I'm assuming the migration will be more difficult with the new major version (currently at 2.6.0).

Edit: Good news! To test my assumption, I replaced the embedded bitarray sources with those from the bitarray-1.6.3 release zip. Everything built properly and I was able to succesfully run my use case of reading a TIFF file into a numpy array. This also worked with bitarray==2.6.0, so maybe it will work with most versions of bitarray?

One thing I noticed is even with the embedded bitarray, it still fails unless you've installed bitarray from pypi. So it's not properly using the embedded version somewhere in there.

Given all of this, I have hope we can completely remove the embedded bitarray version.

Edit 2: accidentally used my work account, whoops

[djhoese]

Great! If you want to try making a PR to remove the bundled one and/or run tests locally with removing it that'd be a good experiment. I see here that bitarray is only used if the user changes a hardcoded value:

pylibtiff/libtiff/lzw.py

Lines 10 to 17 in 51d6f2a

	default_backend = 'bittools'
	# default_backend='bittools'
	if default_backend == 'bitarray':
	from bitarray import bitarray
	if default_backend == 'bittools':
	from bittools import setword, getword

And only for lzw compression. I'm not sure if it is used elsewhere.

[djhoese]

Looks like this is really it and the bundled version is added as part of setup.py. Looks like in #47 @jat255 added the bundled version to support Python 3 in some way, but it isn't clear to me why the bundling was used. Perhaps the upstream bitarray hadn't been updated for Python 3 yet and this bundled version includes those necessary updates?

[djhoese]

Ah, based on the merge gap between 2014 and 2016 in bitarray I'm guessing that's what happened:

https://github.com/ilanschnell/bitarray/pulls?page=3&q=is%3Apr+sort%3Aupdated-desc+is%3Aclosed

[jat255]

@djhoese this is a bit of a blast from the past! IIRC, pylibtiff had a bundled version of bitarray 0.3.5 from numpy that was incompatible with python3, so I updated the bundled version to the latest code at the time (see this change), which was ilanschnell/bitarray@a1646c0 .

My guess (I don't know for sure) is that it was bundled maybe due to difficulties installing at the time (2008, when that version of bitarray is from, was a very different time in the Python packaging world). My suggestion now would be to include it as a dependency instead of bundling a copy, to avoid issues like this in the future.

[gcbrown]

@djhoese I've decided to use the pillow library instead. It works for TIFF, has active devs, and is pulled in by matplotlib anyway. I think packages like this focused on a single format have been gradually swallowed by multipurpose packages like pillow for a good reason. Good luck if you choose to work on this, but for anyone with this issue I would recommend pillow.

[djhoese]

If I remember correctly this library allows things that pillow can't do like custom TIFF tags and others. Glad pillow will work for you.