API Compatibility v0.27 vs v0.27 "dots"

[clanmills]

@nehaljwani shared something about ABI/API compatibility of different releases of Exiv2. I think it was this: https://abi-laboratory.pro/index.php?view=timeline&l=exiv2

In the past, I have not worried about API/ABI compatibility. Every release of Exiv2 (v0.24, v0.25 etc) has similar and slightly different APIs. They have a different set of headers and different library filenames. This is “open-source”, and users are expected to recompile applications for different versions of Exiv2.

However, with “the dots” we have a different scenario. In February, I had an email conversation with Dan Bricklin (Yes, the guy who invented Lotus-1-2-3) and he wanted to use Exiv2 for one of his projects. He had several conditions and one was “he could replace DLLs in an existing application”. Dan had a couple of features he wanted us to add to Exiv2 and we didn’t shake hands on a deal. I couldn’t undertake the necessary work and I can’t ask any member of Team Exiv2 to deliver anything to Dan’s schedule. We didn’t discuss money.

On 2019-05-23, I built Exiv2 0.27.0 and ran the test suite on MacOS-X. Then I build Exiv2 0.27.2.99 (0.27 maintenance development for v0.27.2). I successfully ran the 0.27 test suite using the 0.27.2.99 .dylib. Very good. Very pleased that this is working OK. I didn’t try on other platforms such as Windows or Linux.

I’ve written a 1 line command to extract the API from the library:
$ nm -g --demangle libexiv2.0.27.0.dylib | grep ' T ' | cut -d' ' -f 3- | sort > ~/temp/27.0.txt

This is a 4606 line file. (4610 lines for 0.27.2.99). So small changes in the API. Nothing that seems to be a disaster. I would like folks to be able to replace 0.27 DLLs for existing applications without recompiling anything.

Here's the diff. I'll see if this can be reduced. The remaining differences will be documented in the release notes for v0.27.2

I've invited Nehal to get involved in this effort and it fits well with tasks he has been assigned at work. So, Nehal will be the expert on API/ABI compatibility for "the dots".

I don't see any reason to be concerned about API/ABI compatibility for v0.28.

689 rmills@rmillsmbp:/Applications/Xcode.app $ diff ~/temp/27.0.txt ~/temp/27.2.txt 
774c774
< Exiv2::Internal::LensIdFct const* Exiv2::find<Exiv2::Internal::LensIdFct const, long, 46>(Exiv2::Internal::LensIdFct const (&) [46], long const&)
---
> Exiv2::Internal::LensIdFct const* Exiv2::find<Exiv2::Internal::LensIdFct const, long, 49>(Exiv2::Internal::LensIdFct const (&) [49], long const&)
1088c1088
< Exiv2::Internal::TagDetails const* Exiv2::find<Exiv2::Internal::TagDetails const, Exiv2::Internal::LensTypeAndFocalLengthAndMaxAperture, 352>(Exiv2::Internal::TagDetails const (&) [352], Exiv2::Internal::LensTypeAndFocalLengthAndMaxAperture const&)
---
> Exiv2::Internal::TagDetails const* Exiv2::find<Exiv2::Internal::TagDetails const, Exiv2::Internal::LensTypeAndFocalLengthAndMaxAperture, 360>(Exiv2::Internal::TagDetails const (&) [360], Exiv2::Internal::LensTypeAndFocalLengthAndMaxAperture const&)
1109d1108
< Exiv2::Internal::TagDetails const* Exiv2::find<Exiv2::Internal::TagDetails const, long, 352>(Exiv2::Internal::TagDetails const (&) [352], long const&)
1110a1110
> Exiv2::Internal::TagDetails const* Exiv2::find<Exiv2::Internal::TagDetails const, long, 360>(Exiv2::Internal::TagDetails const (&) [360], long const&)
2403c2403
< Exiv2::WebPImage::decodeChunks(unsigned long long)
---
> Exiv2::WebPImage::decodeChunks(long)
2728a2729
> Exiv2::readChunk(Exiv2::DataBuf&, Exiv2::BasicIo&)
3233a3235
> _exvGettext(char const*)
3235a3238
> bool Safe::Internal::builtin_add_overflow<long>(long, long, long&)
3272a3276
> long Safe::add<long>(long, long)
3977c3981
< std::__1::basic_ostream<char, std::__1::char_traits<char> >& Exiv2::Internal::printTag<352, Exiv2::Internal::canonCsLensType>(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, Exiv2::Value const&, Exiv2::ExifData const*)
---
> std::__1::basic_ostream<char, std::__1::char_traits<char> >& Exiv2::Internal::printTag<360, Exiv2::Internal::canonCsLensType>(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, Exiv2::Value const&, Exiv2::ExifData const*)
690 rmills@rmillsmbp:/Applications/Xcode.app $ 

I will investigate the differences and they will be eliminated or justified in releasenotes/releasenotes.txt

[D4N]

< Exiv2::WebPImage::decodeChunks(unsigned long long)

Exiv2::WebPImage::decodeChunks(long)

This change was introduces in commit 86816c0 (by @kevinbackhouse)

Exiv2::readChunk(Exiv2::DataBuf&, Exiv2::BasicIo&)

This change was introduced in commit d1097fa (by @piponazo). I guess that this one should have been marked with hidden visibility, but we missed that during code review.

_exvGettext(char const*)

Not sure why this shows up in the diff, the header containing this function got touched last in commit 0f01822 (from 2015).

long Safe::add(long, long)

This is a specific instantiation of a template function, it already existed in 0.27, just not this specific instance.

The remaining functions are from the Internal namespace, which I wouldn't strictly consider as part of the API.

[clanmills]

Thanks for those comments @D4N. Very helpful.

You're correct that entry points in the "Internal" namespace aren't in the "public API". My aim is to enable "dynamic library swapping" (DLLs on Windows, .so on Linux/Unix and .dylib on MacOS-X). So far, I've only looked on MacOS-X and will be investigating other platforms for v0.27.2.

We're in Scotland this week on family business. Looking forward to releasing v0.27.2 in June. Aiming for RC1 about 15/6 and GM on 30/6.

[D4N]

My aim is to enable "dynamic library swapping" (DLLs on Windows,
.so on Linux/Unix and .dylib on MacOS-X). So far, I've only looked on MacOS-X
and will be investigating other platforms for v0.27.2.

Dynamic library swapping requires a stable ABI, which is considerably more work
than having a stable API, so I don't think that we'll be able to achieve that before 1.0.

[clanmills]

I did more work on this two weeks ago while on vacation in Scotland. I can copy the Shared objects (DLL, .so, .dylib) built with 0.27.2.99 and run them on 0.27.0 test suite. This runs successfully. It produces some benign warnings for (correct) output changes.

I am going to focus on releasing 0.27.2.1 (v0.27.2-RC1) and will return to this for v0.27.2. I hope that the output of the following command will be identical on 0.27.1 and 0.27.2 on all platforms.

nm -g --demangle libexiv2.0.27.0.dylib | grep ' T ' | cut -d' ' -f 3- | sort

[clanmills]

@cryptomilk has recommended that we use this:
https://github.com/ansasaki/abimap

Thank you @cryptomilk I'm sure this will be very useful.

[cryptomilk]

abimap would make sense for 0.28 and later as this is some work. See

https://git.libssh.org/projects/libssh.git/tree/CMakeLists.txt#n161
https://git.libssh.org/projects/libssh.git/tree/cmake/Modules/GenerateMap.cmake
https://git.libssh.org/projects/libssh.git/tree/cmake/Modules/GetFilesList.cmake
https://git.libssh.org/projects/libssh.git/tree/cmake/Modules/ExtractSymbols.cmake

[cryptomilk]

Btw. I've contacted Andrey and he added exiv2 master to abi checker:

https://abi-laboratory.pro/index.php?view=timeline&l=exiv2

[clanmills]

I'm going to defer this for v0.27.3. This is an important topic. Dan's adding a task to the CI to use abimap and I'll update our status about this for v0.27.3.

[clanmills]

I've run the test suite on Exiv2 v0.27.2 and replaced the dynamic libraries with the 0.27.3.9 versions.

Platform	0.27.3 library	0.27.2 library
macOS	5,515,824 libexiv2.0.27.3.9.dylib	5,269,532 libexiv2.0.27.2.dylib
linux	4,770,184 libexiv2.so.0.27.3.9	4,598,328 libexiv2.so.0.27.2
Cygwin	5,886,262 cygexiv2-27.dll	5,536,628 cygexiv2-27.dll
MSYS	5,888,692 msys-exiv2-27.dll	5,535,525 msys-exiv2-27.dll
msvc	5,196,288 exiv2.dll	5,157,376 exiv2.dll

Success
Suite ran without crashing and with no notable exceptions being reported on all tested platforms. There are of course differences in the output of the test suite as the bugs have been fixed. For example the reporting of GPSLongitude has been changed in 0.27.3.

[clanmills]

Exiv2 v0.27.4

I have run the test suite on Exiv2 v0.27.2 and v0.27.3 using the 0.27.4.9 dynamic libraries.

Platform	0.27.4.9 library	0.27.3 library	0.27.2 library
macOS	5,753,416 libexiv2.0.27.4.dylib	5,567,328 libexiv2.0.27.3.dylib	5,295,248 libexiv2.0.27.2.dylib
linux	5,267,160 libexiv2.so.0.27.4	5,104,400 libexiv2.so.0.27.3	4,715,384 libexiv2.so.0.27.2
Cygwin	6,078,887 cygexiv2-27.dll	5,879,668 cygexiv2-27.dll	5,611,991 cygexiv2-27.dll
MinGW/msys2	7,128,920 libexiv2.dll	6,920,445 libexiv2.dll	6,240,332 libexiv2.dll
msvc	5,572,096 exiv2.dll	5,412,352 exiv2.dll	5,361,664 exiv2.dll

Success

Suite ran without crashing on all tested platforms with no notable exceptions being reported. There are of course differences in the output of the test suite as bugs have been fixed or features added. For example the reporting of Exif.Image.PlanarConfiguration has been changed in 0.27.4 to report "Chunky" instead of 1.

[1div0]

I've noticed wrong versions of the libraries in the 0.27.4.9 library column.

[clanmills]

Oh, this is work in progress, Peter. It'll be correctly finished later today.