Skip to content

Exiv2 RoadMap #1018

Open
Open

Description

@D4N

Recently our main contributor Robin (@clanmills) decided to retire from the project effective immediately. We don't have to fool ourselves: Robin has been the driving force behind this project for roughly a decade now and invested a truly insane amount of time and effort into it. No one of the currently active contributors has either the time or the expertise to match him. This leaves us in a tough spot: how can we keep this project alive and going?

Current state

  • me and @piponazo are unfortunately currently rather busy in our offline lives and cannot invest a lot of time into exiv2. I was unable to keep up with all the bug reports & pull requests, which (as expected) resulted in a lot of frustration from the other contributors.
  • no one of the current contributors has the expertise and endurance matching that of @clanmills
  • without @clanmills we are badly understaffed (more than before) to even perform basic maintenance
  • the exiv2 webpage is run by Robin, as is the Jenkins build server which produces the release binaries

Short term steps

I propose the following to steps in the short term (until the end of 2019):

  • Move the project into (hard) maintenance mode: i.e. no new features, just bugfixes.
  • Ensure that we can produce a 0.27.3 release by the end of 2019: create releases on our current CI without Jenkins. A prerequisite for that is that we can build exiv2 in a reproducible fashion (which is possible, openSUSE is already doing that with one patch).
  • Move the exiv2 webpage to github or gitlab pages.
  • Migrate all existing svn repositories that are still out there to git under the Exiv2 org

Long term steps

I see the following options:

  • We find new contributors. Imho unfortunately rather unlikely, as the past showed. We can try to raise awareness about exiv2, but unfortunately I don't really know how to make this project appealing to new contributors.
  • We find a corporate sponsor, who will sponsor at least some development time. Due to the removal of the (imho) legally questionable commercial licenses, our usage of the GPL2+ and our place in a rather non-existent commercial market (Linux Desktop application dependency), we're in a pretty rough spot for this. We might try our luck with one of the big enterprise Linux vendors (RedHat, SUSE, Canonical, etc.), provided that we are in their enterprise support package.
  • We rewrite the whole thing? I know this sounds crazy (and probably is), but imho exiv2 shows its age and developers like to work on the new hot stuff™ and not maintain an old application. So we might attract a few people if we decide to rewrite in a new language using new features, etc. But, this is going to alienate our current API consumers and it will take years before we achieve feature parity (if we even ever manage to do that). So not sure how realistic this is, probably not much.
  • Drastically reduce the scope of Exiv2 to match only what our API consumers require (cc @cgilles @phako)?
  • ???

What can we do to improve our processes?

The past showed that our current contribution process became rather complicated, which already drove away contributors. This is bad.

How can we improve for one time contributors?

  • Offer patch submission via email? (WIP via sr.ht: https://git.sr.ht/~d4n/exiv2)
  • Don't require strict code review from early contributors, but instead fix contributed code ourselves to reach the desired quality standard.

How can we improve the process for long term contributors?

  • Pull requests don't get reviewed and cannot be merged 😞. => We could specify a ~2 week grace period for a review, where the PR author should send a weekly review reminder. If the PR author is a "proven contributor", then they can exercise their right to merge an unreviewed PR, provided that the CI is green.
  • CI is flaky and breaks, only one person can fix it (@D4N for GitLab, @piponazo for Appveyor). If the person is busy, PRs are red and cannot be merged 😞. => We could move the tests to github actions, so that everyone has access to them and can fix the issues.
  • PR reviews are merciless and annoying because a lot of minor stuff gets flagged. I am the culprit in this case. If this is a stop gap for many, then I will stop reviewing PRs in detail and instead amend them to fix issues that I find need fixing (provided I have time).

How can we attract more people to contribute?

  • Your idea here

Any further ideas/comments/suggestions?

Activity

  1. pinned this issue on Oct 5, 2019
  2. cgilles commented on Oct 5, 2019

    @cgilles
    Collaborator

    Hi all,

    I'm always listening Exiv2 conversation, even if i'm not very active in the project.

    As digiKam use Exiv2 API to handle all metadata excepted video, considerate me as present for some task. one where i'm interested is to support new image format. actually i finalize the HEIC/HEIF support in digiKam. I know that an entry exist to support this format in Exiv2, so i will be ready to check code and test new implementation for a quick move in production.

    https://i.imgur.com/JmEgGNS.png
    https://i.imgur.com/MlNxPLa.png

    I plan also to integrate Webp, Jpegxr, FLIF, support as native loader in digiKam in the future. So All plan to support these file format in Exiv2 are in my mind.

    As i practice cmake since 8 years actively in digiKam and in my office, i can help for CMake problems in Exiv2. Just ask me if necessary.

    In digiKam, i implemented a lots of unit tests to check the large Qt / Exiv2 interface, with a huge collection of images. I already discovered plenty of non re-entrancy in Exiv2 API and i consolidate digiKam code with several lock, as we use multi-threading and multi-core everywhere.
    In digiKam, Exiv2 API is only located in the core Qt interface. So the code in use is limited and not dispatched everywhere. Rememeber that digiKam code is more than 1.5 M of C++ code lines.

    I'm also aware about the MinGW port, as we cross compile ALL the digiKam code + all dependencies for Windows under Linux using MXE. I'm in love with cross compiling, so if something do not work here, i'm ready to report a fix quickly.

    digiKam has a large users base, and we publish weekly a new test version for Linux, Windows, and MacOS. If something is broken or do not work in Exiv2, we are able to report the dysfunction quickly.

    Voilà, I'm here to help you if necessary if time permit.

    Best

    Gilles Caulier

  3. fgeek commented on Oct 5, 2019

    @fgeek

    Thank you Robin Mills for all your efforts towards better open-source software. It was a pleasure to cooperate with you 👍

  4. phako commented on Oct 5, 2019

    @phako
    Contributor

    Mainly some random thoughts there, sorry

    • I'll probably also could check out cmake stuff, it's part of my daily work as well.

    • I can provide some web hosting if necessary

    • I could probably also provide some temporary infrastructure for Jenkins (would have to discuss that beforehand, though)

    • What exactly was the Jenkins doing for releases that cannot be replicated by other means of CI?

    • I'd suggest to drop old unmaintained code, which you've already started on master. Do only meta-data.

    • While the idea of just doing a rewrite sounds always great, especially if the codebase wasn't yours to begin with, the job of doing the rewrite or ever reaching feature parity with the old codebase might be a lost fight from the start

    • A rewrite does not relief you from bugfixing the old code, unfortunately

    • If there's a rewrite, it needs to be consumable by C/C++ which rules out most of the "sexy" languages anyway. The only way I can actually think of would be a gradual conversion to Rust.

    • I'd also be interested in getting HEIC/HEIF support as well; I was following the PR but lost track

    • Attracting contributors for old stuff is usually hard. Best guess would be to get your "downstream" abord contributing, which you are obviously trying ;)

    • Have you ever communicated the current lack of staffing? I wasn't really under the impression that there is any issue there

    • TBH, I found the setup of the exiv2 project a tad weird. It's missing some basic infrastructure I am used to from other projects, like a user's / developers discussion forum (whatever form). Things certainly got more obvious with the move to github, though.

    • Unfortunately for you, the initial decision for GPLv2, while encouraged by the FSF or rather RMS for libraries instead of going for its Lesser cousin limits corporate adoption. I suppose that this was done to encourage the use of the commercial license for those use-cases. Even more unfortunate is that this isn't something that can easily be changed without a massive headache

  5. tester0077 commented on Oct 6, 2019

    @tester0077
    Collaborator
  6. derselbst commented on Oct 6, 2019

    @derselbst
    Collaborator

    Hi,

    I am mainly a user of exiv2 (coming from gwenview) and pretty new to the team; Robin invited me recently. I'm familiar with C++, Cmake, CI. However, I'm also part of two other organizations here at GitHub, which require most of my free time. But I'll try my best to help out where I can, just ping me.

    Regarding how to attract new contributors, I would like to mention that the number of contributors is actually growing. So things don't look too bad to me. Simplifying the review process sounds good to me, @D4N has raised some good ideas. A rewrite IMO would be counterproductive for this project.

  7. phako commented on Oct 6, 2019

    @phako
    Contributor

    Btw, I need to release a new version of gexiv2 soonish. The document mirror on exiv2.dyndns.org is gone, I assume?

  8. clanmills commented on Oct 6, 2019

    @clanmills
    Collaborator

    exiv2.dyndns.org is gone. The release is on https://exiv2.org and the pre-release on https://pre-release.exiv2.org

    A couple of obvious points about v0.27.3 (and later v0.27 dots)

    1. You can publish source releases on GitHub without updating exiv2.org
    2. If you publish releases on GitHub, I'm willing to update exiv2.org and build the binaries with Jenkins on the MacMini, if (and only if) the web and release code is stored on SVN. I've retired because of frustration with review/git/github.

    I wish I could make the fuzzing police disappear or get them to participate in fixing security issues. They appear to have more resources than Team Exiv2. I believe Kevin is the only "fuzzer" who both reported and fixed CVEs.

  9. phako commented on Oct 6, 2019

    @phako
    Contributor

    I ased regarding exvi2.dyndns.org because I received a patch to point all the docs to there. Which I will probably revert now.

    Regarding the fuzzing: Unfortunately, fuzzing is quite use- and helpful. But, looking at eg. #1019 that ticket is basically useless.

    • You should take away reporter's abilites to classify tickets. Classifying tickets is the project's job
    • You should make fuzzing guidelines that embraces those people (I honestly have an issue with people who think writing in l33t is giving them some kind of credibility...) but also gives them a check-list of what to provide. Filing a fuzzing ticket with no reproduction material is basically use-less, it just leaves you the chance of catching such issues by code review. All fuzzers I know give you easy access to the input material that was causing an issue
  10. clanmills commented on Oct 6, 2019

    @clanmills
    Collaborator

    Jens: I seem to recall our conversation about dyndns.org in Exiv2 v0.27-RC days (about one year ago). Since then, thanks to @nehaljwani, we moved exiv2.org to a new AWS and have the pre-release web-site.

  11. clanmills commented on Oct 6, 2019

    @clanmills
    Collaborator

    I agree with you that fuzzing is useful. However the behaviour of the fuzzing police is very unhelpful and demotivating. That bug report today is a good example. I wouldn't be surprised if it's already fixed in v0.27.2 and is actually a waste of time.

  12. piponazo commented on Oct 6, 2019

    @piponazo
    Collaborator

    Hi all,

    Thanks @D4N for the effort of creating this issue and giving details about the current state of the project and proposing some short/long terms actions to try to improve the situation. I find useful to have this open discussion in a github thread instead of having it in a chat.

    Please find my feedback about each of the proposed steps:

    Short term steps

    Move the project into (hard) maintenance mode: i.e. no new features, just bugfixes.

    I guess that here you are referring here about the support we can give to the project given our busy lives. I do not see myself contributing in new features during the next months, due to my personal situation, and I agree that if I find time to contribute to the project, I will do it fixing reported bugs in 0.27-maintenance.

    Nonetheless, I would be more than happy to see people contributing new features to master and I will try to also dedicate time to review those PRs and guide contributors as much as possible.

    Ensure that we can produce a 0.27.3 release by the end of 2019: create releases on our current CI without Jenkins. A prerequisite for that is that we can build exiv2 in a reproducible fashion (which is possible, openSUSE is already doing that with one patch).

    I think it should be possible to generate the binary artifacts for 0.27.3 from travisCI and Appveyor. I could dedicate some to this topic in the next days/weeks. I remember that long ago we discussed about some security issue in travisCI regarding the deployment of binary artifacts and that's we discarded that option. Opinions about this ?

    Move the exiv2 webpage to github or gitlab pages.

    I like this idea. I think it is good to centralise as much as possible all the content/discussions about Exiv2 to a single place.

    Migrate all existing svn repositories that are still out there to git under the Exiv2 org

    I am not sure to what are you referring to here. Would you mind to develop a bit more this?

    Long term steps

    Drastically reduce the scope of Exiv2 to match only what our API consumers require

    I think this is matching we some of the previous actions we have taken on master: i.e, to drop stuff not related with Image Metadata. Of course, any suggestions to keep improving the API would be welcomed.

    We find new contributors. Imho unfortunately rather unlikely, as the past showed. We can try to raise awareness about exiv2, but unfortunately I don't really know how to make this project appealing to new contributors.

    Not sure either about how to find new contributors. Exiv2 has been for me the first open source project in which I participate regularly. Somehow, after working on it for several months I felt responsible of addressing bugs and improving some specific areas ... I guess that newcomers would be interesting in addressing more interesting topics, such as adding support for new formats.

    We rewrite the whole thing?

    I do not think it is realistic.

    Improve processes

    Pull requests don't get reviewed and cannot be merged

    When I had lot of spare time to work on Exiv2 (in a period in which I was switching jobs) this is what I found more annoying about our process. For big changes, I think it is fine to establish some "grace period" for a review. But most of the times we are fixing small bugs or making small improvements here and there which should not be blocked for so long. From my point of view, anyone with admin rights in the Exiv2 group, should have freedom and the good judgement to merge something quickly when the changes are mostly cosmetic or when we are dealing with fixing something in CI (just the first examples that came to my mind). In case one of these persons abuse this power, we could analyse the behaviour and decide what to do.

    CI is flaky and breaks, only one person can fix it (@D4N for GitLab, @piponazo for Appveyor). If the person is busy, PRs are red and cannot be merged disappointed. => We could move the tests to github actions, so that everyone has access to them and can fix the issues.

    Would it not be possible to move the permissions of Gitlab, TravisCI and Appveyor to the Exiv2 team instead of being me and @D4N the owners of these services? I will investigate with TravisCI and Appveyor (I was the one adding those services to Exiv2).

    PR reviews are merciless and annoying because a lot of minor stuff gets flagged. I am the culprit in this case. If this is a stop gap for many, then I will stop reviewing PRs in detail and instead amend them to fix issues that I find need fixing (provided I have time).

    I do not think this is something bad per se. I particularly always appreciate the time people dedicate to review the code I am submitting, and if the suggestions made are good, I always try to address the challenges. I also know by experience that this is not the most common way to react to the code reviews, and some people gets are less open to change things (for whatever reason).

    Something that I proposed in few companies where I have worked is to tag the comments with labels indicating the importance of the comment. Something like:

    • [minor] : Something that could be improved, but that is really a minor thing. Nothing would happen if it is not changed.
    • [style, naming] : Cosmetic things. [naming] is normally about improving code readability. [style] is about following the project guidelines / style. Not considered as blocking issues, but it would be nice to address.
    • [bug/leak] : Problems detected in the new code. Need to be addressed before accepting.
    • [important / blocker] : Not a bug or leak, but something it is extremely important from the point of view of the reviewer, and it should be addressed before the PR is accepted.

    I would say that we could try to be a bit more indulgent in some cases (contributors do not have much spare time or have difficulties with Git), and approve PRs unless there are open comments tagged with [bug/leak/blocker]. Then, for the project maintainers it would take probably few minutes to address the other less important comments.

  13. 36 remaining items

  14. unpinned this issue on Feb 4, 2021
  15. added this to the v1.00 milestone on Apr 13, 2021
  16. modified the milestones: v0.27.4, v1.00 on May 22, 2021
  17. clanmills commented on Aug 9, 2021

    @clanmills
    Collaborator

    Exiv2 v0.27.5 RC3 2021-09-30

    Changes since RC2.

    Group PR Topic Issue
    Build #1938 Exiv2 v0.27.5 RC3 #1935
    Lens #1936 Canon EF 80-200mm f/4.5-5.6 II #1906
    Bug #1925 Fix out of range access in src/tags_int.cpp #1918
    Bug #1924 Fix quadratic loops in pentaxmn_int.cpp #1910
    Bug #1923 Remove bogus code in XMPUtils.cpp #1902
    Build #1937 Fix compiler warning on Apple/M1/Clang #1915 (review)
    Build #1937 Fix linking libiconv on macOS with M1 Requires CMake 3.14+ #1903
    Withdrawn Reading HEIC from iPhone images #1928

    We intend to release Exiv2 v0.27.5 on 2021-10-22.

  18. kevinbackhouse commented on Aug 9, 2021

    @kevinbackhouse
    Collaborator

    Should we add "libFuzzer target (for improved security testing)" to the list of new features?

  19. clanmills commented on Oct 21, 2021

    @clanmills
    Collaborator

    Exiv2 v0.27.5 GM 2021-10-22

    Topic More Info
    Remaining ( 0) https://github.com/Exiv2/exiv2/milestone/9
    Completed (87) https://github.com/Exiv2/exiv2/milestone/9?closed=1

    Exiv2 v0.27.5 Features

    1. BMFF bug fixes including CR3 previews
    2. Security fixes
    3. libFuzzer target (for improved security testing)
    4. Exiv2 monitored by oss-fuzz
    5. Minor bugs and fixes

    Exiv2 v0.27.5 Acknowledgements

    This release is due to Kev and David, supported by the rest of the team.
    Thank You, Kev for your hard work and attention to detail.
    Thank You, David for your effort on CR3 Previews.

    Contributor Activity
    Alex Project Management
    Christoph C++ code
    David CR3 preview C++ code
    Kev Outstanding work on security and other issues
    Luis C++ modernisation
    Miloš C++ code
    Peter K C++ code
    Peter S C++ code
    Robin Release engineering

    If I've forgotten to acknowledge your contribution, I apologise.

    Help Wanted

    Team Exiv2 is a happy band of enthusiastic engineers. We have several tasks for which we're looking for volunteers.

    1. Extended test coverage.
    2. Use of Coverity Scan (static code analysis).
    3. Release Engineer.

    If you'd like to contribute to Exiv2, please talk to us on the chatserver: https://matrix.to/#/#exiv2-chat:matrix.org

    What's next for Exiv2?

    Team Exiv2 is very active and deals with several issues and PRs every week.
    There is no active release schedule at present.
    A reasonable guess about the future is:

    1. Probably another 0.27 dot release. Spring 2022 maybe.
    2. Exiv2 v1.00. Summer 2022 maybe.

    Exiv2 v0.27.5 Release Notes (updated 2021-10-21)

    Group PR Topic Issue
    Bugs &
    Fixes
    #1968 Canon cr3 previews #1893
    #1981 Limit CR3 previews to JPEG only
    #1975 BMFF Performance boost: don't read mdat box
    #1925 Fix out of range access in src/tags_int.cpp #1918
    #1924 Fix quadratic loops in pentaxmn_int.cpp #1909
    #1923 Remove bogus code in XMPUtils.cpp #1901
    #1889 Avoid reading 1 byte past end of no nul string #1888
    #1884 Throw error if preview is greater than 1MB #1882
    #1880 Add some XML validation to avoid xmpsdk bugs #1878
    #1855 Enforce BMFF box nesting
    #1848 Assert->error TiffDirectory::doWriteImage #1847
    #1846 Assert->error TiffDirectory::writeDirEntry #1845
    #1844 malloc->DataBuf WebPImage::decodeChunks #1841
    #1840 Check float within int range before cast #1838
    #1834 Assert->error TiffMnEntry::doCount() #1833
    #1831 Safer cast double to long in value.hpp #1827
    #1828 Check double in range before cast to uint32_t #1827
    #1820 Check string isn't empty #1819
    #1818 Fix memory leak in pngimage.cp #1817
    #1816 check to prevent out-of-bounds read in memcmp #1815
    #1814 jp2image.cpp: check size before allocation #1812
    #1796 Check embedded RAF image is really a TIFF #1791
    #1790 bufRead needs to be adjusted after seek()
    #1789 &bytes[0] (std::vector) will crash if bytes has zero elements
    #1788 Make sure that read is complete to prevent infinite loop
    #1765 Add Exif.Image.PageName tag
    #1758 Check iterator is not end() after findKey
    #1752 Fix incorrect loop condition
    #1745 avoid processing MOV files when BMFF enabled
    #1739 Fix assertion failure in crwimage_int.cpp
    #1737 Zero initialize local variables
    #1735 Use vector::at() rather than operator[]
    #1720 Stricter date parsing in value.cpp
    Security #1882 Large allocation in tiffvisitor_int.cpp #1881
    #1769 Safer std::vector indexing
    #1778 Fix infinite loop Image::printIFDStructure
    #1767 Extra checking to prevent loop counter from wrapping around
    #1759 Better bounds checking in Jp2Image::printStructure
    #1750 Avoid integer divide by zero
    Lens #1936 Canon EF 80-200mm f/4.5-5.6 II #1906
    Build #1983 Exiv2 v0.27.5
    #1972 Only include expat.h when XMP is enabled. #1971
    #1966 Fix macOS build failure
    #1939 Add workaround for conan outage
    #1938 Exiv2 v0.27.5 RC3 #1935
    #1937 Fix compiler warning on Apple/M1/Clang #1915 (review)
    #1937 Fix linking libiconv on macOS with M1 Requires CMake 3.14+ #1903
    #1897 Lock conan version on Windows
    #1894 0.27.5 RC2 #1891
    962ef36 builds fail on gitlab debian CI #1871
    #1864 workaround for softprops/action-gh-release
    #1862 Fix macOS workflow to use gtest 1.8.0
    #1861 add doc to release builds
    #1860 Exiv2 v0.27.5 RC1
    #1859 Enable BMFF in Actions workflows
    #1858 Fix compiler warning on Apple/M1/Clang #1856
    #1854 Actions and fuzzer for 0.27-maintenance
    Test #1853 Use @unittest.skip for ASAN builds #1821
    #1754 Fix Ubuntu 20.04/Release/Sanitizer test breaker
    XMPsdk #1851 memory leak in addBinding (xmlparse.c) #1821
    Localisation
    Documentation #1870 update_27.5_docs_again
    #1868 update release notes for v0.27.5 RC1
    #1865 update_docs_for_0.27.5.1
    Withdrawn
    No action
    improvable error message #1977
    Crash in Debug Mode #1960
    Reading HEIC from iPhone images #1928
    include search path is wrong for exiv2 command-line program #1895
    CI trigger #1873
  20. added a commit that references this issue on Oct 25, 2021
  21. added a commit that references this issue on Aug 14, 2023
  22. modified the milestones: v0.28.0, Backlog on Nov 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions