Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Future of Exiv2? #1018
Recently our main contributor Robin (@clanmills) decided to retire from the project effective immediately. We don't have to fool ourselves: Robin has been the driving force behind this project for roughly a decade now and invested a truly insane amount of time and effort into it. No one of the currently active contributors has either the time or the expertise to match him. This leaves us in a tough spot: how can we keep this project alive and going?
Short term steps
I propose the following to steps in the short term (until the end of 2019):
Long term steps
I see the following options:
What can we do to improve our processes?
The past showed that our current contribution process became rather complicated, which already drove away contributors. This is bad.
How can we improve for one time contributors?
How can we improve the process for long term contributors?
How can we attract more people to contribute?
Any further ideas/comments/suggestions?
I'm always listening Exiv2 conversation, even if i'm not very active in the project.
As digiKam use Exiv2 API to handle all metadata excepted video, considerate me as present for some task. one where i'm interested is to support new image format. actually i finalize the HEIC/HEIF support in digiKam. I know that an entry exist to support this format in Exiv2, so i will be ready to check code and test new implementation for a quick move in production.
I plan also to integrate Webp, Jpegxr, FLIF, support as native loader in digiKam in the future. So All plan to support these file format in Exiv2 are in my mind.
As i practice cmake since 8 years actively in digiKam and in my office, i can help for CMake problems in Exiv2. Just ask me if necessary.
In digiKam, i implemented a lots of unit tests to check the large Qt / Exiv2 interface, with a huge collection of images. I already discovered plenty of non re-entrancy in Exiv2 API and i consolidate digiKam code with several lock, as we use multi-threading and multi-core everywhere.
I'm also aware about the MinGW port, as we cross compile ALL the digiKam code + all dependencies for Windows under Linux using MXE. I'm in love with cross compiling, so if something do not work here, i'm ready to report a fix quickly.
digiKam has a large users base, and we publish weekly a new test version for Linux, Windows, and MacOS. If something is broken or do not work in Exiv2, we are able to report the dysfunction quickly.
Voilà, I'm here to help you if necessary if time permit.
Mainly some random thoughts there, sorry
First of all, a very big "Thank you" to Robin and his work on behalf of Exiv2. I have always found him very responsive and helpful. And the same goes for all of the other maintainers, though it seems there are only two others. :( Having worked a good deal with other people's open source code with multiple dependencies on other similar open source libraries & trying to adapt them to my needs and environment, I understand a good many of the issues they have been facing. As for the future of Eviv2, I am not sure I can offer much help, other than a few comments. My own use of Exiv2 is simply for a couple of pet projects of mine. One, started several years ago to try to analyze and, hopefully, edit image metadata for my family tree project and a second one, which I forked form an existing and apparently abandoned or stalled project, much for a similar reason: collecting all of my images and documents for the family tree project This latter one had already been designed with Exiv2 as a basic and necessary component, but had not been using the latest version, nor was it using many of Exiv2's features as probably intended. The fact that it was using the Exiv2 libraries, was part of my decision to use it as a starting point. There is another, somewhat remotely connected, project, also part of my family tree project, namely the pyexiv?? code used as part of the Gramps project. If exiv2 were to 'stall', for my own projects, I would have to carry on as I have for others: grab the latest code and then work with it on my own and 'beat it into shape' as best I could with my needs, background, time and environment. Only recently have I started to use git, after a relatively long stable period of using SVN, though neither of these to anywhere near their full potential for serious development. Hence, using git and particularly Github and its 'cousins' by myself, requires a lot of effort and time because of their steep learning curve - both of which are always in short supply. As for keeping exiv2 going, I had been wondering to whom the project is or would be important enough to either sponsor the project or take on the job that Robin was doing. The current maintainers might be in a much better position to assess that than myself There definitely seems to be a move towards cloud, web or handheld device based applications, though I doubt that desktops will go away completely. There is no question that another stumbling block, at least for myself, was the fact that the basic roots of both Exiv2 itself and several of the libraries it depends on, are in the Linux tradition with a whole set of different compilers and debugging environments. Porting all of this to the Windows OS involves a tremendous amount of extra software and effort. Not to mention the somewhat erratic update and maintenance cycles and their port to Windows for the libraries Exiv2 depends on. On this issue, I must commend the Exiv2 team on their work to make the porting to Windows a whole lot more convenient with the conan and Cmake utilities. Still, even using those involves considerable effort and time on all sides. As a mostly Windows user, and without wanting to get into any flame wars, I very much appreciate that work by the team. Licensing is another issue, though one which I, as a single developer with no plans to commercialize my applications, am neither qualified not interested in commenting on. Still for some people this will be THE issue and for many it will be a show stopper. Thank you all. Arnold
I am mainly a user of exiv2 (coming from gwenview) and pretty new to the team; Robin invited me recently. I'm familiar with C++, Cmake, CI. However, I'm also part of two other organizations here at GitHub, which require most of my free time. But I'll try my best to help out where I can, just ping me.
Regarding how to attract new contributors, I would like to mention that the number of contributors is actually growing. So things don't look too bad to me. Simplifying the review process sounds good to me, @D4N has raised some good ideas. A rewrite IMO would be counterproductive for this project.
A couple of obvious points about v0.27.3 (and later v0.27 dots)
I wish I could make the fuzzing police disappear or get them to participate in fixing security issues. They appear to have more resources than Team Exiv2. I believe Kevin is the only "fuzzer" who both reported and fixed CVEs.
I ased regarding exvi2.dyndns.org because I received a patch to point all the docs to there. Which I will probably revert now.
Regarding the fuzzing: Unfortunately, fuzzing is quite use- and helpful. But, looking at eg. #1019 that ticket is basically useless.
Thanks @D4N for the effort of creating this issue and giving details about the current state of the project and proposing some short/long terms actions to try to improve the situation. I find useful to have this open discussion in a github thread instead of having it in a chat.
Please find my feedback about each of the proposed steps:
Short term steps
I guess that here you are referring here about the support we can give to the project given our busy lives. I do not see myself contributing in new features during the next months, due to my personal situation, and I agree that if I find time to contribute to the project, I will do it fixing reported bugs in
Nonetheless, I would be more than happy to see people contributing new features to master and I will try to also dedicate time to review those PRs and guide contributors as much as possible.
I think it should be possible to generate the binary artifacts for 0.27.3 from travisCI and Appveyor. I could dedicate some to this topic in the next days/weeks. I remember that long ago we discussed about some security issue in travisCI regarding the deployment of binary artifacts and that's we discarded that option. Opinions about this ?
I like this idea. I think it is good to centralise as much as possible all the content/discussions about Exiv2 to a single place.
I am not sure to what are you referring to here. Would you mind to develop a bit more this?
Long term steps
I think this is matching we some of the previous actions we have taken on master: i.e, to drop stuff not related with Image Metadata. Of course, any suggestions to keep improving the API would be welcomed.
Not sure either about how to find new contributors. Exiv2 has been for me the first open source project in which I participate regularly. Somehow, after working on it for several months I felt responsible of addressing bugs and improving some specific areas ... I guess that newcomers would be interesting in addressing more interesting topics, such as adding support for new formats.
I do not think it is realistic.
When I had lot of spare time to work on Exiv2 (in a period in which I was switching jobs) this is what I found more annoying about our process. For big changes, I think it is fine to establish some "grace period" for a review. But most of the times we are fixing small bugs or making small improvements here and there which should not be blocked for so long. From my point of view, anyone with admin rights in the Exiv2 group, should have freedom and the good judgement to merge something quickly when the changes are mostly cosmetic or when we are dealing with fixing something in CI (just the first examples that came to my mind). In case one of these persons abuse this power, we could analyse the behaviour and decide what to do.
Would it not be possible to move the permissions of Gitlab, TravisCI and Appveyor to the Exiv2 team instead of being me and @D4N the owners of these services? I will investigate with TravisCI and Appveyor (I was the one adding those services to Exiv2).
I do not think this is something bad per se. I particularly always appreciate the time people dedicate to review the code I am submitting, and if the suggestions made are good, I always try to address the challenges. I also know by experience that this is not the most common way to react to the code reviews, and some people gets are less open to change things (for whatever reason).
Something that I proposed in few companies where I have worked is to tag the comments with labels indicating the importance of the comment. Something like:
I would say that we could try to be a bit more indulgent in some cases (contributors do not have much spare time or have difficulties with Git), and approve PRs unless there are open comments tagged with [bug/leak/blocker]. Then, for the project maintainers it would take probably few minutes to address the other less important comments.
Code review is important to keep a high standard of code quality. It is important to ensure that there are no new bugs introduced and things don't break. Yes, it is not a task any developers wants to spent most of if times on. When I request a review for a new merge request I also have to review the code from someone else.
For fuzzing and also security related fixes it is important to have the process documented. You can use: https://www.libssh.org/development/security-process/
Write down which versions you support and communicate that. Example: https://wiki.samba.org/index.php/Samba_Release_Planning
This way you can always point people to your processes. The smaller the team the less versions you should have in maintenance mode.
I hope that helps.
Disclaimer: KDE dev and indirect user of Exiv2 ;)
Hello, another possibility for Exiv2 would be to be integrated into a bigger community (KDE, GNOME or Free Desktop) and use the infrastructure provided by this community.
In the case of KDE, this would allow the Exiv2 developers to use the server infrastructure (Jenkyll CI or gitlab CI, site hosting, release team, promo team, translations, sysadmin, ...). Joining a bigger community could also help because devs from other projects can help with reviewing merge request.
For KDE we have an incubating process, documented here: https://community.kde.org/Incubator. I don't think Exiv2 will have problem passing the incubation, since we are already using Exiv2 in a lot of place: KFileMetadata, Baloo, Dolphin's information panel, Digikam, Gwenview, and probably more :D
Hi KDE guys
Since it is quite widespread indirectly in GNOME, if you can guarantee to keep Qt out of it, KDE might be a good place, otherwise maybe freedesktop is probably better.
Edit: That wasn't meant to bash on Qt. It's more of a practical thing relating to Gtk and Qt mixing together being a bit weird on a technical level
thanks for all these comments!
A few comments to some replies:
Jenkins was/is building the releases of exiv2 that get published. It can surely be replicated with any other CI, we just haven't set this up yet.
That was my idea too, Rust is very tempting, but a rewrite is not realistic unless a bunch of experienced coders step up and drive this forward.
Currently licensing is a blocking that PR (and that it had absolutely zero tests last time I checked).
As a reaction to this, I've created a public chatroom on matrix.org:
Actually we are using GPLv2+, so that would make an integration with e.g. libheif a little simpler, but changing the license is completely off the table as we lack the necessary team of lawyers.
Thanks for the offer, however that will only defer the inevitable: we must be able to create a release without Jenkins. So I'd like to give this a try without Jenkins first, but keep it as a plan B.
Travis has had issues in the past with disabled package verification, meaning that adversaries could (in theory) manipulate your build environment.
Nevertheless, I would never trust a cloud provider as the only source of my binaries, unless they can be built reproducible and I can verify them locally. That's what I'd like to achieve: we built exiv2 inside a trusted container locally and on Travis/GitLab/Github and verify that the results are the same. Then Travis can host the artifacts and we are certain that they haven't been tampered with.
Afaik the website code and some additional tests still exist in svn repos somewhere.
Definitely. Large parts of the API have not been designed with RAII in mind and the io system needs imho a rewrite.
Newcommers like to work on easy bugs, on interesting topics and on clean and well documented code bases. I'm afraid we aren't really shining here.
I agree, if a pull request doesn't get reviewed for n-weeks, then a collaborator can merge it, provided there were no objections and they sent out a few reminders.
To be honest: I don't like this proposal. Merging a PR without a review should imho be the last resort and not something that you are allowed to do on a regular basis. If this is something really urgent (like a broken CI blocking 10 PRs), then so be it, but otherwise it just becomes too tempting to not wait for a review and just merge the PR "because I don't want to wait".
It's not really a problem of permissions (that's trivial to solve), but rather the general knowledge of the underlying platform. The CentOS CI for instance has been broken for a few weeks, because CentOS removed the
So I'd rather like to simplify and unify our CI setup, so that more team members can actually fix the CI.
This sounds like a very good idea!
I'd suggest that we instead fix the issues directly in the PR (usually contributors keep the "allow edits from maintainers" option selected) to prevent a huge backlog of "fix this minor thing" issues.
Sounds good, I'll add us there and to https://www.codetriage.com/ and will try to create a few more easy onboarding issues.
The issue at hand is that we often run into the situation that dev A has time to code but dev B has no time to review and so PRs are left rotting.
I've created #1035 to track progress the progress for this.
Yes, I was thinking about integrating exiv2 tighter into either the Gnome or KDE ecosystem. I've created #1036 to move the discussion there.
About possible Rust port of the library : for me it's a non-sense ! The C++ code base is enough mature, become really stable and C++11 and later come with real improvement with will be enough for the future of the project.
For me Rust is not the future so far... Try to rewrite this project with this language and you will be sure that the project will be forked immediately.
Well, GNOME did that for librsvg and the world didn't end for them. Also, I'd only start with the internals first, so that the API stays the same and you don't even realize that you are using Rust.
But let's not get into bikeshedding here, currently we completely lack the manpower for such an undertaking. On the other hand, I definitely wouldn't be opposed to introducing Rust into our code base.
Maintaining 2 lead code code base will be the hell in time. This will require manpower, and it's not the case for a small open source team. It's already difficult with one lead language, so with 2 it's surrealist.
And i'm not interested by Rust stuff... why to reinvent the wheel again and again with new language. it's a waste of time. Concentrate the effort to fix the bugs and advance the project, this is the plan.