Dependency relations between picard-tools and htsjdk

[vdanjean]

Hi,

I'm one of the packagers of picard-tools and htsjdk in Debian. I would like to know what you (as upstream) expect as relation between these two software.

Currently, in Debian, picard-tools 2.7.1 is packaged but htslib is still at 2.6.1. Is it a combination that is expected to work?

If not, is there a rule? For example picard X.Y.Z should work with htslib X.Y.T (ie same X.Y, the last number would be for bug fix without interface changes)
Or picard X.Y.Z should work with htslib U.V.W with U.V >= X.Y
Or picard X.Y.Z should work with htslib X.V.W with V >= Y (ie htslib keeps ascending compatibility when X do not change)
Or ... ?

Knowing this would allows us to avoid to forget to update htsjdk when this is required. Our build process will force us to respect your conventions.

Regards,
Vincent

[vdauwera]

(I'm assuming you meant htsjdk throughout and not htslib, which you switch to partway through, but correct me if I'm wrong.)

There used to be a direct relation between the version number of picard and of htsjdk, but we've decoupled them so that is no longer the case.

@lbergelson Is there anything else of relevance to say?

[vdanjean]

(Sorry for the confusion between htsjdk and htslib. You are right, I want to talk about htsjdk)

Given a version number for picard, how can we know the minimum version number required for htsjdk?
And given a version number for htsjdk, how can we know the maximum supported version number of picard? (ie how can we know which old versions of picard a new release of htsjdk will break?)

In the classical ELF 'C' world, the SONAME of the library and the symbol analysis help to know these information. In the Java world, there is no such automatic information and we must rely on documentation provided by upstream authors.

[vdauwera]

The Picard build script build.gradle specifies which version of htsjdk should be used in this line:

final htsjdkVersion = System.getProperty('htsjdk.version', 'VERSION')`

where VERSION stands for the version number, as documented in the README.

No other version of htsjdk should be expected to work with Picard than the one specified in build.gradle, unless you're in the process of developing and changing the version of htsjdk intentionally.

I believe the standard build command will retrieve the correct version of htsjdk as a dependency, so there should be no need to fetch any particular version of htsjdk separately.

[vdanjean]

Thank. I missed this information as, in Debian, we do not allow any external download during the build (maven is setup in "local" mode). I will adjust our scripts to ensure that htsjdk and picard have compatible versions.
Note that, from my point of view, having a so tight relationship defect a little bit the separation done between htsjdk and picard. Nevertheless, many thank for this software.

Regards,
Vincent

[lbergelson]

@vdanjean What said @vdauwera said is correct.

You can also see the exact dependencies needed if you run ./gradlew dependencies. That's probably not helpful for your offline only builds though.

We've been talking about switching to semantic versioning for htsjdk so that it's clear what the backwards compatibility situation is. Unfortunately, since htsjdk never made a very clear distinction between it's public API and internal details, it's hard to make any changes to htsjdk without potentially breaking backwards compatibility. Since the API surface is so large, it means that almost every release ends up breaking backwards compatibility in some way. Mostly the breaks are in portions of the api that won't effect most users, but it's very hard to say if that's the case for specific software packages or not. So almost every htsjdk release ends up being technically backwards incompatible.

Like you, we wish the versions were more compatible, and we're trying to improve the situation, but it's not trivial when we regularly need to make pretty invasive changes to a legacy library.