ImageJ-1.51k plugins, URLs, LICENSE

[mmokrejs]

Dear Wayne,
I would like to prepare a package of ImageJ for Gentoo Linux but It is difficult to find the source code of plugins. There used to exist http://rsb.info.nih.gov/ij/download/zips/ but there is no file for 1.50 release. Further, the listing at http://imagej.net/download/zips/ is even more outdated.

It appears I can get the plugins from http://wsr.imagej.net/distros/cross-platform/ files. Is that the right place?

I have system-wide installed jython. Can the code in the below files respect my system-wide jython installation instead of re-fetching its own copy? (sorry, I am a bit lazy now to read the relevant code to see what it is actually needed for)
plugin/frame/Editor.java
plugin/PlugInInterpreter.java
plugin/Macro_Runner.java

I think the page http://imagej.net/download.html lacks a link where to downloadable separate plugin zip files for a given imagej1 version.

Similarly, the page http://imagej.net/plugins/index.html lacks a Download section/pointer.

The page https://imagej.nih.gov/ij/source/ seems to contain a 1.51j version only and the plugins/ subdirectory seems incomplete. Or containing just core plugins?

It would be helpful if the *.zip files contained a plaintext file called LICENSE with the actual license text. Reading https://imagej.net/Licensing does not ensure me ImageJ-1.51j is actually https://choosealicense.com/licenses/bsd-2-clause/ .

Thank you for your help,
Martin

[ctrueden]

ImageJ 1.x is a single JAR file—the plugins are not separate. As far as I know, all source code for ImageJ1 is available from the same Git repository: this one. If you have a counter-example—something which comes with the ImageJ1 distribution, but is not present in this Git repository, feel free to ask about it.

ImageJ2 consists of several JAR files and dependencies, permissively licensed.

Reading https://imagej.net/Licensing does not ensure me ImageJ-1.51j is actually https://choosealicense.com/licenses/bsd-2-clause/ .

ImageJ1—including its built-in plugins—is a public domain project, not BSD. The ImageJ 1.x website has a dedicated copyright disclaimer page. The page https://imagej.net/Licensing describes the license of all components of the SciJava core ecosystem, including ImageJ 1.x, ImageJ2, and more.

Are you looking to package ImageJ 1.x, or ImageJ2? Note that there is already a (third-party) packaging of ImageJ1 for Debian-based systems. It would be awesome to create an ImageJ2 packaging for various flavors of Linux which is well-automated.

[mmokrejs]

First of all, I started with https://bugs.gentoo.org/show_bug.cgi?id=112275 and pushed an ebuild into science overlay: gentoo-mirror/science@395c7e7 . Gentoo compiles everything from sources. It is awkward we have to fetch two zip files and use a portion of each to build ther jar file.

The file in http://wsr.imagej.net/distros/cross-platform/ij150.zip unpacks lots of files into ImageJ/plugins/. On line 109 gentoo-mirror/science@395c7e7#diff-0bd1b5ab55ea5c12014997754f007fc8R109 we copy them to the installation image. Contributions would be welcome through https://github.com/gentoo-science/sci once the imagej-1.51k.ebuild package appears there.

Gentoo does not support officially maven and hence gradle, only Apache ant. I doubt I will bother to make a package for imageJ2 therefore. I had a lot of issues with getting jtreeview, picard, gatk, OWLTools, jannovar, SnpEff to build (and they will never make it into the official tree because of the problems mention further below). The problem is that they fetch lots of pre-compiled jars during compile phase over the network and the stuff gets discarded because Gentoo uses a sandbox to compile packages. So on every attempt to make a package lots of the crap is fetched. More of an issue is that the fetched jar's won't execute on some architectures. Simply, it is a wrong approach.

As far as I remember Debian converts our ebuilds into their packages automagically (or at least Andrey Kysliuk weaver@g.o contributed to Debian some conversion tool like that in the past).

I am a bit reluctanct to inspect in more depth the ImageJ2 and Fiji bundles as it seem I would first have to make new packages for the required java sub-components, then delete from the ImageJ2 and Fiji "sources" their bundled copies, fix the build system and the eventually get the ImageJ2-specific jar file out.

[ctrueden]

I would first have to make new packages for the required java sub-components, then delete from the ImageJ2 and Fiji "sources" their bundled copies, fix the build system and the eventually get the ImageJ2-specific jar file out.

The build system is modular; each component depends on JAR artifacts of lower-level components. Building all of the ImageJ2 components is easy from the bottom up—you can even build all of SciJava+ImageJ2 in "one shot" by restructuring it as a huge multi-module Maven build—although that is probably not the best way to go here. (We actually have a script that does that, although I still need to hook it up to CI to run regularly.)

The trickier part would be building the binary dependencies which are not part of the SciJava ecosystem, since they are structured heterogeneously. So if by "make new packages for the required java sub-components" you mean the non-SciJava dependencies, then yeah, that part is significant work. But I do not understand your comment "delete from the ImageJ2 and Fiji 'sources' their bundled copies". No JAR files are committed to SCM, and no "uber-JARs" (aggregate JAR files consisting of .classes from multiple components) are normally produced.

The other tricky part, I guess, is that all Gentoo builds of these things should be named differently, probably with -gentoo version suffix. Is that how Java projects are typically shipped with Gentoo? Otherwise, you create an artifact which purports to be, e.g., imagej-2.0.0-rc-59.jar, but is actually a Gentoo-specific build of it, with a different JAR manifest than the one originally built and deployed to Maven repositories.

I had a lot of issues with getting jtreeview, picard, gatk, OWLTools, jannovar, SnpEff to build (and they will never make it into the official tree because of the problems mention further below). The problem is that they fetch lots of pre-compiled jars during compile phase over the network and the stuff gets discarded because Gentoo uses a sandbox to compile packages. So on every attempt to make a package lots of the crap is fetched.

Fortunately, ImageJ2 does not use any of those packages. I think we could get around this problem in general by building from the "bottom up" of the dependency chain, installing each compiled binary JAR into the local Maven repository cache. If you do this, then dependencies are not fetched from Maven remote repositories, because they are already present in the cache. And maybe we could do a mass rename of all the artifacts at the end.

More of an issue is that the fetched jar's won't execute on some architectures.

I know that in the case of ImageJ2, we have no platform-specific native binaries in the core (apart from the ImageJ Launcher, which is unnecessary, and slated for demolition), and the software runs on the three "big" platforms (Windows, macOS, Linux) including Raspberry Pis. Are there other architectures of concern? What sorts of problems did you encounter?

Contributions would be welcome through https://github.com/gentoo-science/sci once the imagej-1.51k.ebuild package appears there.

Cool, thanks for the link. I filed an issue: #162. It sounds like a fun project and I wish I had the bandwidth to work on it, but at least this year, I do not. 😞

[ctrueden]

Back to the issue at hand here, of packaging the ImageJ 1.x plugins: sorry I did not look carefully at the ImageJ 1.x distribution to verify whether any external code is present in the plugins/ folder. I guess you will need to hear from @rasband more specifically: where that code comes from, how it gets built, etc.

Further discussion on the ImageJ2 side can take place in that new issue I filed (#162).

[mmokrejs]

The build system is modular; each component depends on JAR artifacts of lower-level components. Building all of the ImageJ2 components is easy from the bottom up—you can even build all of SciJava+ImageJ2 in "one shot" by restructuring it as a huge multi-module Maven build—although that is probably not the best way to go here. (We actually have a script that does that, although I still need to hook it up to CI to run regularly.)

The package should be as small as possible.

So if by "make new packages for the required java sub-components" you mean the non-SciJava dependencies, then yeah, that part is significant work ...

Yes, one example from the past could be attempts to get http://www.cytoscape.org/ into Gentoo. It had hundreds of dependencies. It is doable. Maven-based projects are more difficult. They fetch their own stuff, so it is not about making hundreds of packages before one can make the final top-most package for the application itself, but the difficult work is about overcoming maven's evil behavior (e.g. you can you want to compile current sources which you unpacked from a tar.bz file and maven just running wget to fetch some other stuff). That is not acceptable. During a compile step no downloads are allowed. Not a single bit over a network.

You can search for 'gentoo cytoscape' to get a feeling how many packlages need to be created at first. I see there is somebody still working on it although I am not familiar with current status of the package preparation: https://summerofcode.withgoogle.com/archive/2016/projects/4870044894887936/

Are there other architectures of concern? What sorts of problems did you encounter?

Gentoo runs on much more, see the architectures at https://www.gentoo.org/downloads/ . It runs on even more. I run it on several clusters which are old redHat-based with way too old glibc (I overlaid the Redhat with a self-compiled 'gcc -mtune=native' binaries, even my glibc-2.23, see https://wiki.gentoo.org/wiki/Prefix/libc and https://wiki.gentoo.org/wiki/Project:Prefix for the details how you can get it installed on other UNIXes).

The other tricky part, I guess, is that all Gentoo builds of these things should be named differently, probably with -gentoo version suffix. Is that how Java projects are typically shipped with Gentoo? Otherwise, you create an artifact which purports to be, e.g., imagej-2.0.0-rc-59.jar, but is actually a Gentoo-specific build of it, with a different JAR manifest than the one originally built and deployed to Maven repositories.

Maybe you have a good point here but as far as I know the jars do not have -gentoo substring in their names. Look into some ebuild files or even, install a Gentoo into a virtual box, or even, take the Gentoo::Prefix or Gentoo::RAP approach.

... installing each compiled binary JAR into the local Maven repository cache. If you do this, then dependencies are not fetched from Maven remote repositories, because they are already present in the cache.

I fear there is no way for the "cache" to survive cleanup of the sandbox after a package is unpacked/configured/compile/installed/merged_into_live_system. Actually nobody will want that to be possible.

java 'herd' at Gentoo is understuffed, feel free to poke https://gentoo.org/get-involved/become-developer/ and https://wiki.gentoo.org/wiki/Project:Java .

[ctrueden]

Maven-based projects are more difficult. They fetch their own stuff, so it is not about making hundreds of packages before one can make the final top-most package for the application itself, but the difficult work is about overcoming maven's evil behavior (e.g. you can you want to compile current sources which you unpacked from a tar.bz file and maven just running wget to fetch some other stuff). That is not acceptable. During a compile step no downloads are allowed. Not a single bit over a network.

I understand. But I am telling you: Maven's behavior here is not evil. It only fetches dependencies which are not already available locally. So if you build all the dependencies first, then it does not fetch even a single bit of those dependencies from anywhere else. The only caveat to that is that it does fetch plugins it uses for the build, which of course it needs to bootstrap the first time you run that build. But those bits are not part of the final packages in any way, shape or form.

Maven also has a CLI flag -o which tells it not to fetch things remotely.

Happy to elaborate and discuss further as needed, although again: let's discuss this stuff on #162.

java 'herd' at Gentoo is understuffed, feel free to poke https://gentoo.org/get-involved/become-developer/ and https://wiki.gentoo.org/wiki/Project:Java .

Thank you for the links; I wish I had time for it! ☹️