import --fetch-jars #162

manics · 2020-01-08T12:44:42Z

Thought I'd see how difficult it is to automatically fetch the jars....

$ rm -rf ~/omero/jars/

$ omero import test.fake
Downloading https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip

... import as normal...

Closes #161

manics · 2020-01-09T10:02:53Z

~~--exclude~~

For 5.6.1

joshmoore · 2020-01-09T10:26:11Z

Two pretty self-unsimilar options come to mind:

a more maven-like strategy (see jgo)
doing this without a flag (but perhaps a warning) to a temporary location; a flag could make it permanent: import --fetch-jars=$LOCATION

manics · 2020-01-09T13:14:27Z

OMERO.java is a 125 MB download, I don't think we should download that automatically

joshmoore · 2020-01-09T13:15:56Z

I'd bet most users would prefer for us to just do it, though maybe not repeatedly.

manics · 2020-03-10T10:44:33Z

Hard-coding the OMERO.java version could potentially be avoided by fetching the server version:

        c = self.ctx.conn(args)
        version = c.sf.getConfigService().getVersion()

Though this assumes OMERO.java version == OMERO.server version. It also requires the logic from https://github.com/ome/omego/blob/v0.7.0/omego/artifacts.py#L391 to figure out the build number.

joshmoore · 2020-03-13T14:55:25Z

cc: @sbesson

I'm still for having this somewhat transport rather than a solely being a new argument that needs calling.

manics · 2020-06-03T15:42:53Z

To make this more maintainable we could:

Move the omego artifacts plugin to a new omero cli plugin e.g. package-download
Use that plugin/library to download the OMERO client jars for the latest version (or perhaps the latest major version matching the OMERO-py version?)

manics · 2020-06-30T16:12:57Z

As requested I've made it automatically download the jars if required, the latest https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip will be fetched.

If you want to force a re-download, update or download a different version of the jars pass a version to the --fetch-jars argument:

$ rm -rf ~/omero/jars

$ omero import test.fake
Downloading https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip

<import as normal>

$ cat ~/omero/jars/OMERO.java.txt
OMERO.java-5.6.1-ice36-b225

$ omero import --fetch-jars 5.5
Downloading https://downloads.openmicroscopy.org/omero/5.5/OMERO.java.zip

$ cat ~/omero/jars/OMERO.java.txt
OMERO.java-5.5.1-ice36-b122

$ omero import --fetch-jars latest
Downloading https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip

$ cat ~/omero/jars/OMERO.java.txt
OMERO.java-5.6.1-ice36-b225

The available version strings are whatever https://downloads.openmicroscopy.org/omero/{version}/OMERO.java.zip redirects are in place on our downloads server.

manics · 2020-06-30T16:34:15Z

(Forgot to push the last commit before commenting)

src/omero/plugins/import.py

joshmoore

Thanks for the auto-download 👍 The re-fetching based on versions also looks good. Has a jenv/rbenv vibe to it. Only other thought I had was whether or not the choice could be via symlink, but that would be less fun on Windows, I assume.

src/omero/plugins/import.py

joshmoore · 2020-07-01T10:03:02Z

src/omero/plugins/import.py

@@ -517,6 +542,43 @@ def importer(self, args):
        else:
            self.do_import(command_args, xargs)

+    def _userdir_jars(self, parentonly=False):
+        user_jars = get_omero_userdir() / 'jars'


No objections to this unless we foresee moving away from ~/omero soonish. If so, then perhaps something like user_cache_dir()/omero would be safer (so people don't have to re-download).

It's a fair point, but I think having two different dirs for storing user state is worse than re-downloading. If there's a concrete suggestion for the location of user_cache_dir() shouldn't we modify get_omero_userdir() instead?

My proposal for moving forward would be to place an omero subdirectories into existing directories like .config, .cache. If that's controversial at all, let's not block this PR.

manics · 2020-07-01T13:47:30Z

I had to mess around with the Ice build due to openssl issues. What I think is happening is that https://github.com/ome/zeroc-ice-py-manylinux/releases/download/0.2.0/zeroc_ice-3.6.5-cp36-cp36m-manylinux2010_x86_64.whl included a bundled openssl library which is somehow incompatible with Travis python.

https://github.com/ome/zeroc-ice-ubuntu1804/releases/download/0.3.0/zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl doesn't work, presumably because Travis Python 3.6 is different from the Ubuntu system Python 3.6.

A possible way around this is to build wheels specific to Travis, but for now I've changed this to use the default source package.

Note the reason https requests are being made is that although most tests in https://github.com/ome/omero-py/blob/master/test/unit/clitest/test_import.py that require the jars are disabled with @pytest.mark.skipif(OMERODIR is False, reason="Needs client dir") some of them aren't marked:

omero-py/test/unit/clitest/test_import.py

Line 38 in 41f82a8

help_arguments = ("-h", "--javahelp", "--java-help", "--advanced-help")

and are instead allowed to return a non-zero exit code if the jars are missing:

omero-py/test/unit/clitest/test_import.py

Lines 134 to 138 in 41f82a8

    
           @pytest.mark.parametrize('help_argument', help_arguments) 
        
           def testHelp(self, help_argument): 
        
               """Test help arguments""" 
        
               self.args += [help_argument] 
        
               self.cli.invoke(self.args)

$ omero import --advanced-help
Error: Could not find or load main class ome.formats.importer.cli.CommandLineImporter

With this PR the jars are now automatically downloaded.

If you're wondering whether the other import tests marked @pytest.mark.skipif(OMERODIR is False, reason="Needs client dir") should now pass, some of them do, but some of them are broken, suggesting we never run these unit tests with OMERODIR defined.

manics · 2020-07-01T14:49:52Z

See #231 (which includes the commits from this PR) for the fixed import tests. I can merge that PR into this one if you think it's appropriate.

joshmoore · 2020-07-02T09:22:42Z

I can merge that PR into this one if you think it's appropriate.

👍

Only other comment at this stage is whether or not omero import could print out the used version.

For windows compatiblity a text file is used on all platforms in lieu of a symlink

Currently this is the same as get_omero_userdir but this will change soon as part of ome#210 (comment)

manics · 2020-07-02T10:16:56Z

I've:

rebased
added the 4 commits from Activate test_import unit tests with OMERO_USERDIR #231
added omero.util.get_omero_user_cache_dir. Currently this is the same as get_omero_userdir but will change with OMERO CLI fails if a file ~/omero exists #210 (comment). Since it's quite a big change I don't think we should do that in this PR.

joshmoore

Looking good. Two small code issues. The only other thing I notice is that when using the downloaded files that debugging is skewed since logback-cli.xml is missing (perhaps related to some of your test issues?)

Options I can think of:

move logback-cli.xml to one of the jars
download it separately
bundle it into python (?!)

joshmoore · 2020-07-02T10:27:37Z

src/omero/plugins/import.py

+        if not jars_dir.exists():
+            jars_dir.mkdir()
+        with urlopen(omero_java_zip) as resp:
+            with ZipFile(BytesIO(resp.read())) as zipfile:


It may fall more under #217, but some feedback of the download progress at least in an interactive terminal will probably prevent consternation.

I could look into TQDM https://github.com/tqdm/tqdm

But that's another dependency- should we split omero-cli out of omero-py to avoid installing UI dependencies before going further?

Yeah, I imagine that's one we put on the backburner.

src/omero/plugins/import.py

Print out OMERO.java version if using auto-downloaded jars

manics · 2020-07-02T14:17:34Z

logback-cli.xml: I think it would make sense for it to be part of the OMERO.java zip, I don't know whether it's better to be inside a jar or outside. Since this PR now distinguishes between the downloaded OMERO.java and the default lib/client it should be easy to adjust the logback path depending on where the jars are found.

joshmoore · 2020-07-02T15:07:30Z

Ok. So if we update https://github.com/ome/openmicroscopy/blob/develop/build.xml#L302 we'll get logback-cli.xml into that directory and things should just magically start working... (Worth getting that location set now?)

Otherwise, 👍

manics · 2020-07-02T16:24:23Z

After inserting an extra print to check:

$ git diff
diff --git a/src/omero/plugins/import.py b/src/omero/plugins/import.py
index d23312c1..c890f928 100644
--- a/src/omero/plugins/import.py
+++ b/src/omero/plugins/import.py
@@ -530,6 +530,7 @@ class ImportControl(BaseControl):
                     103, "No JAR files found under '%s'" % client_dir)

         logback = "-Dlogback.configurationFile=%s" % xml_file
+        print(logback)
         return classpath, logback

     def importer(self, args):

$ OMERO_USERDIR=/tmp/xx omero import --advanced-help
Using OMERO.java-5.6.1-ice36-b225
-Dlogback.configurationFile=/tmp/xx/jars/OMERO.java-5.6.1-ice36-b225/libs/logback-cli.xml
...

$ OMERODIR=/openmicroscopy/dist/ omero import --advanced-help
-Dlogback.configurationFile=/openmicroscopy/dist/etc/logback-cli.xml
...

sbesson

Tested using a fresh omero-py virtual environment

$ venv/bin/omero import test.fake -f
Downloading https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip
Using OMERO.java-5.6.1-ice36-b225
...
$ rm -rf ~/omero/jars/
$ venv/bin/omero import test.fake -f
Downloading https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip
Using OMERO.java-5.6.1-ice36-b225
...
$ rm -rf ~/omero/jars/
$ venv/bin/omero import --fetch-jars 5.6
Downloading https://downloads.openmicroscopy.org/omero/5.6/OMERO.java.zip
$ venv/bin/omero import test.fake -f
Using OMERO.java-5.6.1-ice36-b225
...
$ venv/bin/omero import --fetch-jars 5.4
Downloading https://downloads.openmicroscopy.org/omero/5.4/OMERO.java.zip
$ venv/bin/omero import test.fake -f
Using OMERO.java-5.4.10-ice36-b105
...

The feature is very useful and the newly introduced API makes sense.

Are we considering a release omero-py 5.8.0 with this PR included? Shoudl this coordinated with the upcoming release of OMERO.java including the logging configuration file?

src/omero/plugins/import.py

Co-authored-by: Sébastien Besson <seb.besson@gmail.com>

joshmoore · 2020-07-09T12:13:53Z

Thanks again, @manics. I think this is going to make things much nicer for people since the move to OMERODIR. Merging.

(I'd add that as it gets usage there could well be feature requests for the likes of --list-jars, --del-jars=$OLD_VERSION, etc. which may even be best-placed in its own plugin.)

sbesson · 2020-07-09T18:29:56Z

Should we release the current state of master as omero-py 5.8.0 or would you like to include #233 ?

manics · 2020-07-09T19:12:53Z

It'd be nice to include #233, but that may need futher discussions, so I wouldn't say it's a blocker. Feel free to push to it if you want.

PRs merged since 5.7.1:

Using https://gist.github.com/manics/383b4481e1c3a193d224016b01a5d52a
GH_TOKEN=$(git config github.token) GH_REPO=omero-py ../generate_release_notes/generate_release_notes.py v5.7.1 master --version v5.8.0

import --fetch-jars (#162)
fix logging attribute error (#226)
Add PyYAML as a dependency of omero-py (#228)
bin/omero entrypoint (#229)
hql.py: when querying masks, filter bytes field (#230)

#226 is the only bug fix so if that's not urgent I think we can wait a bit.

joshmoore · 2020-09-25T07:33:23Z

src/omero/plugins/import.py

+            return None, omero_java_txt
+
+    def download_omero_java(self, version):
+        omero_java_zip = OMERO_JAVA_ZIP.format(version=version)


It occurs to me for the testing of ome/openmicroscopy#6255 that version should likely have been used directly if it started with http(s)://

joshmoore force-pushed the master branch from 3568210 to 6842ee6 Compare January 10, 2020 13:17

joshmoore mentioned this pull request Mar 10, 2020

scripts: add test for using omero.cli.CLI ome/openmicroscopy#6222

Merged

manics force-pushed the import-fetch-jars branch from 546b2d8 to 0369202 Compare May 20, 2020 13:49

manics force-pushed the import-fetch-jars branch 2 times, most recently from cbe1ca8 to d988b3f Compare June 30, 2020 08:16

jburel reviewed Jul 1, 2020

View reviewed changes

src/omero/plugins/import.py Outdated Show resolved Hide resolved

jburel reviewed Jul 1, 2020

View reviewed changes

src/omero/plugins/import.py Outdated Show resolved Hide resolved

jburel reviewed Jul 1, 2020

View reviewed changes

src/omero/plugins/import.py Outdated Show resolved Hide resolved

joshmoore reviewed Jul 1, 2020

View reviewed changes

manics mentioned this pull request Jul 1, 2020

Activate test_import unit tests with OMERO_USERDIR #231

Closed

manics added 8 commits July 2, 2020 10:58

import --fetch-jars

edaecdf

Use full path in OMERO.java to support multiple versions

0206ccd

Update to OMERO.java-5.6.1-ice36-b225.zip

a0129a0

Use omero/latest/OMERO.java.zip

83a1715

For windows compatiblity a text file is used on all platforms in lieu of a symlink

import --fetch-jars requires a version, can be latest

2386ba0

Automatically download

173a908

import jars: ensure resources are closed

761405e

Refactor classpath check into separate method

c9ab8e3

manics added 6 commits July 2, 2020 10:58

Don't use precompiled wheel for ice

4908ea1

test_import.py: use a temporary OMERO_USERDIR per session

a155a1d

Remove @pytest.mark.skipif(OMERODIR is False

34fa96d

Fix broken import tests

c29b0ee

Disable import tests on Windows

f713d6c

import jars: use get_omero_user_cache_dir

4c68c92

Currently this is the same as get_omero_userdir but this will change soon as part of ome#210 (comment)

manics force-pushed the import-fetch-jars branch from 84fc178 to 4c68c92 Compare July 2, 2020 10:14

joshmoore reviewed Jul 2, 2020

View reviewed changes

If --clientdir or lib/client provided don't auto-download

0245d33

Print out OMERO.java version if using auto-downloaded jars

If downloaded jars used assume logback.xml is next to jars

afc271e

sbesson reviewed Jul 5, 2020

View reviewed changes

src/omero/plugins/import.py Outdated Show resolved Hide resolved

src/omero/plugins/import.py Outdated Show resolved Hide resolved

src/omero/plugins/import.py Show resolved Hide resolved

Apply suggestions from code review

4ae4d25

Co-authored-by: Sébastien Besson <seb.besson@gmail.com>

manics closed this Jul 8, 2020

manics reopened this Jul 8, 2020

manics mentioned this pull request Jul 9, 2020

Tox: Python 3.8, use travis bionic ice-py wheels #232

Merged

1 task

joshmoore merged commit 36b3d6c into ome:master Jul 9, 2020

manics deleted the import-fetch-jars branch July 9, 2020 13:19

manics mentioned this pull request Jul 9, 2020

Replace ~/omero with standard OS-dependent paths #233

Closed

joshmoore mentioned this pull request Aug 11, 2020

Use doConnectionCleanup as statement of intent will-moore/omero-web#1

Merged

manics mentioned this pull request Aug 27, 2020

Changelog notes for 5.8.0 #248

Merged

3 tasks

This was referenced Sep 24, 2020

omero-py>=5.8.0: import automatically downloads JARs ome/omero-documentation#2153

Merged

Include logback-cli.xml in OMERO.java.zip ome/openmicroscopy#6255

Merged

joshmoore reviewed Sep 25, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

import --fetch-jars #162

import --fetch-jars #162

manics commented Jan 8, 2020 •

edited

Loading

manics commented Jan 9, 2020 •

edited

Loading

joshmoore commented Jan 9, 2020

manics commented Jan 9, 2020 •

edited

Loading

joshmoore commented Jan 9, 2020

manics commented Mar 10, 2020

joshmoore commented Mar 13, 2020

manics commented Jun 3, 2020 •

edited

Loading

manics commented Jun 30, 2020 •

edited

Loading

manics commented Jun 30, 2020

joshmoore left a comment

joshmoore Jul 1, 2020

manics Jul 1, 2020

joshmoore Jul 2, 2020

manics commented Jul 1, 2020 •

edited

Loading

manics commented Jul 1, 2020 •

edited

Loading

joshmoore commented Jul 2, 2020

manics commented Jul 2, 2020 •

edited

Loading

joshmoore left a comment

joshmoore Jul 2, 2020

manics Jul 2, 2020 •

edited

Loading

joshmoore Jul 2, 2020

manics commented Jul 2, 2020 •

edited

Loading

joshmoore commented Jul 2, 2020

manics commented Jul 2, 2020

sbesson left a comment

joshmoore commented Jul 9, 2020

sbesson commented Jul 9, 2020

manics commented Jul 9, 2020

joshmoore Sep 25, 2020

joshmoore Sep 7, 2021

import --fetch-jars #162

import --fetch-jars #162

Conversation

manics commented Jan 8, 2020 • edited Loading

manics commented Jan 9, 2020 • edited Loading

joshmoore commented Jan 9, 2020

manics commented Jan 9, 2020 • edited Loading

joshmoore commented Jan 9, 2020

manics commented Mar 10, 2020

joshmoore commented Mar 13, 2020

manics commented Jun 3, 2020 • edited Loading

manics commented Jun 30, 2020 • edited Loading

manics commented Jun 30, 2020

joshmoore left a comment

Choose a reason for hiding this comment

joshmoore Jul 1, 2020

Choose a reason for hiding this comment

manics Jul 1, 2020

Choose a reason for hiding this comment

joshmoore Jul 2, 2020

Choose a reason for hiding this comment

manics commented Jul 1, 2020 • edited Loading

manics commented Jul 1, 2020 • edited Loading

joshmoore commented Jul 2, 2020

manics commented Jul 2, 2020 • edited Loading

joshmoore left a comment

Choose a reason for hiding this comment

joshmoore Jul 2, 2020

Choose a reason for hiding this comment

manics Jul 2, 2020 • edited Loading

Choose a reason for hiding this comment

joshmoore Jul 2, 2020

Choose a reason for hiding this comment

manics commented Jul 2, 2020 • edited Loading

joshmoore commented Jul 2, 2020

manics commented Jul 2, 2020

sbesson left a comment

Choose a reason for hiding this comment

joshmoore commented Jul 9, 2020

sbesson commented Jul 9, 2020

manics commented Jul 9, 2020

PRs merged since 5.7.1:

joshmoore Sep 25, 2020

Choose a reason for hiding this comment

joshmoore Sep 7, 2021

Choose a reason for hiding this comment

manics commented Jan 8, 2020 •

edited

Loading

manics commented Jan 9, 2020 •

edited

Loading

manics commented Jan 9, 2020 •

edited

Loading

manics commented Jun 3, 2020 •

edited

Loading

manics commented Jun 30, 2020 •

edited

Loading

manics commented Jul 1, 2020 •

edited

Loading

manics commented Jul 1, 2020 •

edited

Loading

manics commented Jul 2, 2020 •

edited

Loading

manics Jul 2, 2020 •

edited

Loading

manics commented Jul 2, 2020 •

edited

Loading