Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download releases and non-OMERO artifacts #66

Merged
merged 22 commits into from
Aug 3, 2015

Conversation

manics
Copy link
Member

@manics manics commented Jul 23, 2015

Adds --release as an alias for --branch, and adds support for parsing the downloads release pages. Also adds all artifacts matching 'OMERO\.(\w+).*\.zip$' in addition to the hard-coded ones.

Follow a latest redirect, then parse the downloads page:

  • omego download matlab -v --branch 5.1
  • omego download matlab -v --release 5.1
  • omego download matlab -v --release 5
  • omego download matlab -v --release 4
  • omego download matlab -v --release latest

Request a specific version:

  • omego download matlab -v --release 4.4.8
  • omego download matlab -v --release 5.1.3-m3

Request an artifact which is not explicitly supported in omego:

  • omego download java

Originally I was just downloading the zips with explicit redirects (e.g. .../omero/server.zip, see 0c6a111) but after implementing a downloads page html parser I realised they're no longer necessary since it's enough to follow the redirect to the page.

Note release artifacts are only supported for the latest ice at present.

Updated:
More artifacts can be downloaded, see https://github.com/manics/omego/blob/17bb9e0c961f2f3aef13550a06c740cd952bcd05/omego/artifacts.py#L98 for the specification.

The following should all be equivalent:

  • omego download --release 5 --skipunzip python
  • omego download --release 5 --skipunzip py
  • omego download --release 5 --skipunzip OMERO.py

Partial initial matches are supported in some cases, e.g.:

  • omego download --release 5 --skipunzip se will currently download the server since there are no other matching strings.

Omit the artifact name to list all available:

  • omego download --release 5

Non-OMERO jobs are supported:

  • omego download --branch BIOFORMATS-5.1-merge-build (list artifacts)
  • omego download --branch BIOFORMATS-5.1-merge-build bftools
  • omego download --branch BIOFORMATS-5.1-merge-build bio (partial match, downloads the source)

Jars can be downloaded:

  • omego download --branch BIOFORMATS-5.1-merge-build ij should download ij.jar

As a last resort you can also specify the full filename in the event that the shorter names conflict, or you want to download a non-zip non-jar, e.g.

  • omego download --branch OMERO-5.1-latest GIT_INFO

Example output listing all zips and jars, with the shortnames which can be truncated

$ omego download
Artifacts available for download. Initial partial matching is supported for all except named-components). Alternatively a full filename can be specified to download any artifact, including those not listed.
named-components:
  linux
  mac
  mac6
  matlab
  python
  server
  win
omerozips:
  docs-5.1.3-ice35-b52
  insight-5.1.3-ice35-b52-linux
  insight-5.1.3-ice35-b52-mac_Java6
  insight-5.1.3-ice35-b52-mac_Java7+
  insight-5.1.3-ice35-b52-win
  insight-ij-5.1.3-ice35-b52
  java-5.1.3-ice35-b52
  matlab-5.1.3-ice35-b52
  py-5.1.3-ice35-b52  
  server-5.1.3-ice35-b52
zips:
  OMERO.docs-5.1.3-ice35-b52
  OMERO.insight-5.1.3-ice35-b52-linux
  OMERO.insight-5.1.3-ice35-b52-mac_Java6
  OMERO.insight-5.1.3-ice35-b52-mac_Java7+
  OMERO.insight-5.1.3-ice35-b52-win
  OMERO.insight-ij-5.1.3-ice35-b52
  OMERO.java-5.1.3-ice35-b52
  OMERO.matlab-5.1.3-ice35-b52
  OMERO.py-5.1.3-ice35-b52
  OMERO.server-5.1.3-ice35-b52

Automatically figure out whether the argument is a jobname or release number
Since ReleaseArtifacts parses the downloads pages there's no need to use the individual latest component.zip redirects, instead we can just deference the redirect to the main release page and parse that instead.
@joshmoore
Copy link
Member

Pre-testing, a big 👍 for the general scheme! Default behavior of --release latest?

This is mostly for internal use so its possible to use any of the following without needing additional logic (without this commit the direct server.zip url has to be passed as a separate positional argument):

- omego upgrade --branch OMERO-5.1-latest
- omego upgrade --branch 5.1
- omego upgrade --branch http://example.org/OMERO.server-5.1.1.zip
OMERO.cpp is no longer part of the main jobs
@manics
Copy link
Member Author

manics commented Jul 29, 2015

--release latest is now the default
I've switched to downloading OMERO.py (which is significantly larger) since OMERO.cpp no longer exists.

…mented)"

This reverts commit e9a7260.

Changed my mind on this- if you want to download a custom server you can use the existing method.
@pwalczysko
Copy link
Member

~/Work/omero-setup/omego/main.py download insight -v --release 4.4.5 works.
~/Work/omero-setup/omego/main.py download insight-ij -v --release 4.4.5 does not. What is the list of the aliases here ? (matlab, insight... anything else ?)

From help"

Download an OMERO artifact from a CI server.

positional arguments:
  artifact              The artifact to download e.g. {win,mac,mac6,linux,matl
                        ab,server,python,source,cpp,...}

mac6 ?

@manics
Copy link
Member Author

manics commented Jul 31, 2015

In general XXX in OMERO.<XXX>.zip, with the addition of some custom names e.g. mac/linux/win to specify the different clients. mac6 is for Java 6 in Mac.

@pwalczysko
Copy link
Member

In general all works. @manics will combine this with another PR in order to have more goodies (like insight-ij download) available for testing.

@manics manics changed the title Download releases Download releases and non-OMERO artifacts Jul 31, 2015
@manics
Copy link
Member Author

manics commented Jul 31, 2015

Description updated

@pwalczysko
Copy link
Member

Works as expected.
Except the usual - help, explanations, documenting the complex behaviour.

sbesson added a commit that referenced this pull request Aug 3, 2015
Download releases and non-OMERO artifacts
@sbesson sbesson merged commit 2b0ab33 into ome:master Aug 3, 2015
@pwalczysko
Copy link
Member

@sbesson : I thought it should be possible for @manics to reformat the output lines I commented on here ?

@sbesson
Copy link
Member

sbesson commented Aug 3, 2015

@pwalczysko: sorry for the confusion, I created a Trello card to address your comments in a follow-up PR.

@pwalczysko
Copy link
Member

@sbesson Thanks.

@sbesson sbesson added this to the 0.2.5 milestone Aug 21, 2015
@manics manics deleted the latest_downloads branch November 27, 2015 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants