Create release source artifact containing all submodules #1712

ghost · 2013-11-05T14:04:49Z

Testing:

./build.py release-src

This will create a source release. If the repo is tagged, you should get the correct release version here.

The resulting zip file should contain

all submodules
correct executable flags on all files executable in the git tree
components/antlib/resources/gitversion.xml containing the release version numbers

The unpacked zip file should build all the same targets as for the git repo, the only exception being that the release-src target will not work, so ./build.py release-all will exclude the source zip when run outside git. We could remove this from release-all so that it's only for binary artifacts.

manics · 2013-11-06T11:58:20Z

Generally looks good, source is now in a directory instead at the top level.

Why not replace OMERO.source-5.0.0-beta1-1304-1e8b196-b478.zip completely instead of a separate openmicroscopy-5.0.0-beta1-1304-1e8b196-b478.zip?

build.py has lost its execute permissions

Comparing s local checkout of scc/merge/develop/latest with http://hudson.openmicroscopy.org.uk/job/OMERO-merge-develop/lastSuccessfulBuild/artifact/src/target/openmicroscopy-5.0.0-beta1-1304-1e8b196-b478.zip (using diff -b because my mac has weird issue with line endings):

$ diff -rqb ~/work/openmicroscopy.dev/ openmicroscopy-5.0.0-beta1-1304-1e8b196-b478
Only in /Users/simon/work/openmicroscopy.dev/: .git
Only in /Users/simon/work/openmicroscopy.dev/components/bioformats: .git
Files /Users/simon/work/openmicroscopy.dev/components/bioformats/lib/libJWlz.so and openmicroscopy-5.0.0-beta1-1304-1e8b196-b478/components/bioformats/lib/libJWlz.so differ
Only in /Users/simon/work/openmicroscopy.dev/components/rendering/src/omeis: CVS
Only in /Users/simon/work/openmicroscopy.dev/components/tools/OmeroPy/scripts: .git
Only in /Users/simon/work/openmicroscopy.dev/docs/sphinx: .git

components/bioformats/lib/libJWlz.so is a symlink to libJWlz-1.4.0.so, in the zip it becomes a text file:

cat openmicroscopy-5.0.0-beta1-1304-1e8b196-b478/components/bioformats/lib/libJWlz.so
libJWlz-1.4.0.so

but this probably shouldn't be in the repo in the first place and it seems to build fine anyway. Other differences can be ignored.

joshmoore · 2013-11-06T13:42:18Z

Keeping the OMERO.source was my not knowing what is currently using it. I'm also not sure of the name openmicroscopy-.... If we ever do rename this repo, that becomes a bit too all-encompassing.

manics · 2013-11-06T14:15:18Z

Doesn't the lack of submodules prevent any functional use of OMERO.source?

ghost · 2013-11-06T14:51:27Z

If we rename the repo, then we can simply rename the archive at the same time.

I'll double-check why we're losing the executable bit. Looks like the ant zip task doesn't handle this.

WRT the missing bits, the .git directories are intentional; CVS is unintentional but OK I imagine--legacy from CVS days?

For JWlz.so, I opened #11655.

ghost · 2013-11-06T15:25:42Z

So looking at the ant tar/zip/untar/unzip tasks, it appears that they are completely incapable of handling posix permissions until they are updated to use the Java7 File API which introduces support for them. So if we want to run an external shell script to do the source generation, that's the best way to do it, but at the expense of portability. Mind you, we can't create an archive with correct permissions on Windows anyway, so maybe this isn't such a bad limitation. Thoughts?

joshmoore · 2013-11-07T07:53:50Z

I'm not sure what CVS is about. Worth getting rid of?
re: naming -- looking at this more, I agree. Can we do s/openmicroscopy/OMERO.source/ for the archive names and be done with it? (This is different from the cpp.src file which was intended to provide the generated source, which this archive isn't doing)

joshmoore · 2013-11-14T09:20:48Z

@rleigh-dundee : are you ok to make those changes?

ghost · 2013-11-14T15:03:31Z

The main outstanding blocker is how to keep the executable files executable in the generated zips. We could change from using ant to a shell or python script to run git and zip by hand (which will not work on Windows), but would correctly preserve the execute permissions [packing on Windows would destroy the permissions anyway, so it's not workable there anyway I think].

Insight uses the <zipfileset> mode attribute, but this only works if you know in advance which files have which permissions. We don't know, and we can't preserve them on unpack even if we could make use of this on repack as far as I can tell. We could potentially scan the git tree objects to find the executable blobs, but doing this in ant is likely a tall order.

CVS is an obsolete relic (had empty CVS/Entries); so can definitely be removed.

For the naming, I'm not sure what you mean here; I was suggesting to do the opposite and drop the .source entirely. Naming the source release of openmicroscopy.git "OMERO.source" is counter to what all but a few projects do for their source releases, and is counter to the expectations of people who would want to download and build this source (i.e. it's highly unconventional, and even if it only contained OMERO alone it would still be problematic). As a concrete example, it doesn't meet the package naming conventions used by Debian or RedHat/Fedora (and others), so all these potential downstream consumers would need to unnecessarily repack the archive to correct the name before they could use it.

Fedora does not permit dots as separators in a package name; Debian does not permit uppercase package names, so our being gratuitously different from the rest of the world here does make us incompatible with these two projects (and doubtless many others), so this does have a cost.

joshmoore · 2013-11-15T08:29:47Z

re: permissions. There are far more than I would have expected:

$ for x in `git ls-files`; do test -x $x && echo ${x##*.}; done | sort | uniq -c
      4 bat
      1 bmp
      1 components/tools/OmeroPy/bin/omero
      1 components/tools/OmeroPy/scripts
     46 css
      2 cur
      1 docs/sphinx
     22 gif
      8 html
      1 ico
      4 jar
     29 java
      1 jpg
    153 js
      1 jsb2
      7 m
      3 md
      8 nsh
      1 nsi
      3 omero
     80 png
     79 py
      9 sh
      3 txt

Most likey the goal of this PR shouldn't be to make the new source zip completely build-able, since that's too much work. It didn't build previously, so adding the submodules will already be an improvement. I filed handling the exec bit as: https://trac.openmicroscopy.org.uk/ome/ticket/11593

joshmoore · 2013-11-15T08:51:10Z

Re: zip naming -- I filed https://trac.openmicroscopy.org.uk/ome/ticket/11701

Let's not try to make such decisions here but get several people together to chat. For the moment, please make use of OMERO.source which will also keep jobs green.

manics · 2013-11-18T14:43:38Z

re: permissions. There are far more than I would have expected:

How many of those are required, as opposed to permissions which were inadvertently committed?

The default build target actually works fine after making ./builld.py executable, server runs, however OMERO.web doesn't work due to some version number problems:
Error: Client version does not match server, please contact administrator.
So if anything I think this would be the next priority.

joshmoore · 2013-11-19T07:54:11Z

@manics : not many are needed, I don't think.

joshmoore · 2013-11-25T08:10:59Z

Ping.

ghost · 2013-11-25T16:00:17Z

@joshmoore My question about the execute perms which I've asked twice remains unanswered. We need the permissions preserving, and this isn't possible with ant. We could use either a simple shell script or a python script, for example, run git archive/unzip/zip to do this, but whichever we choose it's not going to be possible for this to (ever) work well on windows as is.

One possibility is that

git ls-tree -r HEAD | grep "^...755" | cut -f2
build.py
components/antlib/nsis/check_ice.nsh
components/antlib/nsis/check_java.nsh
components/antlib/nsis/check_nginx.nsh
components/antlib/nsis/check_pil.nsh
...

gives us a list of files which are executable in git. This would be a means of setting explicit permissions, which would be worth looking at. However, we would need to find a way to exclude these entries from the other lists. Doing this in ant will not be pleasant. It wouldn't be hard with shell or python. This might even work for windows--we can add the execute property with the python "zipfile" module, for example.

If it would be OK to move the zipfile creation out from ant to an external script, I'll be happy to do the above. Where would be the best place to put such a script?

joshmoore · 2013-11-25T20:17:45Z

I have 3 different reactions (at least):

It didn't build before and therefore perhaps we should be happy enough with where we are for the moment
Perhaps we should generate a script (or patch build.py or similar) to fix this after the fact
As @manics pointed out, we don't need many of the permissions and therefore an intermediate fix is likely possible.

Happy to discuss any of them, but since the rest of the build fixes have been ticketed and moved to a later milestone, I would tend to do 1 unless someone has tried 3 and knows it to be easy and feasible.

joshmoore · 2014-01-15T08:30:16Z

@rleigh-dundee : is this something you want to come back to now?

ghost · 2014-01-15T09:42:56Z

Yes. I'll redo this as a python script. shoehorning this into ant works up to a point, but really just isn't good enough.

ghost · 2014-01-15T14:36:12Z

Given the age of the branch, I've rebased it onto current develop. I've added a separate python script to do the archive creation which doesn't have the limitations of the ant-based approach. This now creates a source archive containing all submodules and with correct permissions across the board.

joshmoore · 2014-01-27T15:04:32Z

exclude removed; also tested locally and all builds fine. Only (known) downside is:

jamoore@blue:/tmp/FOO/openmicroscopy-5.0.0-rc1-DEV$ ./build.py version
Buildfile: /tmp/FOO/openmicroscopy-5.0.0-rc1-DEV/build.xml

version:
UNKNOWN
${version.describe}-ice35

joshmoore · 2014-01-27T15:29:17Z

And trying to release-src from a source release also doesn't work:

(ome1)jamoore@blue:/tmp/FOO/openmicroscopy-5.0.0-rc1-DEV$ OMERO_BRANCH=test bash docs/hudson/OMERO.sh
...

release-src:
fatal: Not a git repository (or any of the parent directories): .git
Releasing openmicroscopy-UNKNOWN
Traceback (most recent call last):
  File "/tmp/FOO/openmicroscopy-5.0.0-rc1-DEV/components/antlib/scripts/source-archive", line 41, in <module>
    raise Exception('Failed to create git base archive')
Exception: Failed to create git base archive

BUILD FAILED
/tmp/FOO/openmicroscopy-5.0.0-rc1-DEV/build.xml:443: exec returned: 1

Total time: 3 minutes 12 seconds
(ome1)jamoore@blue:/tmp/FOO/openmicroscopy-5.0.0-rc1-DEV$

Which also seems acceptable. Assuming there are no problems with the builds tomorrow, I'd say let's merge this and make other minor modifications as need be.

ghost · 2014-02-03T14:02:33Z

Version number is now preserved by release-src.

ghost · 2014-02-03T14:53:02Z

Having it in 5.0 would be ideal. I'll be happy to backport as soon as we know all the targets work correctly. Further down the line, many of the jobs which use git clones could be converted to download the source zip from an upstream job, which reduce the diskspace and download times significantly, as well as ensure that the release source is always buildable.

sbesson · 2014-02-03T15:11:54Z

👍 for backporting to 5.0. Happy to test this tomorrow.
A couple of immediate thoughts:

<target name="version" depends="version-embed,version-git"/> in the case of a Git repo, does that involve that if gitversion.xml exists, it will take priority over the git describe output?
related to the previous question, thoughts about moving gitversion.xml to e.g. target so that ./build.py clean can wipe it out (thinking of our daily job specifically)
is is timely to ask @gusferguson if he can generate a shiny release source code icon to expose this new artifact under http://downloads.openmicroscopy.org/omero/5.0.0-rc1/#code ?

ghost · 2014-02-03T16:11:54Z

@sbesson to answer your questions:

if gitversion.xml exists, it will take priority over the git output. However, note that it will never exist in a git tree--it's only created inside the release zip file. If one is ever created by hand, it will be cleanable by git clean.
gitversion.xml should never be cleaned--it's a release file, and there's no way to recreate it, so it would break any future build.py use. From the POV of jobs building from git, this won't be a problem since they will never encounter this file.
An icon for source downloads would be great; this should soon be ready to be used.

joshmoore · 2014-02-04T08:12:16Z

@rleigh-dundee : would you mind enabling release-src either as a part of release-all in build.xml or in docs/hudson/OMERO.sh? This should get it into every daily build. Thanks.

joshmoore · 2014-02-04T08:14:11Z

NB: ./build.py release-src && cd target && unzip target/openmicroscopy*zip && cd openmicroscopy* && ./build.py release-src clearly fails with:

BUILD FAILED
/opt/hudson/workspace/OMERO-5.1-merge-build/src/target/openmicroscopy-5.1.0-m0-DEV/build.xml:448: Releasing is only possible from a git repository

ghost · 2014-02-04T08:43:31Z

@joshmoore Does this work if you unpack outside the git repo?

joshmoore · 2014-02-04T09:32:06Z

Nope:

[hudson@ome-c6100-3 openmicroscopy-5.1.0-m0-DEV]$ pwd
/tmp/openmicroscopy-5.1.0-m0-DEV
[hudson@ome-c6100-3 openmicroscopy-5.1.0-m0-DEV]$ ./build.py release-src
Buildfile: /tmp/openmicroscopy-5.1.0-m0-DEV/build.xml

BUILD FAILED
/tmp/openmicroscopy-5.1.0-m0-DEV/build.xml:448: Releasing is only possible from a git repository

Total time: 0 seconds

but just to be clear: this is my expected, preferred behavior.

ghost · 2014-02-04T10:05:20Z

Ah, OK. I've added release-src to OMERO.sh now.

joshmoore · 2014-02-04T10:18:50Z

Nice. We can look at the artifacts of the next build and then merge. (In the future we'll probably need this in OMERO.bat as well)

ghost · 2014-02-04T10:32:21Z

OK. Note that the source-archive script was also copied into the bioformats PR. Once that's merged, we can move over to using that in openmicroscopy as well.

- Correct version to use plainversion. - Add archive prefix.

…archive

This reverts commit 5b0098333130d3b01eb3aaa081ab188589dfe354.

joshmoore · 2014-02-05T08:25:01Z

Downloaded http://ci.openmicroscopy.org/view/5.1/job/OMERO-5.1-merge-build/ICE=3.5,label=ome-c6100-3/lastSuccessfulBuild/artifact/src/target/openmicroscopy-5.1.0-m0-176-ad21687.zip

Following commands worked fine:

./build.py build-dev
./build.py release-src

Merging.

Create release source artifact containing all submodules

sbesson · 2014-02-24T09:29:01Z

--rebased-to #2064

Roger Leigh added 15 commits February 4, 2014 18:26

build.xml: Package source archive including submodules

eb5adf4

build: Add plainversion

67acc03

build: Restore creation of broken OMERO.source in release-src

5f026b0

- Correct version to use plainversion. - Add archive prefix.

build: release-src ensures target exists

98ba84a

build: Create tar source archive as well as zip

fee1273

omeis: Remove obsolete CVS directory

8d86b1f

build: Move git archive creation to components/antlib/scripts/source-…

b5b4ac0

…archive

build: Embed version as gitversion.xml

f797f5a

build: Embed short version for documentation

35b6680

build: Restore release-src as release-all dependency

0d69c91

build: build.py sets omero.shortversion

04ab4ed

Revert "build: Restore release-src as release-all dependency"

5c06ba2

This reverts commit 5b0098333130d3b01eb3aaa081ab188589dfe354.

build: source-archive: Manually flush stdout

b4c592f

build: source-archive: Generalise version filename

76f1886

OMERO.sh: Create source release

a092b55

ghost mentioned this pull request Feb 4, 2014

Create release source artifact containing all submodules (dev_5_0) #2064

Merged

joshmoore added a commit that referenced this pull request Feb 5, 2014

Merge pull request #1712 from rleigh-dundee/release-zip

119dcfb

Create release source artifact containing all submodules

joshmoore merged commit 119dcfb into ome:develop Feb 5, 2014

sbesson added this to the 5.1.0-m1 milestone Oct 14, 2014

sbesson mentioned this pull request Mar 24, 2015

build: Also provide source zip without binary content ome/bioformats#1686

Merged

rleigh-codelibre deleted the release-zip branch May 29, 2017 10:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create release source artifact containing all submodules #1712

Create release source artifact containing all submodules #1712

ghost commented Nov 5, 2013

manics commented Nov 6, 2013

joshmoore commented Nov 6, 2013

manics commented Nov 6, 2013

ghost commented Nov 6, 2013

ghost commented Nov 6, 2013

joshmoore commented Nov 7, 2013

joshmoore commented Nov 14, 2013

ghost commented Nov 14, 2013

joshmoore commented Nov 15, 2013

joshmoore commented Nov 15, 2013

manics commented Nov 18, 2013

joshmoore commented Nov 19, 2013

joshmoore commented Nov 25, 2013

ghost commented Nov 25, 2013

joshmoore commented Nov 25, 2013

joshmoore commented Jan 15, 2014

ghost commented Jan 15, 2014

ghost commented Jan 15, 2014

joshmoore commented Jan 27, 2014

joshmoore commented Jan 27, 2014

ghost commented Feb 3, 2014

ghost commented Feb 3, 2014

sbesson commented Feb 3, 2014

ghost commented Feb 3, 2014

joshmoore commented Feb 4, 2014

joshmoore commented Feb 4, 2014

ghost commented Feb 4, 2014

joshmoore commented Feb 4, 2014

ghost commented Feb 4, 2014

joshmoore commented Feb 4, 2014

ghost commented Feb 4, 2014

joshmoore commented Feb 5, 2014

sbesson commented Feb 24, 2014

Create release source artifact containing all submodules #1712

Create release source artifact containing all submodules #1712

Conversation

ghost commented Nov 5, 2013

manics commented Nov 6, 2013

joshmoore commented Nov 6, 2013

manics commented Nov 6, 2013

ghost commented Nov 6, 2013

ghost commented Nov 6, 2013

joshmoore commented Nov 7, 2013

joshmoore commented Nov 14, 2013

ghost commented Nov 14, 2013

joshmoore commented Nov 15, 2013

joshmoore commented Nov 15, 2013

manics commented Nov 18, 2013

joshmoore commented Nov 19, 2013

joshmoore commented Nov 25, 2013

ghost commented Nov 25, 2013

joshmoore commented Nov 25, 2013

joshmoore commented Jan 15, 2014

ghost commented Jan 15, 2014

ghost commented Jan 15, 2014

joshmoore commented Jan 27, 2014

joshmoore commented Jan 27, 2014

ghost commented Feb 3, 2014

ghost commented Feb 3, 2014

sbesson commented Feb 3, 2014

ghost commented Feb 3, 2014

joshmoore commented Feb 4, 2014

joshmoore commented Feb 4, 2014

ghost commented Feb 4, 2014

joshmoore commented Feb 4, 2014

ghost commented Feb 4, 2014

joshmoore commented Feb 4, 2014

ghost commented Feb 4, 2014

joshmoore commented Feb 5, 2014

sbesson commented Feb 24, 2014