Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add first stab at a release build script. #302

Merged
merged 37 commits into from Jun 24, 2015

Conversation

@janl
Copy link
Member

commented Feb 11, 2015

Creates a release directory apache-couchdb that we can wrap in a tarball. This is a quick hack to discuss whether this direction is anything we’d like to go towards for releases.

Please be generous with feedback, thank you! :)

make release
cd apache-couchdb
./configure
make dist
./rel/couchdb/bin/couchdb # starts CouchDB

End users will download apache-couchdb-$vsn.tar.gz and then:

tar xzf apache-couchdb-$vsn.tar.gz
cd apache-couchdb-$vsn
./configure
make dist

This builds fauxton and the docs as part of make release, so that end users don’t have to build those themselves.

Preliminary Todo:

  • change ./configure arguments to match 1.x-ish
  • infer CouchDB version from source tree and use in tarball, fauxton and welcome message
  • check for bashisms, maybe test with, say, dash.
  • fix bug noted in build-aux/couchdb-build-release.sh where cloning the current branch doesn’t work when a version tag is specified
  • make docs build understand that it doesn’t have to re-build if targets already exist (https://git-wip-us.apache.org/repos/asf?p=couchdb-documentation.git;h=10828f8)
  • actually create tarball
  • run test/build/*during `make check
  • factor out docs and fauxton build tasks from make release
  • finish make install target and test with various configurations
  • add make uninstall target
  • Windows version
  • test test test

Again, please be generous with feedback!

@janl

This comment has been minimized.

Copy link
Member Author

commented Feb 11, 2015

I also have this patch for couchdb-couch/rebar.config.script so it runs correctly in a non-git environment, but as far as I can tell right now, it is not really needed. It might come in handy later:

diff --git a/rebar.config.script b/rebar.config.script
index 09e11ad..4a1ae87 100644
--- a/rebar.config.script
+++ b/rebar.config.script
@@ -32,7 +32,12 @@ CouchJSName = case os:type() of
         "couchjs"
 end,
 CouchJSPath = filename:join(["priv", CouchJSName]),
-Version = string:strip(os:cmd("git describe --always"), right, $\n),
+Version = case file:open(".git", read) of
+    {error, _Reason}
+      -> "2.0.0"; % TODO: make dynamic
+    _Else ->
+        string:strip(os:cmd("git describe --always"), right, $\n)
+    end.
@sebastianrothbucher

This comment has been minimized.

Copy link
Contributor

commented Feb 17, 2015

Hi Jan,

I don't get through yet (CentOS 7), but here is the first thing that can help:

diff --git a/build-aux/couchdb-build-release.sh b/build-aux/couchdb-build-release.sh
index 50b1897..ce1a8cf 100755
--- a/build-aux/couchdb-build-release.sh
+++ b/build-aux/couchdb-build-release.sh
@@ -6,7 +6,7 @@ rm -rf $RELDIR
mkdir $RELDIR

copy sources over

-git archive build | tar -xC $RELDIR/
+git archive git rev-parse --abbrev-ref HEAD | tar -xC $RELDIR/
mkdir $RELDIR/src
cd src/

(i.e. make the branch name a variable all over the place)

@micah

This comment has been minimized.

Copy link
Contributor

commented Feb 19, 2015

In a typical autotools environment, make clean is no-op on a freshly untarred source dir, as normally, if there is a Makefile, you should be able to do a 'make clean' before a configure has been run. make clean" should always be idempotent, even after you build.

With the above method, we get a nice tarball, but a 'make clean' will fail until you run ./configure to create install.mk first. You shouldn't have to run ./configure first, unless the Makefile doesn't exist. In the debian world, the same thing is required, debian/rules clean needs to be a no-op on a freshly unpacked debian source package (because it is the first thing that is run in order to clean up a previous build, if there was one).

@micah

This comment has been minimized.

Copy link
Contributor

commented Feb 19, 2015

Since I'm on the subject of 'make clean', it seems like if you run this in the release tarball directory, it will fail because of the git pieces:

make -j1 distclean
make[2]: Entering directory '/home/micah/debian/couchdb/apache-couchdb'
ERROR: sh(git describe --always --tags)
failed with return code 128 and the following output:
fatal: Not a git repository (or any of the parent directories): .git
Makefile:26: recipe for target 'clean' failed

The 'make clean' will need to run without the git directories as well.

@janl janl force-pushed the janl:build branch 2 times, most recently from f9bac7e to 7ed8275 Feb 19, 2015

@janl

This comment has been minimized.

Copy link
Member Author

commented Feb 19, 2015

@sebastianrothbucher good catch, got that included now.

@micah The latest push here supports make clean before ./configure, this is more to unblock you and not quite the correct fix, as some of that data needs to be put in there by ./configure on the target system (maybe, still figuring this out).

I think I fixed the make [dist]clean git dependency, needs some work too, but should let you continue.

@janl janl force-pushed the janl:build branch 2 times, most recently from 53399fa to d2d3029 Feb 19, 2015

@janl

This comment has been minimized.

Copy link
Member Author

commented Feb 19, 2015

Latest version builds THANKS file like we did in 1.x.x (it’s not quite correct, but eh)

@micah

This comment has been minimized.

Copy link
Contributor

commented Feb 24, 2015

Yeah, I found that the following file deleted from the tarfile:

THANKS.in
THANKS
share/www/index.html
share/www/img/glyphicons-halflings.png
share/www/img/minilogo.png
share/www/img/glyphicons-halflings-white.png
share/www/img/linen.png
share/www/img/couch-watermark.png
share/www/img/couchdblogo.png
share/www/img/loader.gif
share/www/img/couchdb-site.png
share/www/js/require-0918b2e98537f3ff2f9ec1d4a99f8e03.js
share/www/js/ace/theme-idle_fingers.js
share/www/js/ace/mode-json.js
share/www/js/ace/worker-json.js
share/www/js/ace/worker-javascript.js
share/www/js/ace/mode-javascript.js
share/www/js/zeroclipboard/ZeroClipboard.swf
share/www/fonts/fontawesome-webfont.woff
share/www/fonts/fauxtonicon.woff
share/www/fonts/fontawesome-webfont.ttf
share/www/fonts/fauxtonicon.svg
share/www/fonts/fauxtonicon.eot
share/www/fonts/fontawesome-webfont.eot
share/www/fonts/fontawesome-webfont.svg
share/www/fonts/fauxtonicon.ttf
share/www/css
share/www/css/index-8570a17ed42470d169c1962bf0b250ed.css

One case where there was a mode change:

executable mode 0755 of 'src/couch/priv/couchspawnkillable' will not be represented in diff

and then the following files were changed

 apache-couchdb/build-aux/couchdb-build-release.sh
 apache-couchdb/src/couch/priv/couch_js/config.h
 apache-couchdb/src/couch/priv/couchspawnkillable
@micah

This comment has been minimized.

Copy link
Contributor

commented Feb 24, 2015

The share/www removal is because 'make distclean' is run in a typical build environment to remove all the stuff that ./configure created (typically to build an upstream source release, you would want to do a make distclean first, to clean up anything from before, and then start again. The 'make clean' is what would delete all the files that make created, and 'distclean' the ones that ./configure created).

In the Makefile there is:

distclean: clean
    @rm -rf rel/couchdb
    @rm -rf share/www

In the 'release' target, you are already distributing the pre-made share/www stuff, so removing it in distclean is working against us :)

    cp -r share/www apache-couchdb/share/
@janl

This comment has been minimized.

Copy link
Member Author

commented Feb 24, 2015

@micah good catch, this is an artefact of us using the same Makefile in the git checkout and the release dir / tarball.

A few options:

  • hack the current Makefile so it has context aware targets and does the right thing
  • have two Makefiles, one for the git source and one for the build/dir / tarball

Any preferences either way?

@micah

This comment has been minimized.

Copy link
Contributor

commented Feb 24, 2015

I think that the right thing to do is to change the Makefile so it does the right thing, but to do that might require changing things that I'm unaware of (build scripts, dev processes, muscle memory, etc.).

The typical standard make targets are:

  • make all - Build programs, libraries, documentation, etc. (same as make).
  • make install - Install what needs to be installed, copying the files from the package’s tree to system-wide directories.
  • make uninstall - The opposite of make install: erase the installed files. (This needs to be run from the same build tree that was installed.)
  • make clean - Erase from the build tree the files built by make all.
  • make distclean - Also erase anything ./configure created.
  • make check - Run the test suite, if any.
  • make installcheck - Check the installed programs or libraries, if supported.
  • make dist - Recreate package-version.tar.gz from all the source files.

Sometimes you will find other things like make install-strip, etc. Usually you don't need all of them (uninstall, check, installcheck aren't always there), but it seems like the couchdb Makefile mostly adheres to these defaults, just the release/distclean stuff is deviating a little from the norm and is exposing these issues.

In a way we are stepping one by one into the different issues that autotools was created to solve. I don't want to suggest that you stop and autotoolize couchdb, because that is another world of pain, but there is a point where you end up doing so many of the things that it solves that the pain of doing that is going to be a lot less than solving those again yourself (not to mention the support over time)... just something to keep in mind.

@sebastianrothbucher

This comment has been minimized.

Copy link
Contributor

commented Mar 10, 2015

Hi Jan,

I could do the drill now with make release, create TAR, unpack tar, ./configure, make && make install
(There was just some minor trouble with TeX - which I skipped, with some yum-Install-Erlang stuff, and with Sphinx that renamed the default template we inherit from - but none are TAR-related per se).
Anyhow: running Couch afterwards did fine.

BTW: so did copying in 1.6.1 files, calling them from local port, replicating them up to the cluster port.

That's great so far!
Sebastian

@janl janl force-pushed the janl:build branch from d2d3029 to 0ed6dcd Apr 3, 2015

@janl

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2015

@micah new version, should do the right thing vis-a-vis ./configure && make distclean and make && make clean.

To test this, I added:

@janl

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2015

As for Autotools: we’ve spent quite some time trying to get rid of them as much as possible. I agree that the packaging bits are neat, but they also are not trivial to maintain and it was a repeated time waste to get things going, so we opted for Make + rebar for now.

@janl janl force-pushed the janl:build branch from 0ed6dcd to 807151c Apr 3, 2015

@janl

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2015

@micah on make uninstall what is expected to happen with user-data that was created by the package? Like databases that might still have important data, are they expected to be rm’d? I understand logs could go, but I’m hesitant to include databases.

@janl

This comment has been minimized.

Copy link
Member Author

commented Apr 5, 2015

  1. switched to puhsing commits for progress instead of rebasing into single commit
  2. added version.mk that includes version info variables
  3. added -<githash> suffix if we build from git checkout.
@janl

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2015

  • Added -l (lowercase L) option to ./configure to specify log dir.
  • Cleaned up some internal stuff
  • make make install and make uninstall work
@micah

This comment has been minimized.

Copy link
Contributor

commented Apr 7, 2015

@micah on make uninstall what is expected to happen with user-data that was created by the package? Like databases that might still have important data, are they expected to be rm’d? I understand logs could go, but I’m hesitant to include databases.

regarding user-data, most linux distros follow the File Hierarchy Standard (FHS) and the package would handle the creation, and optional removal, of this data. For example, on debian, it would likely be that the package will create /var/lib/couchdb and the configure script would be run to set that as the directory for this data, and /var/log/couchdb would be be where logs go. The packaging system usually asks you about those things on removal, asking you if you want to remove the databases? are you sure? are you REALLY SURE?! So, I dont think a make distclean/clean etc. should touch that stuff

Regarding the most recent updates, they have eliminated the problems with files changing, being removed from unpacked source that were in the tarball, great! Now the only problem is that after the first build, the following ones fail (even after a make distclean) because these files remain:

tmp/
dev/boot_node.beam
dev/data
dev/lib
dev/pbkdf2.pyc
log/crash.log
dev/logs

The other issue I've been struggling with is that even if I pass a relative path to the -p option to configure, its still trying to use the absolute path on installation. So for example, I've been trying to do ./configure -p debian/couchdb/usr (here because this is where the package wants the resulting build so it can then take everything in there and make a package that would then install into /usr), but during the make install phase this happens:

mkdir: cannot create directory ‘/usr/local/couchdb’: Permission denied

It makes sense that I get a permission denied, because I am building as my unprivileged user, but I dont know why it is trying to use that directory, when I specified the -p option.

@janl

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2015

@micah good findings, I keep chipping away at them. The extra make leftovers should be gone with the latest commit.

Could you post the install.mk in the top level of your tarball extraction here?I don’t see why your code should try to mkdir /usr/local/couchdb, I get mkdir: /var/lib/couchdb: Permission denied with ./configure -p ../somereldir and that’s expected:

At this point, if you change the prefix with -p you also likely want to override the data dir, view dir and log file (-d -v -l) as they default to system paths as per FHS and are not affected by the prefix (see below). I have a todo item to make the ./configure arguments a little more intuitive and work like in 1.x (and other Autotools projects), so expect that to get better.

data_dir = /var/lib/couchdb
view_index_dir = /var/lib/couchdb
log_file = /var/log/couchdb.log
@micah

This comment has been minimized.

Copy link
Contributor

commented May 19, 2015

Hi @janl - sorry I haven't commented here for a while. A few updates here:

. tried to recompile and got this:

==> couchdb-2.0 (clean)
rm: cannot remove ‘dev/data’: Is a directory
rm: cannot remove ‘dev/lib’: Is a directory
rm: cannot remove ‘dev/logs’: Is a directory
Makefile:32: recipe for target 'clean' failed

I suspect that is because Makefile clean target has this:

@rm -f tmp/ dev/boot_node.beam dev/data dev/lib dev/pbkdf2.pyc log/crash.log dev/logs

but those are directories

@janl

This comment has been minimized.

Copy link
Member Author

commented May 21, 2015

@micah fixed!

@janl janl force-pushed the janl:build branch from b48b72a to c3ecbc6 May 21, 2015

@janl

This comment has been minimized.

Copy link
Member Author

commented May 21, 2015

Rebased on master

@micah

This comment has been minimized.

Copy link
Contributor

commented May 28, 2015

Thanks jan for rebasing the branch, I've just tried to compile that and now I'm getting this:

==> couchdb-2.0 (compile)
WARN:  Expected /home/micah/debian/couchdb/couchdb-2.0/src/mango to be an app dir (containing ebin/*.app), but no .app found.
Dependency not available: mango-.* ({git,
                                     "https://git-wip-us.apache.org/repos/asf/couchdb-mango.git",
                                     {branch,"master"}})
ERROR: compile failed while processing /home/micah/debian/couchdb/couchdb-2.0: rebar_abort
Makefile:28: recipe for target 'couch' failed
make[1]: *** [couch] Error 1
make[1]: Leaving directory '/home/micah/debian/couchdb/couchdb-2.0'
@janl

This comment has been minimized.

Copy link
Member Author

commented May 28, 2015

did you run ./configure before make?

janl and others added some commits Jun 18, 2015

New ./configure script!
This work will handle GNU ./configure style configuration of target
directories for installation of the various bits of CouchDB.

Includes a test script(!) for ./configure
allow dynamic variable substitution
E.g. GNU build tools call configure with these parameters:

> ./configure --prefix=/foo --libexecdir=\${prefix}/libexec

Expecting the final value of libexecdir to be /foo/libexec.

This commit makes our ./configure behave like a GNU configure.
Add --infodir --mandir --docdir --htmldir --pdfdir.
Also copy docs into these dirs on make install.

@janl janl force-pushed the janl:build branch from f2cb931 to 1ef7182 Jun 24, 2015

@janl

This comment has been minimized.

Copy link
Member Author

commented Jun 24, 2015

Alright, last push brings a whole lot of little fixes and completions. Also a rebase to master.

I think this is ready to be merged now.

Discussion can continue here, of course.

@asfgit asfgit merged commit 1ef7182 into apache:master Jun 24, 2015

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@yetzt

This comment has been minimized.

Copy link

commented Jun 24, 2015

<3

@janl janl deleted the janl:build branch Jun 24, 2015

@dch

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2015

Nice work!

Here's a nifty trick to get a git version, if this is a git repo, and return true if not. One could tweak the || true bit to return something else of course.

git describe --dirty --abbrev=7 --tags --always --first-parent 2>/dev/null || true
@janl

This comment has been minimized.

Copy link
Member Author

commented Jun 25, 2015

@dch thanks, is this any better or worse than what we have?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.