Integrate prereq in the new build system #15580

jdemeyer · 2013-12-24T09:40:37Z

After downloading Sage 6.0 either from git or from the .tar.gz sdist tarball, the prereq tarball prereq-1.2.tar.gz is missing. This is a huge problem, because many checks are skipped because of this.

Integrating this was probably overlooked in the sage-git build system. I think the best solution is not have prereq as a tarball at all and simply integrate the contents of the former prereq tarball in the git repo.

Depends on #15596

CC: @ohanar @vbraun

Component: distribution

Author: R. Andrew Ohana, Jeroen Demeyer

Branch/Commit: u/jdemeyer/ticket/15580 @ 2e27abb

Reviewer: R. Andrew Ohana, Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/15580

The text was updated successfully, but these errors were encountered:

ohanar · 2013-12-24T11:27:35Z

comment:2

Actually prereq-1.2.tar.gz should work if you drop it into the upstream directory, but since it was different, it didn't receive any of the extra treatment the other packages got.

jdemeyer · 2013-12-24T11:36:35Z

comment:3

Replying to @ohanar:

Actually prereq-1.2.tar.gz should work if you drop it into the upstream directory

OK, I see that. But it's not distributed by default.

jdemeyer · 2013-12-27T12:29:58Z

comment:4

If somebody can merge the prereq tarball contents in the git repo, I can easily do the rest.

ohanar · 2013-12-28T09:23:41Z

comment:5

I've exported prereq-1.2's history to the root directory with no cleanup (e.g. there is still the hgignore).

New commits:

`0881c74`	`Merge in prereq-1.2`
`cb5e072`	`Trac #14406: remove sqrtl() and SAGE_FORTRAN/SAGE_FORTRAN_LIB checks`
`e98db82`	`trac 13385: Remove check for openssl.`
`e63837b`	`Trac #13329: Add -lcrypto and -ldl (needed for -lssl), support SAGE_LOCAL`
`48b54aa`	`Trac #13329: small reviewer fixes`
`ac675c6`	`make dpgk-architecture check error out, its at least in Ubuntu 8.04`
`73f867d`	`added openssl and dpkg-architecture check`
`6b9c3e9`	`Trac #12112: Support --disable-compiler-checks option`
`ea08e9b`	`Initialization of repository`

ohanar · 2013-12-28T09:23:41Z

Branch: u/ohanar/prereq

ohanar · 2013-12-28T09:23:41Z

Commit: 0881c74

jdemeyer · 2013-12-28T17:39:43Z

comment:6

I have a patch ready, but I need to rebuild Sage before it can be uploaded...

jdemeyer · 2013-12-28T17:39:43Z

Author: R. Andrew Ohana, Jeroen Demeyer

jdemeyer · 2013-12-28T21:37:43Z

Changed branch from u/ohanar/prereq to u/jdemeyer/ticket/15580

jdemeyer · 2013-12-28T21:49:58Z

comment:8

Not yet ready for review, requires more work.

jdemeyer · 2013-12-28T21:49:58Z

Changed branch from u/jdemeyer/ticket/15580 to u/ohanar/prereq

jdemeyer · 2013-12-28T23:36:12Z

Changed branch from u/ohanar/prereq to u/jdemeyer/ticket/15580

sagetrac-git · 2013-12-29T00:09:28Z

Changed commit from 0881c74 to 143e5e4

sagetrac-git · 2013-12-29T00:09:28Z

Branch pushed to git repo; I updated commit sha1. New commits:

`143e5e4`	`Don't use cp -p in sage-clone-source`

sagetrac-git · 2013-12-29T10:37:59Z

Branch pushed to git repo; I updated commit sha1. New commits:

`68ac77f`	`Merge branch 'u/jdemeyer/ticket/15596' of git://trac.sagemath.org/sage into ticket/15580`
`c7c0106`	`sage-sdist: copy upstream tarballs using sage-spkg`
`321a9ad`	`Merge remote-tracking branch 'trac/u/ohanar/pillow' into t/15596/sdist_fails_with_capital_p_illow`
`f005905`	`pillow: depends on setuptools`
`adc90ce`	`Pillow -> pillow`
`8987381`	`remove PIL (it has been replaced by Pillow)`
`2c055bd`	`Pillow: fix patch to work against 2.2.2`
`6690d97`	`doc: remove now unused environment variables`
`3778f42`	`Pillow: re-resolve #9864`
`6df0adb`	`replace PIL with Pillow`

sagetrac-git · 2013-12-29T10:37:59Z

Changed commit from 143e5e4 to 68ac77f

jdemeyer · 2013-12-29T10:50:56Z

Dependencies: #15596

ohanar · 2013-12-30T05:33:08Z

Reviewer: R. Andrew Ohana

ohanar · 2013-12-30T05:33:08Z

comment:14

Looks fine to me, so long as someone can review my reviewer commit. (It is just restoring a small regression of sage-sdist.)

New commits:

`43b696f`	`sage-sdist: don't require sage to be built`

ohanar · 2013-12-30T05:33:08Z

Changed branch from u/jdemeyer/ticket/15580 to u/ohanar/prereq

ohanar · 2013-12-30T05:33:08Z

Changed commit from 68ac77f to 43b696f

ohanar · 2013-12-30T05:36:44Z

comment:15

One comment I have is that having a toplevel configure script might confuse those not familiar with sage's odd build system. Maybe we should move it to be under the build directory?

jdemeyer · 2013-12-30T10:07:58Z

comment:16

Replying to @ohanar:

One comment I have is that having a toplevel configure script might confuse those not familiar with sage's odd build system. Maybe we should move it to be under the build directory?

My hope is to converge to a more standard build system, so having ./configure at the top-level makes the most sense. And even if it confuses people, doing ./configure && make like the usual packages still works.

jdemeyer · 2014-01-14T21:02:37Z

Changed commit from 3e5dfa5 to e1df3ce

vbraun · 2014-01-15T03:05:36Z

comment:92

How is that supposed to change anything? The `aclocal.m4' is not in the confball, so you need aclocal to generate it. The job of the "missing" script is to mangle the timestamps as if aclocal has run, but you still need the file whose timestamp is to be changed.

vbraun · 2014-01-15T03:28:24Z

comment:93

PS: What running with bash does give you is the proper warning before it errors out:

WARNING: 'aclocal' is missing on your system.
         You should only need it if you modified 'acinclude.m4' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'aclocal' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>
make[3]: *** [configure] Error 127

vbraun · 2014-01-15T03:29:34Z

comment:94

PPS: any thoughts on enabling AM_MAINTAINER_MODE?

vbraun · 2014-01-15T05:30:17Z

Changed branch from u/jdemeyer/ticket/15580 to u/vbraun/ticket/15580

vbraun · 2014-01-15T05:31:06Z

Changed commit from e1df3ce to 9c13193

vbraun · 2014-01-15T05:31:06Z

comment:96

I guess disabling maintainer mode is only interesting when you actually run the automake-generated makefile. But we probably should, then. I'll put it in for now.

In any case, we should only run autotools from the bootstrap script. As long as we don't use the automake output there are no timestamp issues.

New commits:

`a38f919`	`allow disabling of maintainer mode`
`9c13193`	`only run autotools from the bootstrap script`

jdemeyer · 2014-01-15T07:38:31Z

comment:97

I disagree with ./sage -f configure since I absolutely don't want configure to act like a spkg. I prefer configure to have as few dependencies as possible. I much prefer to not depend on sage-spkg (and preferably also not on build/install nor sage-env, but that's harder for now).

Also, what was wrong with bootstrapping from the top-level Makefile? Doesn't the missing script work properly? I like the fact that configure gets regenerated automatically when I edit configure.ac.

jdemeyer · 2014-01-15T07:52:36Z

comment:98

Replying to @vbraun:

How is that supposed to change anything? The `aclocal.m4' is not in the confball, so you need aclocal to generate it.

OK, I forgot to add that file, no big deal.

jdemeyer · 2014-01-15T08:15:47Z

comment:99

Hmm, it seems that missing has changed functionality. Now it doesn't really do anything, except for printing a more verbose error message. Old versions of missing did indeed touch the files.

jdemeyer · 2014-01-15T12:29:36Z

Changed branch from u/vbraun/ticket/15580 to u/jdemeyer/ticket/15580

jdemeyer · 2014-01-15T13:32:36Z

comment:101

Volker, this patch simply stops using the missing script while keeping the rest of the bootstrap process.

New commits:

`2e27abb`	`Don't use "missing"`

jdemeyer · 2014-01-15T13:32:36Z

Changed commit from 9c13193 to 2e27abb

vbraun · 2014-01-15T14:37:10Z

comment:102

bootstrap shouldn't call make to do its business, usually it would run automake to generate the makefile. But we can also disentangle that later if you prefer.

AFAIK we do guarantee that "sage -f" works even before building Sage, so that you can install replacements for certain optional build-time dependencies (e.g. ccache). Not using that mechanism means that "make download" won't work, for example. Not to mention that it duplicates the sagemath upstream url, doesn't verify the checksum, etc. I don't necessarily care so much about using it, just wanted to make sure that you thought about the potential downsides of downloading the confball manually.

vbraun · 2014-01-15T14:38:28Z

comment:103

PS: I agree that missing changed and should be treated as implementation details for the autotools, i.e., do not call directly.

jdemeyer · 2014-01-15T14:55:30Z

comment:104

Replying to @vbraun:

AFAIK we do guarantee that "sage -f" works even before building Sage, so that you can install replacements for certain optional build-time dependencies (e.g. ccache). Not using that mechanism means that "make download" won't work, for example. Not to mention that it duplicates the sagemath upstream url, doesn't verify the checksum, etc. I don't necessarily care so much about using it, just wanted to make sure that you thought about the potential downsides of downloading the confball manually.

I was thinking of moving towards a more typical "configure + make" model. Then ./sage -f should be guaranteed to work only after ./configure but before make. What we actually want for optional packages would be something like

$ ./configure --enable-ccache --enable-build-gcc --enable-mirror=http://example.com/ --optional-packages=database_gap
$ make

jdemeyer · 2014-01-15T14:57:30Z

comment:105

Replying to @vbraun:

bootstrap shouldn't call make to do its business, usually it would run automake to generate the makefile. But we can also disentangle that later if you prefer.

Perhaps you are right. But I would indeed prefer to do that later, especially since #15606 changes the bootstrap process also.

vbraun · 2014-01-15T15:17:42Z

comment:106

Replying to @jdemeyer:

$ ./configure --enable-ccache --enable-build-gcc --enable-mirror=http://example.com/ --optional-packages=database_gap

Sounds good to me, but does not have to prevent sage -f from working (with sane defaults) before configure is run. If you want to change the mirror url then you just have to have your own set of autotools.

jdemeyer · 2014-01-15T20:19:42Z

comment:107

Replying to @vbraun:

Sounds good to me, but does not have to prevent sage -f from working (with sane defaults) before configure is run.

I'd say: let's see how things evolve. I will not actively prevent sage -f from working.

vbraun · 2014-01-15T20:23:05Z

Changed reviewer from R. Andrew Ohana to R. Andrew Ohana, Volker Braun

vbraun · 2014-01-15T20:23:05Z

comment:108

fine with me...

vbraun · 2014-01-16T04:50:34Z

comment:109

Another side effect of treating the confball different:

vbraun@boxen:~/Sage/git$ git clone ...
vbraun@boxen:~/Sage/git$ ./sage -sdist
Sage version 6.1.beta5, release date 2014-01-15
Error: configure tarball '/home/vbraun/Sage/git/upstream/configure-3.tar.gz' does not exist.

jdemeyer added this to the sage-6.1 milestone Dec 24, 2013

jdemeyer added c: distribution labels Dec 24, 2013

This comment has been minimized.

Sign in to view

jdemeyer changed the title ~~prereq tarball is missing~~ Integrate prereq in the new build system Dec 24, 2013

jdemeyer added the s: needs review label Dec 28, 2013

vbraun added s: positive review and removed s: needs review labels Jan 15, 2014

vbraun removed the s: positive review label Jan 16, 2014

vbraun closed this as completed in ea36c89 Jan 16, 2014

jpflori mentioned this issue Nov 28, 2014

Old FSF config.(guess|sub) scripts do not recognize Cygwin64 #14648

Closed

This was referenced Jan 25, 2014

Move checks from prereq.sh to configure #15606

Closed

Support "make -jN" #15624

Closed

vbraun mentioned this issue Jan 17, 2014

Overzealous compiler check when building docs #15687

Closed

Integrate prereq in the new build system #15580

Integrate prereq in the new build system #15580

Comments

jdemeyer commented Dec 24, 2013

This comment has been minimized.

ohanar commented Dec 24, 2013

jdemeyer commented Dec 24, 2013

jdemeyer commented Dec 27, 2013

ohanar commented Dec 28, 2013

ohanar commented Dec 28, 2013

ohanar commented Dec 28, 2013

jdemeyer commented Dec 28, 2013

jdemeyer commented Dec 28, 2013

jdemeyer commented Dec 28, 2013

jdemeyer commented Dec 28, 2013

jdemeyer commented Dec 28, 2013

jdemeyer commented Dec 28, 2013

sagetrac-git mannequin commented Dec 29, 2013

sagetrac-git mannequin commented Dec 29, 2013

sagetrac-git mannequin commented Dec 29, 2013

sagetrac-git mannequin commented Dec 29, 2013

jdemeyer commented Dec 29, 2013

ohanar commented Dec 30, 2013

ohanar commented Dec 30, 2013

ohanar commented Dec 30, 2013

ohanar commented Dec 30, 2013

ohanar commented Dec 30, 2013

jdemeyer commented Dec 30, 2013

jdemeyer commented Jan 14, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

vbraun commented Jan 15, 2014

jdemeyer commented Jan 15, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 15, 2014

vbraun commented Jan 16, 2014