dev-util/tup - gentoo ebuild

[oakad]

Hi,

I find it unfortunate that there's no ebuild for tup in the gentoo portage, so I endeavored to create one. There are still some shortcomings, unfortunately - some are described in my gentoo bug (enhancement) report.

https://bugs.gentoo.org/show_bug.cgi?id=491734

[eddyp]

The "not a git repository error is due to the rule at Tupfile:28 which assumes the build is done from a git repo, so it can grab the version using git describe. The offending part is the git describe which, for safety, should read test -d .git && git describe || echo "unknown-version" so it can correctly handle the out of tree build.

: src/tup/tup/main.o libtup.a src/lua/liblua.a |> ^ LINK tup^ version=`git describe`; echo "const char *tup_version(void) {return \"$version\";}" | $(CC) -x c -c - -o tup-version.o $(CFLAGS) -Wno-missing-prototypes; $(CC) %f tup-version.o -o tup -lpthread $(LDFLAGS) $(suid) |> tup tup-version.o

@oakad Try with that change and see how it works.
@gittup If there was a script that would run git describe and store the info somewhere, then call git archive to generate a zip/tgz in which the generated file would be added, wouldn't this be a good solution to get the version in the non-versioned archive and use that info instead of the non-working git- describe?

[oakad]

I can, of course, patch this out on the ebuild side, but my impression was that these things ought to be fixed prior to tarball hitting the "releases" list (otherwise, it's not technically a "release").

[lu-zero]

@oakad tup build system is apparently lacking a configure script, so it is impossible to cross compile AND it is impossible to disable hopefully optional features (is fuse compulsory? really??)

[ppannuto]

It's not impossible to cross-compile, you just have to remember that tup is a replacement for make, so it's simply not going to work the same way. Perhaps a script named configure could be created, but it's not really that necessary. Some light modification to build.sh (change CC, add vector table, stdlib references, etc) and tup cross-compiles just fine.

Fuse is obviously compulsory (for linux at least). Tup is built on top of fuse, it's what makes tup work.

[lu-zero]

Then I fail to see how it would work on platforms where fuse is not available (and I thought it uses inotify for that purpose with the normal file-hash/mtime as fallback...)

If it is true tup is to be avoided for projects aiming to be cross platform developed.
(ad no, just changing CC is definitely not enough to cross compile tup itself)

[ppannuto]

Then I fail to see how it would work on platforms where fuse is not available (and I thought it uses inotify for that purpose with the normal file-hash/mtime as fallback...)

Hmm.. perhaps that's true, but I don't see how tup could provide the correctness enforcement that it currently does without using fuse. Is fuse really an unreasonable dependency?

(ad no, just changing CC is definitely not enough to cross compile tup itself)

Correct. As I said, I also had to provide a vector table, link in a cross-compiled stdlib and fuse, and add some stubs for startup and exit routines. Again, not un-doable, particularly if you have some experience cross-compiling. It's just not as simple as it is for more traditional projects. I don't think cross-compiling has been a priority.

[lu-zero]

For the correctness enforcement all you need is a filesystem notification of some kind if you want to track the project state in real-time or, if such feature is missing, mimic what git does for the same purpose on update.

[ppannuto]

Detecting changes isn't sufficient. Tup tracks all of the files that are read from as well and verifies that the dependency relationships expressed in tupfiles are correct. You could possibly try to leverage atime to approximate this, though a lot of filesystems have that disabled nowadays.

Even then, it would only be sufficient for monitoring files in the immediate build tree. With full_deps enabled, tup tracks every file that is read from on the whole system (e.g. including libraries that are linked in). Unless you inotify the entire filesystem (intractable), there is no way to replicate this behavior without hooking every file system access (aka, fuse).

[lu-zero]

Seems similar to libsandbox. Making it optional wouldn't make tup less useful IMHO.

[oakad]

It is similar to libsandbox, but only on windows. :)

On windows it's using dll injection approach, on Linux fuse is probably considered a better performing and easier to implement approach.

[lu-zero]

If you aim for widespread adoption you should make this feature optional and have some easier fallbacks for the users that cannot use fuse.

You aim to replace make, if you manage to have the same requirements you can do easily, otherwise is more than an uphill battle.

[ppannuto]

The aim is not just to replace make but to do it one better. What is the
point of replacing make if you just suffer the same shortcomings?

The beauty of tup is that is proves correctness. This is an incredibly
strong property and one that requires a more invasive approach to achieve.
I don't see this kind of correctness as an optional feature, I see it as
the point. Perhaps Mike has a different opinion on this though.

As an aside, who can't use fuse?

On Sun, Mar 2, 2014 at 7:04 PM, Luca Barbato notifications@github.comwrote:

If you aim for widespread adoption you should make this feature optional
and have some easier fallbacks for the users that cannot use fuse.

You aim to replace make, if you manage to have the same requirements you
can do easily, otherwise is more than an uphill battle.

Reply to this email directly or view it on GitHubhttps://github.com//issues/148#issuecomment-36472707
.

[lu-zero]

Make problems are:

• being unable to scale (seems solved in an elegant way by tup)
• being 1/2 of the solution (and tup still needs a configure complement)
• being hard to master

Make advantages against everything else:

• being ubiquitous
• being easy to grok

Just to compare

cmake:

• all in one with battery and kitchen sink included
• supports lots of builders (albeit all in suboptimal ways, for simple projects ninja or make are equally slow)
• requires more than make (C++ runtime, curl, xmlrpc and such)
• a pain to cross compile with.
• a pain to probe and set options uniformly.

so where tup currently is for me:

• kernel-space, special priv hard dependencies (worse than cmake)
• not providing a standardized interface (as cmake)
• fast and with less rebuilds (custom crafted ninja might rival)

So basically tup would be good for me developing locally on my box with root, between impossible and problematic to use on restricted build-only environment (even within gentoo portage with priv-drop I'd have serious issues getting something tup based to build).

I hope I made my point.

Currently make with a decent configure script works decently at solving cross-compiling and ease of building everywhere (as long you have a working C compiler, more or less).

Despite the wild claims cmake fails here and there but gives you interesting perks for windows builds and an impressive set of ancillary features you might or might not interested on.

Tup might deps make it less portable than cmake and requires the same level of care of make regarding the configure script.

[eddyp]

kernel-space, special priv hard dependencies (worse than cmake)

huh? just being part of the 'fuse' group does not mean you're in kernel space
FUSE = file system in user space

basically tup would be good for me developing locally on my box with root

No, not at all.

[ tup ] [36.789s]
Updated.
130 eddy@heidi ~/usr/src/tup/tup $ whoami
eddy
0 eddy@heidi ~/usr/src/tup/tup $ groups | grep -o fuse
fuse

I have my own set of qualms with tup, too, but let's not invent problems that don't exist.

[lu-zero]

If tup works w/out the fuse kernel module being loaded or accessible, I wonder which part of fuse does it use.

With root, means that I can add myself to the fuse group (special privileges) and load the fuse module.

[anatol]

Some time ago I added tup package to Arch https://projects.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/tup

The main problem for me is that I cannot use fuse in a systemd namespace environment (systemd-nspawn) used during 'clean build' e.g. a new namespace is created then runs the package compilation. At this moment we use ./build.sh that does not use fuse during compilation.

[lu-zero]

That is systemd being difficult, probably could be fixed on their side easily.

Anyway having a fallback would ease adoption. As said, I cannot consider tup for a project that routinely builds over 10 different operating systems (of which only 3 might support fuse somehow) and with lots of restricted environments provided for testing (that would not let you have access to /dev/fuse anyway).

[gittup]

On Fri, Mar 21, 2014 at 6:39 AM, Anatol notifications@github.com wrote:

Some time ago I added tup package to Arch
https://projects.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/tup

The main problem for me is that I cannot use fuse in a systemd namespace
environment (systemd-nspawn) used during 'clean build' e.g. a new namespace
is created then runs the package compilation. At this moment we use
./build.sh that does not use fuse during compilation.

I finally got around to trying a fix for issue #141. Can you guys try that
out to see if it makes it easier to generate a package for tup? Something
like this should work:

./build.sh
./build/tup generate runme.sh
./runme.sh

That will do a one-time build without using FUSE or requiring 'tup init' at
all - after that you'll have to wipe out the tup directory (or 'git clean'
it) if you want to use the 'generate' command again. Since it is using the
rules from the Tupfiles, this will build the tup binary with inotify on
Linux (the one from build.sh is supposed to be a "barebones" one that just
contains the necessary bits to build tup from the Tupfiles).

Some notes:

• The 'tup generate' command will respond to a top-level tup.config, but
  won't do anything with variants
• No support yet for commands that write to generated directories
• No Windows support yet

Since this is primarily for tup packaging at the moment, I'm not too
concerned about the limitations for now.

Let me know if we still need to tweak it, or do something else for
packaging.

Thanks,
-Mike

[gittup]

On Fri, Mar 21, 2014 at 7:17 AM, Luca Barbato notifications@github.comwrote:

That is systemd being difficult, probably could be fixed on their side
easily.

Anyway having a fallback would ease adoption. As said, I cannot consider
tup for a project that routinely builds over 10 different operating systems
(of which only 3 might support fuse somehow) and with lots of restricted
environments provided for testing (that would not let you have access to
/dev/fuse anyway).

Unfortunately tup requires some way to detect file accesses of the
sub-processes automatically, which on Linux is currently done through FUSE.
Although I would love for a cross-platform way of doing this, I don't think
there is one. Please let me know if you have a better solution!

-Mike

[lu-zero]

In gentoo we use libsandbox, but obviously it won't work everywhere since it is using ptrace (that might be disabled as well).

Would be that terrible do w/out that information and provide just the fast resolver and the lean syntax for the users that can't have access to ptrace, /dev/fuse/, dtrace and so on?

[anatol]

Something like this should work:

./build.sh
./build/tup generate runme.sh
./runme.sh

unfortunately it fails on HEAD:

$ ./build/tup generate runme.sh
.tup repository initialized.
Scanning...
Reading tup.config...
Parsing...
 build/luabuiltin
 build
 src/sqlite3
 src/tup/monitor
 src/tup/tup
 src/tup/flock
 src/tup/server
 src/lua
 src/luabuiltin
 src/tup
 src/bsd
tup error: Circular dependency found among Tupfiles. Last directory: src/luabuiltin