[install] fix Issue 11579 - dlang.org repo can't be built without git #422

MartinNowak · 2013-11-29T23:52:56Z

use github API for tags
use tarballs instead of git clone

MartinNowak · 2013-12-03T21:35:20Z

@andralex can you have a look at this? I need it to move on with automatic build scripts.

MartinNowak · 2013-12-05T18:21:05Z

Someone else maybe? @braddr @9rnsr

MartinNowak · 2013-12-13T15:59:15Z

Quick rationale:
The build/installer scripts already fetch sources which should be reused when building documentation.
With the git dependency in the makefile this is not easily possible.

andralex · 2013-12-15T03:57:26Z

posix.mak

@@ -167,7 +167,7 @@ clean:
 	rm -rf $(DOC_OUTPUT_DIR) ${LATEST}.ddoc
 	rm -rf auto dlangspec-tex.d $(addprefix dlangspec,.aux .d .dvi .fdb_latexmk .fls .log .out .pdf .tex)
 	rm -f docs.json docs-prerelease.json
-	@echo You should issue manually: rm -rf ${DMD_DIR}.${LATEST} ${DRUNTIME_DIR}.${LATEST} ${PHOBOS_DIR}.${LATEST}
+	@echo You should issue manually: rm -rf ${DMD_DIR}-${LATEST} ${DRUNTIME_DIR}-${LATEST} ${PHOBOS_DIR}-${LATEST}


What motivates this change?

It's the name of the tarball root directory (e.g. dmd-2.064.2). The same naming scheme is used/expected by rpmbuild and the deb packaging tool.

OK, fine with this

andralex · 2013-12-15T04:05:05Z

I'm not convinced about the rationale. So we have one seemingly unexplained change of naming from . to -, and we switch from requiring git to requiring curl. I don't see progress here.

Why do we need to support machines without git?

Why don't the build/installer scripts use git as well, and if they don't, why can't we make them to? Using git seems like the logical choice instead of adding yet another moving piece - the historical archives.

Without very solid motivation, I am opposed to this. Sorry.

MartinNowak · 2013-12-15T18:22:00Z

Why do we need to support machines without git?

Why don't the build/installer scripts use git as well, and if they don't, why can't we make them to?

Package build scripts are taball centric, for example the rpmbuild tool saves the tarballs in a source rpm (.srpm) file and builds the rpm package out of that. I'd need to add another script that creates tarballs from a temporary git clone, so directly downloading tarballs from github is cleaner.
Now that I already have all the sources I want to build the documentation from them which isn't possible because the folders are named differently and because posix.mak assumes git repos.

we switch from requiring git to requiring curl.

But curl is installed almost everywhere by default.

As a compromise the following would work for me.

Rename the folders to use a hyphen.
Only run git checkout if a .git subdirectory exists.

Another less important point, tarballs are smaller and faster to fetch than a complete git repo.

andralex · 2013-12-18T02:39:50Z

How about we go with the compromise you propose, and consider a more involved solution for later? Thanks!

MartinNowak · 2013-12-19T15:46:16Z

How about we go with the compromise you propose, and consider a more involved solution for later? Thanks!

Done.

andralex · 2013-12-23T07:10:17Z

posix.mak

+	[ -d $@ ] || git clone ${GIT_HOME}/$* $@/
+
+# LATEST
+${DMD_DIR}-${LATEST} ${DRUNTIME_DIR}-${LATEST} ${PHOBOS_DIR}-${LATEST} : ../%-${LATEST} :


Rules with multiple targets are almost never doing the right thing. In this case they're liable to execute the same commands multiple times. I see you have guards to prevent disasters, but I think there are better ways out there, see http://goo.gl/Qi6Qm8

Also, I've tried many times to make rules depending on directories, with bizarre results (the times when directories are written are... somewhat unpredictable). There's some order only prerequisite that may help: http://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html. Simpler, just plant a file .directory or .made to signal an action pertaining to a specific directory has succeeded.

see also http://stackoverflow.com/questions/4440500/depending-on-directories-in-make

There's some order only prerequisite that may help: http://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html.

Didn't knew this, but I went with .clone for now.

I'm not sure I understand what the issue with static pattern rules is.
Is it that because the rule is "generic", it may easier lead to mistakes if $@ et.al. don't get used?
What's the alternative, copying the rule for each target?

andralex · 2013-12-23T07:17:36Z

TL;DR: I think multiple targets should be avoided.

MartinNowak · 2013-12-24T16:28:19Z

updated

andralex · 2013-12-24T19:19:36Z

posix.mak

-	${MAKE} --directory=${DMD_DIR}.${LATEST}/src -f posix.mak clean
-	${MAKE} --directory=${DMD_DIR}.${LATEST}/src -f posix.mak -j 4
+# HEAD
+${DMD_DIR}/.clone ${DRUNTIME_DIR}/.clone ${PHOBOS_DIR}/.clone : ../%/.clone :


How does this pattern rule work? After macro expansion the whole thing is something like:

../dmd/.clone : ../%/.clone [ -d $(@D) ] || git clone ${GIT_HOME}/$* $(@D)/ touch $@ ../druntime/.clone : ../%/.clone [ -d $(@D) ] || git clone ${GIT_HOME}/$* $(@D)/ touch $@ ../phobos/.clone : ../%/.clone [ -d $(@D) ] || git clone ${GIT_HOME}/$* $(@D)/ touch $@

According to http://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html, the % must appear on the target side too. What is the intent here?

It works like a normal pattern rule,

../%/.clone : [ -d $(@D) ] || git clone ${GIT_HOME}/$* $(@D)/ touch $@

But it will only match the targets listed before the first colon.

${DMD_DIR}/.clone ${DRUNTIME_DIR}/.clone ${PHOBOS_DIR}/.clone

We could simply go with ../%/.clone : if that's OK for you.

The fewer subtleties the better. I'd say something like this is the ticket:

../%/.git-cloned : [ ! -e $(@D) ] || git clone ${GIT_HOME}/$* $(@D)/ touch $@

Prolly even simpler:

../%/.git : [ ! -e $(@D) ] || git clone ${GIT_HOME}/$* $(@D)/

This counts on the cloning creating the .git directory.

I used ../%/.cloned, after all not depending on git is the main purpose of this pull request.

.${LATEST} => -${LATEST} - This matches the versioning scheme used by others (github, rpm spec, ...) and simplifies interop.

- check for existence of .git subdir first

MartinNowak · 2013-12-28T18:41:31Z

updated

andralex · 2013-12-28T19:35:45Z

posix.mak

-	${MAKE} --directory=${DMD_DIR}.${LATEST}/src -f posix.mak -j 4
+# HEAD
+../%/.checkout :
+	[ -d $(@D) ] || git clone ${GIT_HOME}/$* $(@D)/


would we care to git pull if the directory does exist?

Only for a clean master maybe, anything else could have unwanted effect.
I don't think it's a good idea.

Then the name of the witness is confusing, should be .cloned or .directory - sorry, I know you've been there before and it's exasperating. But if I see .checkout I think of a witness for the last checkout.

[install] fix Issue 11579 - dlang.org repo can't be built without git

MartinNowak restored the fix11579 branch November 29, 2013 23:52

andralex reviewed Dec 15, 2013
View reviewed changes

andralex reviewed Dec 23, 2013
View reviewed changes

andralex reviewed Dec 24, 2013
View reviewed changes

MartinNowak added 4 commits December 28, 2013 19:21

use GIT_HOME to fetch tags

57b1d0b

separate and reuse targets to clone dmd/druntime/phobos

9731564

use hyphen instead of dot for version suffix

f57d9c4

.${LATEST} => -${LATEST} - This matches the versioning scheme used by others (github, rpm spec, ...) and simplifies interop.

only perform checkout in git repos

fbe585f

- check for existence of .git subdir first

andralex reviewed Dec 28, 2013
View reviewed changes

use /.cloned target for git cloning/checkout

175e653

andralex added a commit that referenced this pull request Dec 29, 2013

Merge pull request #422 from MartinNowak/fix11579

8a6de6e

[install] fix Issue 11579 - dlang.org repo can't be built without git

andralex merged commit 8a6de6e into dlang:master Dec 29, 2013

MartinNowak deleted the fix11579 branch December 29, 2013 21:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[install] fix Issue 11579 - dlang.org repo can't be built without git #422

[install] fix Issue 11579 - dlang.org repo can't be built without git #422

MartinNowak commented Nov 29, 2013

MartinNowak commented Dec 3, 2013

MartinNowak commented Dec 5, 2013

MartinNowak commented Dec 13, 2013

andralex Dec 15, 2013

MartinNowak Dec 15, 2013

andralex Dec 18, 2013

andralex commented Dec 15, 2013

MartinNowak commented Dec 15, 2013

andralex commented Dec 18, 2013

MartinNowak commented Dec 19, 2013

andralex Dec 23, 2013

andralex Dec 23, 2013

andralex Dec 23, 2013

MartinNowak Dec 23, 2013

andralex commented Dec 23, 2013

MartinNowak commented Dec 24, 2013

andralex Dec 24, 2013

MartinNowak Dec 24, 2013

andralex Dec 24, 2013

andralex Dec 24, 2013

MartinNowak Dec 28, 2013

MartinNowak commented Dec 28, 2013

andralex Dec 28, 2013

MartinNowak Dec 28, 2013

andralex Dec 28, 2013

MartinNowak Dec 29, 2013

[install] fix Issue 11579 - dlang.org repo can't be built without git #422

[install] fix Issue 11579 - dlang.org repo can't be built without git #422

Conversation

MartinNowak commented Nov 29, 2013

MartinNowak commented Dec 3, 2013

MartinNowak commented Dec 5, 2013

MartinNowak commented Dec 13, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andralex commented Dec 15, 2013

MartinNowak commented Dec 15, 2013

andralex commented Dec 18, 2013

MartinNowak commented Dec 19, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andralex commented Dec 23, 2013

MartinNowak commented Dec 24, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartinNowak commented Dec 28, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment