Add create_dmd_release utility. #24
Conversation
|
The baseline is that the module is quite complex for what it does and has many external dependencies. fetching sources
extra tools
use std.zip
use std.net.curl
other stuff
|
| --------------------- | ||
| - Git | ||
| - Posix: zip (Info-ZIP) and 7z (p7zip) (On Windows, these will automatically | ||
| be downloaded if necessary.) |
There was a problem hiding this comment.
As written you can get rid of git and zip. I think 7z is a separate topic and converting from zip to 7z is a 2-liner.
|
Building all linux releases in a single distro doesn't work (Issue 10710), so we kind of need to kill the one-for-all zip release. |
|
All your suggestions are good ones, but none of them are blockers for actually using this tool. If this actually gets used, then I'll be happy to improve it by addressing all those issues. Otherwise, there's no point in me pushing this any further and people can just improve it, replace it, or theorize over ideal approaches all they want. So far there doesn't seem to be much interest in it, aside from Walter's desire to have the auto-tester do automated packaging. But for this particular tool, that appears to be a no-go as far as the auto-tester's admin is concerned. So again, either this gets used or it's DOA. If it does get used, then we can worry about making it perfect, in which case I'd be perfectly happy to do so. If not, then screw it we can fix our release process via the makefiles whenever anyone gets around to it. |
| if(do32Bit) | ||
| { | ||
| copyFile(cloneDir~"/dmd/src/dmd32"~exe, releaseBin32Dir~"/dmd"~exe); | ||
| copyDir(cloneDir~"/tools/generated/"~osDirName~"/32", releaseBin32Dir, file => !file.endsWith(obj)); |
There was a problem hiding this comment.
BTW this fails for me on linux 64.
True. I still think this tool is pretty good suited for our current release process, but the process itself is too complex, taking bits from too many locations to stuff everything into an all-in-one zip that nobody needs. |
|
On Mon, 14 Oct 2013 10:29:44 -0700
I do agree with that. The upside is that any improvements to the |
|
I'm unable to build 64-bit Linux on Debian. [...] Until now the binaries contained on the zip release package Walter generates are compiled on an specific machine. So, to avoid mistakes and make it easier it can be a good idea to generate temp zip files containing only the binaries generated on every platform. After that, place all these temp zip files on a single directory and generate the final release zip file using them. Another improvable point is to avoid to download all repositories every time we generate them in a new machine. Something like, if file "git-dmd-2.064.zip" is resent on current directory, use it, if not then download and generate this zip file, for the future buildings on other platforms. |
ghost
assigned 
|
Assigning this task to @dawgfoto. |
|
Could you also include a tar.gz version (and optionally .tar.bz2 too)? Linux distributions usually don't ship with 7z (yet). What you want from an installer is to make people's life easier, and require tools that are not installed everywhere just to save a few bytes seems counterproductive. I know zip works too in Linux, but it kind of sucks to use zip in Linux, the more natural option for Linux is tar.gz. Thanks for addressing this! |
|
I think that we need to replace the multi-platform dmd.zip with a platform specific binary release so that building can be fully automated. |
|
Nick (@Abscissa), can we get this in shape as intermediate solution for the next few releases so that Andrew can take over release building? |
|
On 12/4/2013 3:43 AM, Martin Nowak wrote:
It might take me a little time to get to it, since I've just moved and |
|
I might just make a pull request for your branch. |
|
Just wanted to check if there is any progress on this. I'm planning on producing the 2.065 betas on 17 Dec and this would help out tremendously. |
|
On 12/12/2013 8:28 AM, AndrewEdwards wrote:
I'm re-reading back through the comments/code refreshing my memory on |
- use std.zip for archiving and extracting - rename --archive-zip option to --archive and --combine-zip to --combine
|
Not sure exactly. Maybe my setup is just not accurate. I was able to build on OSX without issues. Not so lucky on Ubuntu or Fedora. I did not try on Windows or FreeBSD yet. On Fedora the following error occurs: Building DMD 32-bit Note that I have installed the gcc toolchain as required by the documentation. Even so, there is no g++ command on Fedora. The error on Ubuntu reads: I've verified that libcurl (first libcurl4-openssl-dev, then libcurl4-gnutls-dev) is installed. However, this did not solve the problem. Would be great if we could run from one OS and generate the binaries for all other OSes but alas that does not seem possible just yet. |
On Fedora you need to install |
|
Got it. Thanks. In that case, the error is the same as with Ubuntu. [andrew@tenken 065]$ sudo yum install libcurl Seems to be searching for a 32bit version of libcurl. Let me install that. Hmmm.... Everything seems to be working. 32 Bit build completed successfully. Now the 64bit version errors out with the same problems as before. That's annoying. Is there no way to install both 64 and 32 bit curl libraries on the same computer? Wait, that can't be the problem: "yum info libcurl" shows that both versions are installed. |
|
What about libcurl-devel? |
|
Ok. That works on Fedora. Not so lucky on Ubuntu. |
|
On Debian/Ubuntu is not possible to install 32 and 64 bit development curl library together due to dependencies, but no problem with run-time libcurl. @AndrewEdwards, can you try to remove any libcurl dev package, install 32 and 64 bit of libcurl3, and then link them with: |
|
Ouch, I've resurrected dlang/phobos#1772 which makes this simpler. |
create dmd release
|
Following errors encountered: create_dmd_release.d(1526): Error: undefined identifier CompressionMethod |
It requires the recent std.zip fixes, so either you can build the tool with a |
Add create_dmd_release utility.
|
@rainers came up with the idea that we can use the old dmd.zip releases to fetch the external binary dependencies. |
|
And we should use the ini files from the dmd repo. |
|
On 12/20/2013 7:16 PM, Martin Nowak wrote:
I specifically avoided making it do that to decrease the chance of Besides, if anything, those files really should just be under version |
|
On 12/21/2013 12:34 AM, Martin Nowak wrote:
Yea, definitely. The ini stuff wasn't in the repo at the time, so that's |
I started to build #!/bin/sh
set -e -o pipefail
# error function
ferror(){
echo "==========================================================" >&2
echo $1 >&2
echo $2 >&2
echo "==========================================================" >&2
exit 1
}
# check if too many parameters
if test $# -ne 1 ;then
ferror "Expected <git-branch-or-tag> as first argument, e.g. v2.064.2." "Exiting..."
fi
ROOT=${PWD}
TMPDIR=$(mktemp -d)
echo "using TMPDIR ${TMPDIR}"
cd ${TMPDIR}
VERSION=$1
for OS in freebsd; do
for MODEL in 32 64; do
cat > Vagrantfile <<EOF
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "create_dmd_release-${OS}-${MODEL}"
config.vm.box_url = "http://dlang.dawg.eu/vagrant/create_dmd_release-${OS}-${MODEL}.box"
# disable shared folders, because the guest additions are missing
config.vm.synced_folder ".", "/vagrant", :disabled => true
end
EOF
vagrant up
vagrant ssh-config > ssh.cfg
ssh -F ssh.cfg default ./create_dmd_release --extras=localextras-${OS} --archive --only-${MODEL} ${VERSION}
scp -F ssh.cfg default:dmd.${VERSION}.${OS}-${MODEL}.zip ${ROOT}/
vagrant destroy -f
done
done
cd ${ROOT}
rm -rf ${TMPDIR} |
Add create_dmd_release utility.
This is a tool to automatically generate both OS-specific and "all OSes" releases of DMD in both zip and 7z format, directly from the github sources.
The "how to" documentation is at the top of the source file:
https://github.com/Abscissa/installer/blob/create_dmd_release/create_dmd_release/create_dmd_release.d
NOTE: This tool "sort of" depends on itself actually being in the official repo. It will work fine before being pulled into the official repo, but the files in the "create_dmd_release/extras" won't be included in the releases it generates. Until this pull request gets merged, you can work around that like this:
After performing step #1 in the "Typical Usage" instructions at the top of the file (ie, extracting dmd-localextras.7z), then download the "create_dmd_release/extras" directory tree in this pull request. Merge both the "extras/all/dmd2" and "extras/[your os]/dmd2" trees into your "localextras-[your os]/dmd2" directory, and then proceed as normal.
I've tested this on the following platforms:
BTW, This won't work out-of-the-box for the current DMD releases, because it relies on several fixes/changes (mainly in makefiles) that currently only exist in master. (Ie, You can compile this tool just fine using DMD v2.063.2, but it won't generate a v2.063.2 release without some hacking.)