Archive sources for integration builds

[ricowind]

For easy access to the source code for a specific revision we should store a tarball of the sources for integration builds.

[ricowind]

cc @mkustermann.

[DartBot]

This comment was originally written by rod...@gmail.com

---

Without source releases, it's increadibly akward to package dart as an official Arch Linux package. Please provide a source tarball per release.

[anders-sandholm]

Removed Area-Build label.
Added Area-Infrastructure label.

[sgjesse]

Issue #15459 has been merged into this issue.

[sgjesse]

Supplying a tarball of the sources for building will be a lot of additional work. Besides the sources from the dart SVN repro there are the third_party directories, and on top of that additional platform specific dependencies are fetched form Google Cloud Storage. Also the third_party directory will contain stuff (like the d8 binaries) which should be filtered out when preparing a tarball.

The next issueare the makefiles. These are generated from the gyp-files also using gclient. The makefiles generated are suitable for distribution - they may contain absolute paths and other specific details. There are no guarantees of portability.

Last but not least we will need to test this additional distribution. This means additional build-bots for tracking this.

We would very much like it to be easy for Linux distributions to include Dart, by making the build process as smooth as possible using the current build-tools. The depot_tools (containing gclient and additional tools) is available both as an SVN checkout and a tarball, and the following commands should build the 64-bit runtime target:

  svn co http://src.chromium.org/svn/trunk/tools/depot_tools
  export PATH=$PATH:pwd/depot_tools
  gclient config http://dart.googlecode.com/svn/branches/bleeding_edge/deps/all.deps
  gclient sync
  gclient runhooks
  cd dart
  tools/build.py -mrelease -ax64 runtime

Building the SDK and editor require installing Java (openjdk-6-jdk) as well.

  sudo install openjdk-6-jdk
  tools/build.py -mrelease -ax64 create_sdk

The above uses bleeding_edge, change accordingly for the other channels:

  gclient config http://dart.googlecode.com/svn/trunk/deps/all.deps
  gclient config http://dart.googlecode.com/svn/branches/1.0/deps/all.deps

---

Removed Type-Defect label.
Added Type-Enhancement label.

[sgjesse]

In comment#­5

sudo install openjdk-6-jdk

should be

  sudo apt-get install openjdk-6-jdk

or

  sudo yum install java-1.6.0-openjdk-devel

depending on the distribution.

[ricowind]

It is possible to get specific revisions by changing:
gclient sync
to
gclient sync --revision REVISION --transitive

[DartBot]

This comment was originally written by beatg...@gmail.com

---

Has there been any update on this?

Dart has been in 1.x for over a year now, so perhaps external dependencies have settled down a bit, especially for just the Dart VM?

Source releases with minimal (hopefully zero) dependencies on forked projects would make porting a lot easier. As I've mentioned in issue #10260, I'm interested in porting to FreeBSD, but working with gclient (and to some extent gyp) is painful, especially since the build process is so bizarre and complicated.

[rmacnak-google]

Do the tarballs at, e.g., http://gsdview.appspot.com/dart-archive/channels/stable/raw/44672/src/ resolve this issue? The third party dependencies are included.

[DartBot]

This comment was originally written by @mulander

---

#­9 looks like it could.

I've been waiting a long time for this. I'm attempting to do an OpenBSD port, starting
as we speak.

The main question I have with your URL:

The chromium project hosts a https://gsdview.appspot.com/chromium-browser-official/ which nicely lists all release tarballs in a single place. This makes it easy for port maintainers to find the proper download links and allows the port infrastructure to do most of the work.

The URL you provided http://gsdview.appspot.com/dart-archive/channels/stable/raw/44672/src/ has an unpredictable path. How do I locate the proper release tarball for a dart version as you seem to include the revision in the URL and some of the other folders I looked at don't even contain the src folder?

[DartBot]

This comment was originally written by @mulander

---

The biggest difference I see between the Chrome build & dart is the handling of makefiles.

Doing the standard:
$ gclient config http://dart.googlecode.com/svn/branches/1.1/deps/all.deps
$ gclient sync
$ gclient runhooks

Results in a source tree that's ready to build with:
$ cd dart
$ tools/build.py --mode=release --arch=x64 create_sdk

Using the provided download URL both on Linux & OpenBSD
we do have a proper checkout with all third parties (major props)
but there's no way to continue with the build without significant
intervention:

Linux:
root@build:~/dart/dart-1.9.1/dart# python2.7 tools/build.py -m release --arch=x64
make -j 4 BUILDTYPE=ReleaseX64 all
make: *** No rule to make target `all'. Stop.
BUILD FAILED

OpenBSD:
$ make build
===> Building for dart-1.9.1
gmake -j 2 BUILDTYPE=ReleaseX64 all
gmake: *** No rule to make target 'all'. Stop.
BUILD FAILED
*** Error 1 in . (Makefile:70 'do-build': @­cd /usr/ports/pobj/dart-1.9.1/dart-1.9.1/dart && /usr/local/bin/python2.7 ./tools/build.py -m rel...)
*** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2747 '/usr/ports/pobj/dart-1.9.1/.build_done')
*** Error 1 in /usr/ports/mystuff/lang/dart (/usr/ports/infrastructure/mk/bsd.port.mk:2476 'build')
$

I cross checked with the www/chromium build on OpenBSD and it seems that Chromium provides a great build/linux/unbundle/replace_gyp_files.py which
allows the port to generate the required makefiles without going through the gclient stuff, from the chromium build:

gyp_chromium generates all the Makefiles from gyp

do-configure:
        cd ${WRKSRC} &&
                build/linux/unbundle/replace_gyp_files.py ${GYP_DEFS} &&
                env -i ${CONFIGURE_ENV} ${MODPY_BIN} build/gyp_chromium
                ${GYP_DEFS} "--depth=${WRKSRC}"

Is there a way to achieve the same with Dart?

[DartBot]

This comment was originally written by @mulander

---

OK I managed to generate the Makefiles without using gclient.

do-configure:
    @cd ${WRKSRC} && /usr/local/bin/python2.7 ./dart/tools/compiler_scripts/generate_my_projects.py &&
    /usr/local/bin/python2.7 ./dart/editor/build/generate_sources.py &&
    /usr/local/bin/python2.7 ./dart/tools/gyp_dart.py

$ make configure
===> Configuring for dart-1.9.1
$

So the tarball indeed looks workable. I guess this issue can be closed though a more predictable path for downloads would be appreciated).

[DartBot]

This comment was originally written by @mulander

---

OK one more issue. Why is third_party/clang not included? Are we forced to build with GCC?

$ ls third_party/clang
ls: third_party/clang: No such file or directory

From a manual gclient checkout:
$ ls third_party/clang/linux/lib/clang/3.4/include/
Intrin.h cpuid.h mm3dnow.h shaintrin.h tmmintrin.h
__wmmintrin_aes.h emmintrin.h mm_malloc.h smmintrin.h unwind.h
__wmmintrin_pclmul.h f16cintrin.h mmintrin.h stdalign.h varargs.h
altivec.h float.h module.map stdarg.h wmmintrin.h
ammintrin.h fma4intrin.h nmmintrin.h stdbool.h x86intrin.h
arm_neon.h fmaintrin.h pmmintrin.h stddef.h xmmintrin.h
avx2intrin.h immintrin.h popcntintrin.h stdint.h xopintrin.h
avxintrin.h iso646.h prfchwintrin.h stdnoreturn.h
bmi2intrin.h limits.h rdseedintrin.h tbmintrin.h
bmiintrin.h lzcntintrin.h rtmintrin.h tgmath.h

[ricowind]

First off, very nice that somebody is doing work the get us going on bsd.

Regarding clang, we explicitly exclude some binary dependencies due to size, see tools/create_tarball.py in a regular dart checkout. If this is a major blocker we can add it to the tarball, but you should be able to easily pull it of svn (will be git soon) if needed:
svn co http://dart.googlecode.com/svn/third_party/clang

Regarding predictable paths for download:
gsutil cat gs://dart-archive/channels/stable/release/*/VERSION
will get you the revision to version mappings
Or if you want urls to version mappings, something like:
gsutil ls gs://dart-archive/channels/stable/release/ | awk '{print "echo " $1 " && gsutil cat " $1 "VERSION | grep version" }' | sh

---

Added Fixed label.