v8 5.4.500.41

[Alhadis]

(Created with brew bump-formula-pr)

This is the V8 version used by the latest stable version of Chrome (54.0.2840.98). Hoping I did everything right.

Would it be possible to add a --HEAD option for the V8 formula? Work on the V8 engine moves fast, and some JS programmers (like me) prefer to benchmark their work against cutting-edge versions.

[DomT4]

They changed the entire build system, which is why nobody has touched this for a while. You'll need to look into that.

[Alhadis]

If I'm obsessed enough to hand-write a manpage for V8's command-line flags, I'm pretty confident that figuring out its build system falls within my motivational scope. :-) Will keep you posted.

[DomT4]

Heh, good luck 😉. If you need any help feel free to prod, I've tended to be the one handling V8 changes in Homebrew for a while, so if there's anything that confuses you in the formula I probably put it there 😅.

[apjanke]

Build failed at an inreplace step on a missing file.

        ==> Using the sandbox
/usr/bin/sandbox-exec -f /tmp/homebrew20161123-73783-qc3if3.sb nice /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/build.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/v8.rb --build-bottle --verbose
==> Downloading https://github.com/v8/v8/archive/5.4.500.41.tar.gz
Already downloaded: /Users/brew/Library/Caches/Homebrew/v8-5.4.500.41.tar.gz
==> Verifying v8-5.4.500.41.tar.gz checksum
tar xzf /Users/brew/Library/Caches/Homebrew/v8-5.4.500.41.tar.gz
Error: No such file or directory - tools/gyp/v8.gyp

This is consistent with the build system change that DomT4 mentioned.

You'll probably want to test and tweak this locally and get it building before resubmitting it to the PR.

[Alhadis]

Aye, I read something about a new build system in another (closed) PR. Will have a closer look at this tonight. =)

This might be the most patriotic PR I've ever proposed. Brews and V8 engines... can't get more Australian than that...

[ilovezfs]

@Alhadis any luck?

[Alhadis]

FUCK! Totally forgot!! Stayed up amending revision requests on a long overdue PR before crashing.

I'm normally pretty good at remembering if there's food on my plate, so to speak, so... embarrassing that I should've forgotten.

A'IGHT GOOGLE, LEMME AT YER SWEET SWEET JS ENGINE.

[Alhadis]

Question: Should I point to a specific commit for Google's depot_tools? I'm guessing the answer is "yes", but I'm not gonna go by guesswork.

[MikeMcQuaid]

Should I point to a specific commit for Google's depot_tools?

Yes, thanks.

[Alhadis]

Do we have facilities for checking out source through a third-party tool? Because this is actually a required part of Google's build step: gn actually refuses to build if it can't locate a "checkout" in the parent directory.

Note that checkout here is referring specifically to a repository that's been cloned using gclient, instead of simply any locally checked-out repo. I made the early mistake of thinking I could build after git cloneing V8, which is exactly what its wiki instructs you not to do:

Don't just git-clone either of these URLs if you want to build V8 from your checkout, instead follow the instructions below to get everything set up correctly.

Still halfway through re-checking-out V8; here's what I'm currently staring at in the terminal window:

λ Google-V8: fetch v8
Running: gclient root
Running: gclient config --spec 'solutions = [
  {
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
'
Running: gclient sync --with_branch_heads
Running: git submodule foreach 'git config -f $toplevel/.git/config submodule.$name.ignore all'
Running: git config --add remote.origin.fetch '+refs/tags/*:refs/tags/*'
Running: git config diff.ignoreSubmodules all

EDIT: Finished a minute or so after posting. Updated to include last three lines of feedback.

[ilovezfs]

system "gclient", "....

[Alhadis]

Ah. So... I can simply omit these lines then?

  url "https://github.com/v8/v8/archive/5.4.500.41.tar.gz"
  sha256 "463ce3e345d30b856b48f0ddf123b48d038c3af8c1098a7a6ad4c0e09c1eae0f"

[Alhadis]

Should I be concerned about pinned formulae (or users wanting specific V8 versions) when modifying the current build steps?

Virtually everything about the build process is different to the current one.

[ilovezfs]

Ah. So... I can simply omit these lines then?

Not unless they were being replaced with something else. It may be that v8 would benefit from a gclient DownloadStrategy.

Should I be concerned about pinned formulae (or users wanting specific V8 versions) when modifying the current build steps?

No, I don't think so.

[Alhadis]

Alright, thanks.

Seems like$PATH-fiddling is needed to get gclient building correctly. From the instructions of depot_tools:

3. Add depot_tools to your PATH:

export PATH=`pwd`/depot_tools:"$PATH"`

I assumed this last step was just a recommendation, but after running my first (successful) approach using relative paths, I was greeted with a handful of errors halfway through gclient sync:

Traceback (most recent call last):
  ...
  File "/Users/johngardner/Mirrors/Google-V8/depot_tools/subprocess2.py", line 262, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory.
Check that /Users/johngardner/Mirrors/Google-V8 or download_from_google_storage exist and have execution permission.

Some cursory snooping through depot_tools suggests the scripts are supposed to be in the user's $PATH, so would tampering with it (briefly) be acceptable?

Also, if it appears this is taking too long, it'll be because I'm waiting on checkouts and builds to finish downloading/compiling.

[ilovezfs]

Yes, you can use ENV.prepend_path or ENV.prepend_create_path

[Alhadis]

I'm not seeing a native-like build option. It's either x64.release or x86.release.

What would you recommend to determine if the formula's being built on a 64-bit or 32-bit machine?

[ilovezfs]

arch = MacOS.prefer_64_bit? ? "x64" : "x86"

[Alhadis]

Another stupid question: could I get clarification on what this is doing?

lib.install Dir["lib*"]
bin.install "d8", "mksnapshot", "process", "shell" => "v8"

First of all, the resulting contents of out.gn/x64.release differ from what the above snippet implies:

-rw-r--r--    1  603K .ninja_deps
-rw-r--r--    1   85K .ninja_log
-rw-r--r--    1   36B args.gn
-rw-r--r--    1  8.1K build.ninja
-rw-r--r--    1  3.3K build.ninja.d
-rwxr-xr-x    1   37M cctest
-rwxr-xr-x    1   17M d8
drwxr-xr-x   15  510B gen
-rwxr-xr-x    1   17M generate-bytecode-expectations
-rw-r--r--    2  9.7M icudtl.dat
-rw-r--r--    1    2B mb_type
-rwxr-xr-x    1   51K mkpeephole
-rwxr-xr-x    1   17M mksnapshot
-rw-r--r--    1  313K natives_blob.bin
drwx------  107  3.6K obj
-rw-r--r--    1  822K snapshot_blob.bin
-rw-r--r--    1   16K toolchain.ninja
-rwxr-xr-x    1   34M unittests
-rw-r--r--    1  248B v8_build_config.json
-rwxr-xr-x    1   17M v8_hello_world
-rwxr-xr-x    1   17M v8_parser_shell
-rwxr-xr-x    1   17M v8_sample_process
-rwxr-xr-x    1   17M v8_shell
-rwxr-xr-x    1   17M v8_simple_json_fuzzer
-rwxr-xr-x    1   17M v8_simple_parser_fuzzer
-rwxr-xr-x    1   17M v8_simple_regexp_fuzzer
-rwxr-xr-x    1   17M v8_simple_wasm_asmjs_fuzzer
-rwxr-xr-x    1   17M v8_simple_wasm_fuzzer

Secondly, I wasn't aware that mksnapshot and process were being installed by Homebrew as part of V8. The output of mksnapshot --help suggests it's not expected to be invoked directly:

SSE3=1 SSE4_1=1 SAHF=1 AVX=1 FMA3=1 BMI1=1 BMI2=1 LZCNT=1 POPCNT=1 ATOM=0
Usage:
  shell [options] -e string
    execute string in V8
  shell [options] file1 file2 ... filek
    run JavaScript scripts in file1, file2, ..., filek
  shell [options]
  shell [options] --shell [file1 file2 ... filek]
    run an interactive JavaScript shell
  d8 [options] file1 file2 ... filek
  d8 [options]
  d8 [options] --shell [file1 file2 ... filek]
    run the new debugging shell

Are these two binaries needed for something?

[ilovezfs]

lib.install Dir["lib*"]

means

mkdir -p /usr/local/Cellar/v8/5.4.500.41/lib
mv lib* /usr/local/Cellar/v8/5.4.500.41/lib

bin.install "d8", "mksnapshot", "process", "shell" => "v8"

means

mkdir -p /usr/local/Cellar/v8/5.4.500.41/bin
mv d8 mksnapshot process /usr/local/Cellar/v8/5.4.500.41/bin
mv shell /usr/local/Cellar/v8/5.4.500.41/bin/v8

Are these two binaries needed for something?

no idea

[Alhadis]

Thanks!

What's the lib folder used for? Static linking with other formula?

[ilovezfs]

Library files which may be dynamic (foo.dylib and occasionally foo.so) or static (foo.a), and which may or may not be used by any other formulae.

brew linkage foo

shows what libraries a given formula uses.

[Alhadis]

Well, these are the only .a files in out.gn/x64_release:

./obj/build/config/sanitizers/liboptions_sources.a
./obj/testing/gmock/libgmock.a
./obj/testing/gmock/libgmock_main.a
./obj/testing/gtest/libgtest.a
./obj/third_party/icu/libicui18n.a
./obj/third_party/icu/libicuuc.a

Whereas the contents of /usr/local/Cellar/v8/5.1.281.47/lib/ are:

libfuzzer_support.a
libjson_fuzzer_lib.a
libparser_fuzzer_lib.a
libregexp_fuzzer_lib.a
libv8.dylib
libv8_base.a
libv8_libbase.a
libv8_libplatform.a
libv8_nosnapshot.a
libv8_snapshot.a
libwasm_asmjs_fuzzer_lib.a
libwasm_fuzzer_lib.a

So I'm unsure what to do. Here're the full contents of out.gn/x64.release.

[ilovezfs]

The old make file had an option library=shared that we were passing, but there aren't any shared libraries in the list in your pastebin. Maybe there's a new way of specifying shared now?

[Alhadis]

Ah, that would be a component build then, I guess.

I'll try rebuilding with it.

[Alhadis]

No luck. Even with is_component_build=true added to the args.gn file, there're still no lib* files.

So far, the only solution seems to be running make native library=shared after the initial GN build, which is essentially repeating the procedure using gyp.

[ilovezfs]

@Alhadis go ahead and push what you have so far please and I'll take a look at what you mean.

[Alhadis]

Scratch what I said about make native library=shared; looks like a dead-end after all:

dyld: Library not loaded: /usr/local/lib/libicui18n.dylib
  Referenced from: /Users/johngardner/Sandbox/v8/out/native/mksnapshot
  Reason: image not found
/bin/sh: line 1: 54184 Abort trap: 6           "/Users/johngardner/Sandbox/v8/out/native/mksnapshot" --random-seed 314159265 --startup_blob "/Users/johngardner/Sandbox/v8/out/native/snapshot_blob.bin" ""
make[1]: *** [/Users/johngardner/Sandbox/v8/out/native/snapshot_blob.bin] Error 134
make: *** [native] Error 2

Nothing to push yet; I wanted to verify the build generated everything before integrating it into the formula. Here're the steps I'm running, which assume depot_tools is installed and available in your $PATH:

# Checkout repository
gclient root
gclient config --spec 'solutions = [
  {
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
target_os = [ "mac" ]
target_os_only = True
'
gclient sync -r 5.4.500.41
git submodule foreach 'git config -f $toplevel/.git/config submodule.$name.ignore all'
git config --add remote.origin.fetch '+refs/tags/*:refs/tags/*'
git config diff.ignoreSubmodules all

# Configure build
tools/dev/v8gen.py x64.release
ninja -C out.gn/x64.release
tools/run-tests.py --outdir out.gn/x64.release

You can omit the last line if you're not interested in running tests.

[Alhadis]

I'll just leave it as-is, actually. I only thought the shared library was getting in the way of the final build, simply because I'd overlooked the fact it relied on the missing dylib at runtime.

I'm still convinced there's a way of generating these files with args.gn. From what I'm getting from GN's documentation, it should be possible to generate static libraries (.a files) if a target requests them:

Static libraries involve duplicating all of the data in the object files that comprise it. This takes more disk space and for certain very large libraries in configurations with very large object files can cause internal limits on the size of static libraries to be exceeded. Source sets do not have this limitation. Some targets switch between source sets and static libraries depending on the build configuration to avoid this problem.

For shared libraries, though, we'll still need a separate build pass.

[Alhadis]

@DomT4 When dealing with V8 builds, have you ever had to deal with it needing natives_blob.bin and
snapshot_blob.bin before?

I'm wondering if this is new to the GN-era build process, because the v8_shell really, really seems to want those bin files nearby (unless v8_use_external_startup_data is off).

[DomT4]

When dealing with V8 builds, have you ever had to deal with it needing natives_blob.bin and
snapshot_blob.bin before?

Yes, been down that path before, which is why:

ENV.append "GYP_DEFINES", "v8_use_external_startup_data=0"

has lived in the formula for a while, since Homebrew/legacy-homebrew#44976. Not only did it need those files, but it was fussy about where they lived to boot.

[DomT4]

From the code comment there you highlighted in # If not specified as a gn arg, use external startup data by default it looks like there's probably an override for that, which you'd likely want to replace the existing ENV from the formula's current gyp-based build.

[Alhadis]

Well, I'm partly relieved to know this isn't a new issue. =)

it looks like there's probably an override for that, which you'd likely want to replace the existing ENV when it still used gyp to build.

To avoid introducing new problems, I wiped the slate clean of ENV-wrangling and kept things simple.

Also, before I forget to bring them up, here're some random things I've learned spelunking through Google's cavernous build-dungeon:

• The ability to point to a system-based ICU is gone. depot_tools is fully responsible for provisioning the icu4c library. We can't simply symlink third_party/icu to a Homebrew-installed icu4c either, because depot_tools adds numerous gn/gyp files to its icu dependency. I read about the kerfuffle with ICU versions in node and icu4c Yosemite bottles have different C++ standard libraries legacy-homebrew#36681, so I'm hoping this won't cause problems...

• Editing the gni/v8.gni file should be fine. Having learned the difference between static libraries and what GN calls source maps, I doubt we're adding build complications by molesting the source code. A more "canon" approach would be to define new targets in a separate .gn file, but screw it, this is easier.

• Checkout through depot_tools is killing me. It's not being cached by Homebrew, so subsequent brew reinstalls require the entire repo to be redownloaded into a separate /tmp/ folder, etc... absolute pain-in-the-khyber. Currently seeing if there's a way to trick gclient into syncing over an existing repository, because this shit's ridiculous. THANK CRAP, it's working. @ilovezfs, no need to cheat, we can keep our integrity after all.

EDIT: Oh yeah, almost forgot this, too: I'm getting the same compiler warning mentioned here while building, but there've been no problems... I dismissed it as a non-issue, but you guys might know if it's worth concern:

ld: warning: directory not found for option '-L../../third_party/libc++-static'

[DomT4]

To avoid introducing new problems, I wiped the slate clean of ENV-wrangling and kept things simple.

IIRC v8 finally recently moved to libc++ rather than libstdc++ by default so certainly we should be able to permanently bin the ENV.cxx11 stuff, as well as the mac_deployment_target which was essentially another buffer to force it to stop linking against libstdc++.

The ability to point to a system-based ICU is gone

This doesn't surprise me. They've been leaning that way for a while. I think it's fine for v8 to statically link in its own icu4c or whatever it's doing if it wants to, personally.

[Alhadis]

Two less headaches to worry about then. =) Good to hear.

I've spent nearly all day digging through GN docs and configs to find a safe way of setting the rpath that's embedded in the dylib files. Which is really all that's stopping me from pushing an updated formula for review.

There's no way to change this at a configuration level, however. Existing targets can't be modified outside their definition. So we're basically reduced to patching a .gn file somewhere instead of tools/gyp/v8.gyp.

So far, that hasn't been turning out too well:

ninja -C out.gn/x64.release
ninja: Entering directory `out.gn/x64.release'
[924/1254] ACTION //:run_mksnapshot(//build/toolchain/mac:clang_x64)
FAILED: gen/snapshot.cc 
python ../../tools/run.py ./mksnapshot --startup_src gen/snapshot.cc --random-seed 314159265
dyld: Library not loaded: @rpath/libicui18n.dylib
  Referenced from: /Users/johngardner/Sandbox/v8/out.gn/x64.release/./mksnapshot
  Reason: image not found
ninja: build stopped: subcommand failed.
λ v8 ((HEAD detached at 5.4.500.41)): 

This snarky bullshit hasn't done anything for my expectations, either:

ERROR at build arg file (use "gn args <out_dir>" to edit):9:1: Invalid token.
	"-Wl,-rpath=/usr/local/opt/v8/lib"
^
You got a tab character in here. Tabs are evil. Convert to spaces.
ninja: error: rebuilding 'build.ninja': subcommand failed

[Alhadis]

Is there a way to download dependent resources to a directory other than buildpath? depot_tools is being downloaded to the same directory as v8...

(If I don't answer, it's because I've finally crashed and fallen asleep)

[Alhadis]

Okay, I tried installing outside buildpath (with /../), and got an "Access forbidden" error. Makes sense.

@ilovezfs Is it possible to download the formula's source to a subdirectory in the buildpath? depot_tools works under the assumption that a .gclient file will be located in the same directory as the v8 directory. Since it's creating the .gclient config inside the V8 repo itself, it's downloading everything twice...

[ilovezfs]

@Alhadis can you come on IRC? (freenode #machomebrew) https://www.irccloud.com/#!/ircs://irc.freenode.net:6697/%23machomebrew

[pinepain]

Any progress on this?

I'm planning to upgrade v8 to 5.7 (sure, not in core, I maintain own formulae for development purposes - https://github.com/pinepain/packaging/tree/master/homebrew) and as build system changed I have to alter packaging scripts, so to not duplicate our efforts it would be wise to coordinate the work. Are there anything I can assist with?

[Alhadis]

Hey mate!

I've actually been waiting on @ilovezfs to give us (hopefully good) news on the progress of the building phase. I kinda entrusted him to finish the build procedure, as he's a hell of a lot more familiar with Homebrew's internals than I am.

@ilovezfs How'd it go with the static libraries?

[pinepain]

@Alhadis are static libraries a must nowadays? In a thread you linked (https://groups.google.com/forum/#!topic/v8-users/Y05xPj956Ys) Jochen Eisinger points that now libplatform could be built as shared. I'll give a try my php-v8 extension to build without static libraries, but before it was not possible because libplatform was built only statically, afair.

Did you go with custom depot download strategy or how you handle sources download process?

[pinepain]

Confirmed with hello_world_build_osx.sh. Now we can build against that shared libraries. It looks like we don't need static libs unless there are some specific requirements (which I'm not aware of).

[Alhadis]

I'm not sure why the static libraries are being used, to be honest... @ilovezfs, status?

I got an e-mail telling me my IRCCloud trial expired and I have to fork out money I don't have in order to upgrade, so, hope you won't be needing too much of my input, haha.

[pinepain]

@Alhadis they are not required anymore, at least I can us dynamic one as of 5.7 with php-v8, try upgrade-to-v8-5.7 branch if you curious (note, it's heavy WIP branch). I'm not sure about 5.6, but if you see sth like libbase.dylib, static is not required. I'll push some of my changes to own co-installable v8 I use for php-v8 later today so you can adapt your changes to it (basically, it's quite the same you already have, but co-installable).

[pinepain]

@ilovezfs @DomT4 We discussed it before, but on more time: what about having multiple v8 versions at once, like php tap does? Make it co-installable, keg-only and let depends on any of v8-<major><minor> formula name rather then rely on formula version. V8 doesn' follow semantic versioning and even within minor version BC-breaking changes occurs, so having single v8 formula just ties all other products that use it together in terms of v8 features to use (or not to use). I already follow this approach in own tap (will push changes later today) and it looks more easier to maintain dependant formulas as well as it very convenient for development purposes.

[ilovezfs]

@pinepain let's not discuss that here please. If you want to discuss versioning, join the party on Homebrew/brew#620.

[pinepain]

@ilovezfs wow, somehow missed that, thanks for pointing me out!

[pinepain]

Hi, as I noted earlier, here is my v8@5.7 formula which works just fine. I also use env variables to specify whether to run or not tests as well as whether to use v8 sources caching (to speedup development and debugging).

Key difference from core v8 formula is that I build vanilla v8 with icu and do not patch anything. d8 installed alongside libs as it doesn't intended for any usage except development purposes.

[mathiasbynens]

#9343 updates V8 to v5.8.48 and adds a --HEAD option.