Benchmark against SWC / pax

[dy]

There are bundlers in Rust: pax and swc.

Would that be interesting to compare their performance too?

[kdy1]

@dy Currently, swc does not have a feature to bundle files.
I'm working on it, but it would take some time.

[evanw]

So I'm not at all familiar with SWC and Pax but it seems like these tools don't do the same thing as the other tools that I'm benchmarking. To be relevant for my benchmark, the build tool must be able to do a full production build.

SWC's minification seems to be whitespace-only as far as I can tell? Pax appears to be only for development and the output is too verbose for a production build.

The smallest size I could get for SWC and Pax with minification enabled is 15mb. This is far bigger than the other tools in my benchmark which all generate bundles around 5.8mb. When it worked, the build time for running SWC followed by Pax was around 2.5 seconds, which seems promising. SWC logs 2,530 errors to the console during the build process though so it's possible that a big part of the build time is being spent doing console I/O, and the build time would be much better if it didn't log so many errors.

SWC also appears to be somewhat unstable. When I run it on my benchmark, around 50% of the time it panics and crashes instead of completing successfully. The specific panic is fatal runtime error: failed to initiate panic, error 5. No idea what that means.

I'm going to close this issue because I believe SWC and Pax are not yet appropriate tools for production builds, and so aren't relevant for my benchmark.

[kdy1]

@evanw Can you share the code which crashed swc?
Also, I think you were targetting es3, and errors comes from the getter / setters.

[evanw]

Can you share the code which crashed swc?

Of course:

$ git clone git@github.com:evanw/esbuild.git
$ make bench/three
$ npm i @swc/core@1.1.24 @swc/cli@0.1.23
$ npx swc bench/three -d dist -s
fatal runtime error: failed to initiate panic, error 5
Abort trap: 6

Also, I think you were targetting es3, and errors comes from the getter / setters.

The errors all looked like this:

error: Expected LBrace, got Some(Word(namespace))
 --> bench/copy1/Three.Legacy.d.ts:6:8
  |
6 | export namespace SceneUtils {
  |        ^^^^^^^^^

error: Unexpected token Some(Colon)
 --> bench/copy1/animation/AnimationAction.d.ts:8:6
  |
8 |     loop: AnimationActionLoopStyles;
  |         ^

error: Expected LBrace, got Some(Word(as))
   --> bench/copy1/Three.d.ts:180:8
    |
180 | export as namespace THREE;
    |        ^^

[kdy1]

@evanw Thanks! I'll look at it.

[kdy1]

I failed to reproduce the crash.

~/projects/esbuild   master ●  npx swc bench/three -d dist -s        
error: Expected Word(from), got Some(LBrace)
 --> bench/three/entry.js:2:17
  |
2 | import * as copy{1..10} from './copy{1..10}/Three.js'; export {copy{1..10}}
  |                 ^

Successfully compiled 609 files with swc.

~/projects/esbuild   master ●  cat ./bench/three/entry.js

import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}
import * as copy23223i from './copy23223i/Three.js'; export {copy23223i}

~/projects/esbuild   master ●  npx swc bench/three -d dist -s
Successfully compiled 610 files with swc.

Can you share more information about your testing? Like how much time you invoked, or operating system you are using.

[evanw]

There's a for loop in the Makefile that copies three.js ten times:

esbuild/Makefile

Lines 101 to 102 in 697e0f5

	for i in {1..10}; do test -d "bench/three/copy$$i" || cp -r github/three/src "bench/three/copy$$i"; done
	for i in {1..10}; do echo "import * as copy$$i from './copy$$i/Three.js'; export {copy$$i}" >> bench/three/entry.js; done

I'm guessing your shell or version of make is different than mine and it's not interpreting the for loop the same way? I'm using bash 3.2.57(1)-release and make 3.81 on macOS 10.14.6. What about you?

[kdy1]

My os is ubuntu and I'm using bash (with zsh). I'll try it on my mac.

[kdy1]

I successfully reproduced it, but it goes away when I invoked with --sync. Anyway, thank you for finding a bug.
Bugs like this are really hard to find (crashes only on osx + async mode)

[evanw]

Cool, glad I could help! It's great that it wasn't hard to reproduce.

And thanks for the info about zsh. I'll modify the for loop so it works in both bash and zsh.

[kdy1]

I fixed the random panic, and ran it.

(Note: swc is a compiler, and it transforms code instead of bundling it)

$ rm -rf ./dist && time npx swc bench/three -d dist -s 
Successfully compiled 6091 files with swc.
npx swc bench/three -d dist -s  10.72s user 2.66s system 245% cpu 5.446 total

5.446 seconds / 6091 file = 0.00089410605 seconds per file = 0.894ms / file.

[ForsakenHarmony]

Is that with es3 compilation now?

[kdy1]

Is that with es3 compilation now?

I targeted es5.

[evanw]

By the way I'm currently tracking adding spack to my benchmarks with #762. I recently posted an update about this in case anyone on this thread is interested: #762 (comment). Follow that thread for further updates.