Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use performance.now when possible, re-disable unconditional perf marking on all Node versions #57875

Merged
merged 9 commits into from
Mar 22, 2024

Conversation

jakebailey
Copy link
Member

@jakebailey jakebailey commented Mar 21, 2024

Now that we're testing different OSs, Node 14 on Windows spuriously fails, like:

diff --git a/./tests/baselines/reference/tsserver/fourslashServer/nodeNextPathCompletions.js b/./tests/baselines/local/tsserver/fourslashServer/nodeNextPathCompletions.js
index 50e476c62..5a9909bcd 100644
--- a/./tests/baselines/reference/tsserver/fourslashServer/nodeNextPathCompletions.js
+++ b/./tests/baselines/local/tsserver/fourslashServer/nodeNextPathCompletions.js
@@ -1215,7 +1215,7 @@ Info seq  [hh:mm:ss:mss] request:
     }
Info seq  [hh:mm:ss:mss] Starting updateGraphWorker: Project: /tsconfig.json
Info seq  [hh:mm:ss:mss] FileWatcher:: Added:: WatchInfo: /node_modules/dependency/lib/package.json 2000 undefined Project: /tsconfig.json WatchType: File location affecting resolution
-Info seq  [hh:mm:ss:mss] Finishing updateGraphWorker: Project: /tsconfig.json projectStateVersion: 2 projectProgramVersion: 1 structureChanged: true structureIsReused:: SafeModules Elapsed:: *ms
+Info seq  [hh:mm:ss:mss] Finishing updateGraphWorker: Project: /tsconfig.json projectStateVersion: 2 projectProgramVersion: 1 structureChanged: true structureIsReused:: SafeModules Elapsed: -1ms
Info seq  [hh:mm:ss:mss] Project '/tsconfig.json' (Configured)
Info seq  [hh:mm:ss:mss]   Files (2)
  /node_modules/dependency/lib/index.d.ts Text-1 "export function fooFromIndex(): void;"
@@ -1242,7 +1242,7 @@ Info seq  [hh:mm:ss:mss] response:
       "request_seq": 28,
       "success": true,
       "performanceData": {
-        "updateGraphDurationMs": *
+        "updateGraphDurationMs": -1
       },
       "body": {
         "isGlobalCompletion": false,
Error: Process completed with exit code 1.

https://github.com/microsoft/TypeScript/actions/runs/8367516104/job/22910040628?pr=57870#step:6:1

Somehow, time went backwards???

My guess is that after #50267, native performance hooks only work in node 16.7+ since we wholesale disable native performance support when some parts of it are missing. This also disables the use of performance.now, meaning Node 14 is falling back to Date.now() for timestamp() even though performance.now() would have worked. Now, why Date.now() would ever go backwards a millisecond, I have no idea, but the flag isn't happening anywhere other than Node 14, and 14 -> 16 is where timestamp() differs too. Surely not a coincidence...

This PR restructures things such that we can still use native performance time functions when available, but still disable the other perf stuff when the mark/measure support is incomplete.

Also, while here, don't check for PerformanceObserver, which we haven't used since #42586.

Funnily, this has exposed the fact that we have been accidentally enabling native perf marking on Node 20+, because it implements all of the global perf hooks we need and so we "think" it's a browser.

This PR started out as slow because I removed the check for PerformanceObserver, which then made Node 18 also satisfy the checks via globals, so we slowed down and that pointed out the problem.

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Mar 21, 2024
@jakebailey
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 21, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this faster ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,619k (± 0.00%) 298,410k (± 0.02%) +2,790k (+ 0.94%) 298,355k 298,474k p=0.005 n=6
Parse Time 2.66s (± 0.28%) 2.68s (± 0.45%) +0.01s (+ 0.56%) 2.66s 2.69s p=0.046 n=6
Bind Time 0.83s (± 0.49%) 0.83s (± 0.76%) ~ 0.82s 0.84s p=0.673 n=6
Check Time 8.22s (± 0.31%) 8.20s (± 0.43%) ~ 8.17s 8.26s p=0.224 n=6
Emit Time 7.04s (± 0.29%) 7.17s (± 0.56%) +0.13s (+ 1.89%) 7.14s 7.25s p=0.005 n=6
Total Time 18.75s (± 0.20%) 18.88s (± 0.22%) +0.13s (+ 0.70%) 18.85s 18.95s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,215k (± 1.03%) 193,229k (± 0.94%) ~ 191,960k 195,682k p=0.471 n=6
Parse Time 1.35s (± 0.90%) 1.37s (± 0.97%) +0.01s (+ 0.98%) 1.36s 1.39s p=0.027 n=6
Bind Time 0.72s (± 0.00%) 0.72s (± 0.00%) ~ 0.72s 0.72s p=1.000 n=6
Check Time 9.54s (± 0.62%) 9.53s (± 1.02%) ~ 9.41s 9.65s p=1.000 n=6
Emit Time 2.65s (± 0.40%) 2.64s (± 0.28%) ~ 2.63s 2.65s p=0.611 n=6
Total Time 14.26s (± 0.45%) 14.26s (± 0.70%) ~ 14.12s 14.40s p=0.936 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,380k (± 0.01%) 348,709k (± 0.00%) +1,329k (+ 0.38%) 348,689k 348,734k p=0.005 n=6
Parse Time 2.48s (± 0.40%) 2.51s (± 0.41%) +0.03s (+ 1.14%) 2.49s 2.52s p=0.005 n=6
Bind Time 0.93s (± 0.56%) 0.93s (± 0.00%) ~ 0.93s 0.93s p=0.174 n=6
Check Time 7.02s (± 0.25%) 7.03s (± 0.28%) ~ 7.01s 7.06s p=0.511 n=6
Emit Time 4.07s (± 0.39%) 4.07s (± 0.29%) ~ 4.06s 4.09s p=0.738 n=6
Total Time 14.50s (± 0.09%) 14.53s (± 0.10%) +0.04s (+ 0.25%) 14.51s 14.55s p=0.010 n=6
TFS - node (v18.15.0, x64)
Memory used 302,732k (± 0.01%) 303,165k (± 0.01%) +433k (+ 0.14%) 303,139k 303,198k p=0.005 n=6
Parse Time 2.01s (± 0.89%) 2.02s (± 0.94%) ~ 2.00s 2.04s p=0.411 n=6
Bind Time 1.00s (± 0.89%) 1.01s (± 1.36%) ~ 0.99s 1.03s p=0.452 n=6
Check Time 6.31s (± 0.36%) 6.33s (± 0.49%) ~ 6.30s 6.38s p=0.372 n=6
Emit Time 3.59s (± 0.52%) 3.61s (± 0.62%) ~ 3.57s 3.63s p=0.222 n=6
Total Time 12.92s (± 0.32%) 12.97s (± 0.16%) ~ 12.94s 13.00s p=0.054 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,877k (± 0.00%) 511,243k (± 0.01%) +1,367k (+ 0.27%) 511,188k 511,319k p=0.005 n=6
Parse Time 2.66s (± 0.48%) 2.68s (± 0.75%) ~ 2.65s 2.70s p=0.121 n=6
Bind Time 0.99s (± 0.64%) 0.99s (± 0.76%) ~ 0.98s 1.00s p=0.718 n=6
Check Time 17.23s (± 0.26%) 17.32s (± 0.48%) ~ 17.20s 17.44s p=0.077 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.88s (± 0.20%) 20.99s (± 0.39%) +0.11s (+ 0.53%) 20.85s 21.09s p=0.037 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,719,650k (± 0.00%) 1,727,300k (± 0.00%) +7,650k (+ 0.44%) 1,727,251k 1,727,317k p=0.005 n=6
Parse Time 6.57s (± 0.21%) 6.67s (± 0.37%) +0.10s (+ 1.55%) 6.64s 6.71s p=0.005 n=6
Bind Time 2.35s (± 0.32%) 2.35s (± 0.95%) ~ 2.33s 2.39s p=1.000 n=6
Check Time 56.24s (± 0.27%) 56.36s (± 0.43%) ~ 56.03s 56.71s p=0.378 n=6
Emit Time 0.13s (± 3.10%) 0.14s (± 4.05%) ~ 0.13s 0.14s p=0.282 n=6
Total Time 65.29s (± 0.26%) 65.51s (± 0.37%) ~ 65.19s 65.88s p=0.093 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,391,694k (± 0.03%) 2,432,691k (± 0.03%) +40,998k (+ 1.71%) 2,431,805k 2,433,855k p=0.005 n=6
Parse Time 5.01s (± 1.43%) 5.10s (± 0.85%) ~ 5.05s 5.18s p=0.066 n=6
Bind Time 1.89s (± 0.85%) 1.88s (± 0.94%) ~ 1.84s 1.89s p=0.152 n=6
Check Time 33.61s (± 0.59%) 33.58s (± 0.44%) ~ 33.38s 33.81s p=0.689 n=6
Emit Time 2.64s (± 1.04%) 2.66s (± 1.53%) ~ 2.61s 2.73s p=0.335 n=6
Total Time 43.17s (± 0.49%) 43.24s (± 0.36%) ~ 43.03s 43.46s p=0.748 n=6
self-compiler - node (v18.15.0, x64)
Memory used 415,080k (± 0.01%) 415,369k (± 0.00%) +289k (+ 0.07%) 415,339k 415,390k p=0.005 n=6
Parse Time 2.83s (± 1.10%) 2.80s (± 2.65%) ~ 2.67s 2.88s p=0.746 n=6
Bind Time 1.07s (± 0.70%) 1.10s (± 6.01%) ~ 1.06s 1.23s p=0.867 n=6
Check Time 15.27s (± 0.34%) 15.27s (± 0.39%) ~ 15.20s 15.34s p=1.000 n=6
Emit Time 1.12s (± 1.49%) 1.12s (± 1.31%) ~ 1.10s 1.14s p=0.802 n=6
Total Time 20.29s (± 0.31%) 20.29s (± 0.20%) ~ 20.24s 20.34s p=0.747 n=6
vscode - node (v18.15.0, x64)
Memory used 2,887,853k (± 0.00%) 2,897,138k (± 0.00%) +9,285k (+ 0.32%) 2,897,047k 2,897,208k p=0.005 n=6
Parse Time 10.86s (± 0.47%) 10.89s (± 0.24%) ~ 10.87s 10.93s p=0.257 n=6
Bind Time 3.47s (± 0.42%) 3.46s (± 0.58%) ~ 3.43s 3.49s p=0.512 n=6
Check Time 61.63s (± 0.66%) 61.60s (± 0.43%) ~ 61.26s 61.88s p=0.810 n=6
Emit Time 17.62s (±10.23%) 16.57s (± 0.91%) ~ 16.35s 16.77s p=1.000 n=6
Total Time 93.58s (± 2.30%) 92.52s (± 0.24%) ~ 92.25s 92.76s p=0.689 n=6
webpack - node (v18.15.0, x64)
Memory used 408,069k (± 0.01%) 409,269k (± 0.02%) +1,200k (+ 0.29%) 409,176k 409,418k p=0.005 n=6
Parse Time 3.22s (± 0.32%) 3.25s (± 0.37%) +0.02s (+ 0.67%) 3.23s 3.26s p=0.018 n=6
Bind Time 1.38s (± 0.71%) 1.40s (± 1.31%) ~ 1.38s 1.42s p=0.059 n=6
Check Time 14.27s (± 0.22%) 14.29s (± 0.36%) ~ 14.21s 14.34s p=0.376 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.87s (± 0.16%) 18.94s (± 0.28%) +0.07s (+ 0.36%) 18.87s 19.02s p=0.029 n=6
xstate - node (v18.15.0, x64)
Memory used 513,012k (± 0.02%) 513,845k (± 0.02%) +833k (+ 0.16%) 513,697k 513,967k p=0.005 n=6
Parse Time 3.28s (± 0.30%) 3.29s (± 0.31%) +0.01s (+ 0.46%) 3.28s 3.31s p=0.028 n=6
Bind Time 1.54s (± 0.54%) 1.54s (± 0.00%) ~ 1.54s 1.54s p=0.176 n=6
Check Time 2.85s (± 0.14%) 2.86s (± 1.04%) ~ 2.83s 2.91s p=0.929 n=6
Emit Time 0.07s (± 0.00%) 0.07s (± 0.00%) ~ 0.07s 0.07s p=1.000 n=6
Total Time 7.75s (± 0.24%) 7.76s (± 0.44%) ~ 7.73s 7.81s p=0.747 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@RyanCavanaugh
Copy link
Member

Many years ago I encountered similar behavior where the time reported by each core on the CPU was slightly different (this was C# code and the behavior didn't repro when I set the thread processor affinity). Unclear if this was a hardware bug or just not an invariant under Windows.

@jakebailey
Copy link
Member Author

Yeah, I'm pretty sure that performance.now() is a completely different clock that's monotomic and everything (hence why it has its own timeOrigin).

@jakebailey
Copy link
Member Author

Going to rerun this because that result does not seem right at all

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 21, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this faster ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,630k (± 0.01%) 298,424k (± 0.01%) +2,794k (+ 0.95%) 298,355k 298,449k p=0.005 n=6
Parse Time 2.67s (± 0.31%) 2.68s (± 0.50%) ~ 2.66s 2.70s p=0.056 n=6
Bind Time 0.83s (± 0.76%) 0.83s (± 0.49%) ~ 0.82s 0.83s p=0.673 n=6
Check Time 8.19s (± 0.16%) 8.21s (± 0.15%) +0.02s (+ 0.22%) 8.19s 8.22s p=0.040 n=6
Emit Time 7.03s (± 0.31%) 7.18s (± 0.53%) +0.15s (+ 2.16%) 7.13s 7.24s p=0.005 n=6
Total Time 18.71s (± 0.16%) 18.90s (± 0.21%) +0.19s (+ 1.01%) 18.85s 18.96s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 192,496k (± 0.74%) 194,429k (± 0.85%) +1,934k (+ 1.00%) 192,282k 195,618k p=0.020 n=6
Parse Time 1.37s (± 0.65%) 1.37s (± 0.92%) ~ 1.36s 1.39s p=0.933 n=6
Bind Time 0.72s (± 0.00%) 0.72s (± 0.57%) ~ 0.72s 0.73s p=0.405 n=6
Check Time 9.50s (± 0.98%) 9.49s (± 0.80%) ~ 9.38s 9.57s p=0.688 n=6
Emit Time 2.64s (± 0.59%) 2.64s (± 0.57%) ~ 2.63s 2.67s p=0.868 n=6
Total Time 14.23s (± 0.58%) 14.23s (± 0.52%) ~ 14.13s 14.31s p=0.873 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,385k (± 0.01%) 348,708k (± 0.00%) +1,323k (+ 0.38%) 348,686k 348,725k p=0.005 n=6
Parse Time 2.49s (± 0.47%) 2.51s (± 0.25%) +0.02s (+ 0.87%) 2.50s 2.52s p=0.007 n=6
Bind Time 0.93s (± 0.00%) 0.93s (± 0.44%) ~ 0.92s 0.93s p=0.405 n=6
Check Time 7.01s (± 0.32%) 7.01s (± 0.25%) ~ 6.99s 7.04s p=0.622 n=6
Emit Time 4.08s (± 0.37%) 4.08s (± 0.43%) ~ 4.05s 4.10s p=0.934 n=6
Total Time 14.49s (± 0.14%) 14.53s (± 0.13%) +0.04s (+ 0.24%) 14.50s 14.55s p=0.019 n=6
TFS - node (v18.15.0, x64)
Memory used 302,721k (± 0.01%) 303,165k (± 0.00%) +444k (+ 0.15%) 303,150k 303,181k p=0.005 n=6
Parse Time 2.01s (± 0.83%) 2.03s (± 0.92%) ~ 2.00s 2.05s p=0.139 n=6
Bind Time 1.00s (± 0.98%) 1.01s (± 0.88%) ~ 1.00s 1.02s p=0.142 n=6
Check Time 6.30s (± 0.37%) 6.31s (± 0.39%) ~ 6.28s 6.34s p=0.507 n=6
Emit Time 3.61s (± 0.64%) 3.60s (± 0.23%) ~ 3.59s 3.61s p=0.933 n=6
Total Time 12.93s (± 0.32%) 12.95s (± 0.15%) ~ 12.92s 12.97s p=0.198 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,890k (± 0.00%) 511,223k (± 0.00%) +1,333k (+ 0.26%) 511,213k 511,250k p=0.005 n=6
Parse Time 2.66s (± 0.57%) 2.68s (± 0.62%) +0.02s (+ 0.88%) 2.65s 2.70s p=0.035 n=6
Bind Time 0.99s (± 0.82%) 0.99s (± 1.18%) ~ 0.98s 1.01s p=0.738 n=6
Check Time 17.29s (± 0.42%) 17.27s (± 0.28%) ~ 17.22s 17.36s p=1.000 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.94s (± 0.29%) 20.95s (± 0.24%) ~ 20.89s 21.03s p=0.746 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,719,659k (± 0.00%) 1,727,314k (± 0.00%) +7,656k (+ 0.45%) 1,727,280k 1,727,349k p=0.005 n=6
Parse Time 6.58s (± 0.40%) 6.68s (± 0.76%) +0.10s (+ 1.49%) 6.61s 6.77s p=0.006 n=6
Bind Time 2.36s (± 0.51%) 2.35s (± 0.23%) ~ 2.35s 2.36s p=0.865 n=6
Check Time 56.28s (± 0.24%) 56.40s (± 0.57%) ~ 55.87s 56.86s p=0.335 n=6
Emit Time 0.13s (± 0.00%) 0.14s (± 2.95%) 🔻+0.01s (+ 6.41%) 0.13s 0.14s p=0.007 n=6
Total Time 65.35s (± 0.21%) 65.57s (± 0.50%) ~ 65.05s 66.02s p=0.173 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,391,720k (± 0.02%) 2,431,999k (± 0.05%) +40,279k (+ 1.68%) 2,430,351k 2,433,781k p=0.005 n=6
Parse Time 5.00s (± 1.39%) 5.07s (± 1.08%) ~ 5.03s 5.16s p=0.077 n=6
Bind Time 1.89s (± 0.99%) 1.90s (± 0.91%) ~ 1.87s 1.92s p=0.256 n=6
Check Time 33.60s (± 0.30%) 33.56s (± 0.34%) ~ 33.38s 33.72s p=0.471 n=6
Emit Time 2.63s (± 1.90%) 2.66s (± 1.46%) ~ 2.62s 2.73s p=0.630 n=6
Total Time 43.14s (± 0.19%) 43.22s (± 0.29%) ~ 43.05s 43.33s p=0.230 n=6
self-compiler - node (v18.15.0, x64)
Memory used 415,085k (± 0.01%) 415,373k (± 0.01%) +288k (+ 0.07%) 415,353k 415,411k p=0.005 n=6
Parse Time 2.85s (± 0.95%) 2.81s (± 1.01%) ~ 2.78s 2.85s p=0.062 n=6
Bind Time 1.07s (± 0.70%) 1.07s (± 0.92%) ~ 1.06s 1.08s p=1.000 n=6
Check Time 15.27s (± 0.26%) 15.30s (± 0.44%) ~ 15.19s 15.37s p=0.294 n=6
Emit Time 1.10s (± 1.10%) 1.15s (± 0.78%) 🔻+0.05s (+ 4.23%) 1.14s 1.16s p=0.005 n=6
Total Time 20.29s (± 0.13%) 20.33s (± 0.39%) ~ 20.21s 20.43s p=0.198 n=6
vscode - node (v18.15.0, x64)
Memory used 2,888,640k (± 0.00%) 2,897,800k (± 0.00%) +9,159k (+ 0.32%) 2,897,660k 2,897,874k p=0.005 n=6
Parse Time 10.81s (± 0.22%) 10.89s (± 0.50%) +0.08s (+ 0.74%) 10.83s 10.96s p=0.010 n=6
Bind Time 3.44s (± 0.36%) 3.46s (± 0.74%) ~ 3.43s 3.50s p=0.220 n=6
Check Time 61.76s (± 0.28%) 61.64s (± 0.29%) ~ 61.37s 61.81s p=0.378 n=6
Emit Time 16.49s (± 0.57%) 16.58s (± 0.50%) ~ 16.50s 16.72s p=0.145 n=6
Total Time 92.50s (± 0.22%) 92.58s (± 0.17%) ~ 92.32s 92.72s p=0.378 n=6
webpack - node (v18.15.0, x64)
Memory used 408,085k (± 0.02%) 409,291k (± 0.01%) +1,206k (+ 0.30%) 409,259k 409,339k p=0.005 n=6
Parse Time 3.24s (± 0.50%) 3.27s (± 0.43%) +0.03s (+ 1.03%) 3.25s 3.29s p=0.012 n=6
Bind Time 1.39s (± 0.98%) 1.42s (± 0.39%) +0.02s (+ 1.56%) 1.41s 1.42s p=0.008 n=6
Check Time 14.23s (± 0.24%) 14.24s (± 0.28%) ~ 14.20s 14.30s p=0.747 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.86s (± 0.25%) 18.93s (± 0.23%) +0.07s (+ 0.37%) 18.90s 18.99s p=0.041 n=6
xstate - node (v18.15.0, x64)
Memory used 513,061k (± 0.01%) 513,757k (± 0.01%) +696k (+ 0.14%) 513,675k 513,880k p=0.005 n=6
Parse Time 3.28s (± 0.19%) 3.29s (± 0.31%) +0.01s (+ 0.41%) 3.28s 3.31s p=0.028 n=6
Bind Time 1.54s (± 0.26%) 1.54s (± 0.34%) ~ 1.53s 1.54s p=0.114 n=6
Check Time 2.86s (± 0.49%) 2.85s (± 0.48%) ~ 2.82s 2.86s p=0.155 n=6
Emit Time 0.07s (± 5.69%) 0.07s (± 0.00%) ~ 0.07s 0.07s p=0.405 n=6
Total Time 7.76s (± 0.24%) 7.75s (± 0.34%) ~ 7.70s 7.77s p=0.569 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Okay this is definitely slower for some reason.

@jakebailey
Copy link
Member Author

I really have no clue what's going on here; I've tried to compare this and there's just no reason for this to be slower.

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 21, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this faster ✅ Started 👀 Results

@fatcerberus
Copy link

Somehow, time went backwards???

Spectre mitigation, perhaps?

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,517k (± 0.01%) 298,287k (± 0.01%) +2,770k (+ 0.94%) 298,251k 298,310k p=0.005 n=6
Parse Time 2.67s (± 0.15%) 2.68s (± 0.51%) +0.02s (+ 0.56%) 2.66s 2.70s p=0.045 n=6
Bind Time 0.83s (± 0.49%) 0.83s (± 0.62%) ~ 0.83s 0.84s p=0.114 n=6
Check Time 8.20s (± 0.22%) 8.22s (± 0.37%) ~ 8.18s 8.26s p=0.244 n=6
Emit Time 7.06s (± 0.28%) 7.17s (± 0.32%) +0.11s (+ 1.58%) 7.15s 7.21s p=0.005 n=6
Total Time 18.75s (± 0.18%) 18.90s (± 0.29%) +0.15s (+ 0.81%) 18.84s 19.00s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,104k (± 0.97%) 193,254k (± 0.98%) ~ 191,927k 195,820k p=0.128 n=6
Parse Time 1.36s (± 1.08%) 1.37s (± 1.13%) ~ 1.36s 1.39s p=0.241 n=6
Bind Time 0.72s (± 0.57%) 0.72s (± 0.00%) ~ 0.72s 0.72s p=0.405 n=6
Check Time 9.50s (± 0.75%) 9.46s (± 0.45%) ~ 9.40s 9.53s p=0.296 n=6
Emit Time 2.63s (± 0.94%) 2.65s (± 0.31%) ~ 2.63s 2.65s p=0.096 n=6
Total Time 14.22s (± 0.37%) 14.20s (± 0.34%) ~ 14.16s 14.29s p=0.630 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,375k (± 0.01%) 348,718k (± 0.01%) +1,343k (+ 0.39%) 348,695k 348,739k p=0.005 n=6
Parse Time 2.49s (± 0.41%) 2.50s (± 0.41%) ~ 2.49s 2.51s p=0.142 n=6
Bind Time 0.93s (± 0.56%) 0.93s (± 0.00%) ~ 0.93s 0.93s p=0.174 n=6
Check Time 7.01s (± 0.42%) 7.04s (± 0.54%) ~ 6.97s 7.08s p=0.146 n=6
Emit Time 4.08s (± 0.40%) 4.07s (± 0.29%) ~ 4.05s 4.08s p=0.462 n=6
Total Time 14.50s (± 0.36%) 14.54s (± 0.28%) ~ 14.46s 14.57s p=0.228 n=6
TFS - node (v18.15.0, x64)
Memory used 302,720k (± 0.01%) 303,171k (± 0.00%) +452k (+ 0.15%) 303,151k 303,191k p=0.005 n=6
Parse Time 2.00s (± 0.93%) 2.02s (± 0.52%) +0.02s (+ 1.08%) 2.01s 2.04s p=0.033 n=6
Bind Time 1.00s (± 1.04%) 1.00s (± 0.51%) ~ 1.00s 1.01s p=0.794 n=6
Check Time 6.31s (± 0.37%) 6.32s (± 0.42%) ~ 6.28s 6.35s p=0.333 n=6
Emit Time 3.60s (± 0.27%) 3.63s (± 0.76%) +0.03s (+ 0.79%) 3.61s 3.68s p=0.026 n=6
Total Time 12.92s (± 0.28%) 12.98s (± 0.30%) +0.06s (+ 0.49%) 12.94s 13.05s p=0.016 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,887k (± 0.00%) 511,261k (± 0.01%) +1,373k (+ 0.27%) 511,214k 511,295k p=0.005 n=6
Parse Time 2.65s (± 0.41%) 2.68s (± 0.38%) +0.03s (+ 1.26%) 2.67s 2.70s p=0.004 n=6
Bind Time 0.99s (± 0.90%) 0.98s (± 0.85%) ~ 0.97s 0.99s p=0.437 n=6
Check Time 17.30s (± 0.60%) 17.30s (± 0.15%) ~ 17.27s 17.33s p=0.747 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.94s (± 0.55%) 20.97s (± 0.18%) ~ 20.93s 21.03s p=0.748 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,737,539k (± 0.00%) 1,745,200k (± 0.00%) +7,661k (+ 0.44%) 1,745,171k 1,745,244k p=0.005 n=6
Parse Time 6.58s (± 0.38%) 6.68s (± 0.60%) +0.10s (+ 1.55%) 6.62s 6.73s p=0.005 n=6
Bind Time 2.36s (± 0.22%) 2.34s (± 0.36%) -0.01s (- 0.50%) 2.33s 2.35s p=0.019 n=6
Check Time 56.25s (± 0.54%) 56.43s (± 0.46%) ~ 56.13s 56.78s p=0.378 n=6
Emit Time 0.13s (± 3.10%) 0.13s (± 0.00%) ~ 0.13s 0.13s p=0.405 n=6
Total Time 65.32s (± 0.48%) 65.57s (± 0.41%) ~ 65.25s 65.92s p=0.128 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,393,696k (± 0.03%) 2,433,825k (± 0.02%) +40,129k (+ 1.68%) 2,433,258k 2,434,796k p=0.005 n=6
Parse Time 5.06s (± 0.60%) 5.09s (± 0.70%) ~ 5.06s 5.14s p=0.258 n=6
Bind Time 1.89s (± 1.03%) 1.78s (± 5.64%) ~ 1.69s 1.91s p=0.052 n=6
Check Time 33.66s (± 0.49%) 33.82s (± 0.72%) ~ 33.54s 34.14s p=0.378 n=6
Emit Time 2.62s (± 0.82%) 2.70s (± 1.39%) +0.08s (+ 3.06%) 2.65s 2.74s p=0.008 n=6
Total Time 43.24s (± 0.41%) 43.42s (± 0.44%) ~ 43.23s 43.68s p=0.066 n=6
self-compiler - node (v18.15.0, x64)
Memory used 415,176k (± 0.01%) 415,466k (± 0.01%) +290k (+ 0.07%) 415,427k 415,523k p=0.005 n=6
Parse Time 2.83s (± 0.55%) 2.78s (± 1.80%) -0.05s (- 1.83%) 2.68s 2.82s p=0.012 n=6
Bind Time 1.07s (± 0.48%) 1.09s (± 5.65%) ~ 1.06s 1.22s p=0.351 n=6
Check Time 15.29s (± 0.33%) 15.30s (± 0.61%) ~ 15.20s 15.45s p=0.936 n=6
Emit Time 1.11s (± 1.33%) 1.13s (± 0.79%) +0.02s (+ 1.95%) 1.12s 1.14s p=0.015 n=6
Total Time 20.30s (± 0.30%) 20.30s (± 0.49%) ~ 20.18s 20.45s p=1.000 n=6
vscode - node (v18.15.0, x64)
Memory used 2,888,678k (± 0.00%) 2,897,892k (± 0.00%) +9,214k (+ 0.32%) 2,897,813k 2,897,968k p=0.005 n=6
Parse Time 10.84s (± 0.16%) 10.90s (± 0.40%) +0.06s (+ 0.57%) 10.83s 10.96s p=0.029 n=6
Bind Time 3.45s (± 0.44%) 3.47s (± 0.56%) ~ 3.45s 3.50s p=0.164 n=6
Check Time 61.56s (± 0.45%) 61.79s (± 0.58%) ~ 61.36s 62.26s p=0.336 n=6
Emit Time 16.52s (± 0.59%) 17.24s (± 8.08%) 🔻+0.72s (+ 4.36%) 16.57s 20.08s p=0.020 n=6
Total Time 92.37s (± 0.33%) 93.40s (± 1.71%) +1.03s (+ 1.12%) 92.50s 96.63s p=0.030 n=6
webpack - node (v18.15.0, x64)
Memory used 408,137k (± 0.01%) 409,299k (± 0.01%) +1,162k (+ 0.28%) 409,243k 409,376k p=0.005 n=6
Parse Time 3.26s (± 0.64%) 3.26s (± 0.34%) ~ 3.24s 3.27s p=0.508 n=6
Bind Time 1.39s (± 1.02%) 1.41s (± 0.29%) +0.02s (+ 1.56%) 1.41s 1.42s p=0.010 n=6
Check Time 14.24s (± 0.28%) 14.29s (± 0.14%) +0.05s (+ 0.33%) 14.27s 14.32s p=0.035 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.89s (± 0.11%) 18.97s (± 0.17%) +0.07s (+ 0.40%) 18.92s 19.01s p=0.006 n=6
xstate - node (v18.15.0, x64)
Memory used 513,073k (± 0.00%) 513,864k (± 0.02%) +791k (+ 0.15%) 513,733k 513,989k p=0.005 n=6
Parse Time 3.28s (± 0.23%) 3.29s (± 0.41%) ~ 3.27s 3.31s p=0.099 n=6
Bind Time 1.54s (± 0.41%) 1.54s (± 0.00%) ~ 1.54s 1.54s p=1.000 n=6
Check Time 2.86s (± 0.29%) 2.87s (± 0.93%) ~ 2.84s 2.90s p=0.677 n=6
Emit Time 0.07s (± 0.00%) 0.07s (± 5.69%) ~ 0.07s 0.08s p=0.405 n=6
Total Time 7.75s (± 0.19%) 7.78s (± 0.44%) ~ 7.73s 7.82s p=0.148 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 22, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this faster ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,522k (± 0.00%) 295,538k (± 0.01%) ~ 295,517k 295,567k p=0.093 n=6
Parse Time 2.65s (± 0.48%) 2.66s (± 0.28%) ~ 2.65s 2.67s p=0.301 n=6
Bind Time 0.83s (± 0.76%) 0.83s (± 1.19%) ~ 0.82s 0.84s p=0.733 n=6
Check Time 8.20s (± 0.30%) 8.21s (± 0.25%) ~ 8.18s 8.23s p=0.685 n=6
Emit Time 7.05s (± 0.60%) 7.04s (± 0.27%) ~ 7.02s 7.06s p=0.627 n=6
Total Time 18.74s (± 0.21%) 18.74s (± 0.12%) ~ 18.70s 18.77s p=0.686 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,036k (± 0.92%) 193,126k (± 0.90%) ~ 191,905k 195,380k p=0.230 n=6
Parse Time 1.36s (± 0.60%) 1.36s (± 1.00%) ~ 1.35s 1.38s p=0.445 n=6
Bind Time 0.72s (± 0.00%) 0.72s (± 0.00%) ~ 0.72s 0.72s p=1.000 n=6
Check Time 9.45s (± 0.86%) 9.45s (± 0.61%) ~ 9.39s 9.56s p=0.686 n=6
Emit Time 2.62s (± 0.57%) 2.63s (± 0.29%) ~ 2.62s 2.64s p=0.797 n=6
Total Time 14.16s (± 0.58%) 14.17s (± 0.44%) ~ 14.09s 14.28s p=0.683 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,386k (± 0.00%) 347,377k (± 0.01%) ~ 347,346k 347,409k p=0.378 n=6
Parse Time 2.48s (± 0.65%) 2.48s (± 0.47%) ~ 2.47s 2.50s p=0.805 n=6
Bind Time 0.92s (± 0.59%) 0.92s (± 0.59%) ~ 0.92s 0.93s p=1.000 n=6
Check Time 7.02s (± 0.57%) 7.01s (± 0.27%) ~ 6.98s 7.03s p=1.000 n=6
Emit Time 4.04s (± 0.26%) 4.06s (± 0.33%) +0.02s (+ 0.41%) 4.04s 4.08s p=0.048 n=6
Total Time 14.47s (± 0.35%) 14.48s (± 0.25%) ~ 14.43s 14.53s p=0.520 n=6
TFS - node (v18.15.0, x64)
Memory used 302,734k (± 0.01%) 302,715k (± 0.01%) ~ 302,678k 302,752k p=0.230 n=6
Parse Time 2.40s (± 0.82%) 2.43s (± 1.34%) ~ 2.37s 2.46s p=0.169 n=6
Bind Time 1.20s (± 0.43%) 1.19s (± 1.30%) ~ 1.18s 1.22s p=0.112 n=6
Check Time 7.47s (± 0.51%) 7.46s (± 0.35%) ~ 7.43s 7.50s p=0.624 n=6
Emit Time 4.28s (± 0.63%) 4.27s (± 0.53%) ~ 4.25s 4.31s p=0.686 n=6
Total Time 15.36s (± 0.19%) 15.35s (± 0.27%) ~ 15.32s 15.42s p=1.000 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,883k (± 0.01%) 509,940k (± 0.01%) ~ 509,871k 510,065k p=0.128 n=6
Parse Time 3.19s (± 0.48%) 3.27s (± 6.06%) ~ 3.16s 3.67s p=0.564 n=6
Bind Time 1.18s (± 1.31%) 1.17s (± 0.35%) ~ 1.17s 1.18s p=0.248 n=6
Check Time 20.56s (± 0.47%) 20.49s (± 0.60%) ~ 20.33s 20.69s p=0.335 n=6
Emit Time 0.00s (±244.70%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=0.405 n=6
Total Time 24.93s (± 0.37%) 24.93s (± 1.03%) ~ 24.69s 25.39s p=0.689 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,737,541k (± 0.00%) 1,737,552k (± 0.00%) ~ 1,737,517k 1,737,582k p=0.470 n=6
Parse Time 7.75s (± 0.64%) 7.80s (± 0.52%) ~ 7.74s 7.86s p=0.148 n=6
Bind Time 2.78s (± 0.42%) 2.77s (± 0.55%) ~ 2.76s 2.80s p=0.410 n=6
Check Time 66.48s (± 0.79%) 66.25s (± 0.20%) ~ 66.06s 66.40s p=0.575 n=6
Emit Time 0.15s (± 2.69%) 0.15s (± 3.53%) ~ 0.15s 0.16s p=0.282 n=6
Total Time 77.17s (± 0.68%) 76.98s (± 0.19%) ~ 76.78s 77.18s p=0.810 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,392,766k (± 0.03%) 2,393,171k (± 0.04%) ~ 2,391,890k 2,394,524k p=0.689 n=6
Parse Time 5.03s (± 0.43%) 5.02s (± 0.75%) ~ 4.97s 5.08s p=0.575 n=6
Bind Time 1.90s (± 1.00%) 1.88s (± 0.52%) ~ 1.87s 1.90s p=0.072 n=6
Check Time 33.61s (± 0.59%) 33.48s (± 0.31%) ~ 33.38s 33.67s p=0.230 n=6
Emit Time 2.63s (± 1.76%) 2.63s (± 1.33%) ~ 2.57s 2.67s p=1.000 n=6
Total Time 43.19s (± 0.48%) 43.05s (± 0.28%) ~ 42.93s 43.24s p=0.199 n=6
self-compiler - node (v18.15.0, x64)
Memory used 415,192k (± 0.01%) 415,137k (± 0.01%) -55k (- 0.01%) 415,089k 415,169k p=0.031 n=6
Parse Time 3.40s (± 0.61%) 3.39s (± 0.71%) ~ 3.36s 3.41s p=0.462 n=6
Bind Time 1.28s (± 0.59%) 1.29s (± 0.63%) +0.01s (+ 0.91%) 1.28s 1.30s p=0.045 n=6
Check Time 18.03s (± 0.58%) 18.00s (± 0.26%) ~ 17.96s 18.06s p=0.747 n=6
Emit Time 1.32s (± 1.49%) 1.33s (± 0.78%) ~ 1.31s 1.34s p=0.413 n=6
Total Time 24.03s (± 0.52%) 24.00s (± 0.27%) ~ 23.94s 24.10s p=0.873 n=6
vscode - node (v18.15.0, x64)
Memory used 2,889,721k (± 0.00%) 2,889,724k (± 0.00%) ~ 2,889,672k 2,889,771k p=0.689 n=6
Parse Time 12.93s (± 0.36%) 12.95s (± 0.46%) ~ 12.88s 13.02s p=0.627 n=6
Bind Time 4.13s (± 0.22%) 4.13s (± 0.32%) ~ 4.11s 4.14s p=0.666 n=6
Check Time 71.27s (± 0.40%) 71.60s (± 0.55%) ~ 71.21s 72.23s p=0.173 n=6
Emit Time 19.36s (± 0.40%) 19.41s (± 0.68%) ~ 19.20s 19.59s p=0.574 n=6
Total Time 107.69s (± 0.29%) 108.09s (± 0.30%) ~ 107.77s 108.63s p=0.128 n=6
webpack - node (v18.15.0, x64)
Memory used 408,092k (± 0.01%) 408,147k (± 0.02%) ~ 408,068k 408,279k p=0.128 n=6
Parse Time 3.89s (± 0.39%) 3.91s (± 0.57%) +0.03s (+ 0.69%) 3.89s 3.95s p=0.040 n=6
Bind Time 1.66s (± 0.74%) 1.67s (± 0.88%) ~ 1.65s 1.69s p=0.085 n=6
Check Time 16.76s (± 0.24%) 16.74s (± 0.45%) ~ 16.64s 16.83s p=0.630 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.30s (± 0.18%) 22.33s (± 0.33%) ~ 22.24s 22.42s p=0.748 n=6
xstate - node (v18.15.0, x64)
Memory used 513,055k (± 0.02%) 513,029k (± 0.01%) ~ 512,953k 513,120k p=0.689 n=6
Parse Time 3.96s (± 0.63%) 3.96s (± 0.67%) ~ 3.91s 3.99s p=0.747 n=6
Bind Time 1.84s (± 0.75%) 1.85s (± 1.25%) ~ 1.83s 1.89s p=0.864 n=6
Check Time 3.38s (± 0.48%) 3.38s (± 0.41%) ~ 3.36s 3.39s p=0.936 n=6
Emit Time 0.09s (± 6.44%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.071 n=6
Total Time 9.27s (± 0.46%) 9.27s (± 0.54%) ~ 9.21s 9.36s p=0.683 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

I understand what happened; PerformanceObserver is only available as a global on Node 20+. This meant that Node 18 was not getting performance hooks at all as the old code was all or nothing. Then I removed that check, which then re-enabled hooks on Node 18, which is what the faster benchmark uses.

Now the order is swapped and so it's no longer enabled in Node at all. Probably this means Node 20 is faster now too.

Will update the description and comments and stuff to reflect that.

@typescript-bot perf test this full

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 22, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this full ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,462k (± 0.01%) 295,482k (± 0.01%) ~ 295,461k 295,518k p=0.378 n=6
Parse Time 3.19s (± 0.65%) 3.20s (± 0.43%) ~ 3.18s 3.21s p=0.370 n=6
Bind Time 0.99s (± 0.41%) 0.99s (± 0.41%) ~ 0.99s 1.00s p=1.000 n=6
Check Time 9.67s (± 0.37%) 9.66s (± 0.35%) ~ 9.62s 9.70s p=0.466 n=6
Emit Time 8.39s (± 0.58%) 8.40s (± 0.60%) ~ 8.35s 8.49s p=0.747 n=6
Total Time 22.24s (± 0.29%) 22.25s (± 0.29%) ~ 22.17s 22.34s p=1.000 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 192,496k (± 0.75%) 193,798k (± 0.96%) +1,302k (+ 0.68%) 192,004k 195,569k p=0.031 n=6
Parse Time 1.64s (± 1.31%) 1.65s (± 1.91%) ~ 1.61s 1.70s p=0.871 n=6
Bind Time 0.85s (± 0.64%) 0.86s (± 1.04%) ~ 0.85s 0.87s p=0.341 n=6
Check Time 11.25s (± 0.84%) 11.28s (± 1.06%) ~ 11.05s 11.39s p=0.687 n=6
Emit Time 3.14s (± 0.35%) 3.13s (± 0.69%) ~ 3.10s 3.16s p=0.681 n=6
Total Time 16.89s (± 0.54%) 16.91s (± 0.84%) ~ 16.65s 17.04s p=0.575 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,413k (± 0.01%) 347,402k (± 0.01%) ~ 347,376k 347,427k p=0.378 n=6
Parse Time 3.00s (± 0.69%) 2.99s (± 0.68%) ~ 2.96s 3.02s p=0.870 n=6
Bind Time 1.11s (± 0.46%) 1.11s (± 0.57%) ~ 1.10s 1.12s p=0.386 n=6
Check Time 8.26s (± 0.33%) 8.26s (± 0.13%) ~ 8.25s 8.28s p=0.744 n=6
Emit Time 4.84s (± 0.48%) 4.85s (± 0.62%) ~ 4.81s 4.90s p=1.000 n=6
Total Time 17.21s (± 0.15%) 17.21s (± 0.29%) ~ 17.14s 17.29s p=0.871 n=6
TFS - node (v18.15.0, x64)
Memory used 302,706k (± 0.00%) 302,693k (± 0.01%) ~ 302,669k 302,719k p=0.378 n=6
Parse Time 2.38s (± 1.70%) 2.44s (± 0.57%) ~ 2.41s 2.45s p=0.052 n=6
Bind Time 1.19s (± 0.43%) 1.19s (± 0.43%) ~ 1.19s 1.20s p=0.069 n=6
Check Time 7.44s (± 0.39%) 7.46s (± 0.19%) ~ 7.44s 7.48s p=0.372 n=6
Emit Time 4.27s (± 0.35%) 4.28s (± 1.07%) ~ 4.22s 4.35s p=0.683 n=6
Total Time 15.28s (± 0.37%) 15.37s (± 0.33%) ~ 15.30s 15.42s p=0.052 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,897k (± 0.01%) 509,898k (± 0.01%) ~ 509,842k 509,938k p=0.936 n=6
Parse Time 3.19s (± 0.43%) 3.18s (± 0.33%) ~ 3.16s 3.19s p=0.249 n=6
Bind Time 1.19s (± 1.37%) 1.17s (± 0.89%) ~ 1.16s 1.19s p=0.073 n=6
Check Time 20.53s (± 0.24%) 20.53s (± 0.66%) ~ 20.35s 20.76s p=0.629 n=6
Emit Time 0.00s (±244.70%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=0.405 n=6
Total Time 24.91s (± 0.26%) 24.88s (± 0.58%) ~ 24.70s 25.13s p=0.470 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,737,541k (± 0.00%) 1,737,554k (± 0.00%) ~ 1,737,520k 1,737,608k p=0.470 n=6
Parse Time 7.77s (± 0.65%) 7.79s (± 0.48%) ~ 7.75s 7.84s p=0.295 n=6
Bind Time 2.80s (± 0.54%) 2.80s (± 0.50%) ~ 2.78s 2.82s p=0.869 n=6
Check Time 66.48s (± 0.36%) 66.46s (± 0.31%) ~ 66.21s 66.82s p=0.873 n=6
Emit Time 0.16s (± 3.29%) 0.15s (± 2.69%) ~ 0.15s 0.16s p=0.112 n=6
Total Time 77.20s (± 0.32%) 77.21s (± 0.31%) ~ 76.91s 77.62s p=0.936 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,394,296k (± 0.03%) 2,394,012k (± 0.04%) ~ 2,392,834k 2,395,241k p=0.575 n=6
Parse Time 6.05s (± 1.21%) 6.06s (± 0.47%) ~ 6.02s 6.10s p=0.688 n=6
Bind Time 2.24s (± 1.07%) 2.25s (± 0.93%) ~ 2.22s 2.27s p=0.468 n=6
Check Time 39.52s (± 0.18%) 39.45s (± 0.41%) ~ 39.25s 39.67s p=0.470 n=6
Emit Time 3.14s (± 0.96%) 3.13s (± 1.32%) ~ 3.07s 3.19s p=0.748 n=6
Total Time 50.96s (± 0.19%) 50.91s (± 0.40%) ~ 50.67s 51.18s p=0.810 n=6
self-compiler - node (v18.15.0, x64)
Memory used 415,188k (± 0.01%) 415,113k (± 0.00%) -75k (- 0.02%) 415,086k 415,141k p=0.006 n=6
Parse Time 3.41s (± 1.10%) 3.40s (± 0.53%) ~ 3.37s 3.42s p=0.808 n=6
Bind Time 1.29s (± 0.80%) 1.30s (± 0.80%) ~ 1.28s 1.31s p=0.134 n=6
Check Time 18.05s (± 0.39%) 18.03s (± 0.32%) ~ 17.96s 18.10s p=0.630 n=6
Emit Time 1.34s (± 1.22%) 1.32s (± 0.92%) ~ 1.31s 1.34s p=0.210 n=6
Total Time 24.08s (± 0.18%) 24.05s (± 0.33%) ~ 23.97s 24.15s p=0.470 n=6
vscode - node (v18.15.0, x64)
Memory used 2,889,651k (± 0.00%) 2,889,712k (± 0.00%) ~ 2,889,621k 2,889,795k p=0.230 n=6
Parse Time 10.82s (± 0.22%) 10.79s (± 0.23%) ~ 10.77s 10.83s p=0.170 n=6
Bind Time 3.45s (± 0.30%) 3.45s (± 0.34%) ~ 3.44s 3.47s p=0.406 n=6
Check Time 61.55s (± 0.46%) 61.56s (± 0.44%) ~ 61.16s 61.89s p=1.000 n=6
Emit Time 16.43s (± 0.59%) 16.42s (± 0.69%) ~ 16.27s 16.59s p=0.688 n=6
Total Time 92.24s (± 0.34%) 92.23s (± 0.29%) ~ 91.86s 92.46s p=0.810 n=6
webpack - node (v18.15.0, x64)
Memory used 408,125k (± 0.01%) 408,129k (± 0.01%) ~ 408,079k 408,219k p=1.000 n=6
Parse Time 3.88s (± 0.81%) 4.08s (± 9.41%) ~ 3.88s 4.86s p=0.053 n=6
Bind Time 1.68s (± 0.81%) 1.75s (± 9.91%) ~ 1.67s 2.10s p=0.676 n=6
Check Time 16.70s (± 0.41%) 16.81s (± 0.99%) ~ 16.67s 17.13s p=0.173 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.27s (± 0.22%) 22.64s (± 3.16%) ~ 22.25s 24.09s p=0.092 n=6
xstate - node (v18.15.0, x64)
Memory used 513,004k (± 0.02%) 513,108k (± 0.02%) ~ 512,997k 513,200k p=0.066 n=6
Parse Time 3.95s (± 0.47%) 3.94s (± 0.67%) ~ 3.92s 3.98s p=0.332 n=6
Bind Time 1.85s (± 0.88%) 1.85s (± 0.93%) ~ 1.83s 1.87s p=0.673 n=6
Check Time 3.36s (± 0.55%) 3.38s (± 0.88%) ~ 3.33s 3.42s p=0.295 n=6
Emit Time 0.09s (± 5.95%) 0.08s (± 4.99%) ~ 0.08s 0.09s p=0.112 n=6
Total Time 9.27s (± 0.23%) 9.26s (± 0.68%) ~ 9.18s 9.34s p=0.936 n=6
Angular - node (v20.5.1, x64)
Memory used 306,972k (± 0.00%) 293,551k (± 0.01%) 🟩-13,421k (- 4.37%) 293,512k 293,578k p=0.005 n=6
Parse Time 3.39s (± 0.64%) 3.42s (± 9.75%) ~ 3.26s 4.10s p=0.066 n=6
Bind Time 1.02s (± 0.96%) 1.06s (± 9.33%) ~ 1.01s 1.26s p=0.611 n=6
Check Time 9.10s (± 0.20%) 9.44s (± 9.36%) ~ 9.06s 11.24s p=0.195 n=6
Emit Time 8.81s (± 0.48%) 8.71s (± 0.21%) -0.10s (- 1.17%) 8.69s 8.74s p=0.005 n=6
Total Time 22.32s (± 0.26%) 22.62s (± 5.76%) ~ 22.05s 25.28s p=0.064 n=6
Compiler-Unions - node (v20.5.1, x64)
Memory used 190,299k (± 0.10%) 191,320k (± 0.96%) ~ 189,956k 193,701k p=0.575 n=6
Parse Time 1.37s (± 0.97%) 1.36s (± 1.20%) ~ 1.34s 1.38s p=0.459 n=6
Bind Time 0.75s (± 0.68%) 0.75s (± 1.77%) ~ 0.73s 0.77s p=0.928 n=6
Check Time 9.03s (± 0.80%) 8.97s (± 1.06%) ~ 8.86s 9.09s p=0.518 n=6
Emit Time 2.67s (± 0.50%) 2.66s (± 0.51%) ~ 2.65s 2.68s p=0.566 n=6
Total Time 13.83s (± 0.52%) 13.74s (± 0.68%) ~ 13.61s 13.86s p=0.128 n=6
Monaco - node (v20.5.1, x64)
Memory used 351,979k (± 0.01%) 345,455k (± 0.01%) -6,524k (- 1.85%) 345,417k 345,514k p=0.005 n=6
Parse Time 2.62s (± 0.79%) 2.53s (± 0.54%) 🟩-0.08s (- 3.12%) 2.52s 2.55s p=0.005 n=6
Bind Time 0.93s (± 0.87%) 0.93s (± 0.88%) ~ 0.92s 0.94s p=0.204 n=6
Check Time 6.97s (± 0.39%) 6.93s (± 0.34%) -0.04s (- 0.60%) 6.89s 6.95s p=0.019 n=6
Emit Time 3.93s (± 0.37%) 3.88s (± 0.31%) -0.05s (- 1.32%) 3.86s 3.89s p=0.005 n=6
Total Time 14.45s (± 0.32%) 14.27s (± 0.17%) -0.18s (- 1.22%) 14.24s 14.30s p=0.005 n=6
TFS - node (v20.5.1, x64)
Memory used 302,521k (± 0.01%) 300,684k (± 0.01%) -1,837k (- 0.61%) 300,650k 300,707k p=0.005 n=6
Parse Time 2.35s (± 0.64%) 2.45s (±10.44%) ~ 2.32s 2.97s p=1.000 n=6
Bind Time 1.03s (± 0.73%) 1.08s (±10.26%) ~ 1.03s 1.31s p=0.150 n=6
Check Time 7.53s (± 0.33%) 7.84s (± 9.40%) ~ 7.52s 9.34s p=0.935 n=6
Emit Time 4.58s (± 0.36%) 4.77s (± 9.23%) ~ 4.54s 5.67s p=0.374 n=6
Total Time 15.50s (± 0.15%) 16.14s (± 9.55%) ~ 15.46s 19.29s p=0.376 n=6
material-ui - node (v20.5.1, x64)
Memory used 514,688k (± 0.01%) 508,144k (± 0.01%) -6,544k (- 1.27%) 508,108k 508,213k p=0.005 n=6
Parse Time 2.86s (± 0.28%) 2.79s (± 0.15%) -0.07s (- 2.50%) 2.79s 2.80s p=0.003 n=6
Bind Time 1.04s (± 0.49%) 1.02s (± 0.80%) -0.02s (- 1.92%) 1.01s 1.03s p=0.004 n=6
Check Time 16.33s (± 0.66%) 16.33s (± 0.27%) ~ 16.26s 16.38s p=0.376 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.24s (± 0.53%) 20.14s (± 0.19%) -0.09s (- 0.45%) 20.08s 20.19s p=0.030 n=6
mui-docs - node (v20.5.1, x64)
Memory used 1,772,401k (± 0.00%) 1,734,625k (± 0.00%) -37,776k (- 2.13%) 1,734,581k 1,734,664k p=0.005 n=6
Parse Time 8.13s (± 0.50%) 7.96s (± 5.50%) ~ 7.77s 8.86s p=0.065 n=6
Bind Time 2.80s (± 5.04%) 2.88s (± 1.05%) ~ 2.83s 2.90s p=0.225 n=6
Check Time 64.24s (± 0.41%) 64.25s (± 0.44%) ~ 63.91s 64.57s p=0.810 n=6
Emit Time 0.15s (± 5.07%) 0.15s (± 2.75%) ~ 0.14s 0.15s p=1.000 n=6
Total Time 75.32s (± 0.45%) 75.23s (± 0.70%) ~ 74.66s 76.10s p=0.810 n=6
self-build-src - node (v20.5.1, x64)
Memory used 2,562,314k (± 0.03%) 2,361,495k (± 0.03%) 🟩-200,820k (- 7.84%) 2,361,048k 2,362,650k p=0.005 n=6
Parse Time 6.29s (± 1.09%) 6.14s (± 2.44%) ~ 5.97s 6.38s p=0.065 n=6
Bind Time 2.42s (± 0.43%) 2.39s (± 1.91%) ~ 2.36s 2.48s p=0.065 n=6
Check Time 38.68s (± 0.25%) 38.67s (± 0.19%) ~ 38.54s 38.75s p=1.000 n=6
Emit Time 3.21s (± 2.13%) 3.10s (± 1.94%) 🟩-0.11s (- 3.58%) 3.02s 3.16s p=0.020 n=6
Total Time 50.59s (± 0.23%) 50.30s (± 0.35%) -0.29s (- 0.57%) 50.13s 50.65s p=0.031 n=6
self-compiler - node (v20.5.1, x64)
Memory used 414,780k (± 0.02%) 412,554k (± 0.01%) -2,226k (- 0.54%) 412,530k 412,627k p=0.005 n=6
Parse Time 3.49s (± 1.03%) 3.56s (± 7.57%) ~ 3.42s 4.11s p=0.170 n=6
Bind Time 1.35s (± 0.73%) 1.36s (± 0.89%) ~ 1.34s 1.37s p=0.507 n=6
Check Time 16.93s (± 0.46%) 16.94s (± 0.36%) ~ 16.85s 17.04s p=0.688 n=6
Emit Time 1.21s (± 2.45%) 1.20s (± 1.97%) ~ 1.17s 1.23s p=0.808 n=6
Total Time 22.97s (± 0.47%) 23.06s (± 1.22%) ~ 22.85s 23.61s p=0.872 n=6
vscode - node (v20.5.1, x64)
Memory used 2,932,959k (± 0.01%) 2,886,900k (± 0.00%) -46,059k (- 1.57%) 2,886,868k 2,886,918k p=0.005 n=6
Parse Time 10.96s (± 0.16%) 10.68s (± 0.20%) -0.28s (- 2.55%) 10.65s 10.71s p=0.005 n=6
Bind Time 3.54s (± 0.34%) 3.47s (± 0.28%) -0.07s (- 1.93%) 3.46s 3.48s p=0.005 n=6
Check Time 59.22s (± 0.43%) 58.92s (± 0.31%) -0.30s (- 0.51%) 58.69s 59.15s p=0.037 n=6
Emit Time 20.21s (± 2.91%) 16.39s (± 0.48%) 🟩-3.82s (-18.91%) 16.29s 16.50s p=0.005 n=6
Total Time 93.93s (± 0.77%) 89.46s (± 0.27%) 🟩-4.47s (- 4.76%) 89.15s 89.83s p=0.005 n=6
webpack - node (v20.5.1, x64)
Memory used 411,511k (± 0.02%) 405,909k (± 0.01%) -5,602k (- 1.36%) 405,868k 405,954k p=0.005 n=6
Parse Time 3.38s (± 0.67%) 3.33s (± 0.79%) -0.05s (- 1.43%) 3.29s 3.35s p=0.005 n=6
Bind Time 1.47s (± 0.43%) 1.46s (± 0.57%) ~ 1.46s 1.48s p=0.226 n=6
Check Time 13.32s (± 0.32%) 13.31s (± 0.40%) ~ 13.23s 13.38s p=0.683 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.17s (± 0.24%) 18.11s (± 0.21%) -0.06s (- 0.36%) 18.04s 18.14s p=0.024 n=6
xstate - node (v20.5.1, x64)
Memory used 512,258k (± 0.02%) 508,674k (± 0.01%) -3,584k (- 0.70%) 508,585k 508,784k p=0.005 n=6
Parse Time 3.54s (± 0.46%) 3.50s (± 0.40%) -0.04s (- 1.27%) 3.48s 3.52s p=0.006 n=6
Bind Time 1.48s (± 0.55%) 1.48s (± 0.00%) ~ 1.48s 1.48s p=0.290 n=6
Check Time 3.20s (± 0.48%) 3.19s (± 0.28%) ~ 3.18s 3.20s p=0.566 n=6
Emit Time 0.12s (± 3.35%) 0.12s (± 6.35%) ~ 0.11s 0.13s p=0.389 n=6
Total Time 8.34s (± 0.36%) 8.28s (± 0.16%) -0.06s (- 0.74%) 8.27s 8.30s p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
  • node (v20.5.1, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
  • Angular - node (v20.5.1, x64)
  • Compiler-Unions - node (v20.5.1, x64)
  • Monaco - node (v20.5.1, x64)
  • TFS - node (v20.5.1, x64)
  • material-ui - node (v20.5.1, x64)
  • mui-docs - node (v20.5.1, x64)
  • self-build-src - node (v20.5.1, x64)
  • self-compiler - node (v20.5.1, x64)
  • vscode - node (v20.5.1, x64)
  • webpack - node (v20.5.1, x64)
  • xstate - node (v20.5.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,856ms (± 0.73%) 2,878ms (± 0.59%) ~ 2,844ms 2,889ms p=0.093 n=6
Req 2 - geterr 7,123ms (± 9.39%) 7,338ms (± 9.61%) ~ 6,673ms 8,036ms p=0.575 n=6
Req 3 - references 440ms (± 8.79%) 426ms (± 9.85%) ~ 383ms 478ms p=0.936 n=6
Req 4 - navto 408ms (± 0.56%) 383ms (± 9.88%) ~ 334ms 409ms p=0.167 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 135ms (± 6.17%) 127ms (±10.39%) ~ 106ms 146ms p=0.293 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,983ms (± 0.91%) 3,146ms (±10.28%) ~ 2,933ms 3,763ms p=0.575 n=6
Req 2 - geterr 5,491ms (±11.65%) 5,306ms (±11.35%) ~ 4,902ms 6,091ms p=1.000 n=6
Req 3 - references 450ms (±11.75%) 450ms (± 7.99%) ~ 402ms 495ms p=0.936 n=6
Req 4 - navto 430ms (± 7.60%) 401ms (±11.07%) ~ 359ms 445ms p=0.228 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 112ms (± 5.08%) 103ms (± 9.78%) 🟩-9ms (- 8.31%) 90ms 110ms p=0.025 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,621ms (± 0.60%) 2,612ms (± 0.43%) ~ 2,595ms 2,624ms p=0.261 n=6
Req 2 - geterr 1,739ms (± 2.10%) 1,719ms (± 2.99%) ~ 1,659ms 1,787ms p=0.630 n=6
Req 3 - references 111ms (± 9.73%) 119ms (± 9.90%) ~ 104ms 130ms p=0.170 n=6
Req 4 - navto 367ms (± 0.86%) 370ms (± 0.37%) ~ 368ms 372ms p=0.106 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 307ms (± 2.21%) 306ms (± 1.47%) ~ 301ms 314ms p=1.000 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 152.60ms (± 0.17%) 153.09ms (± 0.18%) +0.49ms (+ 0.32%) 151.93ms 157.00ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 226.05ms (± 0.16%) 227.07ms (± 0.16%) +1.03ms (+ 0.45%) 225.43ms 232.79ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 221.14ms (± 0.15%) 223.37ms (± 0.15%) +2.23ms (+ 1.01%) 222.09ms 226.06ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 221.07ms (± 0.15%) 223.22ms (± 0.16%) +2.15ms (+ 0.97%) 221.70ms 226.19ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Wow, literally 5% faster vscode build time

@jakebailey
Copy link
Member Author

@typescript-bot perf test bun

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 22, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test bun ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - bun (v1.0.15, x64)
Memory used 313,037k (± 0.19%) 312,711k (± 0.14%) ~ 311,676k 313,836k p=0.525 n=12
Parse Time 2.24s (± 0.58%) 2.23s (± 0.53%) ~ 2.21s 2.27s p=0.515 n=12
Bind Time 0.85s (± 1.05%) 0.84s (± 1.17%) ~ 0.82s 0.87s p=0.495 n=12
Check Time 8.11s (± 0.35%) 8.08s (± 0.36%) ~ 8.02s 8.15s p=0.116 n=12
Emit Time 6.29s (± 0.38%) 6.29s (± 0.46%) ~ 6.22s 6.37s p=0.794 n=12
Total Time 17.49s (± 0.27%) 17.45s (± 0.26%) ~ 17.35s 17.58s p=0.193 n=12
Compiler-Unions - bun (v1.0.15, x64)
Memory used 253,519k (± 2.18%) 252,606k (± 2.68%) ~ 237,008k 267,972k p=0.817 n=12
Parse Time 1.35s (± 2.15%) 1.35s (± 1.63%) ~ 1.31s 1.41s p=0.662 n=12
Bind Time 0.97s (± 1.21%) 0.98s (± 1.68%) ~ 0.94s 1.03s p=1.000 n=12
Check Time 10.38s (± 0.64%) 10.38s (± 0.61%) ~ 10.25s 10.53s p=0.908 n=12
Emit Time 3.21s (± 0.87%) 3.20s (± 0.92%) ~ 3.15s 3.31s p=0.817 n=12
Total Time 15.92s (± 0.53%) 15.91s (± 0.44%) ~ 15.71s 16.08s p=0.885 n=12
Monaco - bun (v1.0.15, x64)
Memory used 372,702k (± 0.32%) 373,158k (± 0.17%) ~ 371,144k 374,617k p=0.237 n=12
Parse Time 2.36s (± 0.59%) 2.35s (± 0.62%) ~ 2.32s 2.39s p=0.704 n=12
Bind Time 1.11s (± 1.39%) 1.12s (± 1.55%) ~ 1.08s 1.16s p=0.684 n=12
Check Time 8.57s (± 0.48%) 8.62s (± 0.29%) +0.05s (+ 0.54%) 8.55s 8.69s p=0.015 n=12
Emit Time 4.48s (± 0.47%) 4.49s (± 0.53%) ~ 4.44s 4.55s p=0.384 n=12
Total Time 16.52s (± 0.27%) 16.58s (± 0.33%) ~ 16.43s 16.69s p=0.093 n=12
TFS - bun (v1.0.15, x64)
Memory used 312,757k (± 0.17%) 312,865k (± 0.19%) ~ 311,722k 314,397k p=0.885 n=12
Parse Time 2.09s (± 0.97%) 2.08s (± 0.50%) ~ 2.05s 2.10s p=0.274 n=12
Bind Time 1.08s (± 2.91%) 1.07s (± 2.83%) ~ 1.02s 1.16s p=0.977 n=12
Check Time 7.90s (± 0.61%) 7.91s (± 0.57%) ~ 7.76s 8.02s p=0.623 n=12
Emit Time 4.08s (± 0.40%) 4.09s (± 0.77%) ~ 4.04s 4.20s p=0.931 n=12
Total Time 15.14s (± 0.23%) 15.15s (± 0.26%) ~ 15.06s 15.25s p=0.840 n=12
material-ui - bun (v1.0.15, x64)
Memory used 553,614k (± 3.85%) 571,718k (± 3.34%) +18,104k (+ 3.27%) 522,010k 592,798k p=0.040 n=12
Parse Time 2.70s (± 1.01%) 2.70s (± 0.60%) ~ 2.66s 2.74s p=0.727 n=12
Bind Time 0.83s (± 2.72%) 0.85s (± 2.46%) ~ 0.79s 0.88s p=0.362 n=12
Check Time 18.88s (± 0.46%) 18.96s (± 0.34%) ~ 18.81s 19.16s p=0.183 n=12
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=12
Total Time 22.42s (± 0.48%) 22.52s (± 0.35%) ~ 22.38s 22.76s p=0.088 n=12
mui-docs - bun (v1.0.15, x64)
Memory used 1,983,666k (± 0.80%) 1,973,926k (± 1.28%) ~ 1,887,969k 1,992,787k p=0.665 n=12
Parse Time 6.63s (± 0.36%) 6.61s (± 0.61%) ~ 6.54s 6.74s p=0.212 n=12
Bind Time 1.77s (± 2.09%) 1.80s (± 1.44%) ~ 1.71s 1.87s p=0.132 n=12
Check Time 65.59s (± 0.65%) 65.52s (± 0.44%) ~ 64.95s 66.44s p=0.751 n=12
Emit Time 0.28s (±12.92%) 0.27s (±13.29%) ~ 0.21s 0.34s p=1.000 n=12
Total Time 74.26s (± 0.56%) 74.20s (± 0.37%) ~ 73.55s 74.99s p=0.840 n=12
self-build-src - bun (v1.0.15, x64)
Memory used 8,758,243k (± 8.46%) 9,018,811k (± 1.00%) ~ 8,719,656k 9,270,756k p=0.470 n=12
Parse Time 4.95s (± 1.14%) 4.88s (± 1.08%) ~ 4.71s 5.02s p=0.088 n=12
Bind Time 2.04s (± 1.02%) 2.05s (± 1.36%) ~ 1.98s 2.12s p=0.908 n=12
Check Time 34.89s (± 3.49%) 33.44s (± 1.86%) ~ 32.69s 36.34s p=0.106 n=12
Emit Time 2.12s (± 1.80%) 2.10s (± 1.18%) ~ 2.04s 2.16s p=0.310 n=12
Total Time 43.99s (± 2.89%) 42.49s (± 1.53%) ~ 41.82s 45.56s p=0.141 n=12
self-compiler - bun (v1.0.15, x64)
Memory used 410,943k (± 0.16%) 411,288k (± 0.40%) ~ 406,411k 415,243k p=0.312 n=12
Parse Time 2.66s (± 0.86%) 2.65s (± 0.49%) ~ 2.62s 2.70s p=0.538 n=12
Bind Time 1.31s (± 1.19%) 1.30s (± 1.39%) ~ 1.26s 1.35s p=0.433 n=12
Check Time 17.10s (± 0.59%) 17.02s (± 0.24%) ~ 16.91s 17.12s p=0.326 n=12
Emit Time 1.08s (± 1.86%) 1.09s (± 2.91%) ~ 1.04s 1.20s p=0.930 n=12
Total Time 22.15s (± 0.50%) 22.07s (± 0.19%) ~ 21.96s 22.18s p=0.418 n=12
vscode - bun (v1.0.15, x64)
Memory used 2,958,434k (± 0.19%) 2,960,693k (± 0.15%) ~ 2,948,549k 2,969,461k p=0.507 n=12
Parse Time 10.55s (± 0.45%) 10.55s (± 0.36%) ~ 10.43s 10.62s p=0.602 n=12
Bind Time 3.50s (± 0.42%) 3.48s (± 0.44%) -0.02s (- 0.69%) 3.43s 3.52s p=0.019 n=12
Check Time 67.00s (± 1.15%) 67.69s (± 0.31%) ~ 67.22s 68.23s p=0.340 n=12
Emit Time 18.45s (± 5.40%) 17.45s (± 0.41%) ~ 17.21s 17.59s p=0.525 n=12
Total Time 99.49s (± 0.31%) 99.17s (± 0.25%) ~ 98.42s 99.65s p=0.157 n=12
webpack - bun (v1.0.15, x64)
Memory used 393,574k (± 2.23%) 390,501k (± 1.66%) ~ 384,825k 422,895k p=0.729 n=12
Parse Time 2.53s (± 0.43%) 2.53s (± 0.46%) ~ 2.51s 2.58s p=0.428 n=12
Bind Time 1.33s (± 1.12%) 1.33s (± 0.92%) ~ 1.30s 1.37s p=0.977 n=12
Check Time 12.34s (± 0.23%) 12.34s (± 0.28%) ~ 12.25s 12.45s p=0.839 n=12
Emit Time 0.01s (±37.93%) 0.00s (±217.90%) 🟩-0.01s (-88.89%) 0.00s 0.01s p=0.001 n=12
Total Time 16.21s (± 0.19%) 16.20s (± 0.22%) ~ 16.11s 16.30s p=0.839 n=12
xstate - bun (v1.0.15, x64)
Memory used 382,543k (± 0.36%) 382,933k (± 0.28%) ~ 379,439k 385,678k p=0.885 n=12
Parse Time 3.22s (± 0.24%) 3.22s (± 0.23%) ~ 3.20s 3.24s p=0.463 n=12
Bind Time 1.22s (± 0.43%) 1.22s (± 0.64%) ~ 1.20s 1.24s p=0.810 n=12
Check Time 3.53s (± 0.23%) 3.53s (± 0.30%) ~ 3.51s 3.55s p=0.480 n=12
Emit Time 0.21s (± 1.54%) 0.21s (± 1.28%) ~ 0.20s 0.22s p=0.684 n=12
Total Time 8.17s (± 0.20%) 8.18s (± 0.21%) ~ 8.14s 8.23s p=0.728 n=12
System info unknown
Hosts
  • bun (v1.0.15, x64)
Scenarios
  • Angular - bun (v1.0.15, x64)
  • Compiler-Unions - bun (v1.0.15, x64)
  • Monaco - bun (v1.0.15, x64)
  • TFS - bun (v1.0.15, x64)
  • material-ui - bun (v1.0.15, x64)
  • mui-docs - bun (v1.0.15, x64)
  • self-build-src - bun (v1.0.15, x64)
  • self-compiler - bun (v1.0.15, x64)
  • vscode - bun (v1.0.15, x64)
  • webpack - bun (v1.0.15, x64)
  • xstate - bun (v1.0.15, x64)
Benchmark Name Iterations
Current pr 12
Baseline baseline 12

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - bun (v1.0.15, x64)
Execution time 170.96ms (± 3.41%) 171.11ms (± 3.42%) +0.15ms (+ 0.09%) 169.88ms 445.44ms p=0.000 n=600
tsserverlibrary-startup - bun (v1.0.15, x64)
Execution time 265.72ms (± 3.40%) 265.68ms (± 3.46%) ~ 263.91ms 697.55ms p=0.050 n=600
typescript-startup - bun (v1.0.15, x64)
Execution time 259.16ms (± 3.49%) 259.01ms (± 3.49%) ~ 257.14ms 684.10ms p=0.116 n=600
System info unknown
Hosts
  • bun (v1.0.15, x64)
Scenarios
  • tsc-startup - bun (v1.0.15, x64)
  • tsserverlibrary-startup - bun (v1.0.15, x64)
  • typescript-startup - bun (v1.0.15, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey jakebailey changed the title Refactor performance hooks to hopefully fix Windows / Node 14 flake, simplify Enable performance.now when possible, re-disable unconditional perf marking on all Node versions Mar 22, 2024
@jakebailey jakebailey changed the title Enable performance.now when possible, re-disable unconditional perf marking on all Node versions Use performance.now when possible, re-disable unconditional perf marking on all Node versions Mar 22, 2024
@@ -116,6 +98,4 @@ export function tryGetNativePerformanceHooks() {
*
* @internal
*/
export const timestamp = nativePerformance ? () => nativePerformance.now() :
Date.now ? Date.now :
() => +(new Date());
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While here, removing this branch. Date.now is most definitely defined.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't guaranteed to be when we targeted ES3, but since we've moved up our minimum target to run tsc this makes sense.

export const timestamp = nativePerformance ? () => nativePerformance.now() :
Date.now ? Date.now :
() => +(new Date());
export const timestamp = nativePerformanceTime ? () => nativePerformanceTime.now() : Date.now;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW this must be a closure because Node 19+ require this to be correct when calling, so we can't just rip off a method.

@jakebailey
Copy link
Member Author

And just to go back to the original Date.now thing, MDN says:

JavaScript Date times are subject to system clock skew or adjustments. This means that the value of time may not always be monotonically increasing. The main purpose of Date objects is to display time and date information to the user and so many operating systems run a daemon which regularly synchronizes time. It might be that the clock is tweaked a few milliseconds several times per hour.

So, this that probably explains things. Thank you random chance for finding this perf win.

@jakebailey jakebailey merged commit 3c63740 into microsoft:main Mar 22, 2024
25 checks passed
@jakebailey jakebailey deleted the perf-hooks branch March 22, 2024 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants