New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports incremental build in tsc --b --w mode #29161

Merged
merged 23 commits into from Jan 17, 2019

Conversation

Projects
None yet
5 participants
@sheetalkamat
Copy link
Member

sheetalkamat commented Dec 26, 2018

Some notes:

  • Since we use builder, the first time program is created, its going to emit all the files and not just changed - just like it does in tsc --watch mode
  • We do not cache source files yet and that's something to be looked at later.
    Two things can be done:
    • To keep source files for program thats being updated in last few cycles
    • To release the source files if size exceeds or program is not used for few build cycles
    • Adding this to #28640
  • We use hash from source file's text as the version.
  • Because in tsbuild we get semantic diagnostics before emit, the EmitAndSemanticDiagnosticsBuilderProgram's existing assumption was not satisfied (that is changed files are already emitted before getting semantic diagnostics). So in such case, we store list of files to emit later so as to help with that case.
  • In this we also have a new Builder method to release the program and its source files to ensure that we aren't holding on program after emit and/or semantic diagnostics are complete. That way the builder state can be in memory for comparison and finding out what are new changes that need emitting while creating new program but we aren't holding onto these costly (memory wise) resources. (all the program methods are thus available only till program is released.

Here are the perf numbers:
Local change::

File change detected. Starting incremental compilation...

Project 'SubProject 4' is out of date because oldest output 'someOutputFile.js' is older than newest input 'SubProject 2'

Building project 'SubProject 4'...

                                Updating unchanged output timestamps of project 'SubProject 4'...

Before                          After
Files:                  8305    Files:                  8305
Lines:                509633    Lines:                509633
Nodes:               1741390    Nodes:               1741390
Identifiers:          563590    Identifiers:          563590
Symbols:              597260    Symbols:              316035
Types:                175190    Types:                   232
Memory used:        1418992K    Memory used:        1859282K
I/O Read time:         2.83s    I/O Read time:         3.85s
Parse time:            2.43s    Parse time:            3.42s
Program time:         23.45s    Program time:         33.07s
Bind time:             1.56s    Bind time:             1.71s
Check time:           11.86s    Check time:            0.04s
transformTime time:   13.72s    transformTime time:    0.25s
Source Map time:       2.58s    Source Map time:       0.01s
commentTime time:      1.46s    commentTime time:      0.00s
printTime time:       27.08s    printTime time:        0.28s
Emit time:            27.08s    Emit time:             0.28s
I/O Write time:       16.16s    I/O Write time:        0.04s
Total time:           63.96s    Total time:           35.09s
Building project 'SubProject 5'...

                                Updating unchanged output timestamps of project 'SubProject 5'...

Before                          After
Files:                  1875    Files:            1875
Lines:                207731    Lines:          207731
Nodes:                632955    Nodes:          632955
Identifiers:          212573    Identifiers:    212573
Symbols:              162334    Symbols:        125245
Types:                 22975    Types:              71
Memory used:        1095817K    Memory used:   311574K
I/O Read time:         1.13s    I/O Read time:   0.74s
Parse time:            2.03s    Parse time:      0.87s
Program time:          9.64s    Program time:    5.97s
Bind time:             0.67s    Bind time:       0.48s
Check time:            2.17s
transformTime time:    1.22s
Source Map time:       0.77s
commentTime time:      0.34s
printTime time:        4.16s
Emit time:             4.16s
I/O Write time:        1.92s
Total time:           16.64s    Total time:      6.45s
Building project 'SubProject 6'...

                                Updating unchanged output timestamps of project 'SubProject 6'...

Before                          After
Files:                  2396    Files:            2396
Lines:                221677    Lines:          221677
Nodes:                669898    Nodes:          669898
Identifiers:          221017    Identifiers:    221017
Symbols:              189874    Symbols:        143939
Types:                 30412    Types:              71
Memory used:        1442324K    Memory used:   333909K
I/O Read time:         0.92s    I/O Read time:   0.91s
Parse time:            1.40s    Parse time:      1.17s
Program time:          8.91s    Program time:    7.73s
Bind time:             0.43s    Bind time:       0.39s
Check time:            1.74s
transformTime time:    0.67s
Source Map time:       0.18s
commentTime time:      0.10s
printTime time:        1.67s
Emit time:             1.67s
I/O Write time:        1.40s
Total time:           12.74s    Total time:      8.12s
Building project 'SubProject 7'...

                                Updating unchanged output timestamps of project 'SubProject 7'...

Before                          After
Files:                  9326    Files:            9326
Lines:                552828    Lines:          552830
Nodes:               1896877    Nodes:         1896879
Identifiers:          605603    Identifiers:    605605
Symbols:              608250    Symbols:        330625
Types:                173986    Types:              71
Memory used:        2367385K    Memory used:   953536K
I/O Read time:         4.22s    I/O Read time:   3.84s
Parse time:            3.18s    Parse time:      2.02s
Program time:         36.41s    Program time:   27.79s
Bind time:             2.73s    Bind time:       1.19s
Check time:           13.15s
transformTime time:   15.36s
Source Map time:       2.35s
commentTime time:      1.52s
printTime time:       28.95s
Emit time:            28.95s
I/O Write time:       13.41s
Total time:           81.23s    Total time:     28.98s
Building project 'SubProject 8'...

                                Updating unchanged output timestamps of project 'SubProject 8'...

Before                          After
Files:                  6146    Files:             6146
Lines:                367180    Lines:           367182
Nodes:               1201703    Nodes:          1201705
Identifiers:          386452    Identifiers:     386454
Symbols:              406182    Symbols:         232990
Types:                 94077    Types:               71
Memory used:        1704251K    Memory used:   1333457K
I/O Read time:         3.06s    I/O Read time:    2.50s
Parse time:            3.61s    Parse time:       2.06s
Program time:         31.86s    Program time:    19.05s
Bind time:             1.76s    Bind time:        0.77s
Check time:           10.90s
transformTime time:    7.64s
Source Map time:       0.96s
commentTime time:      0.91s
printTime time:       13.81s
Emit time:            13.81s
I/O Write time:        6.67s
Total time:           58.33s    Total time:      19.82s
Building project 'SubProject 9'...

                                Updating unchanged output timestamps of project 'SubProject 9'...

Before                          After
Files:                  3781    Files:             3781
Lines:                264899    Lines:           264899
Nodes:                834360    Nodes:           834360
Identifiers:          278052    Identifiers:     278052
Symbols:              261052    Symbols:         178924
Types:                 52797    Types:               71
Memory used:        2175398K    Memory used:   1620896K
I/O Read time:         1.67s    I/O Read time:    1.51s
Parse time:            1.98s    Parse time:       1.19s
Program time:         16.03s    Program time:    11.75s
Bind time:             0.77s    Bind time:        0.49s
Check time:            4.86s
transformTime time:    4.24s
Source Map time:       0.40s
commentTime time:      0.21s
printTime time:        6.63s
Emit time:             6.63s
I/O Write time:        2.03s
Total time:           28.29s    Total time:      12.24s


Solution Total Time:
Before: 261.19, After: 110.7

Local change:

File change detected. Starting incremental compilation...

Project 'SubProject 4' is out of date because oldest output 'someOutputFile.js' is older than newest input 'SubProject 2'

Building project 'SubProject 4'...

                                Updating unchanged output timestamps of project 'SubProject 4'...

Before                          After
Files:                  8305    Files:                 8305
Lines:                509635    Lines:               509635
Nodes:               1741406    Nodes:              1741406
Identifiers:          563597    Identifiers:         563597
Symbols:              597262    Symbols:             315957
Types:                175191    Types:                  114
Memory used:        3243603K    Memory used:        690192K
I/O Read time:         3.23s    I/O Read time:        3.28s
Parse time:            4.00s    Parse time:           2.31s
Program time:         35.91s    Program time:        24.59s
Bind time:             1.75s    Bind time:            1.27s
Check time:           15.76s    Check time:           0.01s
transformTime time:   15.10s    transformTime time:   0.00s
Source Map time:       2.93s    Source Map time:      0.00s
commentTime time:      1.85s    commentTime time:     0.00s
printTime time:       30.68s    printTime time:       0.00s
Emit time:            30.68s    Emit time:            0.00s
I/O Write time:       14.47s    I/O Write time:       0.00s
Total time:           84.10s    Total time:          25.87s
                                Project 'SubProject 5' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 5'...

                                Project 'SubProject 6' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 6'...

                                Project 'SubProject 7' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 7'...

                                Project 'SubProject 8' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 8'...

                                Project 'SubProject 9' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 9'...


Solution Total Time:
Before: 84.1, After: 25.87
Found 0 errors. Watching for file changes.

sheetalkamat added some commits Dec 14, 2018

Use watch factory instead of direct host functions in tsbuild to prov…
…ide detailed information on watch invokations
Revert BuilderProgram to be redirected object to Program in preparati…
…on to set Program in state to undefined for storing.
Use oldProgram to create the new Program.
This helps in storing the semantic diagnostics
Update the timestamps of outputs that dont need to be written because…
… of incremental build

This ensures that after `tsbuild` after incremental build of `tsbuild -w` doesnt result in unnecessary rebuilds
Handle prepend in incremental build. Always emit when program uses pr…
…oject reference with prepend since it cant tell changes in js/map files
Handle declaration emit errors in tsbuild mode by backing up builder …
…state

This helps us revert to state where we pretend as if emit is not done (since we do not do emit if there are errors)
@sheetalkamat

This comment has been minimized.

Copy link
Member Author

sheetalkamat commented Jan 10, 2019

Ping @sandersn @rbuckton @weswigham @DanielRosenwasser Would like to get this in soon.

Show resolved Hide resolved src/compiler/core.ts Outdated

@sandersn sandersn self-assigned this Jan 14, 2019

Show resolved Hide resolved src/compiler/builder.ts Outdated
Show resolved Hide resolved src/compiler/builder.ts Outdated

sheetalkamat added some commits Jan 14, 2019

Show resolved Hide resolved src/compiler/tsbuild.ts Outdated
Show resolved Hide resolved src/compiler/tsbuild.ts Outdated
Show resolved Hide resolved src/compiler/tsbuild.ts Outdated
Show resolved Hide resolved src/compiler/watch.ts Outdated
Show resolved Hide resolved src/compiler/watch.ts Outdated
@sandersn
Copy link
Member

sandersn left a comment

I looked over everything but I need to learn more about watch mode + tsbuild in person before signing off.

Show resolved Hide resolved src/compiler/core.ts Outdated
Show resolved Hide resolved src/compiler/tsbuild.ts Outdated
Show resolved Hide resolved src/compiler/tsbuild.ts Outdated
Show resolved Hide resolved src/compiler/builder.ts Outdated
Show resolved Hide resolved src/compiler/watch.ts Outdated
Show resolved Hide resolved src/compiler/watch.ts Outdated
Show resolved Hide resolved src/compiler/watch.ts Outdated
Show resolved Hide resolved src/compiler/program.ts Outdated
Show resolved Hide resolved src/compiler/builder.ts Outdated
@sheetalkamat

This comment has been minimized.

Copy link
Member Author

sheetalkamat commented Jan 17, 2019

@sandersn comments?

@sheetalkamat sheetalkamat merged commit 9bd2365 into master Jan 17, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
license/cla All CLA requirements met.
Details

@sheetalkamat sheetalkamat deleted the incrementalBuild branch Jan 17, 2019

@xaviergonz

This comment has been minimized.

Copy link

xaviergonz commented Jan 17, 2019

Just wondering, wouldn't it be even faster if the cache was also based on file size as first check before doing the actual hash?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment