Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always fully compute variance structurally #48080

Closed
wants to merge 7 commits into from
Closed

Conversation

ahejlsberg
Copy link
Member

Fixes #44572.

@typescript-bot typescript-bot added the For Milestone Bug PRs that fix a bug with a specific milestone label Mar 1, 2022
@ahejlsberg
Copy link
Member Author

@typescript-bot test this
@typescript-bot user test this inline
@typescript-bot run dt
@typescript-bot perf test faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 1, 2022

Heya @ahejlsberg, I've started to run the extended test suite on this PR at de4a166. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 1, 2022

Heya @ahejlsberg, I've started to run the abridged perf test suite on this PR at de4a166. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 1, 2022

Heya @ahejlsberg, I've started to run the diff-based community code test suite on this PR at de4a166. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 1, 2022

Heya @ahejlsberg, I've started to run the parallelized Definitely Typed test suite on this PR at de4a166. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

@ahejlsberg
Great news! no new errors were found between main..refs/pull/48080/merge

@typescript-bot
Copy link
Collaborator

@ahejlsberg
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..48080

Metric main 48080 Delta Best Worst
Angular - node (v14.15.1, x64)
Memory used 333,049k (± 0.01%) 333,044k (± 0.01%) -5k (- 0.00%) 332,915k 333,125k
Parse Time 2.02s (± 0.47%) 2.02s (± 0.55%) +0.01s (+ 0.30%) 2.00s 2.05s
Bind Time 0.88s (± 0.25%) 0.88s (± 0.96%) -0.00s (- 0.23%) 0.87s 0.91s
Check Time 5.51s (± 0.37%) 5.49s (± 0.48%) -0.02s (- 0.40%) 5.45s 5.58s
Emit Time 6.27s (± 0.56%) 6.26s (± 0.57%) -0.01s (- 0.19%) 6.17s 6.35s
Total Time 14.68s (± 0.31%) 14.65s (± 0.41%) -0.03s (- 0.23%) 14.52s 14.78s
Compiler-Unions - node (v14.15.1, x64)
Memory used 193,823k (± 0.61%) 193,177k (± 0.60%) -646k (- 0.33%) 191,818k 195,128k
Parse Time 0.86s (± 0.75%) 0.85s (± 0.43%) -0.00s (- 0.35%) 0.85s 0.86s
Bind Time 0.56s (± 0.93%) 0.56s (± 0.53%) +0.00s (+ 0.18%) 0.56s 0.57s
Check Time 7.43s (± 0.63%) 7.37s (± 0.52%) -0.06s (- 0.75%) 7.31s 7.45s
Emit Time 2.49s (± 1.12%) 2.47s (± 0.57%) -0.02s (- 0.72%) 2.44s 2.50s
Total Time 11.34s (± 0.59%) 11.26s (± 0.43%) -0.08s (- 0.68%) 11.16s 11.36s
Monaco - node (v14.15.1, x64)
Memory used 325,126k (± 0.01%) 325,131k (± 0.01%) +5k (+ 0.00%) 325,086k 325,168k
Parse Time 1.56s (± 0.38%) 1.56s (± 0.58%) -0.00s (- 0.19%) 1.55s 1.59s
Bind Time 0.77s (± 0.44%) 0.77s (± 0.47%) -0.01s (- 0.91%) 0.76s 0.77s
Check Time 5.40s (± 0.39%) 5.41s (± 0.52%) +0.01s (+ 0.24%) 5.35s 5.46s
Emit Time 3.31s (± 0.83%) 3.29s (± 0.35%) -0.02s (- 0.51%) 3.26s 3.31s
Total Time 11.04s (± 0.43%) 11.03s (± 0.34%) -0.01s (- 0.08%) 10.94s 11.10s
TFS - node (v14.15.1, x64)
Memory used 288,669k (± 0.01%) 288,656k (± 0.01%) -13k (- 0.00%) 288,607k 288,734k
Parse Time 1.33s (± 1.98%) 1.33s (± 1.76%) +0.00s (+ 0.15%) 1.30s 1.41s
Bind Time 0.74s (± 1.37%) 0.73s (± 0.64%) -0.01s (- 0.82%) 0.72s 0.74s
Check Time 5.06s (± 0.57%) 5.06s (± 0.59%) +0.00s (+ 0.02%) 5.01s 5.14s
Emit Time 3.53s (± 1.82%) 3.47s (± 1.98%) -0.06s (- 1.73%) 3.37s 3.62s
Total Time 10.66s (± 0.73%) 10.60s (± 0.84%) -0.06s (- 0.56%) 10.45s 10.81s
material-ui - node (v14.15.1, x64)
Memory used 446,469k (± 0.00%) 446,288k (± 0.06%) -181k (- 0.04%) 445,162k 446,458k
Parse Time 1.85s (± 0.31%) 1.85s (± 0.46%) +0.00s (+ 0.05%) 1.84s 1.87s
Bind Time 0.70s (± 1.15%) 0.70s (± 1.17%) -0.00s (- 0.14%) 0.68s 0.72s
Check Time 12.86s (± 0.57%) 12.80s (± 0.54%) -0.06s (- 0.44%) 12.60s 12.95s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 15.40s (± 0.49%) 15.35s (± 0.49%) -0.06s (- 0.38%) 15.14s 15.51s
xstate - node (v14.15.1, x64)
Memory used 535,257k (± 0.01%) 536,585k (± 0.01%) +1,328k (+ 0.25%) 536,532k 536,656k
Parse Time 2.58s (± 0.35%) 2.57s (± 0.49%) -0.02s (- 0.62%) 2.55s 2.60s
Bind Time 1.16s (± 1.05%) 1.15s (± 0.72%) -0.01s (- 0.60%) 1.14s 1.18s
Check Time 1.50s (± 0.55%) 1.55s (± 0.26%) +0.06s (+ 3.68%) 1.54s 1.56s
Emit Time 0.07s (± 3.14%) 0.07s (± 4.13%) +0.00s (+ 1.41%) 0.07s 0.08s
Total Time 5.32s (± 0.32%) 5.35s (± 0.20%) +0.04s (+ 0.66%) 5.33s 5.37s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-210-generic
Architecturex64
Available Memory16 GB
Available Memory5 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 48080 10
Baseline main 10

Developer Information:

Download Benchmark

@RyanCavanaugh
Copy link
Member

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 2, 2022

Heya @RyanCavanaugh, I've started to run the tarball bundle task on this PR at de4a166. You can monitor the build here.

@ahejlsberg
Copy link
Member Author

@typescript-bot perf test faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 2, 2022

Heya @ahejlsberg, I've started to run the abridged perf test suite on this PR at d059791. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@ahejlsberg
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..48080

Metric main 48080 Delta Best Worst
Angular - node (v14.15.1, x64)
Memory used 333,049k (± 0.01%) 333,025k (± 0.01%) -23k (- 0.01%) 332,980k 333,088k
Parse Time 2.02s (± 0.47%) 2.02s (± 0.70%) +0.01s (+ 0.35%) 2.00s 2.06s
Bind Time 0.88s (± 0.25%) 0.88s (± 0.38%) -0.00s (- 0.11%) 0.87s 0.89s
Check Time 5.51s (± 0.37%) 5.49s (± 0.50%) -0.02s (- 0.33%) 5.43s 5.57s
Emit Time 6.27s (± 0.56%) 6.26s (± 0.39%) -0.01s (- 0.22%) 6.21s 6.32s
Total Time 14.68s (± 0.31%) 14.66s (± 0.26%) -0.03s (- 0.18%) 14.55s 14.75s
Compiler-Unions - node (v14.15.1, x64)
Memory used 193,823k (± 0.61%) 193,115k (± 0.58%) -709k (- 0.37%) 191,870k 195,167k
Parse Time 0.86s (± 0.75%) 0.85s (± 0.52%) -0.01s (- 0.70%) 0.84s 0.86s
Bind Time 0.56s (± 0.93%) 0.56s (± 0.53%) +0.00s (+ 0.18%) 0.56s 0.57s
Check Time 7.43s (± 0.63%) 7.37s (± 0.53%) -0.06s (- 0.75%) 7.32s 7.48s
Emit Time 2.49s (± 1.12%) 2.46s (± 0.87%) -0.02s (- 0.88%) 2.42s 2.51s
Total Time 11.34s (± 0.59%) 11.25s (± 0.48%) -0.08s (- 0.75%) 11.16s 11.37s
Monaco - node (v14.15.1, x64)
Memory used 325,126k (± 0.01%) 325,133k (± 0.00%) +7k (+ 0.00%) 325,090k 325,152k
Parse Time 1.56s (± 0.38%) 1.56s (± 0.51%) -0.00s (- 0.13%) 1.54s 1.58s
Bind Time 0.77s (± 0.44%) 0.77s (± 0.44%) -0.01s (- 0.78%) 0.76s 0.77s
Check Time 5.40s (± 0.39%) 5.37s (± 0.35%) -0.02s (- 0.39%) 5.34s 5.42s
Emit Time 3.31s (± 0.83%) 3.28s (± 0.39%) -0.02s (- 0.73%) 3.25s 3.30s
Total Time 11.04s (± 0.43%) 10.99s (± 0.24%) -0.05s (- 0.43%) 10.94s 11.05s
TFS - node (v14.15.1, x64)
Memory used 288,669k (± 0.01%) 288,667k (± 0.01%) -2k (- 0.00%) 288,619k 288,697k
Parse Time 1.33s (± 1.98%) 1.32s (± 2.40%) -0.01s (- 0.60%) 1.28s 1.42s
Bind Time 0.74s (± 1.37%) 0.74s (± 1.58%) +0.00s (+ 0.00%) 0.72s 0.77s
Check Time 5.06s (± 0.57%) 5.05s (± 0.41%) -0.01s (- 0.22%) 5.03s 5.13s
Emit Time 3.53s (± 1.82%) 3.59s (± 1.55%) +0.06s (+ 1.61%) 3.39s 3.69s
Total Time 10.66s (± 0.73%) 10.70s (± 0.71%) +0.04s (+ 0.38%) 10.46s 10.86s
material-ui - node (v14.15.1, x64)
Memory used 446,469k (± 0.00%) 446,314k (± 0.06%) -155k (- 0.03%) 445,182k 446,488k
Parse Time 1.85s (± 0.31%) 1.85s (± 0.55%) -0.00s (- 0.05%) 1.83s 1.87s
Bind Time 0.70s (± 1.15%) 0.70s (± 0.74%) +0.00s (+ 0.14%) 0.69s 0.71s
Check Time 12.86s (± 0.57%) 12.76s (± 0.56%) -0.10s (- 0.74%) 12.65s 12.90s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 15.40s (± 0.49%) 15.31s (± 0.48%) -0.10s (- 0.62%) 15.20s 15.46s
xstate - node (v14.15.1, x64)
Memory used 535,257k (± 0.01%) 536,574k (± 0.00%) +1,317k (+ 0.25%) 536,539k 536,633k
Parse Time 2.58s (± 0.35%) 2.58s (± 0.32%) -0.01s (- 0.39%) 2.56s 2.60s
Bind Time 1.16s (± 1.05%) 1.15s (± 0.56%) -0.01s (- 0.78%) 1.14s 1.17s
Check Time 1.50s (± 0.55%) 1.55s (± 0.53%) +0.06s (+ 3.88%) 1.54s 1.58s
Emit Time 0.07s (± 3.14%) 0.07s (± 3.14%) 0.00s ( 0.00%) 0.07s 0.08s
Total Time 5.32s (± 0.32%) 5.36s (± 0.27%) +0.04s (+ 0.81%) 5.34s 5.39s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-210-generic
Architecturex64
Available Memory16 GB
Available Memory5 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 48080 10
Baseline main 10

Developer Information:

Download Benchmark

@typescript-bot
Copy link
Collaborator

Hey @RyanCavanaugh, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/121047/artifacts?artifactName=tgz&fileId=A941446E3AD1E9DD2C00F39A3CBA6B60B8F87AE895A0C53B3DB474331FB6762002&fileName=/typescript-4.7.0-insiders.20220302.tgz"
    }
}

and then running npm install.

@ahejlsberg
Copy link
Member Author

Tests are clean. Performance is slightly impacted in xstate but otherwise improved. I think this is an acceptable fix.

Copy link
Member

@weswigham weswigham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, but I wonder if the inconpleteVariancesObserved flag needs to be saved and restored from the cached relation result similarly to variance markers themselves.

@ahejlsberg
Copy link
Member Author

This looks good, but I wonder if the incompleteVariancesObserved flag needs to be saved and restored from the cached relation result similarly to variance markers themselves.

That shouldn't be necessary because we don't cache relations that observed incomplete variances (they result in Ternary.Unknown, and those never get cached).

@Andarist
Copy link
Contributor

Andarist commented Mar 3, 2022

This PR doesn't fix the problem reported here while the previous PR was fixing it.

EDIT:// I've also rechecked my other report that was being fixed by the previous PR and it isn't handled by this PR here, the issue in question: #45859

@ahejlsberg
Copy link
Member Author

ahejlsberg commented Mar 4, 2022

Sadly there are still issues with the fix in this PR as well as the fix in (the now closed) #45628. The core issue looks to be mutually recursive types in which variance is flipped by one of the types. For example:

type Foo<T> = {
  x: T;
  f: FooFn<T>;
}

type FooFn<T> = (foo: Foo<T>) => void;

declare let foo1: Foo<string>;
declare let foo2: Foo<unknown>;

declare let fn1: FooFn<string>;
declare let fn2: FooFn<unknown>;

foo1 = foo2;  // Error
foo2 = foo1;  // Error

fn1 = fn2;  // Error
fn2 = fn1;  // Error

Above, the x property in Foo<T> witnesses T co-variantly and the f property witnesses T recursively and contra-variantly. The combination should cause T to be considered invariant, and it is in the example above. But with a small change to FooFn<T> things fall apart:

type FooFn<T> = (foo: Foo<T[]) => void;

Now, because T isn't referenced as a "naked" type parameter, we don't examine the Foo<T[]> reference structurally, but rather attempt to rely on the computed variance--which is in the process of being computed. So, results come out wrong and are potentially sensitive to order of computation. For example, with this PR we get the following behavior:

foo1 = foo2;  // Error
foo2 = foo1;

fn1 = fn2;
fn2 = fn1;

meaning co-variance for Foo<T> and no witnessing in FooFn<T>. With #45628 we get

foo1 = foo2;  // Error
foo2 = foo1;

fn1 = fn2;
fn2 = fn1;  // Error

i.e. contra-variance for FooFn<T> which is slightly more consistent, but still wrong.

The tricky aspect of this example (and others like it) is that invariance isn't revealed unless we examine multiple circular type references structurally and in-depth. Which of course is exactly what we're trying to avoid with the variance computation in the first place.

@Andarist
Copy link
Contributor

Andarist commented Mar 5, 2022

Now, because T isn't referenced as a "naked" type parameter, we don't examine the Foo<T[]> reference structurally, but rather attempt to rely on the computed variance

Why is this making a difference? Is it purely an optimization for the most common case or something like that? I don't quite understand why naked and not-naked type parameters go through a different code path here 🤔

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 6, 2022

Heya @ahejlsberg, I've started to run the diff-based community code test suite on this PR at a323037. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 6, 2022

Heya @ahejlsberg, I've started to run the parallelized Definitely Typed test suite on this PR at a323037. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

@ahejlsberg
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..48080

Metric main 48080 Delta Best Worst
Angular - node (v14.15.1, x64)
Memory used 332,984k (± 0.01%) 331,371k (± 0.01%) -1,613k (- 0.48%) 331,325k 331,434k
Parse Time 2.03s (± 0.86%) 2.02s (± 0.60%) -0.00s (- 0.15%) 2.01s 2.07s
Bind Time 0.88s (± 0.56%) 0.88s (± 0.56%) 0.00s ( 0.00%) 0.87s 0.89s
Check Time 5.57s (± 0.34%) 5.57s (± 0.48%) -0.00s (- 0.04%) 5.52s 5.63s
Emit Time 6.26s (± 0.91%) 6.25s (± 0.40%) -0.01s (- 0.13%) 6.21s 6.32s
Total Time 14.73s (± 0.51%) 14.72s (± 0.38%) -0.01s (- 0.08%) 14.62s 14.88s
Compiler-Unions - node (v14.15.1, x64)
Memory used 194,104k (± 0.55%) 193,142k (± 0.62%) -962k (- 0.50%) 191,104k 194,498k
Parse Time 0.85s (± 0.80%) 0.86s (± 0.55%) +0.00s (+ 0.35%) 0.85s 0.87s
Bind Time 0.56s (± 1.15%) 0.56s (± 1.04%) -0.01s (- 0.89%) 0.55s 0.57s
Check Time 7.44s (± 0.59%) 7.43s (± 0.48%) -0.01s (- 0.15%) 7.35s 7.49s
Emit Time 2.50s (± 1.21%) 2.47s (± 0.72%) -0.03s (- 1.28%) 2.43s 2.51s
Total Time 11.36s (± 0.63%) 11.31s (± 0.33%) -0.04s (- 0.39%) 11.21s 11.37s
Monaco - node (v14.15.1, x64)
Memory used 325,203k (± 0.00%) 324,930k (± 0.00%) -273k (- 0.08%) 324,903k 324,970k
Parse Time 1.56s (± 0.71%) 1.56s (± 0.71%) 0.00s ( 0.00%) 1.54s 1.59s
Bind Time 0.78s (± 1.07%) 0.77s (± 1.14%) -0.00s (- 0.26%) 0.76s 0.80s
Check Time 5.48s (± 0.42%) 5.48s (± 0.42%) +0.01s (+ 0.13%) 5.43s 5.54s
Emit Time 3.31s (± 0.74%) 3.32s (± 0.70%) +0.01s (+ 0.33%) 3.26s 3.38s
Total Time 11.13s (± 0.40%) 11.14s (± 0.48%) +0.02s (+ 0.16%) 11.02s 11.26s
TFS - node (v14.15.1, x64)
Memory used 288,770k (± 0.01%) 288,666k (± 0.01%) -104k (- 0.04%) 288,606k 288,725k
Parse Time 1.33s (± 1.26%) 1.31s (± 0.81%) -0.02s (- 1.43%) 1.29s 1.33s
Bind Time 0.73s (± 0.67%) 0.74s (± 0.93%) +0.00s (+ 0.14%) 0.72s 0.75s
Check Time 5.14s (± 0.31%) 5.13s (± 0.44%) -0.01s (- 0.21%) 5.07s 5.18s
Emit Time 3.53s (± 1.92%) 3.55s (± 2.00%) +0.02s (+ 0.57%) 3.38s 3.68s
Total Time 10.72s (± 0.85%) 10.72s (± 0.68%) -0.00s (- 0.01%) 10.54s 10.86s
material-ui - node (v14.15.1, x64)
Memory used 453,151k (± 0.00%) 450,137k (± 0.07%) -3,013k (- 0.66%) 449,011k 450,364k
Parse Time 1.86s (± 0.55%) 1.85s (± 0.63%) -0.00s (- 0.22%) 1.84s 1.89s
Bind Time 0.71s (± 1.05%) 0.71s (± 1.07%) -0.00s (- 0.56%) 0.69s 0.72s
Check Time 20.58s (± 0.85%) 20.33s (± 0.75%) -0.25s (- 1.20%) 20.06s 20.71s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 23.15s (± 0.76%) 22.90s (± 0.71%) -0.25s (- 1.08%) 22.62s 23.32s
xstate - node (v14.15.1, x64)
Memory used 537,105k (± 0.00%) 667,197k (± 0.00%) +130,092k (+24.22%) 667,131k 667,227k
Parse Time 2.59s (± 0.52%) 2.59s (± 0.54%) +0.00s (+ 0.08%) 2.55s 2.62s
Bind Time 1.15s (± 0.59%) 1.15s (± 0.65%) +0.00s (+ 0.09%) 1.14s 1.17s
Check Time 1.51s (± 0.57%) 2.96s (± 0.78%) 🔻+1.45s (+96.15%) 2.92s 3.03s
Emit Time 0.07s (± 4.13%) 0.07s (± 0.00%) -0.00s (- 2.78%) 0.07s 0.07s
Total Time 5.32s (± 0.36%) 6.77s (± 0.45%) +1.45s (+27.27%) 6.71s 6.84s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-210-generic
Architecturex64
Available Memory16 GB
Available Memory5 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 48080 10
Baseline main 10

Developer Information:

Download Benchmark

@typescript-bot
Copy link
Collaborator

@ahejlsberg
Great news! no new errors were found between main..refs/pull/48080/merge

@Andarist
Copy link
Contributor

Andarist commented Mar 6, 2022

As mentioned - I'm very open to revisiting XState types to see if we can improve the situation anyhow there. I know that this is just a single library whereas this PR potentially impacts a lot more than that though. However, as a dev, I would always take a slower compilation over random errors - so this PR is a big improvement from my point of view.

Is there any rough description for the overall algorithm of how the variance is computed? If I understand correctly then:

  • each type parameter has its own variance type
  • to compute it we may need to traverse the whole type structurally, because if T is passed down through multiple layers then any layer can impact the variance type of the associated type parameter at the root level (and on each level in between)
  • potentially when assessing that a type param is invariant you can return early from the algorithm since nothing else can change that
  • this information can mostly be cached per uninstantiated type unless conditional types/mapped types do some weird-ish transformations under the hood (not saying that you are caching it, but that such a heuristic could potentially be used, I have no idea if it would be worth it - this is just me trying to understand the problem space better)

I wonder if there is any type construct that makes it impossible for you to take the associated "slot" into consideration? Like - I see how "makes" type params are taken into consideration, even when a type is "boxed" using some other type I can see how the relation gets transitive etc But when I think about crazy conditional & mapped types... I'm getting lost.

In somewhat other words - I wonder if there is anything that we possibly could do to make TS not descend into a particular branch of the structure?

@ahejlsberg
Copy link
Member Author

@typescript-bot test this
@typescript-bot user test this inline
@typescript-bot run dt
@typescript-bot perf test faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 7, 2022

Heya @ahejlsberg, I've started to run the diff-based community code test suite on this PR at 08c0fa9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 7, 2022

Heya @ahejlsberg, I've started to run the parallelized Definitely Typed test suite on this PR at 08c0fa9. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 7, 2022

Heya @ahejlsberg, I've started to run the abridged perf test suite on this PR at 08c0fa9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 7, 2022

Heya @ahejlsberg, I've started to run the extended test suite on this PR at 08c0fa9. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

@ahejlsberg
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..48080

Metric main 48080 Delta Best Worst
Angular - node (v14.15.1, x64)
Memory used 332,984k (± 0.01%) 331,341k (± 0.01%) -1,644k (- 0.49%) 331,287k 331,384k
Parse Time 2.03s (± 0.86%) 2.02s (± 0.55%) -0.00s (- 0.15%) 2.00s 2.05s
Bind Time 0.88s (± 0.56%) 0.87s (± 0.42%) -0.00s (- 0.23%) 0.87s 0.88s
Check Time 5.57s (± 0.34%) 5.56s (± 0.61%) -0.01s (- 0.14%) 5.48s 5.65s
Emit Time 6.26s (± 0.91%) 6.27s (± 0.42%) +0.00s (+ 0.08%) 6.21s 6.32s
Total Time 14.73s (± 0.51%) 14.72s (± 0.39%) -0.01s (- 0.05%) 14.61s 14.86s
Compiler-Unions - node (v14.15.1, x64)
Memory used 194,104k (± 0.55%) 193,782k (± 0.50%) -322k (- 0.17%) 191,195k 194,498k
Parse Time 0.85s (± 0.80%) 0.85s (± 0.58%) +0.00s (+ 0.12%) 0.85s 0.87s
Bind Time 0.56s (± 1.15%) 0.56s (± 0.65%) +0.00s (+ 0.36%) 0.56s 0.57s
Check Time 7.44s (± 0.59%) 7.43s (± 0.72%) -0.01s (- 0.09%) 7.31s 7.54s
Emit Time 2.50s (± 1.21%) 2.46s (± 0.84%) -0.04s (- 1.48%) 2.43s 2.53s
Total Time 11.36s (± 0.63%) 11.32s (± 0.63%) -0.04s (- 0.34%) 11.16s 11.49s
Monaco - node (v14.15.1, x64)
Memory used 325,203k (± 0.00%) 324,923k (± 0.00%) -280k (- 0.09%) 324,892k 324,952k
Parse Time 1.56s (± 0.71%) 1.57s (± 0.56%) +0.01s (+ 0.45%) 1.55s 1.59s
Bind Time 0.78s (± 1.07%) 0.77s (± 0.44%) -0.00s (- 0.26%) 0.77s 0.78s
Check Time 5.48s (± 0.42%) 5.48s (± 0.54%) +0.00s (+ 0.05%) 5.42s 5.55s
Emit Time 3.31s (± 0.74%) 3.30s (± 0.91%) -0.01s (- 0.27%) 3.25s 3.39s
Total Time 11.13s (± 0.40%) 11.13s (± 0.52%) -0.00s (- 0.00%) 11.02s 11.27s
TFS - node (v14.15.1, x64)
Memory used 288,770k (± 0.01%) 288,672k (± 0.01%) -98k (- 0.03%) 288,630k 288,718k
Parse Time 1.33s (± 1.26%) 1.33s (± 2.43%) +0.00s (+ 0.23%) 1.29s 1.44s
Bind Time 0.73s (± 0.67%) 0.74s (± 2.08%) +0.00s (+ 0.68%) 0.73s 0.80s
Check Time 5.14s (± 0.31%) 5.14s (± 0.84%) 0.00s ( 0.00%) 5.06s 5.29s
Emit Time 3.53s (± 1.92%) 3.51s (± 2.26%) -0.01s (- 0.43%) 3.37s 3.65s
Total Time 10.72s (± 0.85%) 10.72s (± 1.31%) -0.00s (- 0.02%) 10.50s 11.07s
material-ui - node (v14.15.1, x64)
Memory used 453,151k (± 0.00%) 450,276k (± 0.03%) -2,874k (- 0.63%) 449,715k 450,378k
Parse Time 1.86s (± 0.55%) 1.86s (± 0.53%) +0.00s (+ 0.16%) 1.84s 1.88s
Bind Time 0.71s (± 1.05%) 0.71s (± 0.63%) -0.00s (- 0.28%) 0.70s 0.72s
Check Time 20.58s (± 0.85%) 20.34s (± 1.00%) -0.24s (- 1.16%) 20.00s 20.78s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 23.15s (± 0.76%) 22.91s (± 0.89%) -0.24s (- 1.05%) 22.58s 23.35s
xstate - node (v14.15.1, x64)
Memory used 537,105k (± 0.00%) 666,424k (± 0.00%) +129,319k (+24.08%) 666,374k 666,467k
Parse Time 2.59s (± 0.52%) 2.59s (± 0.76%) +0.00s (+ 0.15%) 2.56s 2.64s
Bind Time 1.15s (± 0.59%) 1.15s (± 1.02%) +0.00s (+ 0.26%) 1.12s 1.17s
Check Time 1.51s (± 0.57%) 2.95s (± 0.52%) 🔻+1.44s (+95.56%) 2.92s 2.99s
Emit Time 0.07s (± 4.13%) 0.07s (± 0.00%) -0.00s (- 2.78%) 0.07s 0.07s
Total Time 5.32s (± 0.36%) 6.76s (± 0.30%) +1.45s (+27.21%) 6.73s 6.82s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-210-generic
Architecturex64
Available Memory16 GB
Available Memory5 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 48080 10
Baseline main 10

Developer Information:

Download Benchmark

@typescript-bot
Copy link
Collaborator

@ahejlsberg
Great news! no new errors were found between main..refs/pull/48080/merge

@ahejlsberg
Copy link
Member Author

@Andarist Here's a quick overview of the variance computation logic.

For a generic type T<A, B>, we determine the variance of A by structurally relating instantiations T<Sub, B> and T<Super, B>, where Sub and Super are internal marker types that have a known type relationship (i.e. Sub is a subtype of Super). Then

  • if T<Sub, B> is related to T<Super, B>, but not vice versa, we know A to be covariant,
  • if T<Super, B> is related to T<Sub, B>, but not vice versa, we know A to be contravariant,
  • if neither is related to the other, we know A to be invariant, and
  • if both are related to the other, we know A to be bivariant.

To determine the variance of B, we similarly structurally relate instantiations T<A, Sub> and T<A, Super>.

As we're relating these marker type instantiations, we may come upon other type instantiations for which we need to know the variance. If so, we recursively start a variance computation and use the results of that to relate the type arguments. We may also come upon a circular reference to an instantiation of a type for which we are already computing variance information. If so, we again structurally relate the instantiations, but only if some of the type arguments contain marker types. Ultimately, we stop relating after three levels of recursion.

In other to understand where time is spent in the variance computation logic you can use the --generateTrace compiler option. See instructions here. The xstate types appear to have a very deep dependency graph (some 35 levels), which generates quite a lot of work. Anything you can do to simplify that will likely help greatly.

This is what the recursive invocation tree looks like for getVariancesWorker when compiling xstate:

StateSchema [
  Partial [
  ]
]
StatesConfig [
  StateNodeConfig [
    InvokeConfig [
      InvokeCreator [
        InvokeCallback [
        ]
        PromiseLike [
        ]
        StateMachine [
          StateNodesConfig [
            StateNode [
              ActionObject [
                ActionFunction [
                  ActionMeta [
                    State [
                      Mapper [
                      ]
                      PropertyMapper [
                      ]
                      MachineOptions [
                        Record [
                        ]
                        ConditionPredicate [
                          GuardMeta [
                            GuardPredicate [
                              ActivityDefinition [
                                InvokeDefinition [
                                  ActionFunctionMap [
                                    ActivityConfig [
                                      DelayFunctionMap [
                                        DelayExpr [
                                          SCXMLEventMeta [
                                            Event [
                                            ]
                                          ]
                                        ]
                                      ]
                                      MachineSchema [
                                      ]
                                      Map [
                                        IterableIterator [
                                          IteratorYieldResult [
                                          ]
                                        ]
                                      ]
                                      TransitionDefinition [
                                        IteratorResult [
                                          IteratorReturnResult [
                                          ]
                                        ]
                                        Omit [
                                          Exclude [
                                          ]
                                        ]
                                        TransitionDefinitionMap [
                                          TransitionConfig [
                                            DelayedTransitionDefinition [
                                              SendAction [
                                                ExprWithMeta [
                                                ]
                                                SendExpr [
                                                ]
                                                AssignAction [
                                                  Assigner [
                                                    AssignMeta [
                                                      PropertyAssigner [
                                                        PartialAssigner [
                                                          LogAction [
                                                            LogExpr [
                                                            ]
                                                            StopAction [
                                                              Expr [
                                                              ]
                                                              ChooseAction [
                                                                ChooseConditon [
                                                                  Readonly [
                                                                  ]
                                                                  StateNodeDefinition [
                                                                    StatesDefinition [
                                                                      StateTransition [
                                                                      ]
                                                                    ]
                                                                  ]
                                                                ]
                                                              ]
                                                            ]
                                                          ]
                                                        ]
                                                      ]
                                                    ]
                                                  ]
                                                ]
                                              ]
                                            ]
                                          ]
                                        ]
                                      ]
                                    ]
                                  ]
                                ]
                              ]
                            ]
                          ]
                        ]
                      ]
                    ]
                  ]
                ]
              ]
            ]
          ]
        ]
      ]
    ]
    SimpleActionsFrom [
    ]
  ]
]
Model [
  MachineConfig [
  ]
  ExtractEvent [
  ]
  Prop [
  ]
]
Typestate [
]
RaiseAction [
]
Guard [
]
RaiseActionObject [
]
Action [
]
Interpreter [
  Set [
  ]
  StateListener [
  ]
  ContextListener [
  ]
  Observer [
  ]
  SendActionObject [
  ]
  Subscribable [
  ]
]
BaseActorRef [
]
ActorRef [
  Sender [
  ]
]
Extract [
]
Event [
]
EventListener [
]
Promise [
]
Behavior [
  ActorContext [
  ]
]
Iterable [
  Iterator [
  ]
]
AdjList [
]
StateConfig [
]
ServiceConfig [
]
DelayConfig [
]
InvokeActionObject [
]
SingleOrArray [
]
AtomicStateNodeConfig [
]

@ahejlsberg
Copy link
Member Author

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 7, 2022

Heya @ahejlsberg, I've started to run the tarball bundle task on this PR at 367ef6d. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 7, 2022

Hey @ahejlsberg, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/121427/artifacts?artifactName=tgz&fileId=3564D9FE4E925DEB072B7F2EA049AAB996BB172F4FA500BD881022D7F059E67E02&fileName=/typescript-4.7.0-insiders.20220307.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/pr-build@4.7.0-pr-48080-37".;

@Andarist
Copy link
Contributor

Andarist commented Mar 8, 2022

I can confirm that both of my issues related to this are being fixed with the latest build of this PR. However, I don't see an explicit test case for this being introduced here. The minimal repro created by me can be found here. From what I understand this has been fixed within this diff but as we can see there - no new tests have been added as part of this. Perhaps this is actually tested by the changed baselines - but assessing that is over my head.

@sandersn sandersn added this to Not started in PR Backlog via automation Mar 9, 2022
@sandersn sandersn moved this from Not started to Waiting on reviewers in PR Backlog Mar 9, 2022
@ahejlsberg
Copy link
Member Author

Sadly we won't be able to merge this PR. While it improves accuracy of variance measurement, the adverse effects on type checking performance are simply too great. For example, the check time for the fairly popular immutable.js framework degrades by 50x with this PR in its current form. For now, we'll have to accept that variance measurement for circular types is sometimes inaccurate and/or dependent on declaration order. However, with #48240 it becomes possible to add variance annotations to establish the correct variance in such cases.

@ahejlsberg ahejlsberg closed this Mar 13, 2022
PR Backlog automation moved this from Waiting on reviewers to Done Mar 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Milestone Bug PRs that fix a bug with a specific milestone
Projects
Archived in project
PR Backlog
  
Done
Development

Successfully merging this pull request may close these issues.

Check order dependence with mutually-recursive non-unary generics
6 participants