Skip to content

Commit

Permalink
[dagit] Fix Backfills page locking up when displaying large backfills…
Browse files Browse the repository at this point in the history
…, improve "target" column (#11693)

### Summary & Motivation

I created a backfill of 180,000 partition keys and found that Dagit is
unable to render the Overview > Backfills page. It seems this is due to
several things:

- The API is returning a `PartitionStatuses` object for each backfill
containing an object for each partition. It'd be great to move this to
the ranges format that @clairelin135 is putting together which will pass
straight through to the rendering!

- Apollo cache is trying to normalize all of these 160k objects into
some sort of fancy cache, which it "deep partial updates" with the new
data. We can disable this behavior for just `PartitionStatus` with an
entry in AppCache.tsx, but I have mixed feelings about making global
changes. Maybe we do it for now and remove it when we move this status
data to a smaller / faster range format. It reduces the "page lock time"
after data is received from 7s to about 2s on my machine.

- The Backfill page is using the `<PartitionStatus />` component in
"separated bars" mode, which actually tries to render 180k divs for the
180k partitions.

This PR addresses the last two - if there are more than 1k partition
keys in the backfill, we now just render a box that is the same shape of
the PartitionStatus component but contains just the counts:


![image](https://user-images.githubusercontent.com/1037212/213035809-0e2638a6-c31c-4418-8879-f68e3fb71d56.png)


I also observed a lot of funkiness in the Backfill Target column because
I have both asset and job backfills for both loaded and unloaded
repositories in my test set. I changed the code so that it omits
partition set names that are part of hidden asset jobs, and made it show
asset keys even when the partition set isn't part of a loaded repo.

### How I Tested These Changes

Page loads!

For the target backfill changes, I attached before and after screenshots
and I think my setup tests all five cases (assets / non-assets + jobs /
asset graph / asset partition set).


![image](https://user-images.githubusercontent.com/1037212/212241840-0421a01d-d151-4843-89cf-f32a5a2350ed.png)

Co-authored-by: bengotow <bgotow@elementl.com>
  • Loading branch information
bengotow and bengotow committed Jan 18, 2023
1 parent dde417c commit 7e582d7
Show file tree
Hide file tree
Showing 12 changed files with 269 additions and 137 deletions.
7 changes: 7 additions & 0 deletions js_modules/dagit/packages/core/src/app/AppCache.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ export const createAppCache = () =>
new InMemoryCache({
addTypename: true,
possibleTypes,
typePolicies: {
PartitionStatus: {
keyFields: false,
},
},
dataIdFromObject: (object: any) => {
if (
object.name &&
Expand All @@ -17,6 +22,8 @@ export const createAppCache = () =>
return 'Instance';
} else if (object.__typename === 'Workspace') {
return 'Workspace';
} else if (object.__typename === 'PartitionBackfill') {
return object.backfillId;
} else {
return defaultDataIdFromObject(object);
}
Expand Down
6 changes: 6 additions & 0 deletions js_modules/dagit/packages/core/src/graphql/schema.graphql

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions js_modules/dagit/packages/core/src/graphql/types.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 7e582d7

Please sign in to comment.