DataFlow backfill - disabled but most of the functionality is there #1261

travjenkins · 2024-09-19T20:33:53Z

Issues

completes #1267
contributes #1220
contributes #310

Changes

1220

Started making calls in almost all the steps
Better error handling VS server failures
Updating content as requested
Adding a new review selections step
Adding the job status into the step
Showing draft errors

1267

Updated the count to just be the enabled collections

310

More work done on the spec diff and showing a "What's changed" table that is the basis of more comparison

misc

Some quick refactoring on how we store entity properties
scrolling down chip lists a tiny bit when showing all
Removing description from the details form. This will eventually be put into the presave prompt

Tests

Manually tested

While working on the new feature - a TON of testing the new modal.
After disabling new stuff just ran through Capture and Materialization creation and editing

Automated tests

unit testing covered

Playwright tests ran locally

Screenshots

When opening chiplist we will scroll down a few pixels to try to get more chips to slightly show to hopefully make it clear to users that they can scroll

Added some clarification text

Calling intl for translations

Moving label into store to reduce duplication Adding a running handler into parent

Switching to uses as single status property Getting capture disable wired up

handling errors a bit better updating enum

Closing when "back" button is clicked on first step No longer making steps options just defaulting to empty Just setting show to false is enough for closing dialog Checking the shallow `onFirstStep` to reduce renders Moving context out to the main store Only running initialize once per hydration

…twork Fetching live spec based on spec id

Wiring up most of the steps (without waiting for no data)

Support passing in 'sx' so we can override styling when needed Scrolling the element down a little bit when all items are shown

below

Making consts for important keys

Starting work on handling errors better Trying to get publication populated correctly

…losing

travjenkins · 2024-09-25T03:43:35Z

src/components/shared/Entity/LogDialog/Content.tsx

Thought I was going to reuse this and ended up not. However, no real harm in keeping it broken out.

travjenkins · 2024-09-25T03:45:02Z

src/components/shared/Entity/LogDialog/index.tsx

Moved from a single file to a component folder and using the new "shared" Content up above.

travjenkins · 2024-09-25T03:47:24Z

src/context/LoopIndex/index.tsx

New concept to keep the index while in a loop. So you don't have to drill index down

travjenkins · 2024-09-25T03:48:06Z

src/hooks/useDetailsNavigator.ts

Allowing to override because the presave prompt will have a link to details for a capture, collection, and materialization.

travjenkins · 2024-09-25T03:48:29Z

src/hooks/useLiveSpecsExt.ts

Currently going to only allow matching on the sourceCapture as that is the safest query

travjenkins · 2024-09-25T03:48:50Z

src/services/supabase.ts

@@ -311,18 +311,35 @@ export function invokeSupabase<T>(fn: FUNCTIONS, body: any) {
    );
 }

-export const insertSupabase = (
+export const insertSupabase = <T = any>(


Starting to add some support for typing

travjenkins · 2024-09-25T03:49:14Z

src/services/supabase.ts

+        if (!noResponse) {
+            return query.select();


Where we handle not always including the select to fetch everything like V1

travjenkins · 2024-09-25T03:49:34Z

src/services/supabase.ts

+export const JOB_TYPE_EMPTY = 'emptyDraft';
+export const JOB_TYPE_FAILURE = 'buildFailed';
+export const JOB_TYPE_SUCCESS = 'success';


Want to be able to check specific statuses if needed.

travjenkins · 2024-09-25T03:50:30Z

src/stores/Binding/Store.ts

+    getEnabledCollections: () =>
+        Object.values(get().resourceConfigs)
+            .filter(({ meta }) => !meta.disable)
+            .map(({ meta }) => meta.collectionName),


We are not really using the collectionName mapping part - but left it in so it matches the getCollections up above.

travjenkins · 2024-09-25T03:52:46Z

src/components/shared/Entity/prompts/PreSave/index.tsx

-            <Title />
-            <Content />
-            <Actions />
+            <PromptsHydrator>


Putting the hydrator inside the dialog so that it is always reset when it is shown/hidden

travjenkins · 2024-09-25T03:53:18Z

src/components/shared/Entity/prompts/steps/dataFlowReset/DisableCapture/definition.ts

+// interface DisableCaptureStepContext {
+//     pubId: string | null;
+//     captureSpec: JSON | null;
+//     captureName: string | null;
+// }


These will eventually be combined into the context in the store

travjenkins · 2024-09-25T03:54:26Z

src/components/shared/Entity/prompts/steps/dataFlowReset/SelectMaterialization/Selector.tsx

+    if (loading) {
+        return <Skeleton />;
+    }


Moved the loading state to inside the selector. This way the selector can also handle the "empty" state which needs to eventually allow steps to be skipped.

kiahna-tucker

There is an issue with the backfilled binding logic that needs to be addressed before this PR can be merged.

src/components/editor/Bindings/Backfill/index.tsx

src/components/editor/Bindings/Backfill/BackfillCount.tsx

Typing the ref Including the current top height when scrolling

Switching out format components

kiahna-tucker

There is a small, content typo called out below.

src/lang/en-US/Workflows.ts

…er-calls

kiahna-tucker

Approved with moderate, happy path testing.

travjenkins added 30 commits September 16, 2024 13:26

Turning everything back on

a4cf14b

Moving steps stuff into a manual machine state in a store

48a53e8

Getting actions buttons wired up

89736ca

Calling intl for translations

Moving more stuff into content

9a03b21

Moving label into store to reduce duplication Adding a running handler into parent

checking first step

46775ca

Updating typing a lot

c0491d3

Switching to uses as single status property Getting capture disable wired up

typing tweaks

f820b36

handling errors a bit better updating enum

Storing off step definition

38ee682

Updating key to keep consistent

6ffd672

Adding support for a context to store data... but this feels wrong

afebcf9

adding quick support to store off things in a keyed / machine way

d5b94e1

cleaning up handled as we are not using this

1cf48d3

Forcing to pass an index when updating a step

1cf6db5

quick hack and typing update

2ce4b59

Adding handled to get no response when an insert is done to reduce ne…

0e329e9

…twork Fetching live spec based on spec id

Storing stuff to re-enable the capture

8cacc59

Wiring up most of the steps (without waiting for no data)

marking valid so the next step works

232e8b9

Starting to wire up the last step

9503ce3

Updating ChipList to:

cdb05a9

Support passing in 'sx' so we can override styling when needed Scrolling the element down a little bit when all items are shown

Making code a bit safer

e94ff64

Typing for chiplist changes

fd5ad1f

styling the options better so the collections don't overlap the option

c46a656

below

Reset the state when leaving the flow

653a701

Probably best to just store the publication status as a whole

67a86e6

Making consts for important keys

Adding a "review selections" step as an idea

39e905f

Starting work on handling errors better Trying to get publication populated correctly

Refactoring to reduce need of functions

3e43464

Resetting when closing as doing it in store is not working

6238b84

Moving hydrator within modal so it is always reset when opening and c…

bdedc89

…losing

Breaking out the review table

3a9d9fb

travjenkins commented Sep 25, 2024

View reviewed changes

PR: commenting out to disable this

7e7e6fe

travjenkins commented Sep 25, 2024

View reviewed changes

travjenkins added the change:planned This is a planned change label Sep 25, 2024

travjenkins requested a review from kiahna-tucker September 25, 2024 03:55

travjenkins marked this pull request as ready for review September 25, 2024 03:55

travjenkins requested a review from a team as a code owner September 25, 2024 03:55

kiahna-tucker requested changes Sep 25, 2024

View reviewed changes

src/components/editor/Bindings/Backfill/index.tsx Show resolved Hide resolved

src/components/editor/Bindings/Backfill/BackfillCount.tsx Outdated Show resolved Hide resolved

travjenkins added 4 commits September 25, 2024 11:16

Making code a bit simpler

e2efcb6

Typing the ref Including the current top height when scrolling

Getting the chips aligned

a89a4fe

PR: putting original count back in

e901568

Updating content to be more clear

c4e48f8

Switching out format components

travjenkins requested a review from kiahna-tucker September 25, 2024 15:39

kiahna-tucker reviewed Sep 25, 2024

View reviewed changes

src/lang/en-US/Workflows.ts Outdated Show resolved Hide resolved

travjenkins added 3 commits September 25, 2024 11:54

PR: typo and small content change

6750662

PR: content typo

8396d89

Merge branch 'main' into travjenkins/feature/full-entity-refresh/serv…

2e80049

…er-calls

kiahna-tucker approved these changes Sep 25, 2024

View reviewed changes

travjenkins merged commit 7bbb7d6 into main Sep 25, 2024
3 checks passed

travjenkins deleted the travjenkins/feature/full-entity-refresh/server-calls branch September 25, 2024 17:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DataFlow backfill - disabled but most of the functionality is there #1261

DataFlow backfill - disabled but most of the functionality is there #1261

travjenkins commented Sep 19, 2024 •

edited

Loading

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

travjenkins Sep 25, 2024

kiahna-tucker left a comment

kiahna-tucker left a comment

kiahna-tucker left a comment

DataFlow backfill - disabled but most of the functionality is there #1261

DataFlow backfill - disabled but most of the functionality is there #1261

Conversation

travjenkins commented Sep 19, 2024 • edited Loading

Issues

Changes

1220

1267

310

misc

Tests

Manually tested

Automated tests

Playwright tests ran locally

Screenshots

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kiahna-tucker left a comment

Choose a reason for hiding this comment

kiahna-tucker left a comment

Choose a reason for hiding this comment

kiahna-tucker left a comment

Choose a reason for hiding this comment

travjenkins commented Sep 19, 2024 •

edited

Loading