Skip to content

Commit

Permalink
feat: Implement airbyte form builder (#101)
Browse files Browse the repository at this point in the history
Because

- #55 
- Implement Airbyte form builder

This commit

- Loop though json schema and generate formTree
- Generate form UI from formTree
  • Loading branch information
EiffelFly committed Jul 3, 2022
1 parent e66e5f5 commit d39b4a4
Show file tree
Hide file tree
Showing 20 changed files with 1,325 additions and 39 deletions.
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@
},
"dependencies": {
"@amplitude/analytics-browser": "^0.4.1",
"@instill-ai/design-system": "^0.0.73",
"@instill-ai/design-system": "^0.0.75",
"@types/json-schema": "^7.0.11",
"axios": "^0.27.2",
"clsx": "^1.1.1",
"cookie": "^0.5.0",
Expand Down
20 changes: 15 additions & 5 deletions src/components/forms/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,17 @@ In every component we separately function with sections, this sections have seve
- Submit the form using formik onSubmit handler and validator

// ###################################################################
// # #
// # 1 - <_title_> #
// # #
// # #
// # 1 - <_title_> #
// # #
// ###################################################################
//
// <_comment_>

### About the naming convention

- Action verifirer: can____ (canDeployModel, canSetupModel)
- Action: handle__ (handleDeployModel, handleSetupModel)
- Action verifirer: can\_\_\_\_ (canDeployModel, canSetupModel)
- Action: handle\_\_ (handleDeployModel, handleSetupModel)

## Pipeline form

Expand All @@ -37,4 +37,14 @@ When you read though pipeline form, it now separatelt to 5 step
- Setup pipeline destination step
- Setup pipeline details step

## About the complicated form generation

We are using airbyte potocol to control our destination connectors, they are using quite complicated yaml to generate their form. They generate all the whole formik component by digesting the JsonSchema ([ServiceForm](https://github.com/airbytehq/airbyte/blob/8076b56f3718d6fe054b660a838f2c1c6890ffc2/airbyte-webapp/src/views/Connector/ServiceForm/ServiceForm.tsx)). In my opinion this is not flexible and our form's structure is much complicated than airbyte due to we have the pipeline concept on top of their connection.

The solution will be cell-design in normal formik's form's flow. Take `CreatePipelineForm` for example, We have a giant formik form that hold the full state of the flow, each step have its own logic and upon finish the step, they will fill in the value into formik state and call it the day. Here comes a problem, if we want to validate at the end of the flow, we have to write a giant validation schema to validate every possibility when create the model or the destination connector. This is appearantly not ideal. Below are the proposal for better implementation.

- We should trust the validation of every step and not validate at the end of the flow.
- We will have a flag for every step's validation like `validModel`, at the end of the flow, it only needs to check the value of this kind of flag.
- About the complicated form like model definition and destination connection, we generate them from json-schema and compose a block(not a formik container), it's not a `<form></form>` HTML tag, but a pure functional component. It will digest the user's input, validate the input then submit. After the request is complete, it will fill in the return value(most of the cases, it will fill in the identifier of the resource, take connector for example, it may fill in `source-connectors/hi`) and move on.

In this implementation we could have very flexible block that can install in near every form. and we will call each of this kind of component FormCell.
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
import { FC, useState, useMemo, Fragment } from "react";
import {
BasicSingleSelect,
SingleSelectOption,
} from "@instill-ai/design-system";
import { useDestinationDefinitions } from "@/services/connector";
import { airbyteSchemaToAirbyteFormTree } from "@/lib/airbytes/airbyteSchemaToAirbyteFormTree";
import useBuildForm from "@/lib/airbytes/useBuildForm";
import { AirbyteFormErrors, AirbyteFormValues } from "@/lib/airbytes";

/* eslint-disable @typescript-eslint/no-explicit-any */

export type AsyncDestinationFormCellProps = Record<string, any>;

export type AsyncDestinationFormCellState = Record<string, any> & {
destinationDefinition: string;
};

const AsyncDestinationFormCell: FC<AsyncDestinationFormCellProps> = () => {
const [cellState, setCellState] = useState<AsyncDestinationFormCellState>();

const destinationDefinitions = useDestinationDefinitions();

const destinationOptions = useMemo(() => {
if (!destinationDefinitions.isSuccess) return [];

const options: SingleSelectOption[] = [];

for (const definition of destinationDefinitions.data) {
options.push({
label: definition.id,
value: definition.name,
});
}

return options;
}, [destinationDefinitions.isSuccess, destinationDefinitions.data]);

const selectedDestinationOption = useMemo(() => {
if (!cellState?.destinationDefinition || !destinationOptions) return null;
return (
destinationOptions.find(
(e) => e.value === cellState.destinationDefinition
) || null
);
}, [cellState?.destinationDefinition, destinationOptions]);

const selectedDestinationFromTree = useMemo(() => {
if (!selectedDestinationOption || !destinationDefinitions.isSuccess) {
return null;
}

const selectedDestination = destinationDefinitions.data.find(
(e) => e.name === selectedDestinationOption.value
);

if (!selectedDestination) {
return null;
}

const formTree = airbyteSchemaToAirbyteFormTree(
selectedDestination.connector_definition.spec.connection_specification
);

return formTree;
}, [selectedDestinationOption]);

const [fieldValues, setFieldValues] = useState<AirbyteFormValues>({});
const [fieldErrors, setFieldErrors] = useState<AirbyteFormErrors>({});

const fields = useBuildForm(
selectedDestinationFromTree,
false,
fieldValues,
setFieldValues,
fieldErrors
);

console.log(fields);

return (
<Fragment>
<BasicSingleSelect
id="destinationDefinition"
instanceId="destinationDefinition"
menuPlacement="auto"
label="Destination type"
additionalMessageOnLabel={null}
description={""}
disabled={false}
readOnly={false}
required={false}
error={null}
value={selectedDestinationOption}
options={destinationOptions}
onChangeInput={(_, option) => {
setCellState((prev) => {
return {
...prev,
destinationDefinition: option.value,
};
});
}}
/>
{fields}
</Fragment>
);
};

export default AsyncDestinationFormCell;
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
export { default } from "./AsyncDestinationFormCell";
export * from "./AsyncDestinationFormCell";
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,15 @@ import {
SingleSelectOption,
} from "@instill-ai/design-system";

import { SingleSelect } from "../../../../formik/FormikField";
import { TextField } from "../../../../formik/FormikField";
import { FormBase } from "@/components/formik";
import { ConnectorIcon, PrimaryButton } from "@/components/ui";
import { CreateDestinationPayload } from "@/lib/instill";
import { Nullable } from "@/types/general";
import { useCreateDestination, useDestinations } from "@/services/connector";
import { useAmplitudeCtx } from "context/AmplitudeContext";
import { sendAmplitudeData } from "@/lib/amplitude";
import AsyncDestinationFormCell from "../AsyncDestinationFormCell";

export type CreateDestinationFormValues = {
id: string;
Expand Down Expand Up @@ -163,33 +164,23 @@ const CreateDestinationForm: FC = () => {
{(formik) => {
return (
<FormBase marginBottom={null} gapY="gap-y-5" padding={null}>
{/* <TextField
<TextField
id="id"
name="id"
label="Id"
label="ID"
additionalMessageOnLabel={null}
description="Pick a name to help you identify this destination in Instill"
disabled={allSyncDestinationCreated ? true : false}
disabled={false}
readOnly={false}
required={true}
placeholder=""
type="text"
autoComplete="off"
value={formik.values.id || ""}
/> */}
<SingleSelect
id="destinationDefinition"
name="destinationDefinition"
label="Destination type"
additionalMessageOnLabel={null}
options={syncDestinationDefinitionOptions}
value={selectedSyncDestinationDefinitionOption}
additionalOnChangeCb={destinationDefinitionOnChange}
error={formik.errors.destinationDefinition || null}
disabled={false}
readOnly={false}
required={true}
description="Setup Guide"
menuPlacement="auto"
error={null}
additionalOnChangeCb={null}
/>
<AsyncDestinationFormCell />
<div className="flex flex-row">
{createDestinationError ? (
<BasicProgressMessageBox width="w-[216px]" status="error">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ const ConfigurePipelineForm: FC<ConfigurePipelineFormProps> = ({
disabled={true}
readOnly={false}
required={true}
description={null}
description=""
/>
<TextArea
id="pipelineDescription"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ const DeleteResourceModal: FC<DeleteResourceModalProps> = ({
disabled={false}
readOnly={false}
required={false}
description={null}
description=""
/>
<div className="grid grid-cols-2 gap-x-5">
<OutlineButton
Expand Down
22 changes: 22 additions & 0 deletions src/lib/airbytes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# About the flow of generating Airbyte connector's form

We are using Airbyte protocol for generating, maintain, create our connectors, frontend need to come up with a way that have backward compatability and onward support of any Airbyte connectors. Here is how we accomplishing it and the principle behind these implementation.

## Principles and implementation

- **Loose couple to all the involved library**
- We use Formik just as Airbyte, but they had closely coupled with Formik to generate the whole form ([ref](https://github.com/airbytehq/airbyte/blob/59e20f20de73ced59ae2c782612fa7554fc1fced/airbyte-webapp/src/views/Connector/ServiceForm/ServiceForm.tsx)). Every details are controled by the Formik from form's state, validation schema and submit action. In this way they may have hard time if they have much complicated form structure.
- **We try to isolated Airbyte connector's logic**
- We separate all the types and functions into the folder
- We make the form act like a small island, they have their own state, action and validation besides from our main form.

## Useful refernece

- [Airbyte - ServiceForm](https://github.com/airbytehq/airbyte/blob/59e20f20de73ced59ae2c782612fa7554fc1fced/airbyte-webapp/src/views/Connector/ServiceForm/ServiceForm.tsx)
- [Airbyte - Form's types](https://github.com/airbytehq/airbyte/blob/master/airbyte-webapp/src/core/form/types.ts)
- [Airbyte - SchemaToUiWidget](https://github.com/airbytehq/airbyte/blob/59e20f20de73ced59ae2c782612fa7554fc1fced/airbyte-webapp/src/core/jsonSchema/schemaToUiWidget.ts)
- How they loop though json schema and construct FormBlock(In our terms, it's FormTree)
- [Airbyte - FormSection](https://github.com/airbytehq/airbyte/blob/master/airbyte-webapp/src/views/Connector/ServiceForm/components/Sections/FormSection.tsx)
- The FormSection is how they control their UI widget
- [Airbyte - Control](https://github.com/airbytehq/airbyte/blob/master/airbyte-webapp/src/views/Connector/ServiceForm/components/Property/Control.tsx)
- How they generate field based on their type, array, boolean, string, integer, array
Loading

0 comments on commit d39b4a4

Please sign in to comment.