feat: update tests to skip long-running operation related tests (#355)

Because

- #353

This commit

- Further disable tests

README.md

@@ -104,48 +104,9 @@ For server-side code, because we will alter the ./.env file in the image. It can
 
 - We currently use self-signed certification in all our backend so the default of env variable `NEXT_PUBLIC_INTEGRATION_TEST_ENABLED` will be true. Which means that the Axios won't complaint the `Unauthorized` request. Please do not set this env variable to true on production. 
 
-## About the test container
-
-We maintain another container `Dockerfile.playwright` specifically for test usage. This container has some caveats that maintainers of this repo should notice
-
-- The playwright version should align with the pulled playwright image version. Two should be the same.
-- If you want to debug in the test container, you should grant this container the root user permission. (default is specific user: playwright) with `--build-arg TEST_USER='root'` in you `docker build` command
-
-### How to test at local
-
-- In VDP folder: `git pull --rebase` to pull the up-to-date code.
-- In VDP folder: `make build PROFILE=all` 
-- In console folder: `pnpm dev` to setup local dev server
-- If you want to test in your host (Run the app with pnpm dev in the console folder) 
-  - In VDP folder: `make dev PROFILE=console ITMODE=true`
-  - In console folder: `pnpm dev`
-  - In console folder: `pnpm integration-test`
-- If you want to test in the docker-container (Run the app with VDP) 
-  - In VDP folder: `make dev PROFILE=all ITMODE=true CONSOLE_BASE_URL_HOST=console CONSOLE_BASE_API_GATEWAY_URL_HOST=api-gateway`
-  - In console folder: `pnpm docker-build-test`
-  - In console folder: `pnpm docker-run-test`
-
-### How to debug the test in the docker
-
-- when build the text container, please alter `pnpm docker-build-test` to `docker build -f Dockerfile.playwright --build-arg TEST_USER='root' -t instill-console-integration .`
-- Then the user of this container will have the permission to write test-result into container.
-- you could look up the container's id by `docker cp` then copy the test-results into host folder by `docker cp <container_id>:/app/test-results .`
-
-### About the manual test 
-
-Currently we have several steps that require manually testing.
-
-- Please follow the step above to setup the VDP backend
-- Please create all the model list [here](https://github.com/instill-ai/vdp#model-hub)
-- In the `/models` page, please make sure all the created models are correctly listed
-- Please click every listed model to go to the model details page `/models/:model_id`
-- Please check the model state on the `/models/:model_id` page, you should select the model_instance_tag in the dropdown to the one you just deployed. The model state should be `Online`
-- Please switch the state toggle button, the model_state should be `Unspecified` and after a while it should be Offline. Please do it again to check whether it can switch back to the Online state.
-- Please use the model you just created to create a sync pipeline.
-- Go to the `pipelines/:pipeline_id` page you just created.
-- If you switch the state toggle button, it should warn you `Pipeline pipeline-hehe is in the SYNC mode, which is always active`
-- Please create a async pipeline with local json destination
-- Please go to `pipeline/:pipeline_id` page to check it has correctly displayed data
+## About how to test the console
+
+Please refer to the console integration test's [document](/integration-test/README.md)
 
 ## Other Caveats
 

integration-test/README.md

@@ -3,6 +3,56 @@
 
 - Use page object model
 
+## About the test container
+
+We maintain another container `Dockerfile.playwright` specifically for test usage. This container has some caveats that maintainers of this repo should notice
+
+- The playwright version should align with the pulled playwright image version. Two should be the same.
+- If you want to debug in the test container, you should grant this container the root user permission. (default is specific user: playwright) with `--build-arg TEST_USER='root'` in you `docker build` command
+
+### How to test at local
+
+- In VDP folder: `git pull --rebase` to pull the up-to-date code.
+- In VDP folder: `make build PROFILE=all` 
+- In console folder: `pnpm dev` to setup local dev server
+- If you want to test in your host (Run the app with pnpm dev in the console folder) 
+  - In VDP folder: `make dev PROFILE=console ITMODE=true`
+  - In console folder: `pnpm dev`
+  - In console folder: `pnpm integration-test`
+- If you want to test in the docker-container (Run the app with VDP) 
+  - In VDP folder: `make dev PROFILE=all ITMODE=true CONSOLE_BASE_URL_HOST=console CONSOLE_BASE_API_GATEWAY_URL_HOST=api-gateway`
+  - In console folder: `pnpm docker-build-test`
+  - In console folder: `pnpm docker-run-test`
+
+### How to debug the test in the docker
+
+- when build the text container, please alter `pnpm docker-build-test` to `docker build -f Dockerfile.playwright --build-arg TEST_USER='root' -t instill-console-integration .`
+- Then the user of this container will have the permission to write test-result into container.
+- you could look up the container's id by `docker cp` then copy the test-results into host folder by `docker cp <container_id>:/app/test-results .`
+
+### About the manual test 
+
+Currently we have several steps that require manually testing.
+
+#### Model
+
+- Please follow the step above to setup the VDP backend
+- Please create all the model list [here](https://github.com/instill-ai/vdp#model-hub)
+- In the `/models` page, please make sure all the created models are correctly listed
+- Please click every listed model to go to the model details page `/models/:model_id`
+- Please check the model state on the `/models/:model_id` page, you should select the model_instance_tag in the dropdown to the one you just deployed. The model state should be `Online`
+- Please switch the state toggle button, the model_state should be `Unspecified` and after a while it should be Offline. Please do it again to check whether it can switch back to the Online state.
+- Please delete all the models and make sure they are not displayed on `/models` page
+
+#### Pipeline
+
+- Please use the model you just created to create a sync pipeline.
+- Go to the `pipelines/:pipeline_id` page you just created.
+- If you switch the state toggle button, it should warn you `Pipeline pipeline-hehe is in the SYNC mode, which is always active`
+- Please create a async pipeline with local json destination
+- Please go to `pipeline/:pipeline_id` page to check it has correctly displayed data
+- Please delete all the pipelines and make sure they are not displayed on `/pipelines` page
+
 ## Caveats
 
 ### Use `locator.fill()` instead of `locator.type()`
@@ -17,17 +67,6 @@
   - [subtree intercepts pointer events, Unable to click](https://github.com/microsoft/playwright/issues/13576)
   - [Chromium: Cannot click, element intercepts pointer events](https://github.com/microsoft/playwright/issues/12821)
 
-### About the env variables
-
-- We use `.env` and `dotenv` to digest environment variables.
-- Make sure your `.env` file have following variables:
-  - NEXT_PUBLIC_CONSOLE_BASE_URL=http://localhost:3000
-  - NEXT_PUBLIC_MGMT_BACKEND_BASE_URL=http://localhost:8080
-  - NEXT_PUBLIC_PIPELINE_BACKEND_BASE_URL=http://localhost:8081
-  - NEXT_PUBLIC_CONNECTOR_BACKEND_BASE_URL=http://localhost:8082
-  - NEXT_PUBLIC_MODEL_BACKEND_BASE_URL=http://localhost:8083
-  - NEXT_PUBLIC_API_VERSION=v1alpha
-  - NEXT_PUBLIC_INSTILL_AI_USER_COOKIE_NAME=instill-ai-user
 
 ### About the config `fullyParallel`
 
@@ -56,4 +95,5 @@ const saveButton = page.locator("button", { hasText: "Save" });
 expect(await saveButton.isEnabled()).toBeTruthy();
 const succeedMessage = page.locator("h3", { hasText: "Succeed" });
 await Promise.all([saveButton.click(), succeedMessage.isVisible()]);
-```
+```
+

integration-test/common/model.ts

@@ -2,6 +2,8 @@ import { env, delay, expectToSelectReactSelectOption } from "../helper";
 import { Page, expect, Locator } from "@playwright/test";
 
 export const expectToDeleteModel = async (page: Page, modelId: string) => {
+  delay(20000);
+
   await page.goto(`/models/${modelId}`, { waitUntil: "networkidle" });
 
   // Should enable open delete model modal button

integration-test/common/pipeline.ts

@@ -1,4 +1,4 @@
-import { env } from "../helper";
+import { delay, env } from "../helper";
 import { Page, expect } from "@playwright/test";
 
 export const openPipelinesPage = async (page: Page) => {

integration-test/destination-sync.spec.ts

@@ -1,4 +1,8 @@
-import { env , deleteDestination, expectToSelectReactSelectOption } from "./helper";
+import {
+  env,
+  deleteDestination,
+  expectToSelectReactSelectOption,
+} from "./helper";
 import { test, expect } from "@playwright/test";
 import { expectToDeleteConnector } from "./common/connector";
 

integration-test/model-artivc.spec.ts

@@ -104,7 +104,9 @@ test.describe.serial("Artivc model", () => {
     await expectToUpdateModelDescription(page, modelId, "new");
   });
 
-  test("should have proper delete model modal and delete this model", async ({
+  // Disable test related to long-running operation
+
+  test.skip("should have proper delete model modal and delete this model", async ({
     page,
   }) => {
     await expectToDeleteModel(page, modelId);

integration-test/model-github.spec.ts

@@ -94,7 +94,9 @@ test.describe.serial("GitHub model", () => {
     await expectToUpdateModelDescription(page, modelId, "new");
   });
 
-  test("should have proper delete model modal and delete this model", async ({
+  // Disable test related to long-running operation
+
+  test.skip("should have proper delete model modal and delete this model", async ({
     page,
   }) => {
     await expectToDeleteModel(page, modelId);

integration-test/model-huggingface.spec.ts

@@ -103,7 +103,9 @@ test.describe.serial("Hugging face model", () => {
     await expectToUpdateModelDescription(page, modelId, "new");
   });
 
-  test("should have proper delete model modal and delete this model", async ({
+  // Disable test related to long-running operation
+
+  test.skip("should have proper delete model modal and delete this model", async ({
     page,
   }) => {
     await expectToDeleteModel(page, modelId);

integration-test/model-local.spec.ts

@@ -97,7 +97,9 @@ test.describe.serial("Local model", () => {
     await expectToUpdateModelDescription(page, modelId, "new");
   });
 
-  test("should have proper delete model modal and delete this model", async ({
+  // Disable test related to long-running operation
+
+  test.skip("should have proper delete model modal and delete this model", async ({
     page,
   }) => {
     await expectToDeleteModel(page, modelId);

integration-test/pipeline-async.spec.ts

@@ -1,4 +1,5 @@
-import { env ,
+import {
+  env,
   createInstillAxiosTestClient,
   deleteDestination,
   deleteModel,
@@ -219,7 +220,9 @@ test.describe
     await expectToUpdatePipelineDescription(page, pipelineId, "");
   });
 
-  test("should have proper delete pipeline modal and delete this pipeline", async ({
+  // Disable test related to long-running operation
+
+  test.skip("should have proper delete pipeline modal and delete this pipeline", async ({
     page,
   }) => {
     await expectToDeletePipeline(page, pipelineId);

integration-test/pipeline-sync.spec.ts

@@ -193,7 +193,9 @@ test.describe
     await expectToUpdatePipelineDescription(page, pipelineId, "");
   });
 
-  test("should have proper delete pipeline modal and delete this pipeline", async ({
+  // Disable test related to long-running operation
+
+  test.skip("should have proper delete pipeline modal and delete this pipeline", async ({
     page,
   }) => {
     await expectToDeletePipeline(page, pipelineId);