UI Automation Tests for LLM Studio #561

itsmunishbhardwaj · 2024-01-12T10:59:23Z

How to run the tests

To run tests locally

 export LOCAL_LOGIN=True
 export PYTEST_BASE_URL=localhost:10101
 make llmstudio
 make test-ui

To run tests on platform with okta login like: https://internal-genai.dedicated.h2o.ai/

 export OKTA_USER=
 export OKTA_PASSWORD=
 export PYTEST_BASE_URL=
 make test-ui

To run tests on platform with keycloak login like: https://cloud-qa.h2o.ai/

 export KEYCLOAK_USER=
 export KEYCLOAK_PASSWORD=
 export PYTEST_BASE_URL=
 make test-ui

Test results are stored at reports/junit_ui.xml

mnezh · 2024-01-15T10:58:11Z

Makefile

@@ -142,4 +172,4 @@ build-doc-locally:  # Bundles your website into static files for production
 	cd documentation && npm run build

 serve-doc-locally:  # Serves the built website locally 
-	cd documentation && npm run serve
+	cd documentation && npm run serve


missing new line

mnezh · 2024-01-15T10:58:23Z

Pipfile

+pytest-bdd = "==6.1.1"
+playwright = "==1.40.0"
+hac-playwright = { file = "http://h2o-public-test-data.s3.amazonaws.com/e2e-testing/hac_playwright-1.40.0-py3-none-any.whl" }
+pytest-base-url = "==2.0.0"


missing new line

mnezh · 2024-01-15T10:59:13Z

tests/ui/llm_studio.feature

+        And I run the experiment
+        Then I should see the test-experiment should finish successfully 
+        When I delete experiment test-experiment
+        Then I should not see the experiment test-experiment


missing new line

mnezh · 2024-01-15T11:02:47Z

tests/ui/utils.py

+        # If the heading is present, click the "I agree" button
+        page.get_by_role("button", name="I agree").click()
+    else:
+        return 1


this method has None return type

the return value is not used anywhere in the code
thus entire section could be removed:

else: return 1

Makefile

maxjeblick

I still have issues running the tests locally (make test-ui-local), in particular it fails on page.goto(base_url) with base_url being an empty string.

tests/ui/test.py

maxjeblick · 2024-01-15T12:30:24Z

tests/ui/llm_studio_page.py

+from hac_playwright.pages.base import BasePage
+from playwright.sync_api import expect
+
+CLOUD_FILESYSTEM_PATH = "/home/llmstudio/mount/data/user/"


As discussed, this won't work for local tests.
Probably an env variable in the makefile together with an if-else condition will do the job (unless there's a more direct way of detecting running on the cloud):

test-ui-local: $LOCAL_TEST=true \ (PW_DEBUG) $(PIPENV) run pytest \ ...

Within the python code:
if "LOCAL_TEST" in os.environ

I can create an env variable to store the local directory to run the tests locally. WDYT? eg:
export FILESYSTEM_PATH=<path to local filesystem>
you can manually export this env var to add path which shall trump $CLOUD_FILESYSTEM_PATH

There may be other places where one may need to distinguish between local vs. cloud (e.g. disable login for local), having a flag may be more flexible.

makes sense yess

In general, Page Objects should not contain any test logic and test data, like specific paths, names etc.
It's just an abstraction layer to express your tests steps.
So all paths, names belong to test cases/fixtures only.

mnezh · 2024-01-16T10:14:50Z

tests/ui/llm_studio.feature

+        Then I should see the test-experiment should finish successfully 
+        When I delete experiment test-experiment
+        Then I should not see the experiment test-experiment
+


Line 25 contains spaces " ", but it should be just new line character at the end of 24

mnezh · 2024-01-16T10:16:25Z

tests/ui/llm_studio_page.py

+from hac_playwright.pages.base import BasePage
+from playwright.sync_api import expect
+
+CLOUD_FILESYSTEM_PATH = "/home/llmstudio/mount/data/user/"


In general, Page Objects should not contain any test logic and test data, like specific paths, names etc.
It's just an abstraction layer to express your tests steps.
So all paths, names belong to test cases/fixtures only.

tests/ui/llm_studio_page.py

mnezh · 2024-01-16T10:18:41Z

tests/ui/llm_studio_page.py

+
+    def view_experiment_page(self):
+        locator = self.page.get_by_role("button", name="View experiments")
+        locator.click()


self.page.get_by_role("button", name="View experiments").click()

shouldn't we wait for the experiments page to be displayed?

We dont have to

Is it failing for you?

mnezh · 2024-01-16T10:19:16Z

tests/ui/llm_studio_page.py

+        self.page.wait_for_timeout(3000)
+
+    def experiment_name(self, name: str):
+        locator = self.get_by_test_id(self.EXPERIMENT_NAME_SELECTOR)


self.get_by_test_id(self.EXPERIMENT_NAME_SELECTOR).fill(name)

mnezh · 2024-01-16T10:19:33Z

tests/ui/llm_studio_page.py

+        locator.fill(name)
+
+    def llm_backbone(self, value: str):
+        locator = self.page.get_by_role("combobox", name="LLM Backbone")


self.page.get_by_role("combobox", name="LLM Backbone").fill(value)

mnezh · 2024-01-16T10:21:10Z

tests/ui/test.py

+
+@when("I login to LLM Studio", target_fixture="llm_studio")
+def login_to_llm_studio(logger: logging.Logger, page: Page, base_url: str):
+    okta_user = os.environ.get("OKTA_USER")


what if we deploy LLM Studio to GCP env with keycloak instead of Okta?

We need account for that yes. Do you know how to achieve this?

jenkins has secrets for cloud-qa and managed cloud

maxjeblick · 2024-01-29T16:34:44Z

Pipfile

@@ -68,3 +68,7 @@ types-pyyaml = ">=6.0"
 types-requests = ">=2.31"
 types-toml = ">=0.10"
 wheel = "==0.41.0"
+pytest-bdd = "==6.1.1"
+playwright = "==1.40.0"
+hac-playwright = { file = "http://h2o-public-test-data.s3.amazonaws.com/e2e-testing/hac_playwright-1.40.0-py3-none-any.whl" }


This is currently blocking using 1.40

okay, installation the previous available version 1.38.0 instead

pascal-pfeiffer · 2024-02-02T08:36:26Z

Makefile

+	$(PIP) install pip --upgrade
+	$(PIP) install "pipenv>=2023.11.15"


let's stick to what is on main branch, if not causing troubles.
Fixed pip,pipenv version and newer than your min. pipenv

pascal-pfeiffer · 2024-02-02T08:40:07Z

Makefile


 .PHONY: setup-no-flash
 setup-no-flash: pipenv
 	$(PIPENV) install --verbose --python $(PYTHON_VERSION)

+setup-ui: pipenv


Why does it need another make command, why is -dev not enough? It should get some coverage in the README.md or any other appropriate docs how the tests are to be used.

Probably designed to be run on a remote machine connecting to another instance? Again, should get coverage in docs why it has been designed this way and how it is to be used.

precisely, its designed to be able to run from a remote machine. I'll add documentation coverage

pascal-pfeiffer · 2024-02-02T08:41:41Z

Pipfile

@@ -68,3 +68,7 @@ types-pyyaml = ">=6.0"
 types-requests = ">=2.31"
 types-toml = ">=0.10"
 wheel = "==0.41.0"
+pytest-bdd = "==6.1.1"
+playwright = "==1.40.0"


afaik, playwright is not needed when you already specify hac-playwright. Please confirm.

youre right, works without playwright

Makefile

# Conflicts: # Pipfile.lock # requirements.txt

maxjeblick

Thanks a lot for adding the missing Ui tests!
PR looks go to me, we can address subsequent refinements in upcoming PRs.

itsmunishbhardwaj and others added 12 commits January 11, 2024 20:10

Add make commands

c2e730e

Update Pipfile

d36635c

Modify Makefile

5fa4348

Create ui test suite

b36a326

Update requirements.txt

1755869

Fix ci failure

57a4444

Fix make style error

56038bb

Modify conftest

8632189

Modify conftest

83302f3

Ignore [attr-defined]

203110e

Ignore [attr-defined]

f7808f0

Ignore attribute error

47e8639

itsmunishbhardwaj mentioned this pull request Jan 14, 2024

UI Automation tests for LLM Studio #549

Closed

itsmunishbhardwaj requested review from maxjeblick, pascal-pfeiffer and mnezh January 14, 2024 12:22

itsmunishbhardwaj marked this pull request as ready for review January 14, 2024 12:27

mnezh suggested changes Jan 15, 2024

View reviewed changes

maxjeblick reviewed Jan 15, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

maxjeblick suggested changes Jan 15, 2024

View reviewed changes

maxjeblick reviewed Jan 15, 2024

View reviewed changes

tests/ui/test.py Outdated Show resolved Hide resolved

maxjeblick reviewed Jan 15, 2024

View reviewed changes

Modify to accomodate feedback

d555bb4

mnezh suggested changes Jan 16, 2024

View reviewed changes

itsmunishbhardwaj added 2 commits January 18, 2024 17:46

Modify code to make it work locally

e9b4b6b

Modify PR with feedback

7475516

maxjeblick reviewed Jan 29, 2024

View reviewed changes

itsmunishbhardwaj and others added 3 commits February 2, 2024 10:06

Resolve conflicts and merge main

fe6dcd9

Update requirements.txt

deb7701

revert requirements.txt to original

9778cf4

itsmunishbhardwaj and others added 2 commits February 2, 2024 13:56

Resolve merge conflicts

5aa4737

Update requirements.txt

1538f31

pascal-pfeiffer reviewed Feb 2, 2024

View reviewed changes

maxjeblick reviewed Feb 2, 2024

View reviewed changes

Makefile Show resolved Hide resolved

itsmunishbhardwaj added 3 commits February 8, 2024 17:30

Modify make command and update pipfile

ae32939

Fix local ui testing

33bf1af

Add keycloak login support

d66e540

itsmunishbhardwaj requested review from maxjeblick and mnezh February 9, 2024 05:22

maxjeblick and others added 3 commits February 12, 2024 10:39

Merge branch 'main' into munish/ui

b49f8fe

# Conflicts: # Pipfile.lock # requirements.txt

fix pipfile merge conflicts

8175a51

Update requirements.txt

dc65af5

mnezh approved these changes Feb 12, 2024

View reviewed changes

maxjeblick approved these changes Feb 12, 2024

View reviewed changes

maxjeblick merged commit 1442394 into main Feb 12, 2024

maxjeblick deleted the munish/ui branch February 12, 2024 10:36

psinger mentioned this pull request Apr 22, 2024

[CODE IMPROVEMENT] Playwright test suite for LLM Studio #524

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UI Automation Tests for LLM Studio #561

UI Automation Tests for LLM Studio #561

itsmunishbhardwaj commented Jan 12, 2024 •

edited

mnezh Jan 15, 2024

mnezh Jan 15, 2024

mnezh Jan 15, 2024

mnezh Jan 15, 2024

maxjeblick left a comment

maxjeblick Jan 15, 2024

itsmunishbhardwaj Jan 15, 2024

maxjeblick Jan 15, 2024

itsmunishbhardwaj Jan 15, 2024

mnezh Jan 16, 2024

mnezh Jan 16, 2024

mnezh Jan 16, 2024

mnezh Jan 16, 2024

itsmunishbhardwaj Jan 18, 2024

itsmunishbhardwaj Jan 18, 2024

mnezh Jan 16, 2024

mnezh Jan 16, 2024

mnezh Jan 16, 2024

itsmunishbhardwaj Jan 18, 2024

mnezh Jan 18, 2024

maxjeblick Jan 29, 2024 •

edited

itsmunishbhardwaj Feb 8, 2024

pascal-pfeiffer Feb 2, 2024

pascal-pfeiffer Feb 2, 2024 •

edited

pascal-pfeiffer Feb 2, 2024

itsmunishbhardwaj Feb 2, 2024

pascal-pfeiffer Feb 2, 2024

itsmunishbhardwaj Feb 8, 2024

maxjeblick left a comment

		$(PIP) install pip --upgrade
		$(PIP) install "pipenv>=2023.11.15"

UI Automation Tests for LLM Studio #561

UI Automation Tests for LLM Studio #561

Conversation

itsmunishbhardwaj commented Jan 12, 2024 • edited

How to run the tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxjeblick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxjeblick Jan 29, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pascal-pfeiffer Feb 2, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxjeblick left a comment

Choose a reason for hiding this comment

itsmunishbhardwaj commented Jan 12, 2024 •

edited

maxjeblick Jan 29, 2024 •

edited

pascal-pfeiffer Feb 2, 2024 •

edited