UI e2e testing framework #7350

satkunas · 2022-11-07T19:51:03Z

Description

Integrate cypress end-to-end testing framework

Impacts

Venom workflows
CI/CD results

NEW Package(s) required

RHEL

xorg-x11-server-Xvfb gtk2-devel gtk3-devel libnotify-devel GConf2 nss libXScrnSaver alsa-lib

Debian

libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb

Delete branch after merge

YES

Checklist

(REQUIRED) - [yes, no or n/a]

Document the feature
Add unit tests
Add acceptance tests (TestLink)

NEWS file entries

(REQUIRED, but may be optional...)

New Features

Adding end-to-end testing framework to UI for CI/CD pipelines

.gitlab-ci.yml

t/html/pfappserver/Makefile

t/html/pfappserver/README.md

t/venom/scenarios/pfappserver/playbooks/run_tests.yml

t/venom/vars/all.yml

nqb · 2023-05-31T11:57:47Z

Difference between `pfappserver_configurator` and `configurator` test suites

It looks like there are differences between pfappserver_configurator and configurator (existing) test suites.

Step 1: configurator will configure dhcp-listener, registration and isolation interfaces, DNS servers
Step 2: configurator will configure alerting and send an email
Post-configurator: configurator will use new admin password defined at step 2 to login against API

[X] Packaging

We need to package t/html/ directory in packetfence-test package. Not sure if this is the case at the moment.

EDIT: packaging checked and t/html is part of packetfence-test on RPM and Deb packages.

JeGoi

Some modifications/answers on my sides

.gitlab-ci.yml

t/venom/Makefile

JeGoi · 2023-06-02T15:43:41Z

t/html/pfappserver/Makefile

+	@echo "make test-x11: run tests interactively (w/ X11 \$$DISPLAY)"
+	@echo ""
+	@echo "make [target] DEBUG=cypress:* (see https://docs.cypress.io/guides/references/troubleshooting#Log-sources)"
+	@echo "make [target] BROWSER=firefox:nightly"


Suggested change

@echo "make [target] BROWSER=firefox:nightly"

@echo "make [target] BROWSER= chrome" # or for ex: firefox:nightly

To be consistant with default values

The default was changed, firefox is preferred due to lower memory usage. It also gives us better coverage that will prevent firefox-only issues in the future.

JeGoi · 2023-06-02T15:58:55Z

t/html/pfappserver/cypress/config/cypress.config-configurator.js

+      'cypress/specs/e2e/*-configurator/**/*.cy.{js,jsx,ts,tsx}',
+    ],
+  }
+};


Same as the other one => t/html/pfappserver/cypress/config/cypress.config-configuration.js
Duplicate?

That's a stub for differentiation so it's more explicit for the next person to setup a new scenario using the same framework.

JeGoi · 2023-06-02T16:00:15Z

t/html/pfappserver/cypress/config/cypress.config.js

+module.exports = {
+//  defaultCommandTimeout: 10000, // 10s
+  e2e: {
+    baseUrl: 'https://localhost:1443',


Isn't it supposed to be variables? or there is an override system

Cypress uses NodeJS, so this is what's expected

I misunderstood which line you were referring to. I understand now.

This is the default value. Our vars are injected in the scenario through make using the BASE_URL environmental value.

t/venom/scenarios/pfappserver/playbooks/run_tests.yml

t/venom/scenarios/pfappserver/playbooks/localdev.yml

JeGoi · 2023-06-02T17:45:33Z

t/html/pfappserver/cypress/support/commands.js

+  /*
+  cy.visit('https://localhost:1443/admin#/login').then(() => {
+    cy.readFile('/usr/local/pf/conf/unified_api_system_pass').then((password) => {
+      cy.get('form input#username').first().type('system')
+      cy.get('form input#password').first().type(password)
+      cy.get('form button[type="submit"]').first().click()
+    })
+  })
+  */
+  return cy.pfUnifiedSystemPassword().then(password => {
+    return cy.request('POST', '/api/v1/login', { username: 'system', password }).then(response => {
+      return window.localStorage.setItem('user-token', response.body.token)
+    })
+  })
+})


Are we using API or the browser?

The configurator is using the Browser, which doesn't require /login

This was created during the PoC to provide a support helper for writing tests. I left it there to help with future development.

nqb · 2023-06-06T11:58:50Z

I resolved some conversation.
Two items remaining for me:

Difference between pfappserver_configurator and configurator test suites
Tag and Tags

JeGoi

Move t/venom/scenarios/pfappserver/README.md to t/venom/test_suites/pfappserver_configurator/TESTSUITE.md to be consistant with the other tests.

t/venom/scenarios is more for ansible to prepare the environment and the test and it can be shared with multiple test_suites.

satkunas · 2023-06-06T19:11:41Z

@nqb repo rebased w/ devel.

JeGoi

Ok, looks good to me.

nqb · 2023-06-07T07:07:28Z

Running a pipeline to see if artifacts are saved correctly: https://gitlab.com/inverse-inc/packetfence/-/pipelines/891933439

nqb · 2023-06-07T07:48:21Z

Regarding Difference between pfappserver_configurator and configurator test suites, @satkunas is aware that test suites are different. We will keep things as is. Once we want to deprecate configurator test suite, we will extend what pfappserver_configurator is doing.

nqb

@satkunas, I identified few issues:

We need to configure 2nd interface (eth1) as management during configurator and not 1st interface (eth0). Otherwise, it can cause issues because iptables will reject SSH requests on eth1
Artifacts were not correctly pulled using Ansible because hostname is changed during configurator. This issue should be fixed now
I'm not able to see if screenshots are available in GitLab because it looks like current Cypress test doesn't generate screenshots, could you tell me how I can enable this ?
There is a firefox error when we run tests on Debian (I provided logs in DM)

Also I opened #7704 because I got it while testing.

satkunas · 2023-06-07T16:48:36Z

@nqb

We need to configure 2nd interface (eth1) as management during configurator and not 1st interface (eth0). Otherwise, it can cause issues because iptables will reject SSH requests on eth1

Adjustments have been made

I'm not able to see if screenshots are available in GitLab because it looks like current Cypress test doesn't generate screenshots, could you tell me how I can enable this ?

This is automatic. By default at least 1 screenshot should be created w/ a failed test. This is where I mentioned I had some trouble extracting the them from the VM.

There is a firefox error when we run tests on Debian (I provided logs in DM)

I added a -headless Cypress command-line flag for firefox. My reasoning is based on libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) in venom.log

If the test still fails, we may need to revert back to using the default browser.

diff --git a/t/venom/test_suites/pfappserver_configurator/run_e2e.yml b/t/venom/test_suites/pfappserver_configurator/run_e2e.yml
index 7d0c2ece7a..f8c498b75c 100644
--- a/t/venom/test_suites/pfappserver_configurator/run_e2e.yml
+++ b/t/venom/test_suites/pfappserver_configurator/run_e2e.yml
@@ -5,7 +5,6 @@ testcases:
   steps:
   - type: html_e2e
     base_url: "{{.html.pfappserver_baseurl}}"
-    browser: firefox
     config_file: "cypress/config/cypress.config-configurator.js"
     tag: "pfappserver_configurator"
     info:

nqb · 2023-06-08T05:20:55Z

Running a new pipeline to see if it's better: https://gitlab.com/inverse-inc/packetfence/-/pipelines/893321590

…scaling

…detect

satkunas force-pushed the feature/ui-e2e branch from 7ff0f5c to 292a7d9 Compare January 13, 2023 20:35

satkunas force-pushed the feature/ui-e2e branch from d32384e to d6b3303 Compare March 28, 2023 19:15

satkunas requested review from nqb and JeGoi May 16, 2023 18:27

satkunas added this to the PacketFence-13.0 milestone May 16, 2023

satkunas added Status: For review Status: In progress labels May 16, 2023

nqb suggested changes May 31, 2023

View reviewed changes

JeGoi requested changes Jun 2, 2023

View reviewed changes

satkunas requested review from nqb and JeGoi June 2, 2023 18:12

JeGoi requested changes Jun 6, 2023

View reviewed changes

satkunas force-pushed the feature/ui-e2e branch from 61ff913 to 89b72ac Compare June 6, 2023 19:10

JeGoi approved these changes Jun 6, 2023

View reviewed changes

nqb suggested changes Jun 7, 2023

View reviewed changes

satkunas requested a review from nqb June 7, 2023 16:49

nqb force-pushed the feature/ui-e2e branch from 36baefa to a51a5fc Compare June 8, 2023 09:01

satkunas added 6 commits June 9, 2023 12:29

initial e2e uio commit

0608bb6

add some tests

5032d52

split configs, add Makefile targets, begin composite scaffolding for …

f807524

…scaling

move to t/html/pfappserver/

3f5000e

improve Makefile

e5339ae

add venom

9b5ce13

satkunas and others added 26 commits June 9, 2023 12:29

working localdev with external sources

e35accc

add scenario to gitlab runners

6ffa386

split localdev

4080019

deprecate cypress cloud, remove psono vars

9726bda

tune test for CI

dffefaf

temporarily disable all other test scenarios

93cc8e4

revert temp disable

51a16ac

use firefox

23bad61

remove block

8af9180

Add -y to update

c60d99e

add docs on tags

8138bba

improve tags docs

682cd0f

add devel and maintenance branches

c2a0eff

move cypress install to configure

c9938e0

fix tests for devel and maintenance branches

e07b726

reorder test

dc5e445

move readme

fd14097

save results using current FQDN of VM

aa1672d

use headless firefox to avoid libva aGetDriverNameByIndex() error

2b95773

rework interfaces, use 2nd interface for management, stop using auto-…

68a7136

…detect

reword cloud as project

6028854

code is already available inside VM

d473fb8

wiat for cancel button to be enabled

a98fbc3

typo

b65a75d

split wait and click

1807566

remove unneeded click() and wait for API

f632528

satkunas force-pushed the feature/ui-e2e branch from 8df11f6 to f632528 Compare June 9, 2023 12:29

nqb merged commit 33aa736 into devel Jun 12, 2023

nqb deleted the feature/ui-e2e branch June 12, 2023 08:16

nqb added a commit that referenced this pull request Jun 12, 2023

news for #7350

1b4a20d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UI e2e testing framework #7350

UI e2e testing framework #7350

satkunas commented Nov 7, 2022 •

edited

Loading

nqb commented May 31, 2023 •

edited

Loading

JeGoi left a comment

JeGoi Jun 2, 2023

satkunas Jun 2, 2023

JeGoi Jun 2, 2023

satkunas Jun 2, 2023

JeGoi Jun 2, 2023

satkunas Jun 2, 2023

satkunas Jun 2, 2023 •

edited

Loading

JeGoi Jun 2, 2023

satkunas Jun 2, 2023

nqb commented Jun 6, 2023

JeGoi left a comment

satkunas commented Jun 6, 2023

JeGoi left a comment

nqb commented Jun 7, 2023

nqb commented Jun 7, 2023 •

edited

Loading

nqb left a comment •

edited

Loading

satkunas commented Jun 7, 2023 •

edited

Loading

nqb commented Jun 8, 2023

	@echo "make [target] BROWSER=firefox:nightly"
	@echo "make [target] BROWSER= chrome" # or for ex: firefox:nightly

UI e2e testing framework #7350

UI e2e testing framework #7350

Conversation

satkunas commented Nov 7, 2022 • edited Loading

Description

Impacts

NEW Package(s) required

RHEL

Debian

Delete branch after merge

Checklist

NEWS file entries

New Features

nqb commented May 31, 2023 • edited Loading

Difference between pfappserver_configurator and configurator test suites

[X] Packaging

JeGoi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

satkunas Jun 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nqb commented Jun 6, 2023

JeGoi left a comment

Choose a reason for hiding this comment

satkunas commented Jun 6, 2023

JeGoi left a comment

Choose a reason for hiding this comment

nqb commented Jun 7, 2023

nqb commented Jun 7, 2023 • edited Loading

nqb left a comment • edited Loading

Choose a reason for hiding this comment

satkunas commented Jun 7, 2023 • edited Loading

nqb commented Jun 8, 2023

satkunas commented Nov 7, 2022 •

edited

Loading

nqb commented May 31, 2023 •

edited

Loading

Difference between `pfappserver_configurator` and `configurator` test suites

satkunas Jun 2, 2023 •

edited

Loading

nqb commented Jun 7, 2023 •

edited

Loading

nqb left a comment •

edited

Loading

satkunas commented Jun 7, 2023 •

edited

Loading