Provide initial system test suite #2092

de-jcup · 2023-03-20T07:11:22Z

This is a subissue of

System testing test suite #453

Situation

We want to have a system test suite which can be used by Java developers (as JUnit tests), but also by people who do not write java Junit tests but want to create or test just PDS solutions.

Solution

We use a datamodel for junit tests which can be serialized and deserialized as JSON
Inside the PDS tool, we also provide access to this model by loading from file system
The java side (junit test) will have a SystemTestAPI class which is the central point for test development (similar to TestAPI in sechub-integration test)

The text was updated successfully, but these errors were encountered:

- introduce model - introduce first version of API - added some unit tests - added one integration test

- introduce model - introduce first version of API - added some unit tests - added one integration test - implemented config features: - configrue url + credentials for sechub(remote) - configure start stop for sechub (local) - configure start stop for pds solutions (local) - configure executors for sechub - configure tests + asserts - implemented runtime features - start /stop of sechub (local) - start /stop of pds-solutions (local) - environment variables on script launch (local) - fallback default-profile and default-project when missing in configuration + link to executors - missing - pds server configuration must be moved to pds-commons so is readable by system tests (necessary to check if config file is wrong configured, and to fetch scanType etc.) - check pds, sechub started - create local profiles, executor configs on sechub automatically - create local project - test and admin user for local machine (it is normally not integration test mode - do we need credentials here as well?) - connect user with project - connect project with profiles - testing mechanism - prepare phase - upload - sechub job creation - client usage (?) - variable support - automatic variables (RUNTIME parts from ConfigConstants - e.g. runtime.workspaceRoot etc.)

- introduce model - introduce first version of API - added some unit tests - added one integration test - implemented config features: - configrue url + credentials for sechub(remote) - configure start stop for sechub (local) - configure start stop for pds solutions (local) - configure executors for sechub - configure tests + asserts - automatically set calculated parts into configuration (altering configuration, but this is very useful for debugging) - automatically calculate pds server config file location - the wrong configuration exception does now always contain a stacktrace with the altered JSON - implemented runtime features - start /stop of sechub (local) - start /stop of pds-solutions (local) - environment variables on script launch (local) - fallback default-profile and default-project when missing in configuration + link to executors - missing - pds server configuration must be moved to pds-commons so is readable by system tests (necessary to check if config file is wrong configured, and to fetch scanType etc.) - check pds, sechub started - create local profiles, executor configs on sechub automatically - create local project - test and admin user for local machine (it is normally not integration test mode - do we need credentials here as well?) - connect user with project - connect project with profiles - testing mechanism - prepare phase - upload - sechub job creation - client usage (?) - variable support - automatic variables (RUNTIME parts from ConfigConstants - e.g. runtime.workspaceRoot etc.)

- introduce model - introduce first version of API - added some unit tests - added one integration test - implemented config features: - configrue url + credentials for sechub(remote) - configure start stop for sechub (local) - configure start stop for pds solutions (local) - configure executors for sechub - configure tests + asserts - automatically set calculated parts into configuration (altering configuration, but this is very useful for debugging) - automatically calculate pds server config file location - the wrong configuration exception does now always contain a stacktrace with the altered JSON - implemented runtime features - start /stop of sechub (local) - start /stop of pds-solutions (local) - environment variables on script launch (local) - fallback default-profile and default-project when missing in configuration + link to executors

- phases introduced with dedicated classes - environment variables are now replaced in template - user defined variables are resolved now - currently many test still do fail

- variable handling is now inside integration tests and works - added possibility to add parameters to script execution - fixed problems

- runtime variables introduced - secretEnv variables introduced - runtime.workspaceRoot available - if there is a usage of an unknown variable a proposal for the correct variable definition is done. For example: if user uses an unknown ${runtime.XYZ} expression, this will result in an error with a proposal containing at least "runtime.workspaceRoot" - systemtests documentation - initial asciidoc generator for system test parts - refactorings

- stages introduced - stages wait for script processes whern necessary - added documentation - some refactorings

- when no build stage is defined, the api parts are not built - no build stage setup necessary for "normal" builds - adopted release and gradle github workflow - updated documentation

- Reduced OpenAPITestTool - SecHubClient renamed to SecHubAccess - SecHubAccess builds internally http client and even provides trust all SSL dummy - SecHubAccess provides getters to access different api parts - added NotFound to AbstractTestExecutable of a flaky integration test so it shall retry until found...

- currently there are problems with UUID results (instead of correct JSON).

- projects are created - profiles are created - dry run possible

- health check parts added - auto prepare some parts added - executors, profiles and projects are now configured completely (and it is possible to restart the setup, old data is automatically removed except old but now unused executor configurations) - credentials are still missing for PDS solution

- added credentials handling in executor configuration - moved all fallback parts into preparation phase, no longer field initializations (avoids confusion for users) - new default fallbacks introduced - wait for available implemented for SecHub - some minor changes

- implementation is now able to start test prepare scripts - for prepare scripts the framework does always wait - added illegal state handling (scripts not existing, scripts cannot be executed) - added test parts into SystemTestDryRunIntegrationTest

- Added upload definitions - github workflow integration test gradle calls do now set SecHub build stage to "all" which will execute system integration test as well - Moved archive support into systemtest and used this one now in PDS tools - added internal test for illegal states - improved health check - system test document now inside techdoc

- fixed failing part (variant names - improved log output for rest doc parts

- secretEnv is not usable for credentials (PDS and sechub side + executor configuration setup) - enhanced jsonconverter ,it is now possible to add a custom mapper - use generated example in documentation - reduced output of generated examples - system test api builders do now provide comments method as well - added unit tests

- reduced assert sechub result: not longer an array but an optional..

- changed generators, now internal access and no longer abstract variant --> composition vs. inheritance - fixed failing test (rename of parameters was reason)

- remote setup had a bug (local setup was used and a skip was not skipping as well) - refactoring: constants extracted and added javadoc parts using now always TestConfigUtil to create environment providers - remote system test does now work

- enhanced fake bash scripts for dry run test

- log outputs - at the end the there is now a log info with details and path to workspace location - runtime contains now artifacts - filenames + structure in runtime changed

- Provide initial system test suite #2092 - introduce build stage because of build lifecycle problems - Introduce 'sechub-pds-commons-core' #2118 - provide wrapper api in sechub-api-java to stabilize usage #2184 - fix wrong OpenAPI generation for adminAssignsUserToProject #2191 - tests are not marked as integration tests but they are #2204 - Archive support is not able to compress multiple folders or single files #2359 - SecHubConfigurationModel has no excludes/includes #2363 - Improve log output for PDS execution failures #2398

de-jcup self-assigned this Mar 20, 2023

de-jcup added a commit that referenced this issue Mar 20, 2023

Initial setup for the system test api #2092

9d62463

- introduce model - introduce first version of API - added some unit tests - added one integration test

de-jcup added the testing label Mar 22, 2023

de-jcup added a commit that referenced this issue Mar 23, 2023

Initial setup for the system test api #2092

49374c7

- introduce model - introduce first version of API - added some unit tests - added one integration test

This was referenced Mar 28, 2023

Introduce 'sechub-pds-commons-core' #2118

Closed

Introduce system test api #2092 #2119

Merged

de-jcup added a commit that referenced this issue Apr 4, 2023

Improved error handling, integration tests do now check results #2092

ce9debe

- variable handling is now inside integration tests and works - added possibility to add parameters to script execution - fixed problems

de-jcup added a commit that referenced this issue Apr 4, 2023

Improved error handling, integration tests do now check results #2092

afa06c8

- variable handling is now inside integration tests and works - added possibility to add parameters to script execution - fixed problems

de-jcup added a commit that referenced this issue Apr 6, 2023

Introducing stages and changed error handling #2092

a37a4a7

- stages introduced - stages wait for script processes whern necessary - added documentation - some refactorings

de-jcup added a commit that referenced this issue Apr 12, 2023

Improved runtime process execution #2092

31a9e3c

de-jcup mentioned this issue Apr 13, 2023

Make sechub-api-java available as project dependency inside build #2162

Open

de-jcup added a commit that referenced this issue Apr 17, 2023

Added adminApi parts to OpenApiTestTool #2092

84ee9ab

- currently there are problems with UUID results (instead of correct JSON).

de-jcup mentioned this issue Apr 21, 2023

Fix wrong OpenAPI generation for adminAssignsUserToProject #2191

Closed

de-jcup added a commit that referenced this issue Apr 21, 2023

Improved wrapper parts #2092

0978332

de-jcup added a commit that referenced this issue Apr 21, 2023

Implemented local sechub configuration parts #2092

c6c6463

- projects are created - profiles are created - dry run possible

de-jcup added a commit that referenced this issue Jul 19, 2023

Added sechub debug logging for checkmarx.sh #2092

dcfb694

de-jcup mentioned this issue Jul 19, 2023

Improve log output for PDS execution failures #2398

Closed

de-jcup added a commit that referenced this issue Jul 19, 2023

Improved error handling in integration test script to find errors #2092

8fa1cde

de-jcup added a commit that referenced this issue Jul 19, 2023

No longer absolute paths in created archives #2092

a4f0ceb

de-jcup added a commit that referenced this issue Jul 20, 2023

Fixed integration test + replaced deprecated call #2092

ceecee3

de-jcup added a commit that referenced this issue Jul 21, 2023

Further fixes for build #2092

a9f23c5

- fixed failing part (variant names - improved log output for rest doc parts

de-jcup added a commit that referenced this issue Jul 27, 2023

Improvements, manual gosec parts, documentation etc. #2092

40f12f9

- reduced assert sechub result: not longer an array but an optional..

de-jcup added a commit that referenced this issue Jul 28, 2023

Start/Stop final termination now only when init phase not broken #2092

0eb426f

de-jcup added a commit that referenced this issue Jul 28, 2023

Fixed failing documentation build #2092

dedaa64

de-jcup added a commit that referenced this issue Jul 28, 2023

Minor changes #2092

a60320e

de-jcup added a commit that referenced this issue Aug 8, 2023

Bugfix for pds tools cli and added tests #2092

c82a219

de-jcup added a commit that referenced this issue Aug 8, 2023

Review changes #2092

29a7088

- changed generators, now internal access and no longer abstract variant --> composition vs. inheritance - fixed failing test (rename of parameters was reason)

de-jcup added a commit that referenced this issue Aug 9, 2023

Review and imporovements for #2092

1350354

- enhanced fake bash scripts for dry run test

de-jcup added a commit that referenced this issue Aug 10, 2023

Improvements #2092

67b7269

- log outputs - at the end the there is now a log info with details and path to workspace location - runtime contains now artifacts - filenames + structure in runtime changed

de-jcup added a commit that referenced this issue Aug 10, 2023

Improvement - wait for SecHub before PDS start #2092

0d43774

de-jcup added a commit that referenced this issue Aug 10, 2023

Improvement - wait for SecHub admin available #2092

1ca0ec1

de-jcup added a commit that referenced this issue Aug 11, 2023

Review changes + documentation updates #2092

c13cee8

de-jcup added a commit that referenced this issue Aug 11, 2023

Introduce reduced-openapi3.json #2092

2137d2a

de-jcup added a commit that referenced this issue Aug 11, 2023

Jenkins build failure fix #2092

0c359ca

de-jcup added a commit that referenced this issue Aug 11, 2023

Added some files with unicode and space characters to tests #2092

e1c3f1d

de-jcup added a commit that referenced this issue Aug 11, 2023

Fix jenkins build problems #2092

c44c3d8

de-jcup added a commit that referenced this issue Aug 11, 2023

Renamed unicode parts (build problems (fs/gradle) #2092

4e6cf43

de-jcup added a commit that referenced this issue Aug 11, 2023

Some fixes #2092

7d13401

de-jcup mentioned this issue Aug 11, 2023

Jenkins build does not execute integration test from system test framework #2470

Open

de-jcup closed this as completed in #2119 Aug 11, 2023

Jeeppler mentioned this issue Aug 22, 2023

Replace manually written API calls in WebUI with SecHub Java API Client #2513

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide initial system test suite #2092

Provide initial system test suite #2092

de-jcup commented Mar 20, 2023 •

edited

Provide initial system test suite #2092

Provide initial system test suite #2092

Comments

de-jcup commented Mar 20, 2023 • edited

Situation

Solution

de-jcup commented Mar 20, 2023 •

edited