Disabled test results #212

jhughesbiot · 2023-12-27T19:39:03Z

Adds better descriptions for tests that are disabled
Detects and ignores disabled tests in report instead of failing \
Adds back TLS and DNS tests removed in a disabled state

Add dns and tls tests back in disabled state

jboddey · 2024-01-02T10:18:05Z

The output of this PR is as below:

If a test is disabled, we do not want it to be present in the test list - especially as the user does not have the ability to enable or disable tests from the UI. Instead, disabled tests should not be present or run at all.

jhughesbiot · 2024-01-02T14:50:23Z

I think we need to differentiate here what a user can do vs what the UI can do. The user has every capability of disabling tests, the UI does not. Skipping a test within a module and indicating it makes sense. Removing any indication a disabled test exists in the system seems like something that would cause confusion in the results.

jboddey · 2024-01-03T13:18:26Z

Our primary user is manufacturers who use the user interface. With this current setup, we are telling the user about tests which do not exist within qualification (or will not in 4.5). I am not opposed to have the tests available for labs or other users to run but it seems counter productive to have them present.

jhughesbiot · 2024-01-03T15:26:59Z

Maybe we just need to add this as a configurable option in the report to show/hide disabled tests hiding by default. I can look into that and add to this PR.

Not sure we should remove any tests that might get removed from 4.5 until it's finalized but should continue to use 4.4 as the reference for test generation until then.

jboddey · 2024-01-04T10:15:57Z

Maybe we just need to add this as a configurable option in the report to show/hide disabled tests hiding by default. I can look into that and add to this PR.

Not sure we should remove any tests that might get removed from 4.5 until it's finalized but should continue to use 4.4 as the reference for test generation until then.

With this in mind, I'm happy to have the tests re-added and enabled by default - provided that we add an option in the config to hide disabled tests (prevent them from running at all) which should be enabled by default. Once 4.5 is released, we will set the test status to disabled.

jboddey · 2024-02-07T17:08:12Z

These re-added tests are no longer present in the device qualification test plan so I will close this PR now. In the future, I would like to add a module with just informational tests that provide us with additional info about devices. I will leave the branch present.

jhughesbiot · 2024-02-07T17:11:27Z

Re-opening as this is not just adding tests in disabled state, it is fixing disable tests state detection. Regardless of current test plan, we shouldn't remove tests if they are not part of the test plan, we should only ever disable them unless they are fully removed for some other reason.

jboddey · 2024-02-07T17:24:17Z

Re-opening as this is not just adding tests in disabled state, it is fixing disable tests state detection. Regardless of current test plan, we shouldn't remove tests if they are not part of the test plan, we should only ever disable them unless they are fully removed for some other reason.

If this is the case, could we hide disabled tests from the report (with a config option to toggle this behavior - which will be set to hide disabled tests by default)?

jhughesbiot · 2024-02-07T18:29:13Z

Right now it marks the tests as disabled in their description and with a skipped state. If we want them hidden from the report, we should probably move that to it's own update so we can prevent merge conflicts with all the other recent updates to the report format.

framework/python/src/test_orc/test_orchestrator.py

* Better handling of disbled tests Add dns and tls tests back in disabled state * pyling fixes * Fix subscriptable error * Re-add old tests as informational --------- Co-authored-by: J Boddey <boddey@google.com>

* added tests for profile endpoints * Modified test_start_testrun_started_successfully payload to match the expected json format, updated the profile endpoints tests * fixed the pylint errors from test_api.py * fixed few more pylint errors * Expired profile profile (#619) * Use device mac_addr if report mac_addr is missing (#622) * Fix some pylint issues (#620) * Disabled test results (#212) * Better handling of disbled tests Add dns and tls tests back in disabled state * pyling fixes * Fix subscriptable error * Re-add old tests as informational --------- Co-authored-by: J Boddey <boddey@google.com> * added try-except block to delete_all_profiles() * Revert session file back * Add new line * Update api testing * updated the tests for profile endpoint, added a new fixture (add_profile) to create a profile * Updated profile endpoints: created a new fixture add_profile for creating profiles * Fix pylint * Fix pylint * Fix pylint issues --------- Co-authored-by: Sofia Kurilova <kurilova@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com>

* Update requirements.txt (#429) Signed-off-by: J Boddey <boddey@google.com> * Remove in progress status after discovery (#422) * Add steps to resolve to PDF report (#411) * Add progress for steps to resolve * Formatting * Remove print statements * Allow for no steps to resolve * Add multipage * Fix multipage * Fix config * update report unit tests --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Bump ejs in /modules/ui in the npm_and_yarn group across 1 directory (#428) Bumps the npm_and_yarn group with 1 update in the /modules/ui directory: [ejs](https://github.com/mde/ejs). Updates `ejs` from 3.1.9 to 3.1.10 - [Release notes](https://github.com/mde/ejs/releases) - [Commits](mde/ejs@v3.1.9...v3.1.10) --- updated-dependencies: - dependency-name: ejs dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * For delete report take mac_addr from upper level (#436) * Update dependencies (#435) * Add new mac addr field for report deleting (#432) * Bump version to v1.2.2 (#433) * fix: replace Feature Not Present to Feature Not Detected (#445) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix PDF alignment (#441) * Do not remove form control on destroy as it causes error; call system/config on settings open (#442) * Fix some pylint issues (#437) * Update documentation (#448) * Update documentation * Update docs * V1.3 (#393) * Adds version analytics event (#306) * Techdebt: adds state for testrun page (#392) * Fix tests (#397) * Fix tests * Update node version * 331379891: (feat) disable connection settings when testrun is in progress (#371) * 331379891: (feat) disable connection settings when testrun is in progress * 331379891: (fix) include more testrun results as progress * 331379891: (fix) fix spelling * 333349715: (fix) GAR 1.3 The disabled system settings panel contains a focusable element (#388) Co-authored-by: Volha Mardvilka <mardvilka@google.com> --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Disable device item if device in progress (#370) * Adds ga to track testrun initiation (#415) Adds ga ti track testrun initiation * Fix function return value (#421) * Fix redirect + adds google analytics (#420) * Adds certificates drawer component with list of certificates (#414) Adds certificates drawer component with list of certificates. Upload and delete is out of scope * List modules (#425) * Adds delete certificate (#424) Adds delete certificate * Adds class for links to track report type on download; adds event when download report is clicked on progress page (#434) * Adds upload certificate (#431) * Adds upload certificate * move delete functionality to certificates store * Catch error on report/certificate deletion (#438) * Fix button size; fix text-overflow (#440) * Adds focus on next or close button when certificate is deleted (#439) * Adds focus on next or close button when certificate is deleted * Enable test modules for initiate test run dialog (#400) Enable test modules for initiate test run dialog * Add steps to resolve to PDF report (#411) * Add progress for steps to resolve * Formatting * Remove print statements * Allow for no steps to resolve * Add multipage * Fix multipage * Fix config * update report unit tests --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Add new mac addr field for report deleting (#432) * Bump version to v1.2.2 (#433) * 337012359: (feat) add snackBar with wait or stop testrun (#444) * 337012359: (feat) add snackBar with wait or stop testrun * 337012359: (fix) update to fix failed tests * 337012359: (fix) add fix for deletCertificate test --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Feature/port stats (#430) * Add port speed and duplex tests Add unit tests for port stats testing Add place holder for ip port link test * Add ethtool to system dependencies Restructure conn stats methods and tests Resolve port stat information for link test * Fix runtime issue * Implement link test Fix unit tests Misc port stats updates * fix pylint issues * update readme * pylint fixes --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> * 339315842: (feat) add risk assessment tab (#450) * 339315842: (feat) add risk assessment tab * 339315842: (feat) add risk assessment tab --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Add steps to resolve to PDF report (#411) * Add progress for steps to resolve * Formatting * Remove print statements * Allow for no steps to resolve * Add multipage * Fix multipage * Fix config * update report unit tests --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Add new mac addr field for report deleting (#432) * Remove rebase error --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Olga Mardvilko <omardvilko@mail.ru> Co-authored-by: Volha Mardvilka <mardvilka@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> Co-authored-by: jhughesbiot <50999916+jhughesbiot@users.noreply.github.com> * Rename "history" to "reports" (#456) * Rename "device-repository" to "devices" (#455) * Change certificates endpoints (#458) * 339311887: (feat) display saved risk profile (#460) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix ui defects (#459) * 340835710: (fix) [Risk assessment] change page view when callout is visible (#461) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix style to allow screen reader to read label (#462) * Updaate device test module configuration from api start endpoint (#463) * Add get profiles format endpoint (#465) * 339311250: (feat) delete risk profile (#466) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 341254121: (fix) [a11y] add item name for aria-labels delete and copy buttons (#467) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Make testing statuses available outside of Testing Tab (#468) * Adds timeout token * Status is available on all pages; Removed unused field isTestrunStarted - actual status is available * Validate CA certificate on FE (#464) * Adds validation on certificate upload * Show notification when error happens * 341901606: (fix) [GAR 1.6]: update for arrows usage on risk profile (#471) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Adds space in certificate name regexp (#469) * 342096458: (fix) [GAR 1.3] add focus trap to certificates panel to prevent move focus to risk profile (#472) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * The focus is not moved to the snackbar with certificate validation rule (#470) * Adds class to fix focus flow * Changes the delay according to string length * Adds file name in error message (#473) * Add certificate endpoints (#451) * Adds version analytics event (#306) * Techdebt: adds state for testrun page (#392) * Fix tests (#397) * Fix tests * Update node version * 331379891: (feat) disable connection settings when testrun is in progress (#371) * 331379891: (feat) disable connection settings when testrun is in progress * 331379891: (fix) include more testrun results as progress * 331379891: (fix) fix spelling * 333349715: (fix) GAR 1.3 The disabled system settings panel contains a focusable element (#388) Co-authored-by: Volha Mardvilka <mardvilka@google.com> --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Disable device item if device in progress (#370) * Adds ga to track testrun initiation (#415) Adds ga ti track testrun initiation * Certs progress * Add list and upload cert endpoint * Add delete certificate endpoint * Update cert response codes * Upgrade dependency --------- Co-authored-by: Sofia Kurilova <sonnycactus@gmail.com> Co-authored-by: Olga Mardvilko <omardvilko@mail.ru> Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix modify devices test (#449) * Fix modify devices * Remove excess logging * "Waiting for Device" :the snack bar appears on all pages (#474) * Make waiting for device snackbar global * Do not dismiss snack bar onDestroy * Rename testrun component (#480) * Rename settings component (#476) * Rename testrun component (#477) * Rename testrun component * Improve API test coverage (#291) * Improve API test coverage * Add timeouts * Improve coverage * Increase API timeout Signed-off-by: J Boddey <boddey@google.com> * Pylint fixes * Disable broken tests * Fix skip * Disable broken tests * Fix test --------- Signed-off-by: J Boddey <boddey@google.com> * Remove certificate if BE error happens (#484) * Add cert status (#478) * Add ethtool to make and docs (#483) * Add exception handling to certificate upload (#479) * Add cert status * Add exception handling to cert upload * Add get profiles format endpoint (#475) * Add get profiles format endpoint * Update risk assessment format * Re-add modules endpoint * Update session.py Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> * Reduce locations of Testrun version (#453) * Reduce locations of Testrun version * Update from comments * Resolve make control file --------- Signed-off-by: J Boddey <boddey@google.com> * Allow stop testrun from any other page (#487) * Feature/tls client protocols (#485) * Add methods to allow known protocols that dont fit into the strict tls client detection methods Add QUIC protocol to approved protocols Start moving unit tests to a docker container for consistency * Fix client connections protocol method script Downgrade cryptography and pyOpenSSL libraries Move unit testing into docker for TLS module * Misc cleanup and pylint issues --------- Co-authored-by: J Boddey <boddey@google.com> * The focus goes to the main page element after adding the ceriticate (#489) * Focus first interactive element in cert container when cert snack bar closed * The snack bar with test attempt status appears if the user is not on Testing page (#488) * Remove testrun status snack bar * Update consent form (#490) * Adds opt out checkbox * Adds announce when settings or certificate panel is opened (#493) * Set status code on failed cert upload (#491) * Add risk profiles (#486) * Add get profiles format endpoint * Update risk assessment format * Re-add modules endpoint * Work on profiles * Add load profiles format * Pylint * Add check status --------- Signed-off-by: J Boddey <boddey@google.com> * Add test count to PDF report (#482) * Add test count to PDF report * Fix pylint issue * Exclude error --------- Signed-off-by: J Boddey <boddey@google.com> * Change network mode on network modules to fix gateway routing (#495) * 342365574: (feat) display info about Risk Assessment during testing (#492) * 342365574: (feat) display info about Risk Assessment during testing * 342365574: (fix) change to show Risk Assessment callout only on InProgress --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Updates status with data from start response (#494) * 341966862: (feat) display info about risk assessment in welcome modal (#496) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Adds download zip window (#497) * 344874424: (fix) disable device tile in Canceling status (#500) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix/UI/345164706 (#501) * 345164706: (fix) disable start testrun btn in canceling state * 345164706: (fix) remove commented code --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * GA option issues on the Welcome modal (#502) * Fix ga initial value * Fix label * 345202815: (fix) callout with the RA message is shown on the RA page (#503) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 345203686: (fix) update callout block view on the welcome modal (#504) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Allow UI to specify modules (#505) * Update risk assessment format (#499) * Fix issue with checking for error result (#498) * Fix issue with checking for error result * Resolve issue * Update html in test report Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> * Announce disabled state of settings panel (#506) * The Downlaod ZIP action can not be performed using the keaboard (#508) * Changes aria-label for Download Anyway button * Fix keyboard navigation for Download zip component * Add create and delete profile endpoints (#507) * Work towards creating profiles * Add delete profile endpoint * Exclude link local for arp (#418) * Add feature not detected test result (#396) * Add feature not present test result * Remove changes to tls module * Further pylint fixes * Add Feature Not Present to relevant tests * Reduce pylint limit * Change to feature not detected * Modify bacnet result * Update DNS test --------- Signed-off-by: J Boddey <boddey@google.com> * Add extension for cert upload (#510) * 340859666: (feat): display risk profile form with name field (#514) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fixes some pylint issues (#511) * Fix some pylint issues * Fix more pylint issues * bug/test_baseline (#513) * Update mac address for baseline config dynamically * fix line endings * Update device config via Testrun api * Add API to no ui mode * Fix API endpoint call * Remove mac address update Change mac address in container * Add container start command back * Skip DNS tests * Cleanup --------- Co-authored-by: J Boddey <boddey@google.com> * Attach profile to ZIP report (#518) * Work towards creating profiles * Add delete profile endpoint * Work on attaching profile to zip * Pylint * Fix zipping * Pylint fixes --------- Signed-off-by: J Boddey <boddey@google.com> * Generate Risk profile from json (#515) Generate form from json * The Downlaod ZIP action can not be performed using the keyboard (#517) * Stop propagating event to eliminate the error * Fix tests * Adds tooltip * Fix export endpoint to use POST request * Fix zip download --------- Co-authored-by: Jacob Boddey <boddey@google.com> * Add required if applicable (#519) * WIP: Add required if applicable * Fix bug with TLS client test * Update TLS results * Fix styles for helperbird (#524) * Form from json validation (#523) * Adds field validation * Fix vulnerabilities in dependencies (#526) * Enable draft button when profile name is present; enable save button when form is valid; remove discard button (#530) * Correct result on tls client test (#528) * Add informational and fnd to report (#527) * Mark fields required when trimmed value is empty (#529) * Update requests dependency (#525) * Remove debug artifact (#531) * Fix scroll area on reports page (#532) * Announce risk form open; focus first element in container (#536) * Fix validation; change element for text-long (#538) * 345258435: (feat) display expired certificate (#539) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348187954: (fix) update callouts position on the small window size (#540) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348356236: (fix) add tooltips for icons without any accompanying text (#541) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348353479: (fix) add tooltip for the download zip button using the keyboard (#542) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348361925: (fix) add helper text for mandatory profile name field (#543) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Save new risk profile (#533) * Save new risk profile * Clear and close form after profile saved * Adds status valid for save profile * Fix DNS report when DNS packet is missing the qname property (#546) * Adds edit risk profile (#544) * Refactor delete form: rename it to simple dialog as it is not used for delete anymore * Adds edit risk profile * Fix autozise (#547) * Feature/risk profile (#522) * Update risk profile * Add unit tests for risk profile Fix service nmap unit test * Add unit tests for risk profile Fix service nmap unit test * Update api and session for new risk profile methods * pylint * pylint * Fix some pylint * Fix more pylint * Fix some bugs * Update risk profile logic (#535) * Update risk profile logic * Update test profiles Remove duplicate test file Cleanup temp test files * Remove categories from profile * Update expiration check to account for leap years * Update risk assessment questions * Update dependency * Add missing question * Update format and add error handling --------- Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Jacob Boddey <boddey@google.com> * Adds save draft (#549) Co-authored-by: Sofia Kurilova <kurilova@google.com> * 346351108: (feat) display risk assessment result (#553) * Close form after selected risk profile was deleted (#554) Co-authored-by: Sofia Kurilova <kurilova@google.com> * Fix error when updating profile (#555) * Render informational result correctly (#551) * Re-add method for exporting profle (#550) * Fix bug when failed to fetch latest version (#548) * 349769454: (fix) update size on the empty reports page to prevent overlapping (#557) * Fix bad multiple ip report when no ip requested (#556) * Fix Duplicate Certificate Names (#545) * Check for existing common name before uploading new cert Misc pylint updates * Re-add missing session content --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Jacob Boddey <boddey@google.com> * Changes the icon; adds create date (#552) * 349783464: (fix) add styles for selected elements and fix form size on RA (#558) * 349793005: (fix) change focus to profile form to scroll up on opening profile (#559) * Return focus on create button when new profile created; return focus on profile is profile was edited (#560) * Small refactoring; update validators on selected profile update and on profile list update (#564) * Enables save and draft button for edit mode (#565) * Validate multi select form group on last checkbox tab press (#566) * Remove copy button (#567) * Bug/bacnet device (#568) * Change BACnet device detection validation Add unit tests * Fix runtime * cleanup * Update unit tests to match runtime types * Show only valid profile in modal; fix condition to not download profile if redirect button clicked (#569) * Fix bug when saving draft profile (#563) * Fix bug when saving draft * Remove risk when status changes to draft * Re-add exception handling * Fix formatting : * Update questions before calculating risk * Update unit testing * Update unit testing * Invoke risk profile update method in favor of manually updating properties in session * Update risk after update and fix type error * Update risk correctly --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Focus title or first element in container when navigation is triggered by enter (#570) * Adds aria label and tooltip to risk profile icons (#572) * Add profile PDF (#562) * Work on pdf * Work on profile PDF * Fix risk answer formatting * Downgrade weasyprint * Remove duplicate line * Update risk assessment after review * Fix profile format undefined * Update zip file to use /tmp directory (#571) Update device report folder Update file paths to use old and new patterns Co-authored-by: J Boddey <boddey@google.com> * Remove cert from session after delete request (#575) * Fix error when only draft profiles are exist (#576) * Focus fix; fix profile status icon (#579) * Fix focus after page is opened * Fix profile status * Support longer string answers in profile PDF (#578) * Add extra spacing for long string answers * Format JSON * Cleanup old test devices from runtime (#583) * 351758698: (fix) update app version styles to meet design on expanded nav (#586) * Update risk profile description (#582) * Bump version for release (#584) * Remove skipped result (#580) * Prevent creating device with duplicate manufacturer and model (#581) * Prevent duplicate mf and model * Change error message to be more precise * Fix logic * Remove profile from runtime once included in ZIP (#577) * Remove profile after ZIP creation * Create /tmp dir for results Copy test results and profile to results dir Zip results dir and cleanup --------- Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Do not show settings callout if there are no interfaces and saved settings (#587) * Fix modbus results (#588) * Don't show error if config is empty; don't show settings callout when settings are not empty after saving (#589) * Catch error to proceed with device creation/editing (#592) * Remove output logging from OS level commands (#594) * Fix cancel after monitor bug and add testrun.log (#595) * Fix step 1 callout error (#593) * Fix GAR bug with cert upload (#590) * Change exception logic on cert upload * Fix error logic * Update documentation (#591) * Work on docs changes * Update docs * Update roadmap * Update docs * Update docs * Remove dev README.md * Remove skipped from docs * Add exception handling to timestamp parsing (#598) * Change exception logic on cert upload * Fix error logic * Add exception handling to timestamp parse * Stick button to the bottom of risk page (#574) * Adds copy of risk profile (#573) * Lint fix * Set testrun IDLE status if report of finished test run is removed (#585) * Adds Discard risk profile (#596) * Load test modules dynamically (#597) * Update download zip modal: add link to Risk Profiles, remove redirect button, rename download button, add No Profile option (#599) * 351338001: (feat) update rule for the third step message (#600) * 347009372: (feat) add selector for profile options for GA4 (#604) * Adds tooltip for copy and delete; show "same name" error when more than one copy is created (#606) * 346999760: (feat) [GA4] Track CA Certificates (#607) * 353476778: (fix) GAR 2.11 change for prevent risk profile tiles overlap (#609) * If profile is editing, return focus on profile (#612) * Change title for save profile dialog according to type of profile (#610) * Get reports when app is opened and when report page is opened; change status if testrun does not exist in reports; do not change status if testrun is just finished but not in reports yet (#613) * Show tooltip only on keyboard or hover (#614) * Update condition as report field is unique (#615) * Expired profile profile (#619) * Use device mac_addr if report mac_addr is missing (#622) * get system network interfaces util func * test get_sys_interfaces * pylint * get all network interfaces on session start * dicts diff * detect network adapter change * pylint * logging * Add ws server * Add MQTT protocol * Upgrade ws server * paho-mqtt dependency * getting docker container IP by container name * MQTT client class * Fix pylint * rename mqtt client logger * APScheduler * initializing the client inside the testrun object * pylint * check networks adapters in background * remove extra lines at the end of a file * rename mgtt logger * move network_adapters_checker to network_orchestrator * Adds mqtt client, adds pop up when new adapter is available (#603) * Fix some pylint issues (#620) * Disabled test results (#212) * Better handling of disbled tests Add dns and tls tests back in disabled state * pyling fixes * Fix subscriptable error * Re-add old tests as informational --------- Co-authored-by: J Boddey <boddey@google.com> * remove get ip of docker container because it is not necessary * Fix individual test disabling when run from the UI (#629) * Fix individual test disabling when run from the UI * Remove disabled result for now --------- Co-authored-by: Jacob Boddey <boddey@google.com> * Update package.yml (#624) Signed-off-by: J Boddey <boddey@google.com> * Added risk profile api testing (#628) * added tests for profile endpoints * Modified test_start_testrun_started_successfully payload to match the expected json format, updated the profile endpoints tests * fixed the pylint errors from test_api.py * fixed few more pylint errors * Expired profile profile (#619) * Use device mac_addr if report mac_addr is missing (#622) * Fix some pylint issues (#620) * Disabled test results (#212) * Better handling of disbled tests Add dns and tls tests back in disabled state * pyling fixes * Fix subscriptable error * Re-add old tests as informational --------- Co-authored-by: J Boddey <boddey@google.com> * added try-except block to delete_all_profiles() * Revert session file back * Add new line * Update api testing * updated the tests for profile endpoint, added a new fixture (add_profile) to create a profile * Updated profile endpoints: created a new fixture add_profile for creating profiles * Fix pylint * Fix pylint * Fix pylint issues --------- Co-authored-by: Sofia Kurilova <kurilova@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Adds close button for expired profile (#635) * Copy changes from hotfix 1.3.1 to dev (#631) * Copy changes from hotfix 1.3.1 to dev * Add pydyf version Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> * Inform FE about a new network adapter discovered( rename mqtt topic according to naming convention) (#634) * rename mqtt topic * Rename topic in ui --------- Co-authored-by: kurilova <sonnycactus@gmail.com> Co-authored-by: J Boddey <boddey@google.com> * Fix network only mode issues (#617) Co-authored-by: J Boddey <boddey@google.com> * Update all unit tests to work within the runtime environment (#611) * Update all unit tests to work within the runtime environment * Fix some formatting * fix line endings * update gitignore * Fix binary files in dockerfile * Change unit tests to run from testrun root directory * Run unit tests in actions * Fix pylint issues * Change command in actions * Update testing.yml Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Jacob Boddey <boddey@google.com> * Feature/dns report update (#637) * Update DNS module report Downgrade python packages for tls module * Fix header * pylint fixes * refactor func to handle case when network interface not exists * set test result "Error" * check device connected * thread for monitoring device connection * Minor changes * check the device connection only before each test * Adds tooltip (#638) Adds tooltip * send testrun status using mqtt * remove duplicatied line * refactor setting remaining tests to error * pylint * Fix focus after profile delete - track by name (#640) Fix focus after profile delete - track by name * Update the requests dependency (#643) * Update requests dependency * Update requests dependency * Update dependency in TLS test * Update docker dependency --------- Signed-off-by: J Boddey <boddey@google.com> * Revert "Expired profile (#619)" (#645) Prevent opening of Expired risk profile * Improve documentation (#639) * Improve docs * Remove paragraph * Text changes * Fix text for the BE error * Change tooltip (#650) * Change tooltip * Allows draft profiles to become expired (#636) * Allow draft profiles to expire * Move status method into risk profile class * Use existing method * Check for expiry in validate method * Remove unused variable * Build UI during package instead of install (#621) * Build UI during package * Fix local build * Install npm * Remove duplicate build message * Fix ESLint * Fix script * Modify scripts * Improve scripts * Fix copy command * Try installing package * Depend on package job * Add sudo * Add sudo * Troubleshoot * Fix workflow * Checkout source for prepare command * Built ui within a container * Mount src files for build instead of static copy in build image * Attempt to fix actions * Remove manual build container cleanup methods * undo failed attempts to fix actions * Fix path * Remove -it flag --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: kurilova <sonnycactus@gmail.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Feature/risk in selected (#654) * Adds risk to selected value * Adds risk to selected value --------- Co-authored-by: J Boddey <boddey@google.com> * Show risk for each question in the Risk profile (#647) * Show risk for each question in the Risk profile * set top position to 0 --------- Co-authored-by: J Boddey <boddey@google.com> * Show internet connection (#653) * MQTT show internet connection * remove unused method * Change tooltip for internet icon (#656) * Change tooltip for internet icon * Remove unused import' --------- Co-authored-by: Jacob Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: Sofia Kurilova <kurilova@google.com> * bug/modbus_constructor (#657) * Pin all required packages Update modbus constructor to prevent error Add full trace logging for general errors in tests * Fix pylint issue --------- Co-authored-by: Jacob Boddey <boddey@google.com> * Use mqtt service instead of calling GET /status every 5 seconds. (#644) * Use mqtt service instead of calling GET /status every 5 seconds. * Adds tooltip (#638) Adds tooltip * Fix focus after profile delete - track by name (#640) Fix focus after profile delete - track by name * Update the requests dependency (#643) * Update requests dependency * Update requests dependency * Update dependency in TLS test * Update docker dependency --------- Signed-off-by: J Boddey <boddey@google.com> * remove unused output * encode mqtt message to json * Revert "Expired profile (#619)" (#645) Prevent opening of Expired risk profile * Improve documentation (#639) * Improve docs * Remove paragraph * Text changes * Fix text for the BE error * Change tooltip (#650) * Change tooltip * Allows draft profiles to become expired (#636) * Allow draft profiles to expire * Move status method into risk profile class * Use existing method * Check for expiry in validate method * Remove unused variable * Build UI during package instead of install (#621) * Build UI during package * Fix local build * Install npm * Remove duplicate build message * Fix ESLint * Fix script * Modify scripts * Improve scripts * Fix copy command * Try installing package * Depend on package job * Add sudo * Add sudo * Troubleshoot * Fix workflow * Checkout source for prepare command * Built ui within a container * Mount src files for build instead of static copy in build image * Attempt to fix actions * Remove manual build container cleanup methods * undo failed attempts to fix actions * Fix path * Remove -it flag --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: kurilova <sonnycactus@gmail.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Feature/risk in selected (#654) * Adds risk to selected value * Adds risk to selected value --------- Co-authored-by: J Boddey <boddey@google.com> * Show risk for each question in the Risk profile (#647) * Show risk for each question in the Risk profile * set top position to 0 --------- Co-authored-by: J Boddey <boddey@google.com> * Use mqtt service instead of calling GET /status every 5 seconds. * Use mqtt service instead of calling GET /status every 5 seconds. * Use mqtt service instead of calling GET /status every 5 seconds. * pylint --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: Aliaksandr Nikitsin <aliaksandrn@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Adds test statuses (#661) * Ignore folders when loading certs (#660) * Remove scorecard schedule and bump version (#659) * Allow ICMP response to DHCP messages in DHCP snooping test (#608) * Allow ICMP response to DHCP messages * Bug/unit test runtime (#655) * Change base test module startup to allow setup script to run independent of module startup process Update connection_module to allow for unit testing Update unit test run script to use new process * enable all unit tests update google cert * Remove binary fix lines from docker files pylint updates * pylint updates --------- Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * The risk profile saved with old format is shown improperly while loading based on a new format (#664) * Fill only fields that are present in profile * GAR : The alt text for the expired risk profile should be communicated on Enter key (#662) * Change Expired profile title on Enter; announce Expired profile title on Enter * Update wording of tls cipher results (#671) * Show error message if provided; show default message if no (#680) * Test install on supported operating systems (#675) * Test install on multiple versions * Update step names * Remove recommendations on error (#674) * Tests for API (#649) * changed the tests order in test_api.py * added tests for '/system/config' POST endpoint * added the tests for 'system/shutdown' endpoint * added the test for GET '/reports' endpoint, updated 'test_update_system_config_invalid_config' to return error 400 * Check for missing fields Signed-off-by: J Boddey <boddey@google.com> * added tests for delete profile (404, 400), added tests for create and update profile (400), added test 'run_test_and_get_report' skipped due to blocking during testing phase * added a new json file in '/testing/api/' used in 400 error tests * added error handling if 'name' and 'questions' not in profile json * fixed pylint * Added tests when update is available and 500 status code for '/system/version', test for system/modules * added responses library in requirements.txt * fixed the requested changes in api.py * Renamed the load_profile method to load_json and changed the logic to allow to load any json based on file name and relative path, corrected the new lines issues * updated restore_config fixture to run after the test * added test for create/update profile (500 error) * fixed pylint * fixed spacing, removed get_report_one_report * added tests: 500 error for delete '/profiles', 500 error for 'profles/format', 400, 404, 409 for '/system/start' * modified the tests for 500 response * added new profile with missing 'answer' * removed the tests with mock response * Update NTP report (#666) * Update NTP report * cleanup imports * pylint updates * modified update profile for bad request * changed validate_profile_json: handling empty spaces in name and question, handle if 'risk' field missing if status is 'Valid' * updated the requested changes * Add further profile validation * Fix pylint issues * Fix profile tests * Move validation to session * Fix pylint issue Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Fix a delay in the internet connectivity check (#669) * Add a timeout to the command * move single-intf check to scheduler * add timeout arg to run_command * move jobs to constructor * fping for internet connection checking * Update internet connection when device is In Progress, Monitoring, Waiting for Device status (#677) * Update internet connection when device is In Progress, Monitoring, Waiting for Device status * check if interface physically connected * Revert "fping for internet connection checking" --------- Co-authored-by: Aliaksandr Nikitsin <aliaksandrn@google.com> Co-authored-by: Sofia Kurilova <kurilova@google.com> * Check if device folder already exists (#678) * Check if device folder already exists * Fix error message * Fix pylint issue * Remove debug mqtt logs (#692) --------- Signed-off-by: J Boddey <boddey@google.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sofia Kurilova <sonnycactus@gmail.com> Co-authored-by: Olga Mardvilko <omardvilko@mail.ru> Co-authored-by: Volha Mardvilka <mardvilka@google.com> Co-authored-by: jhughesbiot <50999916+jhughesbiot@users.noreply.github.com> Co-authored-by: Sofia Kurilova <kurilova@google.com> Co-authored-by: Aliaksandr Nikitsin <aliaksandrn@google.com> Co-authored-by: Marius <86727846+MariusBaldovin@users.noreply.github.com>

* Release v1.4 into main (#687) * Update requirements.txt (#429) Signed-off-by: J Boddey <boddey@google.com> * Remove in progress status after discovery (#422) * Add steps to resolve to PDF report (#411) * Add progress for steps to resolve * Formatting * Remove print statements * Allow for no steps to resolve * Add multipage * Fix multipage * Fix config * update report unit tests --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Bump ejs in /modules/ui in the npm_and_yarn group across 1 directory (#428) Bumps the npm_and_yarn group with 1 update in the /modules/ui directory: [ejs](https://github.com/mde/ejs). Updates `ejs` from 3.1.9 to 3.1.10 - [Release notes](https://github.com/mde/ejs/releases) - [Commits](mde/ejs@v3.1.9...v3.1.10) --- updated-dependencies: - dependency-name: ejs dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * For delete report take mac_addr from upper level (#436) * Update dependencies (#435) * Add new mac addr field for report deleting (#432) * Bump version to v1.2.2 (#433) * fix: replace Feature Not Present to Feature Not Detected (#445) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix PDF alignment (#441) * Do not remove form control on destroy as it causes error; call system/config on settings open (#442) * Fix some pylint issues (#437) * Update documentation (#448) * Update documentation * Update docs * V1.3 (#393) * Adds version analytics event (#306) * Techdebt: adds state for testrun page (#392) * Fix tests (#397) * Fix tests * Update node version * 331379891: (feat) disable connection settings when testrun is in progress (#371) * 331379891: (feat) disable connection settings when testrun is in progress * 331379891: (fix) include more testrun results as progress * 331379891: (fix) fix spelling * 333349715: (fix) GAR 1.3 The disabled system settings panel contains a focusable element (#388) Co-authored-by: Volha Mardvilka <mardvilka@google.com> --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Disable device item if device in progress (#370) * Adds ga to track testrun initiation (#415) Adds ga ti track testrun initiation * Fix function return value (#421) * Fix redirect + adds google analytics (#420) * Adds certificates drawer component with list of certificates (#414) Adds certificates drawer component with list of certificates. Upload and delete is out of scope * List modules (#425) * Adds delete certificate (#424) Adds delete certificate * Adds class for links to track report type on download; adds event when download report is clicked on progress page (#434) * Adds upload certificate (#431) * Adds upload certificate * move delete functionality to certificates store * Catch error on report/certificate deletion (#438) * Fix button size; fix text-overflow (#440) * Adds focus on next or close button when certificate is deleted (#439) * Adds focus on next or close button when certificate is deleted * Enable test modules for initiate test run dialog (#400) Enable test modules for initiate test run dialog * Add steps to resolve to PDF report (#411) * Add progress for steps to resolve * Formatting * Remove print statements * Allow for no steps to resolve * Add multipage * Fix multipage * Fix config * update report unit tests --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Add new mac addr field for report deleting (#432) * Bump version to v1.2.2 (#433) * 337012359: (feat) add snackBar with wait or stop testrun (#444) * 337012359: (feat) add snackBar with wait or stop testrun * 337012359: (fix) update to fix failed tests * 337012359: (fix) add fix for deletCertificate test --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Feature/port stats (#430) * Add port speed and duplex tests Add unit tests for port stats testing Add place holder for ip port link test * Add ethtool to system dependencies Restructure conn stats methods and tests Resolve port stat information for link test * Fix runtime issue * Implement link test Fix unit tests Misc port stats updates * fix pylint issues * update readme * pylint fixes --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> * 339315842: (feat) add risk assessment tab (#450) * 339315842: (feat) add risk assessment tab * 339315842: (feat) add risk assessment tab --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Add steps to resolve to PDF report (#411) * Add progress for steps to resolve * Formatting * Remove print statements * Allow for no steps to resolve * Add multipage * Fix multipage * Fix config * update report unit tests --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Add new mac addr field for report deleting (#432) * Remove rebase error --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Olga Mardvilko <omardvilko@mail.ru> Co-authored-by: Volha Mardvilka <mardvilka@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> Co-authored-by: jhughesbiot <50999916+jhughesbiot@users.noreply.github.com> * Rename "history" to "reports" (#456) * Rename "device-repository" to "devices" (#455) * Change certificates endpoints (#458) * 339311887: (feat) display saved risk profile (#460) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix ui defects (#459) * 340835710: (fix) [Risk assessment] change page view when callout is visible (#461) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix style to allow screen reader to read label (#462) * Updaate device test module configuration from api start endpoint (#463) * Add get profiles format endpoint (#465) * 339311250: (feat) delete risk profile (#466) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 341254121: (fix) [a11y] add item name for aria-labels delete and copy buttons (#467) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Make testing statuses available outside of Testing Tab (#468) * Adds timeout token * Status is available on all pages; Removed unused field isTestrunStarted - actual status is available * Validate CA certificate on FE (#464) * Adds validation on certificate upload * Show notification when error happens * 341901606: (fix) [GAR 1.6]: update for arrows usage on risk profile (#471) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Adds space in certificate name regexp (#469) * 342096458: (fix) [GAR 1.3] add focus trap to certificates panel to prevent move focus to risk profile (#472) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * The focus is not moved to the snackbar with certificate validation rule (#470) * Adds class to fix focus flow * Changes the delay according to string length * Adds file name in error message (#473) * Add certificate endpoints (#451) * Adds version analytics event (#306) * Techdebt: adds state for testrun page (#392) * Fix tests (#397) * Fix tests * Update node version * 331379891: (feat) disable connection settings when testrun is in progress (#371) * 331379891: (feat) disable connection settings when testrun is in progress * 331379891: (fix) include more testrun results as progress * 331379891: (fix) fix spelling * 333349715: (fix) GAR 1.3 The disabled system settings panel contains a focusable element (#388) Co-authored-by: Volha Mardvilka <mardvilka@google.com> --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Disable device item if device in progress (#370) * Adds ga to track testrun initiation (#415) Adds ga ti track testrun initiation * Certs progress * Add list and upload cert endpoint * Add delete certificate endpoint * Update cert response codes * Upgrade dependency --------- Co-authored-by: Sofia Kurilova <sonnycactus@gmail.com> Co-authored-by: Olga Mardvilko <omardvilko@mail.ru> Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix modify devices test (#449) * Fix modify devices * Remove excess logging * "Waiting for Device" :the snack bar appears on all pages (#474) * Make waiting for device snackbar global * Do not dismiss snack bar onDestroy * Rename testrun component (#480) * Rename settings component (#476) * Rename testrun component (#477) * Rename testrun component * Improve API test coverage (#291) * Improve API test coverage * Add timeouts * Improve coverage * Increase API timeout Signed-off-by: J Boddey <boddey@google.com> * Pylint fixes * Disable broken tests * Fix skip * Disable broken tests * Fix test --------- Signed-off-by: J Boddey <boddey@google.com> * Remove certificate if BE error happens (#484) * Add cert status (#478) * Add ethtool to make and docs (#483) * Add exception handling to certificate upload (#479) * Add cert status * Add exception handling to cert upload * Add get profiles format endpoint (#475) * Add get profiles format endpoint * Update risk assessment format * Re-add modules endpoint * Update session.py Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> * Reduce locations of Testrun version (#453) * Reduce locations of Testrun version * Update from comments * Resolve make control file --------- Signed-off-by: J Boddey <boddey@google.com> * Allow stop testrun from any other page (#487) * Feature/tls client protocols (#485) * Add methods to allow known protocols that dont fit into the strict tls client detection methods Add QUIC protocol to approved protocols Start moving unit tests to a docker container for consistency * Fix client connections protocol method script Downgrade cryptography and pyOpenSSL libraries Move unit testing into docker for TLS module * Misc cleanup and pylint issues --------- Co-authored-by: J Boddey <boddey@google.com> * The focus goes to the main page element after adding the ceriticate (#489) * Focus first interactive element in cert container when cert snack bar closed * The snack bar with test attempt status appears if the user is not on Testing page (#488) * Remove testrun status snack bar * Update consent form (#490) * Adds opt out checkbox * Adds announce when settings or certificate panel is opened (#493) * Set status code on failed cert upload (#491) * Add risk profiles (#486) * Add get profiles format endpoint * Update risk assessment format * Re-add modules endpoint * Work on profiles * Add load profiles format * Pylint * Add check status --------- Signed-off-by: J Boddey <boddey@google.com> * Add test count to PDF report (#482) * Add test count to PDF report * Fix pylint issue * Exclude error --------- Signed-off-by: J Boddey <boddey@google.com> * Change network mode on network modules to fix gateway routing (#495) * 342365574: (feat) display info about Risk Assessment during testing (#492) * 342365574: (feat) display info about Risk Assessment during testing * 342365574: (fix) change to show Risk Assessment callout only on InProgress --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Updates status with data from start response (#494) * 341966862: (feat) display info about risk assessment in welcome modal (#496) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Adds download zip window (#497) * 344874424: (fix) disable device tile in Canceling status (#500) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fix/UI/345164706 (#501) * 345164706: (fix) disable start testrun btn in canceling state * 345164706: (fix) remove commented code --------- Co-authored-by: Volha Mardvilka <mardvilka@google.com> * GA option issues on the Welcome modal (#502) * Fix ga initial value * Fix label * 345202815: (fix) callout with the RA message is shown on the RA page (#503) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 345203686: (fix) update callout block view on the welcome modal (#504) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Allow UI to specify modules (#505) * Update risk assessment format (#499) * Fix issue with checking for error result (#498) * Fix issue with checking for error result * Resolve issue * Update html in test report Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> * Announce disabled state of settings panel (#506) * The Downlaod ZIP action can not be performed using the keaboard (#508) * Changes aria-label for Download Anyway button * Fix keyboard navigation for Download zip component * Add create and delete profile endpoints (#507) * Work towards creating profiles * Add delete profile endpoint * Exclude link local for arp (#418) * Add feature not detected test result (#396) * Add feature not present test result * Remove changes to tls module * Further pylint fixes * Add Feature Not Present to relevant tests * Reduce pylint limit * Change to feature not detected * Modify bacnet result * Update DNS test --------- Signed-off-by: J Boddey <boddey@google.com> * Add extension for cert upload (#510) * 340859666: (feat): display risk profile form with name field (#514) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Fixes some pylint issues (#511) * Fix some pylint issues * Fix more pylint issues * bug/test_baseline (#513) * Update mac address for baseline config dynamically * fix line endings * Update device config via Testrun api * Add API to no ui mode * Fix API endpoint call * Remove mac address update Change mac address in container * Add container start command back * Skip DNS tests * Cleanup --------- Co-authored-by: J Boddey <boddey@google.com> * Attach profile to ZIP report (#518) * Work towards creating profiles * Add delete profile endpoint * Work on attaching profile to zip * Pylint * Fix zipping * Pylint fixes --------- Signed-off-by: J Boddey <boddey@google.com> * Generate Risk profile from json (#515) Generate form from json * The Downlaod ZIP action can not be performed using the keyboard (#517) * Stop propagating event to eliminate the error * Fix tests * Adds tooltip * Fix export endpoint to use POST request * Fix zip download --------- Co-authored-by: Jacob Boddey <boddey@google.com> * Add required if applicable (#519) * WIP: Add required if applicable * Fix bug with TLS client test * Update TLS results * Fix styles for helperbird (#524) * Form from json validation (#523) * Adds field validation * Fix vulnerabilities in dependencies (#526) * Enable draft button when profile name is present; enable save button when form is valid; remove discard button (#530) * Correct result on tls client test (#528) * Add informational and fnd to report (#527) * Mark fields required when trimmed value is empty (#529) * Update requests dependency (#525) * Remove debug artifact (#531) * Fix scroll area on reports page (#532) * Announce risk form open; focus first element in container (#536) * Fix validation; change element for text-long (#538) * 345258435: (feat) display expired certificate (#539) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348187954: (fix) update callouts position on the small window size (#540) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348356236: (fix) add tooltips for icons without any accompanying text (#541) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348353479: (fix) add tooltip for the download zip button using the keyboard (#542) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * 348361925: (fix) add helper text for mandatory profile name field (#543) Co-authored-by: Volha Mardvilka <mardvilka@google.com> * Save new risk profile (#533) * Save new risk profile * Clear and close form after profile saved * Adds status valid for save profile * Fix DNS report when DNS packet is missing the qname property (#546) * Adds edit risk profile (#544) * Refactor delete form: rename it to simple dialog as it is not used for delete anymore * Adds edit risk profile * Fix autozise (#547) * Feature/risk profile (#522) * Update risk profile * Add unit tests for risk profile Fix service nmap unit test * Add unit tests for risk profile Fix service nmap unit test * Update api and session for new risk profile methods * pylint * pylint * Fix some pylint * Fix more pylint * Fix some bugs * Update risk profile logic (#535) * Update risk profile logic * Update test profiles Remove duplicate test file Cleanup temp test files * Remove categories from profile * Update expiration check to account for leap years * Update risk assessment questions * Update dependency * Add missing question * Update format and add error handling --------- Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Jacob Boddey <boddey@google.com> * Adds save draft (#549) Co-authored-by: Sofia Kurilova <kurilova@google.com> * 346351108: (feat) display risk assessment result (#553) * Close form after selected risk profile was deleted (#554) Co-authored-by: Sofia Kurilova <kurilova@google.com> * Fix error when updating profile (#555) * Render informational result correctly (#551) * Re-add method for exporting profle (#550) * Fix bug when failed to fetch latest version (#548) * 349769454: (fix) update size on the empty reports page to prevent overlapping (#557) * Fix bad multiple ip report when no ip requested (#556) * Fix Duplicate Certificate Names (#545) * Check for existing common name before uploading new cert Misc pylint updates * Re-add missing session content --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Jacob Boddey <boddey@google.com> * Changes the icon; adds create date (#552) * 349783464: (fix) add styles for selected elements and fix form size on RA (#558) * 349793005: (fix) change focus to profile form to scroll up on opening profile (#559) * Return focus on create button when new profile created; return focus on profile is profile was edited (#560) * Small refactoring; update validators on selected profile update and on profile list update (#564) * Enables save and draft button for edit mode (#565) * Validate multi select form group on last checkbox tab press (#566) * Remove copy button (#567) * Bug/bacnet device (#568) * Change BACnet device detection validation Add unit tests * Fix runtime * cleanup * Update unit tests to match runtime types * Show only valid profile in modal; fix condition to not download profile if redirect button clicked (#569) * Fix bug when saving draft profile (#563) * Fix bug when saving draft * Remove risk when status changes to draft * Re-add exception handling * Fix formatting : * Update questions before calculating risk * Update unit testing * Update unit testing * Invoke risk profile update method in favor of manually updating properties in session * Update risk after update and fix type error * Update risk correctly --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Focus title or first element in container when navigation is triggered by enter (#570) * Adds aria label and tooltip to risk profile icons (#572) * Add profile PDF (#562) * Work on pdf * Work on profile PDF * Fix risk answer formatting * Downgrade weasyprint * Remove duplicate line * Update risk assessment after review * Fix profile format undefined * Update zip file to use /tmp directory (#571) Update device report folder Update file paths to use old and new patterns Co-authored-by: J Boddey <boddey@google.com> * Remove cert from session after delete request (#575) * Fix error when only draft profiles are exist (#576) * Focus fix; fix profile status icon (#579) * Fix focus after page is opened * Fix profile status * Support longer string answers in profile PDF (#578) * Add extra spacing for long string answers * Format JSON * Cleanup old test devices from runtime (#583) * 351758698: (fix) update app version styles to meet design on expanded nav (#586) * Update risk profile description (#582) * Bump version for release (#584) * Remove skipped result (#580) * Prevent creating device with duplicate manufacturer and model (#581) * Prevent duplicate mf and model * Change error message to be more precise * Fix logic * Remove profile from runtime once included in ZIP (#577) * Remove profile after ZIP creation * Create /tmp dir for results Copy test results and profile to results dir Zip results dir and cleanup --------- Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Do not show settings callout if there are no interfaces and saved settings (#587) * Fix modbus results (#588) * Don't show error if config is empty; don't show settings callout when settings are not empty after saving (#589) * Catch error to proceed with device creation/editing (#592) * Remove output logging from OS level commands (#594) * Fix cancel after monitor bug and add testrun.log (#595) * Fix step 1 callout error (#593) * Fix GAR bug with cert upload (#590) * Change exception logic on cert upload * Fix error logic * Update documentation (#591) * Work on docs changes * Update docs * Update roadmap * Update docs * Update docs * Remove dev README.md * Remove skipped from docs * Add exception handling to timestamp parsing (#598) * Change exception logic on cert upload * Fix error logic * Add exception handling to timestamp parse * Stick button to the bottom of risk page (#574) * Adds copy of risk profile (#573) * Lint fix * Set testrun IDLE status if report of finished test run is removed (#585) * Adds Discard risk profile (#596) * Load test modules dynamically (#597) * Update download zip modal: add link to Risk Profiles, remove redirect button, rename download button, add No Profile option (#599) * 351338001: (feat) update rule for the third step message (#600) * 347009372: (feat) add selector for profile options for GA4 (#604) * Adds tooltip for copy and delete; show "same name" error when more than one copy is created (#606) * 346999760: (feat) [GA4] Track CA Certificates (#607) * 353476778: (fix) GAR 2.11 change for prevent risk profile tiles overlap (#609) * If profile is editing, return focus on profile (#612) * Change title for save profile dialog according to type of profile (#610) * Get reports when app is opened and when report page is opened; change status if testrun does not exist in reports; do not change status if testrun is just finished but not in reports yet (#613) * Show tooltip only on keyboard or hover (#614) * Update condition as report field is unique (#615) * Expired profile profile (#619) * Use device mac_addr if report mac_addr is missing (#622) * get system network interfaces util func * test get_sys_interfaces * pylint * get all network interfaces on session start * dicts diff * detect network adapter change * pylint * logging * Add ws server * Add MQTT protocol * Upgrade ws server * paho-mqtt dependency * getting docker container IP by container name * MQTT client class * Fix pylint * rename mqtt client logger * APScheduler * initializing the client inside the testrun object * pylint * check networks adapters in background * remove extra lines at the end of a file * rename mgtt logger * move network_adapters_checker to network_orchestrator * Adds mqtt client, adds pop up when new adapter is available (#603) * Fix some pylint issues (#620) * Disabled test results (#212) * Better handling of disbled tests Add dns and tls tests back in disabled state * pyling fixes * Fix subscriptable error * Re-add old tests as informational --------- Co-authored-by: J Boddey <boddey@google.com> * remove get ip of docker container because it is not necessary * Fix individual test disabling when run from the UI (#629) * Fix individual test disabling when run from the UI * Remove disabled result for now --------- Co-authored-by: Jacob Boddey <boddey@google.com> * Update package.yml (#624) Signed-off-by: J Boddey <boddey@google.com> * Added risk profile api testing (#628) * added tests for profile endpoints * Modified test_start_testrun_started_successfully payload to match the expected json format, updated the profile endpoints tests * fixed the pylint errors from test_api.py * fixed few more pylint errors * Expired profile profile (#619) * Use device mac_addr if report mac_addr is missing (#622) * Fix some pylint issues (#620) * Disabled test results (#212) * Better handling of disbled tests Add dns and tls tests back in disabled state * pyling fixes * Fix subscriptable error * Re-add old tests as informational --------- Co-authored-by: J Boddey <boddey@google.com> * added try-except block to delete_all_profiles() * Revert session file back * Add new line * Update api testing * updated the tests for profile endpoint, added a new fixture (add_profile) to create a profile * Updated profile endpoints: created a new fixture add_profile for creating profiles * Fix pylint * Fix pylint * Fix pylint issues --------- Co-authored-by: Sofia Kurilova <kurilova@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Adds close button for expired profile (#635) * Copy changes from hotfix 1.3.1 to dev (#631) * Copy changes from hotfix 1.3.1 to dev * Add pydyf version Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> * Inform FE about a new network adapter discovered( rename mqtt topic according to naming convention) (#634) * rename mqtt topic * Rename topic in ui --------- Co-authored-by: kurilova <sonnycactus@gmail.com> Co-authored-by: J Boddey <boddey@google.com> * Fix network only mode issues (#617) Co-authored-by: J Boddey <boddey@google.com> * Update all unit tests to work within the runtime environment (#611) * Update all unit tests to work within the runtime environment * Fix some formatting * fix line endings * update gitignore * Fix binary files in dockerfile * Change unit tests to run from testrun root directory * Run unit tests in actions * Fix pylint issues * Change command in actions * Update testing.yml Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: Jacob Boddey <boddey@google.com> * Feature/dns report update (#637) * Update DNS module report Downgrade python packages for tls module * Fix header * pylint fixes * refactor func to handle case when network interface not exists * set test result "Error" * check device connected * thread for monitoring device connection * Minor changes * check the device connection only before each test * Adds tooltip (#638) Adds tooltip * send testrun status using mqtt * remove duplicatied line * refactor setting remaining tests to error * pylint * Fix focus after profile delete - track by name (#640) Fix focus after profile delete - track by name * Update the requests dependency (#643) * Update requests dependency * Update requests dependency * Update dependency in TLS test * Update docker dependency --------- Signed-off-by: J Boddey <boddey@google.com> * Revert "Expired profile (#619)" (#645) Prevent opening of Expired risk profile * Improve documentation (#639) * Improve docs * Remove paragraph * Text changes * Fix text for the BE error * Change tooltip (#650) * Change tooltip * Allows draft profiles to become expired (#636) * Allow draft profiles to expire * Move status method into risk profile class * Use existing method * Check for expiry in validate method * Remove unused variable * Build UI during package instead of install (#621) * Build UI during package * Fix local build * Install npm * Remove duplicate build message * Fix ESLint * Fix script * Modify scripts * Improve scripts * Fix copy command * Try installing package * Depend on package job * Add sudo * Add sudo * Troubleshoot * Fix workflow * Checkout source for prepare command * Built ui within a container * Mount src files for build instead of static copy in build image * Attempt to fix actions * Remove manual build container cleanup methods * undo failed attempts to fix actions * Fix path * Remove -it flag --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: kurilova <sonnycactus@gmail.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Feature/risk in selected (#654) * Adds risk to selected value * Adds risk to selected value --------- Co-authored-by: J Boddey <boddey@google.com> * Show risk for each question in the Risk profile (#647) * Show risk for each question in the Risk profile * set top position to 0 --------- Co-authored-by: J Boddey <boddey@google.com> * Show internet connection (#653) * MQTT show internet connection * remove unused method * Change tooltip for internet icon (#656) * Change tooltip for internet icon * Remove unused import' --------- Co-authored-by: Jacob Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: Sofia Kurilova <kurilova@google.com> * bug/modbus_constructor (#657) * Pin all required packages Update modbus constructor to prevent error Add full trace logging for general errors in tests * Fix pylint issue --------- Co-authored-by: Jacob Boddey <boddey@google.com> * Use mqtt service instead of calling GET /status every 5 seconds. (#644) * Use mqtt service instead of calling GET /status every 5 seconds. * Adds tooltip (#638) Adds tooltip * Fix focus after profile delete - track by name (#640) Fix focus after profile delete - track by name * Update the requests dependency (#643) * Update requests dependency * Update requests dependency * Update dependency in TLS test * Update docker dependency --------- Signed-off-by: J Boddey <boddey@google.com> * remove unused output * encode mqtt message to json * Revert "Expired profile (#619)" (#645) Prevent opening of Expired risk profile * Improve documentation (#639) * Improve docs * Remove paragraph * Text changes * Fix text for the BE error * Change tooltip (#650) * Change tooltip * Allows draft profiles to become expired (#636) * Allow draft profiles to expire * Move status method into risk profile class * Use existing method * Check for expiry in validate method * Remove unused variable * Build UI during package instead of install (#621) * Build UI during package * Fix local build * Install npm * Remove duplicate build message * Fix ESLint * Fix script * Modify scripts * Improve scripts * Fix copy command * Try installing package * Depend on package job * Add sudo * Add sudo * Troubleshoot * Fix workflow * Checkout source for prepare command * Built ui within a container * Mount src files for build instead of static copy in build image * Attempt to fix actions * Remove manual build container cleanup methods * undo failed attempts to fix actions * Fix path * Remove -it flag --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: kurilova <sonnycactus@gmail.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Feature/risk in selected (#654) * Adds risk to selected value * Adds risk to selected value --------- Co-authored-by: J Boddey <boddey@google.com> * Show risk for each question in the Risk profile (#647) * Show risk for each question in the Risk profile * set top position to 0 --------- Co-authored-by: J Boddey <boddey@google.com> * Use mqtt service instead of calling GET /status every 5 seconds. * Use mqtt service instead of calling GET /status every 5 seconds. * Use mqtt service instead of calling GET /status every 5 seconds. * pylint --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: Aliaksandr Nikitsin <aliaksandrn@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Adds test statuses (#661) * Ignore folders when loading certs (#660) * Remove scorecard schedule and bump version (#659) * Allow ICMP response to DHCP messages in DHCP snooping test (#608) * Allow ICMP response to DHCP messages * Bug/unit test runtime (#655) * Change base test module startup to allow setup script to run independent of module startup process Update connection_module to allow for unit testing Update unit test run script to use new process * enable all unit tests update google cert * Remove binary fix lines from docker files pylint updates * pylint updates --------- Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * The risk profile saved with old format is shown improperly while loading based on a new format (#664) * Fill only fields that are present in profile * GAR : The alt text for the expired risk profile should be communicated on Enter key (#662) * Change Expired profile title on Enter; announce Expired profile title on Enter * Update wording of tls cipher results (#671) * Show error message if provided; show default message if no (#680) * Test install on supported operating systems (#675) * Test install on multiple versions * Update step names * Remove recommendations on error (#674) * Tests for API (#649) * changed the tests order in test_api.py * added tests for '/system/config' POST endpoint * added the tests for 'system/shutdown' endpoint * added the test for GET '/reports' endpoint, updated 'test_update_system_config_invalid_config' to return error 400 * Check for missing fields Signed-off-by: J Boddey <boddey@google.com> * added tests for delete profile (404, 400), added tests for create and update profile (400), added test 'run_test_and_get_report' skipped due to blocking during testing phase * added a new json file in '/testing/api/' used in 400 error tests * added error handling if 'name' and 'questions' not in profile json * fixed pylint * Added tests when update is available and 500 status code for '/system/version', test for system/modules * added responses library in requirements.txt * fixed the requested changes in api.py * Renamed the load_profile method to load_json and changed the logic to allow to load any json based on file name and relative path, corrected the new lines issues * updated restore_config fixture to run after the test * added test for create/update profile (500 error) * fixed pylint * fixed spacing, removed get_report_one_report * added tests: 500 error for delete '/profiles', 500 error for 'profles/format', 400, 404, 409 for '/system/start' * modified the tests for 500 response * added new profile with missing 'answer' * removed the tests with mock response * Update NTP report (#666) * Update NTP report * cleanup imports * pylint updates * modified update profile for bad request * changed validate_profile_json: handling empty spaces in name and question, handle if 'risk' field missing if status is 'Valid' * updated the requested changes * Add further profile validation * Fix pylint issues * Fix profile tests * Move validation to session * Fix pylint issue Signed-off-by: J Boddey <boddey@google.com> --------- Signed-off-by: J Boddey <boddey@google.com> Co-authored-by: J Boddey <boddey@google.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> * Fix a delay in the internet connectivity check (#669) * Add a timeout to the command * move single-intf check to scheduler * add timeout arg to run_command * move jobs to constructor * fping for internet connection checking * Update internet connection when device is In Progress, Monitoring, Waiting for Device status (#677) * Update internet connection when device is In Progress, Monitoring, Waiting for Device status * check if interface physically connected * Revert "fping for internet connection checking" --------- Co-authored-by: Aliaksandr Nikitsin <aliaksandrn@google.com> Co-authored-by: Sofia Kurilova <kurilova@google.com> * Check if device folder already exists (#678) * Check if device folder already exists * Fix error message * Fix pylint issue * Remove debug mqtt logs (#692) --------- Signed-off-by: J Boddey <boddey@google.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sofia Kurilova <sonnycactus@gmail.com> Co-authored-by: Olga Mardvilko <omardvilko@mail.ru> Co-authored-by: Volha Mardvilka <mardvilka@google.com> Co-authored-by: jhughesbiot <50999916+jhughesbiot@users.noreply.github.com> Co-authored-by: Sofia Kurilova <kurilova@google.com> Co-authored-by: Aliaksandr Nikitsin <aliaksandrn@google.com> Co-authored-by: Marius <86727846+MariusBaldovin@users.noreply.github.com> * Resolve merge conflicts --------- Signed-off-by: J Boddey <boddey@google.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: jhughesbiot <jonathan.hughes@buildingsiot.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sofia Kurilova <sonnycactus@gmail.com> Co-authored-by: Olga Mardvilko <omardvilko@mail.ru> Co-authored-by: Volha Mardvilka <mardvilka@google.com> Co-authored-by: jhughesbiot <50999916+jhughesbiot@users.noreply.github.com> Co-authored-by: Sofia Kurilova <kurilova@google.com> Co-authored-by: Aliaksandr Nikitsin <aliaksandrn@google.com> Co-authored-by: Marius <86727846+MariusBaldovin@users.noreply.github.com>

Better handling of disbled tests

9dd819f

Add dns and tls tests back in disabled state

jhughesbiot requested a review from jboddey December 27, 2023 19:39

jhughesbiot changed the base branch from main to dev December 27, 2023 19:39

jboddey changed the base branch from dev to release/v1.1.1 January 8, 2024 20:48

Merge branch 'release/v1.1.1' into disabled_test_results

6939f68

jboddey closed this Feb 7, 2024

jhughesbiot reopened this Feb 7, 2024

Base automatically changed from release/v1.1.1 to dev February 7, 2024 22:03

jboddey and others added 3 commits February 15, 2024 17:23

Merge branch 'dev' into disabled_test_results

613044d

merge dev

6bc9dd6

pyling fixes

7c14378

jboddey reviewed Apr 18, 2024

View reviewed changes

framework/python/src/test_orc/test_orchestrator.py Outdated Show resolved Hide resolved

jboddey added 4 commits June 5, 2024 11:22

Merge branch 'dev' into disabled_test_results

7d6064b

Fix subscriptable error

695bfb8

Merge branch 'dev' into disabled_test_results

bc02b43

Re-add old tests as informational

fd2e7a3

jboddey self-requested a review July 22, 2024 12:32

jboddey approved these changes Jul 22, 2024

View reviewed changes

jboddey merged commit 86d0583 into dev Jul 22, 2024
10 checks passed

jboddey deleted the disabled_test_results branch July 22, 2024 19:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disabled test results #212

Disabled test results #212

jhughesbiot commented Dec 27, 2023 •

edited

Loading

jboddey commented Jan 2, 2024

jhughesbiot commented Jan 2, 2024

jboddey commented Jan 3, 2024

jhughesbiot commented Jan 3, 2024

jboddey commented Jan 4, 2024

jboddey commented Feb 7, 2024

jhughesbiot commented Feb 7, 2024

jboddey commented Feb 7, 2024

jhughesbiot commented Feb 7, 2024

Disabled test results #212

Disabled test results #212

Conversation

jhughesbiot commented Dec 27, 2023 • edited Loading

jboddey commented Jan 2, 2024

jhughesbiot commented Jan 2, 2024

jboddey commented Jan 3, 2024

jhughesbiot commented Jan 3, 2024

jboddey commented Jan 4, 2024

jboddey commented Feb 7, 2024

jhughesbiot commented Feb 7, 2024

jboddey commented Feb 7, 2024

jhughesbiot commented Feb 7, 2024

jhughesbiot commented Dec 27, 2023 •

edited

Loading