[b/memory_compare.py] consider fw allocated memory for pci #32
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
"new upstream version"
…ronix_0.6.0~rc1.orig.tar.gz and merge it
"new upstream version"
…ronix_0.6.0.orig.tar.gz and merge it
"new upstream version"
…t_0.4.0.orig.tar.gz and merge it
"new upstream version"
Release_2020_Week16
Release_2020_Week16
…ase-layout-launchers
Now the PTS tests will determine by themselves which build of cache to use and also will not run if the cache environment variable is not defined.
This commit updates the db tests as follows: - adds a common test_db_type function to utils.sh, which checks consul config for active services, and local configuration.toml for disabled services - inverts the two db test scenarios as redis is now the default db for Geneva - addresses shellcheck issues
The sys-mgmt-agent is disabled by default in Geneva due to the following issue: edgexfoundry/edgex-go#2486
Improve snap building process
1. Consider fw allocated memory for pci devices to fix the false alarm on some tight RAM system or a system allocated huge memory for PCI devices (e.g. graphic cards). (LP: #1941854) 2. Remove pci reserved memory if other than gpu 2.1. For amdgpu, read sysfs entries. 2.2. For non-amdgpu, read the reserved BAR memory.
pieqq
requested changes
Feb 8, 2023
| try: | ||
| lspci = check_output(['lspci', '-v', '-s', addr], universal_newlines=True) | ||
| except CalledProcessError as exc: | ||
| return exc.returncode |
There was a problem hiding this comment.
This will return the error code as an int. This is a problem since it can be confused with an actual size, which is what we expect.
I think it's better to let the exception bubble up.
(the same advice apply for other parts in the code where an int is returned as an error)
There was a problem hiding this comment.
+1.
The same type is used for two different meanings. This return code would later be added to the total size.
| size = 0 | ||
|
|
||
| for line in lspci.split('\n'): | ||
| # Skip G, K and lower than K, if lspci changes the format and |
There was a problem hiding this comment.
You mention skipping G, but below you add another regexp to search for sizes in G. Is this part of the comment still relevant?
There was a problem hiding this comment.
But if the info is there why not do proper job and parse it? it's trivial. Also running match twice instead of using KMGT(?) regex alternatives is wasteful.
| # this needs to be adjusted. So far, in 4G allocation, it shows | ||
| # in M (mega-bytes). | ||
| match = re.search('(.*)prefetchable\)(.*)\[size=(.*)M\](.*)', line) | ||
| if match is not None: |
There was a problem hiding this comment.
is not None is not necessary here (and in other case below)
| @@ -118,6 +199,8 @@ def main(): | |||
| print("Results:") | |||
| print("\t/proc/meminfo reports:\t{}".format(visible_memory)) | |||
| print("\tlshw reports:\t{}".format(installed_memory)) | |||
| print("\tFound FW allocated %d MB for PCI GPUs" % | |||
| fw_allocated_memory_for_pci) | |||
| print("\nPASS: Meminfo reports %s less than lshw, a " | |||
There was a problem hiding this comment.
The difference might be from meminfo, but it might also be from meminfo and the memory allocated for the GPU, so this message should be modified to fit the reported info.
(the same applies below when there is an error)
kissiel
suggested changes
Feb 8, 2023
There was a problem hiding this comment.
This code needs multiple fixes.
Was this tested on systems with discrete a AMD GPU? Cause it seems that the VRAM is substracted from the pool, and not the reserved system memory. Meaning, if I have a PC with 8GB of system memory, and a radeon with 8GB of vram, this program will yield the system as having negative available ram :D
As for the implementation:
the code mixes return types (error codes with sizes and so on).
there's many inefficiencies (both in runtime and code complexity and size). (see inline)
there are no tests :(
|
|
||
| # Return 0, if not found the memory for the pci device | ||
| # Return >0, reserved memory for the pci device | ||
| def get_allocated_pci_memory(addr=None): |
There was a problem hiding this comment.
calling this function with no address seems to be a callers logic error, so I'd recommend ditching the default val, and the check in the function.
| size = 0 | ||
|
|
||
| for line in lspci.split('\n'): | ||
| # Skip G, K and lower than K, if lspci changes the format and |
There was a problem hiding this comment.
But if the info is there why not do proper job and parse it? it's trivial. Also running match twice instead of using KMGT(?) regex alternatives is wasteful.
| except CalledProcessError as exc: | ||
| return exc.returncode | ||
|
|
||
| # If ARM, then exit |
There was a problem hiding this comment.
!= 'x86_64' doesn't mean it's arm. It can be multiple different things
|
|
||
| # If ARM, then exit | ||
| if arch != 'x86_64': | ||
| return 1 |
There was a problem hiding this comment.
this states the allocated memory for a device is 1B if the arch is not x86_64
| return size | ||
| with open('%s/mem_info_vram_total' % path) as vram: | ||
| size = int(vram.read()) | ||
| size = int(size / (1024 * 1024)) |
There was a problem hiding this comment.
use integer division '//' instead of casting.
| difference = HumanReadableBytes(installed_memory - visible_memory) | ||
| if fw_allocated_memory_for_pci >= 0: | ||
| difference = HumanReadableBytes(installed_memory - visible_memory - | ||
| fw_allocated_memory_for_pci * 1024 * 1024) |
There was a problem hiding this comment.
so you previously divide the size by 1024*1024 to once again multiply it here?
I recommend using basic units everywhere.
| visible_memory = HumanReadableBytes(get_visible_memory_size()) | ||
| threshold = get_threshold(installed_memory) | ||
|
|
||
| difference = HumanReadableBytes(installed_memory - visible_memory) | ||
| if fw_allocated_memory_for_pci >= 0: |
|
Following @kissiel comment, I tried the script on my desktop (i5 CPU, AMD RX580 discrete GPU): So it passes, but with very strange results :) |
|
Please rebase your work on top of current main branch |
|
Closing this for now |
|
For a bit of background, we reviewed the proposed solution, and it seems too flaky to rely on some of the assumptions made in the patch, depending on the type of device being tested. @os369510 if you can think of a better way to solve this, we'll be happy to review a new PR. |
pieqq
pushed a commit
that referenced
this pull request
Jan 12, 2024
* Fix tox test environment error Each tox test environment won't be reseted, therefore removing all providers before test is needed. After checking with checkbox repo, checkbox-ng has to use different install method to install. * Follow checkbox repo to update pyYAML version to 6.0.1 for python 3.10+ to solve tox issue
pieqq
pushed a commit
that referenced
this pull request
Jan 12, 2024
* Fix tox test environment error Each tox test environment won't be reseted, therefore removing all providers before test is needed. After checking with checkbox repo, checkbox-ng has to use different install method to install. * Follow checkbox repo to update pyYAML version to 6.0.1 for python 3.10+ to solve tox issue
kissiel
pushed a commit
that referenced
this pull request
Jan 12, 2024
* Fix tox test environment error Each tox test environment won't be reseted, therefore removing all providers before test is needed. After checking with checkbox repo, checkbox-ng has to use different install method to install. * Follow checkbox repo to update pyYAML version to 6.0.1 for python 3.10+ to solve tox issue
kissiel
pushed a commit
that referenced
this pull request
Jan 12, 2024
* Initial commit * Migrate ARM generic jobs from launchpad * Initial commit * Clean up this repo, and up it to date. * Change: job id for cpu cases * Change: id of rpmsg * Add: manage.py Since we need this repo as a provider * Change: separate jobs into generic and non-generic * Add: checkbox-provider-arm folder Following the structure of checkbox * Change: the access of tool not created by us To prevent those tools been packaged into checkbox * Add: test plan for ARM generic jobs * Change: test method for NPU TensorFlow Change the test method by using test snap build by Isaac * Change: MTD test method Since we don't have reliable way to indentify the type of MTD. * Change: test method for buzzer Change the test method for buzzer that probe as an input PC speaker * Change: chunkfs test method * Add: dump DTB file * Add: manifest for the jobs * Add: rs485 remote test * Remove: non-generic jobs Since we need to move generic parts to github Co-authored-by: Nara Huang <nara.huang@canonical.com> * Add snapcraft yaml for uc20 (#2) * Add: snapcraft.yaml for uc20 * Add: README.md * Fix: export PYTHONPATH in override-build section * Add: libc6-dev for build-packages of input-pcspkr Since snapcraft7.x will need extra lib from libc6-dev * Fix cpu and device tree cases (#3) * Fix: cpu offlining test * Change: requires of cpu offlineing test * Add: stage-package linuxptp * Fix: test plan for dtb * Add eeprom test and change name of the provider (#6) * Add: eeprom test cases * Change: provider name from arm to ce-oem * Change name space and README.md (#9) * Change: name space * Change: README.md * Fix: wording of checkbox interface snap * Remove: snapcraft.yaml (#8) * Add: led generic test (#7) * Add: support cold reboot tests by pdu (#10) * Add: support cold reboot tests by pdu add cold reboot tests by network PDU for those system which not able to wakeup by rtc * Fix: fix bugs add following changes in this commit - check supported pdu type inside of main function - sync data after dump device information - correct command for init-boot-loop-data job * Fix: fix bug - fix the error reported by shellcheck * Change prefixes (#11) * Change: prefixes for plans and jobs * Change: rename test plan and add PDU related plan * Change: name of category * Fix: missing prefixes of serial test job and plan * Add: informations about used config in readme (#13) * Change: sysfs led method (#12) * Fix: rtc wakealarm checking to right attribute (#14) * Fix: remove ce-oem-power-automated-by-pdu Remove the ce-oem-power-automated-by-pdu from the ce-oem-automated The ce-oem-power-automated-by-pud will always be executed even the user not intend to reboot the DUT by PDU. And the checkbox session will stuck after send out a snmpset command. So remove this test first, and we need to integrate the cold boot related tests to PPC to support cold reboot test for those DUT which not able to wake up by rtc. * Add: add a generic iot test plans Adding a generic IoT test plans, it's a full testing set include tests in certification test plan and ce-oem-generic test plan * Improve the MTD script Before doing the read and write testing of specific mtd partition, we need to know the MTD is read-only or writable, therefore, a new variable called is_mtd_writable to help us to know it. If this mtd is read-only, I only read its content to a file. If this mtd is writable, backup the its original content to a file, then do the compare procedure, finally, recover its content. * Add: CAAM related test (#21) * Add: CAAM related test * Add: generic crypto test * Fix naming with ce-oem * Add: add digital-io tests (#24) * Add: add digital-io tests add digital-io tests * Fix: fixed scripts and test plan - fixed the format string style - fixed the typo in docstring - fixed the variable type for TEST_STATES - remove useless test jobs * Fix: fixed typo fixed typo * Add: add readme (#23) * Fix: updated README.md add a section about how to use checkbox-ce-oem as a checkbox launcher * Fix: updated the installation steps updated the installation steps for Ubuntu classic and Ubuntu core * fix: fixed typo fixed typo * Fix: updated digital-io tests (#25) - corrected digital-io tests in post-suspend stage - import manifest job * Add: cpufreq governors test for 6 governors (#19) * Add: cpufreq governors test for 6 governors Add a resource job to check system supported governors, and test jobs for all 6 governors. These tests are only applicable to the platform using acpi_cpufreq related drivers. Newer intel CPU using intel_pstate driver is NOT applicable. * Fix: typo and apply suggestions * Remove: ce-oem-cpu/offlining_test from the plan Since there is already a fix for cpu/offlining_test in checkbox base provider, this ce-oem-cpu/offlining_test is no longer required. #522 * Modify CAAM and installation time related jobs. (#26) * Modify: modify hwrng and caam algo check to require manifest * Add: requires for installation related jobs Make those jobs run only in ubuntu core * Fix: revert summary section for snapd_installation_time The previous commit accidentally removed the section * Fix script for pep8 check (#27) * Remove: test job/script that not in use. Since we have new script for cpu freq governors. And also cpu_offline test has been updated in base provider. * Fix: python script to fit flake8 * Fix: manifest shouldn't place in resource job Move the manifest to test job. To prevent manifest being executed at bootstrap stage. * Add code check by tox just like checkbox did (#28) * Add code check by tox just like checkbox did This commit following what checkbox did, and add pep8-naming check for python 3.8+. * Change the fork checkbox git repo url to offical one * Fix tox test environment error (#32) * Fix tox test environment error Each tox test environment won't be reseted, therefore removing all providers before test is needed. After checking with checkbox repo, checkbox-ng has to use different install method to install. * Follow checkbox repo to update pyYAML version to 6.0.1 for python 3.10+ to solve tox issue * Fix for shellcheck (#29) * Fix: shell scrip to fit shellcheck * Fix: audio jobs to fit shellcheck * Fix: caam jobs to fit shellcheck * Fix: device jobs to fit shellcheck * Fix: digital-io jobs to fit shellcheck * Change: nested EEPROM test from checkbox base Since EEPROM jobs merged back to the base provider already. * Fix: power-manager jobs to fit shellcheck * Change: nested RTC test from checkbox base Since RTC jobs merged back to the base provider already. * Fix: stress jobs to fit shellcheck * Fix: theraml-sensor jobs to fit shellcheck * Fix: Nested plan to ce-oem Follow the structure as a client-cert plan * Add multiple usb otg test (#33) * Remove: OTG related jobs Those OTG jobs have been migrated back to base provider. * Add: multiple OTG test * Add: readme.md * Refactor cpufreq governor test (#36) * Change: refactor cpufreq governor test CQT-2906 The original tests did not handle the following 2 conditions: 1. CPUfreq scaling driver not found (no files under cpufreq folder) 2. On ARM device, system using multiple policies to handle different CPUs group This commit refactor the tests to make them policies-based tests. Also adding a driver detect job that can be use on all platforms. * Add optee test (#34) * Add: optee test * Add: README.md * Include led test from base (#35) * Add: nested led job from checkbox-base * Change: README.md for led * Change: PTP command (#37) * Change: PTP command lp:2030942, workaround for the current version of ptp4l * Fix: apply musical scales for pwm buzzer test (#40) * Fix: apply musical scales for pwm buzzer test apply musical scales for pwm buzzer test * Fix: added description for test_pwm_buzzer func added description for test_pwm_buzzer function * Fix: add touchscreen tests (#39) * Fix: add touchscreen tests validate the required touch events for touchscreen also add new test to cover maximum fingers support on touchscreen * Change: hardware PTP support keywords For PTP capabilities, ethtool v5.4 (on Ubuntu 20.04) shows `hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)` But ethtool v5.16 (on Ubuntu 22.04) only shows `hardware-transmit` The test case was using `SOF_TIMESTAMPING_TX_HARDWARE` as keywords, This commit changes it to use `hardware-transmit` for adapting new version of ethtool. * Remove: nested ptp-manual test plans The current manual test job steps are duplicated with the auto test job. Currently we only need to run the auto test job. The manual test steps here is used as a reference. If there are new jobs in th future, the ptp-manual test plan can be nested back. * Add: LD_PATH information in readme.md (#41) * Change: additional patterns to map UDC and USB node (#42) * Change: additional patterns to map UDC and USB node * Fix: the if else logic for map UDC * Fix: include python3-systemd in tox.ini (#48) include the python3-systemd package in tox.ini * Fix: updated the related packages for tox (#50) updated the related packages for tox * Fix: nested plan for server manual (#44) * Fix: nested plan for server manual * Fix: errors when mtd count is 0 (#55) * Fix: errors when mtd count is 0 * Add shellcheck ignore 2126 * Add: rpmsg tests for i.MX and TI processors (#46) * Fix: updated the related packages for tox updated the related packages for tox * Add: rpmsg tests for i.MX and TI processors adding rpmsg tests for i.MX and TI processors * Fix: updated rpmsg_tests.py scripts updated rpmsg_tests.py scripts * Fix: fixed bugs fixed bugs * Modify optee test run out of gadget (#45) * Add: support outsourcing optee-test * 230921 crypto test (#52) * Fix: updated the related packages for tox updated the related packages for tox * Fix: update the crypto tests seperate the crypto tests to the generic crypto tests and crypto accelerator tests * Fix: bypass if failed to set extra seed the extra seed might not be supported for all platform * Fix: fixed bugs fixed bugs * Fix: refactor check_crypto_driver_priority func refactor check_crypto_driver_priority function * Fix: mock systemd-python package and remove packages in tox mock systemd-python package and remove packages in tox * Revamp otg_ports resource job should not fail (#53) * Revamp otg_ports resource job should not fail Add a if condition to handle when the OTG config variable is not set. * Revise the echo string * Add: cupfreq module probe (#51) * Add: cupfreq module probe * Check hwrng test (#57) * Modify hwrng test more flexible * Add README.md * Make ce-oem-info jobs be attachment jobs (#58) ce-oem-info jobs are aimed at collecting the system bootup information from snapd and cloud-init to help developers analyze bugs. Therefore, these jobs are more suitable to be attachment jobs and should not fail even if some information can not be found. For example, some systems may not have cloud-init. In these systems, the job cannot gather the information from cloud-init, but this should not be a failure. * Modify the device tree dump job (#59) The current ce-oem-device-tree/dump job does not match the resource job format and will show warnings when running it. The original plan is to parse this job to match the resource format to let any other jobs able to get the dts info conveniently. As parse the dts to resource job format is a huge work, and will not be done in the near future. This commit excludes the ce-oem-device-tree/dump job from the test plan and add comments to explain the purpose. Also keep the attachment job to make sure we could get the device tree information when checking the report. * Revamp caam_hwrng_test by looping w/ smaller size (#60) Some of the project devices' caam_hwrng have buffer, so previously the dd size is increased to 10M to exceed the buffer size to make the interrupt increase. It will make the device without buffer take a very long time (over 30 mins) to do the dd command. Therefore, this commit separated the size to 512K and looped it 20 times. On the device without buffer it could do 512K and detect the increased interrupt to get the pass result, which is much faster. On the device with buttfer it could also exceed the buffer by dd 20 times and detect the increased interrupt to get the pass result. * Fix the nested crypto test plan name (#61) Previously the caam test plan was revamped to accelerator test plan, but forgot to update the name nested in ce-oem-automated. Fix the name in this commit. * Add: new socketcan stress test (#62) * Add: new socketcan tests add new socketcan stress test (100000 loop echo test) add bus-off test and fixed bug * Add: include new socketcan tests (#63) include new socketecan test plan into ce-oem test plans * Add ubuntu-frame and glmark2 tests (#64) * Add ubuntu-frame and glmark2 tests * Modify the way checking ubuntu-frame active * Modify: Add more details in README * Add: implement the RPMSG-tty tests implement the RPMSG-tty tests for i.MX series * Fix: update tox configuration update tox configuration to perform tests with specific Python version * Add: check VPU device tests (#68) * Add: check VPU device tests adding new test cases about identify VPU devices * Fix: fixed the issues from comments fixed the issues from comments * Fix: bug fixed bug fixed * Fix: fixed issues fixed issues * Fix: refactor the thermal tests (#69) * Fix: refactor the thermal tests refactor the thermal tests * Fix: correct the argument name in socketcan_test (#71) corrected the argument name in socketcan_test.py * Add dbus reboot stress (#70) * Add script for installing test snap and connect interfaces * Add reboot via dbus command This test is for snap strict confinement mode * Add tcp test (#65) * Add TCP test * Move ce-oem README inside the provider directory * Change the ce-oem provider namespace to the contrib namespace * Add README for the contrib area * Remove .gitignore from contrib/ Artefact left from the ce-oem provider migration --------- Co-authored-by: stanley31huang <stanley.huang@canonical.com> Co-authored-by: rickwu666666 <98441647+rickwu666666@users.noreply.github.com> Co-authored-by: Nara Huang <nara.huang@canonical.com> Co-authored-by: rickwu4444 <rick.wu@canonical.com> Co-authored-by: baconyao <patrick.chang@canonical.com> Co-authored-by: liaou3 <vincent.liao@canonical.com> Co-authored-by: patliuu <111331153+patliuu@users.noreply.github.com> Co-authored-by: Patrick Liu <patrick.liu@canonical.com> Co-authored-by: hanhsuan <32028620+hanhsuan@users.noreply.github.com>
LiaoU3
added a commit
to LiaoU3/checkbox
that referenced
this pull request
Mar 20, 2024
…al#932) * Initial commit * Migrate ARM generic jobs from launchpad * Initial commit * Clean up this repo, and up it to date. * Change: job id for cpu cases * Change: id of rpmsg * Add: manage.py Since we need this repo as a provider * Change: separate jobs into generic and non-generic * Add: checkbox-provider-arm folder Following the structure of checkbox * Change: the access of tool not created by us To prevent those tools been packaged into checkbox * Add: test plan for ARM generic jobs * Change: test method for NPU TensorFlow Change the test method by using test snap build by Isaac * Change: MTD test method Since we don't have reliable way to indentify the type of MTD. * Change: test method for buzzer Change the test method for buzzer that probe as an input PC speaker * Change: chunkfs test method * Add: dump DTB file * Add: manifest for the jobs * Add: rs485 remote test * Remove: non-generic jobs Since we need to move generic parts to github Co-authored-by: Nara Huang <nara.huang@canonical.com> * Add snapcraft yaml for uc20 (canonical#2) * Add: snapcraft.yaml for uc20 * Add: README.md * Fix: export PYTHONPATH in override-build section * Add: libc6-dev for build-packages of input-pcspkr Since snapcraft7.x will need extra lib from libc6-dev * Fix cpu and device tree cases (canonical#3) * Fix: cpu offlining test * Change: requires of cpu offlineing test * Add: stage-package linuxptp * Fix: test plan for dtb * Add eeprom test and change name of the provider (canonical#6) * Add: eeprom test cases * Change: provider name from arm to ce-oem * Change name space and README.md (canonical#9) * Change: name space * Change: README.md * Fix: wording of checkbox interface snap * Remove: snapcraft.yaml (canonical#8) * Add: led generic test (canonical#7) * Add: support cold reboot tests by pdu (canonical#10) * Add: support cold reboot tests by pdu add cold reboot tests by network PDU for those system which not able to wakeup by rtc * Fix: fix bugs add following changes in this commit - check supported pdu type inside of main function - sync data after dump device information - correct command for init-boot-loop-data job * Fix: fix bug - fix the error reported by shellcheck * Change prefixes (canonical#11) * Change: prefixes for plans and jobs * Change: rename test plan and add PDU related plan * Change: name of category * Fix: missing prefixes of serial test job and plan * Add: informations about used config in readme (canonical#13) * Change: sysfs led method (canonical#12) * Fix: rtc wakealarm checking to right attribute (canonical#14) * Fix: remove ce-oem-power-automated-by-pdu Remove the ce-oem-power-automated-by-pdu from the ce-oem-automated The ce-oem-power-automated-by-pud will always be executed even the user not intend to reboot the DUT by PDU. And the checkbox session will stuck after send out a snmpset command. So remove this test first, and we need to integrate the cold boot related tests to PPC to support cold reboot test for those DUT which not able to wake up by rtc. * Add: add a generic iot test plans Adding a generic IoT test plans, it's a full testing set include tests in certification test plan and ce-oem-generic test plan * Improve the MTD script Before doing the read and write testing of specific mtd partition, we need to know the MTD is read-only or writable, therefore, a new variable called is_mtd_writable to help us to know it. If this mtd is read-only, I only read its content to a file. If this mtd is writable, backup the its original content to a file, then do the compare procedure, finally, recover its content. * Add: CAAM related test (canonical#21) * Add: CAAM related test * Add: generic crypto test * Fix naming with ce-oem * Add: add digital-io tests (canonical#24) * Add: add digital-io tests add digital-io tests * Fix: fixed scripts and test plan - fixed the format string style - fixed the typo in docstring - fixed the variable type for TEST_STATES - remove useless test jobs * Fix: fixed typo fixed typo * Add: add readme (canonical#23) * Fix: updated README.md add a section about how to use checkbox-ce-oem as a checkbox launcher * Fix: updated the installation steps updated the installation steps for Ubuntu classic and Ubuntu core * fix: fixed typo fixed typo * Fix: updated digital-io tests (canonical#25) - corrected digital-io tests in post-suspend stage - import manifest job * Add: cpufreq governors test for 6 governors (canonical#19) * Add: cpufreq governors test for 6 governors Add a resource job to check system supported governors, and test jobs for all 6 governors. These tests are only applicable to the platform using acpi_cpufreq related drivers. Newer intel CPU using intel_pstate driver is NOT applicable. * Fix: typo and apply suggestions * Remove: ce-oem-cpu/offlining_test from the plan Since there is already a fix for cpu/offlining_test in checkbox base provider, this ce-oem-cpu/offlining_test is no longer required. canonical#522 * Modify CAAM and installation time related jobs. (canonical#26) * Modify: modify hwrng and caam algo check to require manifest * Add: requires for installation related jobs Make those jobs run only in ubuntu core * Fix: revert summary section for snapd_installation_time The previous commit accidentally removed the section * Fix script for pep8 check (canonical#27) * Remove: test job/script that not in use. Since we have new script for cpu freq governors. And also cpu_offline test has been updated in base provider. * Fix: python script to fit flake8 * Fix: manifest shouldn't place in resource job Move the manifest to test job. To prevent manifest being executed at bootstrap stage. * Add code check by tox just like checkbox did (canonical#28) * Add code check by tox just like checkbox did This commit following what checkbox did, and add pep8-naming check for python 3.8+. * Change the fork checkbox git repo url to offical one * Fix tox test environment error (canonical#32) * Fix tox test environment error Each tox test environment won't be reseted, therefore removing all providers before test is needed. After checking with checkbox repo, checkbox-ng has to use different install method to install. * Follow checkbox repo to update pyYAML version to 6.0.1 for python 3.10+ to solve tox issue * Fix for shellcheck (canonical#29) * Fix: shell scrip to fit shellcheck * Fix: audio jobs to fit shellcheck * Fix: caam jobs to fit shellcheck * Fix: device jobs to fit shellcheck * Fix: digital-io jobs to fit shellcheck * Change: nested EEPROM test from checkbox base Since EEPROM jobs merged back to the base provider already. * Fix: power-manager jobs to fit shellcheck * Change: nested RTC test from checkbox base Since RTC jobs merged back to the base provider already. * Fix: stress jobs to fit shellcheck * Fix: theraml-sensor jobs to fit shellcheck * Fix: Nested plan to ce-oem Follow the structure as a client-cert plan * Add multiple usb otg test (canonical#33) * Remove: OTG related jobs Those OTG jobs have been migrated back to base provider. * Add: multiple OTG test * Add: readme.md * Refactor cpufreq governor test (canonical#36) * Change: refactor cpufreq governor test CQT-2906 The original tests did not handle the following 2 conditions: 1. CPUfreq scaling driver not found (no files under cpufreq folder) 2. On ARM device, system using multiple policies to handle different CPUs group This commit refactor the tests to make them policies-based tests. Also adding a driver detect job that can be use on all platforms. * Add optee test (canonical#34) * Add: optee test * Add: README.md * Include led test from base (canonical#35) * Add: nested led job from checkbox-base * Change: README.md for led * Change: PTP command (canonical#37) * Change: PTP command lp:2030942, workaround for the current version of ptp4l * Fix: apply musical scales for pwm buzzer test (canonical#40) * Fix: apply musical scales for pwm buzzer test apply musical scales for pwm buzzer test * Fix: added description for test_pwm_buzzer func added description for test_pwm_buzzer function * Fix: add touchscreen tests (canonical#39) * Fix: add touchscreen tests validate the required touch events for touchscreen also add new test to cover maximum fingers support on touchscreen * Change: hardware PTP support keywords For PTP capabilities, ethtool v5.4 (on Ubuntu 20.04) shows `hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)` But ethtool v5.16 (on Ubuntu 22.04) only shows `hardware-transmit` The test case was using `SOF_TIMESTAMPING_TX_HARDWARE` as keywords, This commit changes it to use `hardware-transmit` for adapting new version of ethtool. * Remove: nested ptp-manual test plans The current manual test job steps are duplicated with the auto test job. Currently we only need to run the auto test job. The manual test steps here is used as a reference. If there are new jobs in th future, the ptp-manual test plan can be nested back. * Add: LD_PATH information in readme.md (canonical#41) * Change: additional patterns to map UDC and USB node (canonical#42) * Change: additional patterns to map UDC and USB node * Fix: the if else logic for map UDC * Fix: include python3-systemd in tox.ini (canonical#48) include the python3-systemd package in tox.ini * Fix: updated the related packages for tox (canonical#50) updated the related packages for tox * Fix: nested plan for server manual (canonical#44) * Fix: nested plan for server manual * Fix: errors when mtd count is 0 (canonical#55) * Fix: errors when mtd count is 0 * Add shellcheck ignore 2126 * Add: rpmsg tests for i.MX and TI processors (canonical#46) * Fix: updated the related packages for tox updated the related packages for tox * Add: rpmsg tests for i.MX and TI processors adding rpmsg tests for i.MX and TI processors * Fix: updated rpmsg_tests.py scripts updated rpmsg_tests.py scripts * Fix: fixed bugs fixed bugs * Modify optee test run out of gadget (canonical#45) * Add: support outsourcing optee-test * 230921 crypto test (canonical#52) * Fix: updated the related packages for tox updated the related packages for tox * Fix: update the crypto tests seperate the crypto tests to the generic crypto tests and crypto accelerator tests * Fix: bypass if failed to set extra seed the extra seed might not be supported for all platform * Fix: fixed bugs fixed bugs * Fix: refactor check_crypto_driver_priority func refactor check_crypto_driver_priority function * Fix: mock systemd-python package and remove packages in tox mock systemd-python package and remove packages in tox * Revamp otg_ports resource job should not fail (canonical#53) * Revamp otg_ports resource job should not fail Add a if condition to handle when the OTG config variable is not set. * Revise the echo string * Add: cupfreq module probe (canonical#51) * Add: cupfreq module probe * Check hwrng test (canonical#57) * Modify hwrng test more flexible * Add README.md * Make ce-oem-info jobs be attachment jobs (canonical#58) ce-oem-info jobs are aimed at collecting the system bootup information from snapd and cloud-init to help developers analyze bugs. Therefore, these jobs are more suitable to be attachment jobs and should not fail even if some information can not be found. For example, some systems may not have cloud-init. In these systems, the job cannot gather the information from cloud-init, but this should not be a failure. * Modify the device tree dump job (canonical#59) The current ce-oem-device-tree/dump job does not match the resource job format and will show warnings when running it. The original plan is to parse this job to match the resource format to let any other jobs able to get the dts info conveniently. As parse the dts to resource job format is a huge work, and will not be done in the near future. This commit excludes the ce-oem-device-tree/dump job from the test plan and add comments to explain the purpose. Also keep the attachment job to make sure we could get the device tree information when checking the report. * Revamp caam_hwrng_test by looping w/ smaller size (canonical#60) Some of the project devices' caam_hwrng have buffer, so previously the dd size is increased to 10M to exceed the buffer size to make the interrupt increase. It will make the device without buffer take a very long time (over 30 mins) to do the dd command. Therefore, this commit separated the size to 512K and looped it 20 times. On the device without buffer it could do 512K and detect the increased interrupt to get the pass result, which is much faster. On the device with buttfer it could also exceed the buffer by dd 20 times and detect the increased interrupt to get the pass result. * Fix the nested crypto test plan name (canonical#61) Previously the caam test plan was revamped to accelerator test plan, but forgot to update the name nested in ce-oem-automated. Fix the name in this commit. * Add: new socketcan stress test (canonical#62) * Add: new socketcan tests add new socketcan stress test (100000 loop echo test) add bus-off test and fixed bug * Add: include new socketcan tests (canonical#63) include new socketecan test plan into ce-oem test plans * Add ubuntu-frame and glmark2 tests (canonical#64) * Add ubuntu-frame and glmark2 tests * Modify the way checking ubuntu-frame active * Modify: Add more details in README * Add: implement the RPMSG-tty tests implement the RPMSG-tty tests for i.MX series * Fix: update tox configuration update tox configuration to perform tests with specific Python version * Add: check VPU device tests (canonical#68) * Add: check VPU device tests adding new test cases about identify VPU devices * Fix: fixed the issues from comments fixed the issues from comments * Fix: bug fixed bug fixed * Fix: fixed issues fixed issues * Fix: refactor the thermal tests (canonical#69) * Fix: refactor the thermal tests refactor the thermal tests * Fix: correct the argument name in socketcan_test (canonical#71) corrected the argument name in socketcan_test.py * Add dbus reboot stress (canonical#70) * Add script for installing test snap and connect interfaces * Add reboot via dbus command This test is for snap strict confinement mode * Add tcp test (canonical#65) * Add TCP test * Move ce-oem README inside the provider directory * Change the ce-oem provider namespace to the contrib namespace * Add README for the contrib area * Remove .gitignore from contrib/ Artefact left from the ce-oem provider migration --------- Co-authored-by: stanley31huang <stanley.huang@canonical.com> Co-authored-by: rickwu666666 <98441647+rickwu666666@users.noreply.github.com> Co-authored-by: Nara Huang <nara.huang@canonical.com> Co-authored-by: rickwu4444 <rick.wu@canonical.com> Co-authored-by: baconyao <patrick.chang@canonical.com> Co-authored-by: liaou3 <vincent.liao@canonical.com> Co-authored-by: patliuu <111331153+patliuu@users.noreply.github.com> Co-authored-by: Patrick Liu <patrick.liu@canonical.com> Co-authored-by: hanhsuan <32028620+hanhsuan@users.noreply.github.com>
binli
pushed a commit
to binli/checkbox
that referenced
this pull request
Mar 22, 2024
…al#932) * Initial commit * Migrate ARM generic jobs from launchpad * Initial commit * Clean up this repo, and up it to date. * Change: job id for cpu cases * Change: id of rpmsg * Add: manage.py Since we need this repo as a provider * Change: separate jobs into generic and non-generic * Add: checkbox-provider-arm folder Following the structure of checkbox * Change: the access of tool not created by us To prevent those tools been packaged into checkbox * Add: test plan for ARM generic jobs * Change: test method for NPU TensorFlow Change the test method by using test snap build by Isaac * Change: MTD test method Since we don't have reliable way to indentify the type of MTD. * Change: test method for buzzer Change the test method for buzzer that probe as an input PC speaker * Change: chunkfs test method * Add: dump DTB file * Add: manifest for the jobs * Add: rs485 remote test * Remove: non-generic jobs Since we need to move generic parts to github Co-authored-by: Nara Huang <nara.huang@canonical.com> * Add snapcraft yaml for uc20 (canonical#2) * Add: snapcraft.yaml for uc20 * Add: README.md * Fix: export PYTHONPATH in override-build section * Add: libc6-dev for build-packages of input-pcspkr Since snapcraft7.x will need extra lib from libc6-dev * Fix cpu and device tree cases (canonical#3) * Fix: cpu offlining test * Change: requires of cpu offlineing test * Add: stage-package linuxptp * Fix: test plan for dtb * Add eeprom test and change name of the provider (canonical#6) * Add: eeprom test cases * Change: provider name from arm to ce-oem * Change name space and README.md (canonical#9) * Change: name space * Change: README.md * Fix: wording of checkbox interface snap * Remove: snapcraft.yaml (canonical#8) * Add: led generic test (canonical#7) * Add: support cold reboot tests by pdu (canonical#10) * Add: support cold reboot tests by pdu add cold reboot tests by network PDU for those system which not able to wakeup by rtc * Fix: fix bugs add following changes in this commit - check supported pdu type inside of main function - sync data after dump device information - correct command for init-boot-loop-data job * Fix: fix bug - fix the error reported by shellcheck * Change prefixes (canonical#11) * Change: prefixes for plans and jobs * Change: rename test plan and add PDU related plan * Change: name of category * Fix: missing prefixes of serial test job and plan * Add: informations about used config in readme (canonical#13) * Change: sysfs led method (canonical#12) * Fix: rtc wakealarm checking to right attribute (canonical#14) * Fix: remove ce-oem-power-automated-by-pdu Remove the ce-oem-power-automated-by-pdu from the ce-oem-automated The ce-oem-power-automated-by-pud will always be executed even the user not intend to reboot the DUT by PDU. And the checkbox session will stuck after send out a snmpset command. So remove this test first, and we need to integrate the cold boot related tests to PPC to support cold reboot test for those DUT which not able to wake up by rtc. * Add: add a generic iot test plans Adding a generic IoT test plans, it's a full testing set include tests in certification test plan and ce-oem-generic test plan * Improve the MTD script Before doing the read and write testing of specific mtd partition, we need to know the MTD is read-only or writable, therefore, a new variable called is_mtd_writable to help us to know it. If this mtd is read-only, I only read its content to a file. If this mtd is writable, backup the its original content to a file, then do the compare procedure, finally, recover its content. * Add: CAAM related test (canonical#21) * Add: CAAM related test * Add: generic crypto test * Fix naming with ce-oem * Add: add digital-io tests (canonical#24) * Add: add digital-io tests add digital-io tests * Fix: fixed scripts and test plan - fixed the format string style - fixed the typo in docstring - fixed the variable type for TEST_STATES - remove useless test jobs * Fix: fixed typo fixed typo * Add: add readme (canonical#23) * Fix: updated README.md add a section about how to use checkbox-ce-oem as a checkbox launcher * Fix: updated the installation steps updated the installation steps for Ubuntu classic and Ubuntu core * fix: fixed typo fixed typo * Fix: updated digital-io tests (canonical#25) - corrected digital-io tests in post-suspend stage - import manifest job * Add: cpufreq governors test for 6 governors (canonical#19) * Add: cpufreq governors test for 6 governors Add a resource job to check system supported governors, and test jobs for all 6 governors. These tests are only applicable to the platform using acpi_cpufreq related drivers. Newer intel CPU using intel_pstate driver is NOT applicable. * Fix: typo and apply suggestions * Remove: ce-oem-cpu/offlining_test from the plan Since there is already a fix for cpu/offlining_test in checkbox base provider, this ce-oem-cpu/offlining_test is no longer required. canonical#522 * Modify CAAM and installation time related jobs. (canonical#26) * Modify: modify hwrng and caam algo check to require manifest * Add: requires for installation related jobs Make those jobs run only in ubuntu core * Fix: revert summary section for snapd_installation_time The previous commit accidentally removed the section * Fix script for pep8 check (canonical#27) * Remove: test job/script that not in use. Since we have new script for cpu freq governors. And also cpu_offline test has been updated in base provider. * Fix: python script to fit flake8 * Fix: manifest shouldn't place in resource job Move the manifest to test job. To prevent manifest being executed at bootstrap stage. * Add code check by tox just like checkbox did (canonical#28) * Add code check by tox just like checkbox did This commit following what checkbox did, and add pep8-naming check for python 3.8+. * Change the fork checkbox git repo url to offical one * Fix tox test environment error (canonical#32) * Fix tox test environment error Each tox test environment won't be reseted, therefore removing all providers before test is needed. After checking with checkbox repo, checkbox-ng has to use different install method to install. * Follow checkbox repo to update pyYAML version to 6.0.1 for python 3.10+ to solve tox issue * Fix for shellcheck (canonical#29) * Fix: shell scrip to fit shellcheck * Fix: audio jobs to fit shellcheck * Fix: caam jobs to fit shellcheck * Fix: device jobs to fit shellcheck * Fix: digital-io jobs to fit shellcheck * Change: nested EEPROM test from checkbox base Since EEPROM jobs merged back to the base provider already. * Fix: power-manager jobs to fit shellcheck * Change: nested RTC test from checkbox base Since RTC jobs merged back to the base provider already. * Fix: stress jobs to fit shellcheck * Fix: theraml-sensor jobs to fit shellcheck * Fix: Nested plan to ce-oem Follow the structure as a client-cert plan * Add multiple usb otg test (canonical#33) * Remove: OTG related jobs Those OTG jobs have been migrated back to base provider. * Add: multiple OTG test * Add: readme.md * Refactor cpufreq governor test (canonical#36) * Change: refactor cpufreq governor test CQT-2906 The original tests did not handle the following 2 conditions: 1. CPUfreq scaling driver not found (no files under cpufreq folder) 2. On ARM device, system using multiple policies to handle different CPUs group This commit refactor the tests to make them policies-based tests. Also adding a driver detect job that can be use on all platforms. * Add optee test (canonical#34) * Add: optee test * Add: README.md * Include led test from base (canonical#35) * Add: nested led job from checkbox-base * Change: README.md for led * Change: PTP command (canonical#37) * Change: PTP command lp:2030942, workaround for the current version of ptp4l * Fix: apply musical scales for pwm buzzer test (canonical#40) * Fix: apply musical scales for pwm buzzer test apply musical scales for pwm buzzer test * Fix: added description for test_pwm_buzzer func added description for test_pwm_buzzer function * Fix: add touchscreen tests (canonical#39) * Fix: add touchscreen tests validate the required touch events for touchscreen also add new test to cover maximum fingers support on touchscreen * Change: hardware PTP support keywords For PTP capabilities, ethtool v5.4 (on Ubuntu 20.04) shows `hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)` But ethtool v5.16 (on Ubuntu 22.04) only shows `hardware-transmit` The test case was using `SOF_TIMESTAMPING_TX_HARDWARE` as keywords, This commit changes it to use `hardware-transmit` for adapting new version of ethtool. * Remove: nested ptp-manual test plans The current manual test job steps are duplicated with the auto test job. Currently we only need to run the auto test job. The manual test steps here is used as a reference. If there are new jobs in th future, the ptp-manual test plan can be nested back. * Add: LD_PATH information in readme.md (canonical#41) * Change: additional patterns to map UDC and USB node (canonical#42) * Change: additional patterns to map UDC and USB node * Fix: the if else logic for map UDC * Fix: include python3-systemd in tox.ini (canonical#48) include the python3-systemd package in tox.ini * Fix: updated the related packages for tox (canonical#50) updated the related packages for tox * Fix: nested plan for server manual (canonical#44) * Fix: nested plan for server manual * Fix: errors when mtd count is 0 (canonical#55) * Fix: errors when mtd count is 0 * Add shellcheck ignore 2126 * Add: rpmsg tests for i.MX and TI processors (canonical#46) * Fix: updated the related packages for tox updated the related packages for tox * Add: rpmsg tests for i.MX and TI processors adding rpmsg tests for i.MX and TI processors * Fix: updated rpmsg_tests.py scripts updated rpmsg_tests.py scripts * Fix: fixed bugs fixed bugs * Modify optee test run out of gadget (canonical#45) * Add: support outsourcing optee-test * 230921 crypto test (canonical#52) * Fix: updated the related packages for tox updated the related packages for tox * Fix: update the crypto tests seperate the crypto tests to the generic crypto tests and crypto accelerator tests * Fix: bypass if failed to set extra seed the extra seed might not be supported for all platform * Fix: fixed bugs fixed bugs * Fix: refactor check_crypto_driver_priority func refactor check_crypto_driver_priority function * Fix: mock systemd-python package and remove packages in tox mock systemd-python package and remove packages in tox * Revamp otg_ports resource job should not fail (canonical#53) * Revamp otg_ports resource job should not fail Add a if condition to handle when the OTG config variable is not set. * Revise the echo string * Add: cupfreq module probe (canonical#51) * Add: cupfreq module probe * Check hwrng test (canonical#57) * Modify hwrng test more flexible * Add README.md * Make ce-oem-info jobs be attachment jobs (canonical#58) ce-oem-info jobs are aimed at collecting the system bootup information from snapd and cloud-init to help developers analyze bugs. Therefore, these jobs are more suitable to be attachment jobs and should not fail even if some information can not be found. For example, some systems may not have cloud-init. In these systems, the job cannot gather the information from cloud-init, but this should not be a failure. * Modify the device tree dump job (canonical#59) The current ce-oem-device-tree/dump job does not match the resource job format and will show warnings when running it. The original plan is to parse this job to match the resource format to let any other jobs able to get the dts info conveniently. As parse the dts to resource job format is a huge work, and will not be done in the near future. This commit excludes the ce-oem-device-tree/dump job from the test plan and add comments to explain the purpose. Also keep the attachment job to make sure we could get the device tree information when checking the report. * Revamp caam_hwrng_test by looping w/ smaller size (canonical#60) Some of the project devices' caam_hwrng have buffer, so previously the dd size is increased to 10M to exceed the buffer size to make the interrupt increase. It will make the device without buffer take a very long time (over 30 mins) to do the dd command. Therefore, this commit separated the size to 512K and looped it 20 times. On the device without buffer it could do 512K and detect the increased interrupt to get the pass result, which is much faster. On the device with buttfer it could also exceed the buffer by dd 20 times and detect the increased interrupt to get the pass result. * Fix the nested crypto test plan name (canonical#61) Previously the caam test plan was revamped to accelerator test plan, but forgot to update the name nested in ce-oem-automated. Fix the name in this commit. * Add: new socketcan stress test (canonical#62) * Add: new socketcan tests add new socketcan stress test (100000 loop echo test) add bus-off test and fixed bug * Add: include new socketcan tests (canonical#63) include new socketecan test plan into ce-oem test plans * Add ubuntu-frame and glmark2 tests (canonical#64) * Add ubuntu-frame and glmark2 tests * Modify the way checking ubuntu-frame active * Modify: Add more details in README * Add: implement the RPMSG-tty tests implement the RPMSG-tty tests for i.MX series * Fix: update tox configuration update tox configuration to perform tests with specific Python version * Add: check VPU device tests (canonical#68) * Add: check VPU device tests adding new test cases about identify VPU devices * Fix: fixed the issues from comments fixed the issues from comments * Fix: bug fixed bug fixed * Fix: fixed issues fixed issues * Fix: refactor the thermal tests (canonical#69) * Fix: refactor the thermal tests refactor the thermal tests * Fix: correct the argument name in socketcan_test (canonical#71) corrected the argument name in socketcan_test.py * Add dbus reboot stress (canonical#70) * Add script for installing test snap and connect interfaces * Add reboot via dbus command This test is for snap strict confinement mode * Add tcp test (canonical#65) * Add TCP test * Move ce-oem README inside the provider directory * Change the ce-oem provider namespace to the contrib namespace * Add README for the contrib area * Remove .gitignore from contrib/ Artefact left from the ce-oem provider migration --------- Co-authored-by: stanley31huang <stanley.huang@canonical.com> Co-authored-by: rickwu666666 <98441647+rickwu666666@users.noreply.github.com> Co-authored-by: Nara Huang <nara.huang@canonical.com> Co-authored-by: rickwu4444 <rick.wu@canonical.com> Co-authored-by: baconyao <patrick.chang@canonical.com> Co-authored-by: liaou3 <vincent.liao@canonical.com> Co-authored-by: patliuu <111331153+patliuu@users.noreply.github.com> Co-authored-by: Patrick Liu <patrick.liu@canonical.com> Co-authored-by: hanhsuan <32028620+hanhsuan@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
2.1. For amdgpu, read sysfs entries.
2.2. For non-amdgpu, read the reserved BAR memory.
[Clone from https://code.launchpad.net/~os369510/plainbox-provider-checkbox/+git/plainbox-provider-checkbox/+merge/416966]
Resolved issues
Fixes #191
Documentation
Tests