Initial (bare-bones) implementation of the infrastructure for end-to-end tests #2691

narrieta · 2022-10-25T02:36:33Z

This is the very first basic implementation of the end-to-end tests, based in the POC replacement for DCR under the 'dcr' subdirectory.

At this point, it simply executes 2 skeleton tests using the LISA framework. Additional functionality will be implemented in future PRs. The present PR starts defining patterns for implementing end-to-end tests.

narrieta · 2022-10-25T02:37:13Z

tests_e2e/azure-pipelines.yml

@@ -0,0 +1,21 @@
+variables:


Entry point for Azure Pipelines

narrieta · 2022-10-25T02:38:40Z

tests_e2e/lisa/runbook/azure.yml

@@ -0,0 +1,82 @@
+name: azure


Currently, the interface with LISA is under the 'lisa' subdirectory (it will be moved one level up in a future PR). This YML file is the entry point to LISA.

narrieta · 2022-10-25T02:40:53Z

tests_e2e/lisa/tests/agent_bvt/check_agent_version.py

@@ -0,0 +1,23 @@
+#!/usr/bin/env python


Skeleton for the current scenarios\agent-bvt\01_check_waagent_version.py test in DCR

narrieta · 2022-10-25T02:41:27Z

tests_e2e/lisa/tests/agent_bvt/custom_script.py

@@ -0,0 +1,45 @@
+import argparse


Skeleton for the current scenarios\agent-bvt\06_customscript_20.py test in DCR

narrieta · 2022-10-25T02:42:18Z

tests_e2e/lisa/testsuites/agent-bvt.py

@@ -0,0 +1,41 @@
+from assertpy import assert_that


Skeleton to define the 2 basic patterns for the end-to-end tests. Currently hardcoded to specific tests, but will be abstracted in future PRs.

codecov · 2022-10-25T02:42:24Z

Codecov Report

Merging #2691 (37f03a1) into develop (527443c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #2691   +/-   ##
========================================
  Coverage    72.06%   72.06%           
========================================
  Files          103      103           
  Lines        15702    15702           
  Branches      2237     2237           
========================================
  Hits         11315    11315           
  Misses        3872     3872           
  Partials       515      515

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

narrieta · 2022-10-25T02:43:18Z

tests_e2e/lisa/testsuites/agent-bvt.py

+)
+class AgentBvt(TestSuite):
+    @TestCaseMetadata(description="", priority=0)
+    def check_agent_version(self, node: Node, log: Logger) -> None:


Pattern 1: Execute a test remotely over SSH

narrieta · 2022-10-25T02:44:04Z

tests_e2e/lisa/testsuites/agent-bvt.py

+        assert_that(result.exit_code).is_equal_to(0)
+
+    @TestCaseMetadata(description="", priority=0)
+    def custom_script(self, node: Node) -> None:


Pattern 2: Execute a test locally (local to the orchestrator) to trigger extensions on the remote test VM

At what point is the remote test VM created?

LISA does that. In the output of the task you will see something similar to

2022-10-25 15:13:49.326[140692391196416][INFO] lisa.[azure].deploy[generated_0] vm setting: AzureNodeSchema(name='lisa-azure-20221025-151340-053-e0-n0', short_name='lisa--053-e0-n0', vm_size='Standard_DS2_v2', maximize_capability=False, location='westus2', subscription_id='8e037ad4-****-****-****-********c15b', marketplace_raw={'publisher': 'canonical', 'offer': 'ubuntuserver', 'sku': '18.04-lts', 'version': '18.04.202210180'}, shared_gallery_raw=None, vhd='', hyperv_generation=1, purchase_plan=None, is_linux=True) 2022-10-25 15:13:53.044[140692391196416][INFO] lisa.[azure].deploy[generated_0] resource group 'lisa-azure-20221025-151340-053-e0' deployment is in progress... 2022-10-25 15:15:26.749[140692391196416][INFO] lisa.[azure].deploy[generated_0] deployed in 99.972 sec 2022-10-25 15:15:26.754[140692483270400][INFO] lisa.env[generated_0].node[0] initializing node 'lisa-azure-20221025-151340-053-e0-n0' w****t@20.230.174.145:22

narrieta · 2022-10-25T02:45:16Z

tests_e2e/lisa/testsuites/agent-bvt.py

+    @TestCaseMetadata(description="", priority=0)
+    def custom_script(self, node: Node) -> None:
+        node_context = get_node_context(node)
+        subscription_id = node.features._platform.subscription_id


Currently reaching into the internal "_platform" object to get the Subscription ID. This should be done by implementing a LISA feature that exposes subscription ID/resource group name/vm name.

Once we are creating multiple test VMs with different images, how will LISA know which nodes this test should be ran with? Will that be specified in the runbook?

Yes, we will create a combinator to execute each test on each distro and add it to the runbook

I believe the node details that it's trying to fetch for the actual test vm not the orchestrator vm right?

yes, the Node represents the VM created by LISA

narrieta · 2022-10-25T02:46:22Z

tests_e2e/requirements.txt

@@ -0,0 +1,13 @@
+# This is a list of pip packages that will be installed on both the orchestrator and the test VM


Requirements for the orchestrator/test VM. Comes directly from the 'dcr' POC, but should be splitted in 2 (one for orchestrator, one for test VM)

narrieta · 2022-10-25T02:48:47Z

tests_e2e/scenario_utils/agent_log_parser.py

@@ -0,0 +1,99 @@
+from __future__ import print_function


All the files under the "scenario_utils" provide infrastructure to execute the tests. They come directly from the 'dcr' POC, which in turn copied them from the DCR repo.

These modules require significant improvements, specially around error handling and diagnostic logging. These improvements will come on future PRs. Feel free to skip those in this initial review.

narrieta · 2022-10-25T03:12:33Z

tests_e2e/scenario_utils/azure_models.py

@@ -0,0 +1,239 @@
+import time


The files under 'scenario_utils' come straight from the 'dcr' POC, which in turn copied them from the current DCR implementation. They need significant improvements on error handling and diagnostics logging, feel free to skip them for this review.

maddieford

Left some comments for my own understanding

maddieford · 2022-10-25T17:52:20Z

tests_e2e/templates/execute-tests.yml

+        name: sshKey
+        displayName: "Download SSH key"
+        inputs:
+          secureFile: 'id_rsa'


Where does this task download the secureFile 'id_rsa' from?

The Azure Pipeline includes a Library where one uploads secure files to, and then this task downloads it from there.

BTW, a nice feature of these tasks is that their output in the pipeline includes a link to the documentation for the task. e.g.

============================================================================== Task : Download secure file Description : Download a secure file to the agent machine Version : 1.200.0 Author : Microsoft Corporation Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/download-secure-file ==============================================================================

maddieford · 2022-10-25T18:29:33Z

tests_e2e/lisa/runbook/azure.yml

@@ -0,0 +1,82 @@
+name: azure
+extension:
+  - "../testsuites"


Is this the directory all of the LISA tests need to be in? Or will LISA look for tests in the entire /lisa directory

The LISA runbook points to the test location

extension: - "../testsuites"

maddieford · 2022-10-25T18:40:41Z

tests_e2e/lisa/testsuites/agent-bvt.py

+        assert_that(result.exit_code).is_equal_to(0)
+
+    @TestCaseMetadata(description="", priority=0)
+    def custom_script(self, node: Node) -> None:


At what point is the remote test VM created?

maddieford · 2022-10-25T18:45:43Z

tests_e2e/lisa/testsuites/agent-bvt.py

+    @TestCaseMetadata(description="", priority=0)
+    def custom_script(self, node: Node) -> None:
+        node_context = get_node_context(node)
+        subscription_id = node.features._platform.subscription_id


Once we are creating multiple test VMs with different images, how will LISA know which nodes this test should be ran with? Will that be specified in the runbook?

maddieford · 2022-10-25T18:56:56Z

tests_e2e/lisa/tests/agent_bvt/custom_script.py

+        cse.get_ext_props(settings={'commandToExecute': "echo \'Hello again\'"})
+    ]
+
+    cse.run(ext_props=ext_props)


I noticed all of the logging from cse.run is being logged as lisa.stderr. Why is lisa capturing these as stderr instead of stdout?

The output is coming from BaseExtensionTestClass (in the scenatio_utils directory). This code comes straight from the current DCR and will need a lot of improvements, for example writing to the log file instead of stdout/stderr.

nagworld9

Looks like some of the places we pinned/hardcoded, may need abstraction. I believe you would change in later PRs.

nagworld9 · 2022-10-26T15:39:36Z

tests_e2e/lisa/testsuites/agent-bvt.py

+    @TestCaseMetadata(description="", priority=0)
+    def custom_script(self, node: Node) -> None:
+        node_context = get_node_context(node)
+        subscription_id = node.features._platform.subscription_id


I believe the node details that it's trying to fetch for the actual test vm not the orchestrator vm right?

nagworld9 · 2022-10-26T15:43:24Z

tests_e2e/lisa/tests/agent_bvt/custom_script.py

+
+    cse = CustomScriptExtension(extension_name="testCSE")
+
+    ext_props = [


why do we rewrite here? it already there in add_cse() function in Customerscript extension. Can we reuse that to run it?

add_cse() does more than I need for this prototype (slowing the execution down), and, in any case, we will need to do a major rewrite of the BaseExtensionTest class we brought from DCR. In my oipinion the abstraction it presents is not appropriate and, more importantly, it handles errors poorly and does not output enough diagnostic info to debug failures.

nagworld9 · 2022-10-26T15:48:36Z

tests_e2e/scripts/execute_tests.sh

+cd $BUILD_SOURCESDIRECTORY/lisa
+
+# LISA needs both the public and private keys; generate the former
+chmod 700 $SSHKEY_SECUREFILEPATH


are these some sort of environment vars?

Yes, they are defined by the tasks in the Azure pipeline. Many tasks expose their output/parameters as environment variables (those variables are defined in the documentation for each task).

In this case, SSHKEY_SECUREFILEPATH comes from the "sshkey" task that downloads the private ssh key. The task appends the path for the download to the name and capitalizes, hence the variable name.

BTW, the DCR pipeline requires you to add the sshPublicKey environment variable when you create a new instance of the pipeline. Now we generate the public key from the private key at run time (as an alternative we could also have uploaded it as a secure file and then add another task to download it, but that does not seem to maintain the mode of the file, which need to be 700, so I just did it as code.)

nagworld9 · 2022-10-26T16:04:06Z

tests_e2e/lisa/runbook/azure.yml

+  - name: vm_name
+    value: ""
+  - name: marketplace_image
+    value: "Canonical UbuntuServer 18.04-LTS latest"


looks like this is not in urn format. Is this how we suppose to define images?
How can we pass multiple images? how each scenario override this?

We'll define a combinator to pass multiple distros. The LISA documentation has some info on how to do that, but we can always work with them if we need more info or features.

nagworld9 · 2022-10-26T16:05:40Z

tests_e2e/azure-pipelines.yml

+pr: none
+
+pool:
+  vmImage: ubuntu-latest


is this pool for orchestrator vms?

Yes, that is the pool of Azure Pipelines VMs where we run the orchestrator (singular, now we have only 1 orchestrator)

you mean you plan to keep only 1 for all scenarios or at this point we have one?

1 at this point. the actual number will depend on how we organize the test runs and how much we can take advantage of LISA's capabilities.

narrieta added 6 commits October 18, 2022 15:32

Prototype for end-to-end tests

9c72135

Add agent BVT

00aa11f

Cleanup SSH key; delete cleanup pipeline

14be306

Delete unused references

4e1da04

Delete unused references - Part 2

998e7a1

Remove local configuration

4c1ecc4

narrieta requested review from ZhidongPeng, nagworld9 and maddieford as code owners October 25, 2022 02:36

narrieta commented Oct 25, 2022

View reviewed changes

tests_e2e/azure-pipelines.yml

@@ -0,0 +1,21 @@

variables:

Copy link

Member Author

narrieta Oct 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Entry point for Azure Pipelines

narrieta commented Oct 25, 2022

View reviewed changes

Remove unused references, Part 3

147e1fb

narrieta commented Oct 25, 2022

View reviewed changes

Disable proxy

37f03a1

maddieford approved these changes Oct 25, 2022

View reviewed changes

nagworld9 reviewed Oct 26, 2022

View reviewed changes

narrieta merged commit a1f3049 into Azure:develop Oct 26, 2022

narrieta deleted the tests-e2e branch October 26, 2022 16:25

		@@ -0,0 +1,13 @@
		# This is a list of pip packages that will be installed on both the orchestrator and the test VM


		cse = CustomScriptExtension(extension_name="testCSE")

		ext_props = [

Initial (bare-bones) implementation of the infrastructure for end-to-end tests #2691

Initial (bare-bones) implementation of the infrastructure for end-to-end tests #2691

Conversation

narrieta commented Oct 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Oct 25, 2022 • edited

Choose a reason for hiding this comment

codecov bot commented Oct 25, 2022 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maddieford left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagworld9 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Oct 26, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagworld9 Oct 26, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Oct 25, 2022 •

edited

codecov bot commented Oct 25, 2022 •

edited

narrieta Oct 26, 2022 •

edited

nagworld9 Oct 26, 2022 •

edited