test-driver.py: delete VM state directory after test run

Keeping the VM state test across several run sometimes lead to subtle
and hard to spot errors in practice. We delete the VM state which
contains (among other things) the qcow volume.

We also introduce a -K (--keep-vm-state) flag making VM state to
persist after the test run. This flag makes test-driver.py to match
its previous behaviour.

nixos/doc/manual/development/running-nixos-tests-interactively.xml

@@ -6,8 +6,7 @@
  <title>Running Tests interactively</title>
 
  <para>
-  The test itself can be run interactively. This is particularly useful when
-  developing or debugging a test:
+  The test itself can be run interactively. This is particularly useful when developing or debugging a test:
 <screen>
 <prompt>$ </prompt>nix-build nixos/tests/login.nix -A driver
 <prompt>$ </prompt>./result/bin/nixos-test-driver
@@ -21,10 +20,7 @@ starting VDE switch for network 1
 <prompt>&gt;</prompt> machine.succeed("touch /tmp/foo")
 <prompt>&gt;</prompt> print(machine.succeed("pwd")) # Show stdout of command
 </screen>
-  The function <command>test_script</command> executes the entire test script
-  and drops you back into the test driver command line upon its completion.
-  This allows you to inspect the state of the VMs after the test (e.g. to debug
-  the test script).
+  The function <command>test_script</command> executes the entire test script and drops you back into the test driver command line upon its completion. This allows you to inspect the state of the VMs after the test (e.g. to debug the test script).
  </para>
 
  <para>
@@ -33,12 +29,14 @@ starting VDE switch for network 1
 <prompt>$ </prompt>nix-build nixos/tests/login.nix -A driver
 <prompt>$ </prompt>./result/bin/nixos-run-vms
 </screen>
-  The script <command>nixos-run-vms</command> starts the virtual machines
-  defined by test.
+  The script <command>nixos-run-vms</command> starts the virtual machines defined by test.
  </para>
 
  <para>
-  The machine state is kept across VM restarts in
-  <filename>/tmp/vm-state-</filename><varname>machinename</varname>.
+  If you can keep the VMs states by using the <command>--keep-vm-state</command> flag.
+<screen>
+<prompt>$ </prompt>./result/bin/nixos-run-vms/ --keep-vm-state
+</screen>
+  The machine state will then be stored in the <filename>/tmp/vm-state-</filename><varname>machinename</varname> directory.
  </para>
 </section>

nixos/doc/manual/release-notes/rl-2009.xml

@@ -637,6 +637,18 @@ systemd.services.nginx.serviceConfig.ReadWritePaths = [ "/var/www" ];
 }</programlisting>
     </para>
    </listitem>
+   <listitem>
+    <para>
+      <package>netbeans</package> was upgraded to 12.0 and now defaults to OpenJDK 11. This might cause problems if your projects depend on packages that were removed in Java 11.
+    </para>
+   </listitem>
+   <listitem>
+     <para>
+       The <literal>nixos-run-vms</literal> script now deletes the machines states by default.
+       You can use the <literal>--keep-vm-state</literal> flag to match the previous behaviour and keep
+       the VM state between different test runs.
+     </para>
+   </listitem>
   </itemizedlist>
  </section>
 </section>

nixos/lib/test-driver/test-driver.py

@@ -4,6 +4,7 @@
 from typing import Tuple, Any, Callable, Dict, Iterator, Optional, List
 from xml.sax.saxutils import XMLGenerator
 import _thread
+import argparse
 import atexit
 import base64
 import codecs
@@ -751,6 +752,10 @@ def process_serial_output() -> None:
 
         self.log("QEMU running (pid {})".format(self.pid))
 
+    def cleanup_statedir(self) -> None:
+        self.log("Delete the VM state directory.")
+        shutil.rmtree(self.state_dir)
+
     def shutdown(self) -> None:
         if not self.booted:
             return
@@ -889,14 +894,22 @@ def subtest(name: str) -> Iterator[None]:
 
 
 if __name__ == "__main__":
+    arg_parser = argparse.ArgumentParser()
+    arg_parser.add_argument(
+        "-K",
+        "--keep-vm-state",
+        help="Do not delete the VM state after the test run",
+        action="store_true",
+    )
+    (cli_args, vm_scripts) = arg_parser.parse_known_args()
+
     log = Logger()
 
     vlan_nrs = list(dict.fromkeys(os.environ.get("VLANS", "").split()))
     vde_sockets = [create_vlan(v) for v in vlan_nrs]
     for nr, vde_socket, _, _ in vde_sockets:
         os.environ["QEMU_VDE_SOCKET_{}".format(nr)] = vde_socket
 
-    vm_scripts = sys.argv[1:]
     machines = [create_machine({"startCommand": s}) for s in vm_scripts]
     machine_eval = [
         "{0} = machines[{1}]".format(m.name, idx) for idx, m in enumerate(machines)
@@ -911,6 +924,8 @@ def clean_up() -> None:
                     continue
                 log.log("killing {} (pid {})".format(machine.name, machine.pid))
                 machine.process.kill()
+                if not cli_args.keep_vm_state:
+                    machine.cleanup_statedir()
 
             for _, _, process, _ in vde_sockets:
                 process.terminate()