Run faucet integration tests in CI

.github/actions/run-faucet-tests/action.yaml

@@ -0,0 +1,50 @@
+name: "Faucet Integration Tests"
+description: |
+  Runs the tests for the Aptos faucet against a local testnet built from a particular release branch
+inputs:
+  NETWORK:
+    description: "The network branch for running the local testnet: devnet or testnet."
+    required: true
+
+runs:
+  using: composite
+  steps:
+    # Create the bindmount directory.
+    - name: Create bindmount directory
+      run: mkdir -p ${{ runner.temp }}/testnet
+      shell: bash
+
+    # Run a Redis server.
+    - name: Run Redis server
+      uses: shogo82148/actions-setup-redis@v1
+      with:
+        redis-version: '6.x'
+
+    # Set up Rust for running the integration tests.
+    - name: Set up Rust
+      uses: aptos-labs/aptos-core/.github/actions/rust-setup@main
+
+    # Install Poetry.
+    - uses: snok/install-poetry@v1
+      with:
+        version: 1.2.2
+
+    # Install the script dependencies.
+    - name: Install script dependencies
+      working-directory: crates/aptos-faucet/integration-tests
+      shell: bash
+      run: poetry install
+
+    # Run the faucet integration tests. This script will handle starting the local
+    # testnet, moving the mint key where the tests expect it to be, and running the
+    # integration tests.
+    - name: Run integration tests
+      run: poetry run python main.py --base-network ${{ inputs.NETWORK }} --external-test-dir ${{ runner.temp }}/testnet
+      working-directory: crates/aptos-faucet/integration-tests
+      shell: bash
+
+    # Print the logs from the local testnet if the tests failed.
+    - name: Print local testnet logs if something failed
+      run: docker logs local-testnet-${{ inputs.NETWORK }}
+      shell: bash
+      if: ${{ failure() }}

.github/workflows/faucet-tests.yaml

@@ -0,0 +1,24 @@
+name: "Faucet Integration Tests"
+on:
+  pull_request:
+  push:
+    branches:
+      - devnet
+      - testnet
+
+jobs:
+  run-tests-devnet:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      # - uses: aptos-labs/aptos-core/.github/actions/run-faucet-tests@main
+      - uses: ./.github/actions/run-faucet-tests
+        with:
+          NETWORK: devnet
+  run-tests-testnet:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/run-faucet-tests
+        with:
+          NETWORK: testnet

crates/aptos-faucet/README.md

@@ -64,28 +64,7 @@ cargo build
 ```
 
 ## Testing
-If you want to run the tests manually, follow these steps. Note that this is **not necessary** for release safety as the tests are run as part of continuous integration (CI) already.
-
-For the test suite to pass, you must spin up a local testnet, notably without a faucet running (since we're testing the faucet here):
-```
-cargo run -p aptos -- node run-local-testnet --force-restart --assume-yes
-```
-
-You must then copy the mint key for that local testnet to the location the tests expect:
-```
-cp ~/.aptos/testnet/mint.key /tmp
-```
-
-As well as spin up a local Redis 6 ([installation guide](https://redis.io/docs/getting-started/)):
-```
-redis-server
-redis-cli flushall
-```
-
-Finally you can run the tests:
-```
-cargo test -p aptos-faucet-core --features integration-tests
-```
+If you want to run the tests manually, follow the steps in [integration-tests/README.md](integration-tests/README.md). Note that this is **not necessary** for release safety as the tests are run as part of continuous integration (CI) already.
 
 ## Validating configs
 To ensure all the configs are valid, run this:

crates/aptos-faucet/integration-tests/README.md

@@ -0,0 +1,37 @@
+# Faucet integration tests
+This directory contains Python code to help with running the faucet integration tests. It takes care of spinning up a local testnet, moving the mint key where it is expected, checking that a Redis server is up, and running the integration tests.
+
+## Requirements
+We use [Poetry](https://python-poetry.org/docs/#installation) for packaging and dependency management:
+
+```
+curl -sSL https://install.python-poetry.org | python3 -
+```
+
+Once you have Poetry, you can install the dependencies for the testing framework like this:
+```
+poetry install
+```
+
+## Running
+First, run a local Redis 6 server ([installation guide](https://redis.io/docs/getting-started/)).
+```
+redis-server
+redis-cli flushall
+```
+
+To learn how to use the testing framework, run this:
+```
+poetry run python main.py -h
+```
+
+For example:
+```
+poetry run python main.py --base-network mainnet
+```
+
+## Formatting:
+```
+poetry run isort .
+poetry run black .
+```

crates/aptos-faucet/integration-tests/common.py

@@ -0,0 +1,18 @@
+# Copyright © Aptos Foundation
+# SPDX-License-Identifier: Apache-2.0
+
+from enum import Enum
+
+NODE_PORT = 8080
+
+
+class Network(Enum):
+    DEVNET = "devnet"
+    TESTNET = "testnet"
+
+    def __str__(self):
+        return self.value
+
+
+def build_image_name(image_repo_with_project: str, tag: str):
+    return f"{image_repo_with_project}/tools:{tag}"

crates/aptos-faucet/integration-tests/local_testnet.py

@@ -0,0 +1,81 @@
+# Copyright © Aptos Foundation
+# SPDX-License-Identifier: Apache-2.0
+
+# This file contains functions for running the local testnet.
+
+import logging
+import subprocess
+import time
+
+import requests
+from common import NODE_PORT, Network, build_image_name
+
+LOG = logging.getLogger(__name__)
+
+# Run a local testnet in a docker container. We choose to detach here and we'll
+# stop running it later using the container name. For an explanation of these
+# arguments, see the argument parser in main.py.
+def run_node(network: Network, image_repo_with_project: str, external_test_dir: str):
+    image_name = build_image_name(image_repo_with_project, network)
+    container_name = f"local-testnet-{network}"
+    internal_mount_path = "/mymount"
+    LOG.info(f"Trying to run local testnet from image: {image_name}")
+
+    # First delete the existing container if there is one with the same name.
+    subprocess.run(
+        ["docker", "rm", "-f", container_name],
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+    )
+
+    # Run the container.
+    subprocess.check_output(
+        [
+            "docker",
+            "run",
+            "--name",
+            container_name,
+            "--detach",
+            # Expose the API port.
+            "-p",
+            f"{NODE_PORT}:{NODE_PORT}",
+            # Mount the external test directory into the container.
+            "-v",
+            f"{external_test_dir}:{internal_mount_path}",
+            image_name,
+            "aptos",
+            "node",
+            "run-local-testnet",
+            "--test-dir",
+            internal_mount_path,
+        ],
+    )
+    LOG.info(f"Running local testnet from image: {image_name}")
+    return container_name
+
+
+# Stop running the detached node.
+def stop_node(container_name: str):
+    LOG.info(f"Stopping container: {container_name}")
+    subprocess.check_output(["docker", "stop", container_name])
+    LOG.info(f"Stopped container: {container_name}")
+
+
+# Query the node until the API comes up, or we timeout.
+def wait_for_startup(container_name: str, timeout: int):
+    LOG.info(f"Waiting for node API for {container_name} to come up")
+    count = 0
+    api_response = None
+    while True:
+        try:
+            api_response = requests.get(f"http://127.0.0.1:{NODE_PORT}/v1")
+            if api_response.status_code != 200:
+                raise RuntimeError(f"API not ready. API response: {api_response}")
+            break
+        except Exception:
+            if count >= timeout:
+                LOG.error(f"Timeout while waiting for node to come up")
+                raise
+            count += 1
+            time.sleep(1)
+    LOG.info(f"Node API for {container_name} came up")

crates/aptos-faucet/integration-tests/main.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+# Copyright © Aptos Foundation
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+This script is how we orchestrate running a local testnet and then the faucet
+integration tests against it.
+
+Example invocation:
+  python3 main.py --local-testnet-network devnet
+
+This would run a local testnet built from the devnet release branch and then run the
+faucet integration tests against it.
+
+The script confirms that pre-existing conditions are suitable, e.g. checking that a
+Redis instance is alive.
+"""
+
+import argparse
+import logging
+import os
+import shutil
+import sys
+
+from common import Network
+from local_testnet import run_node, stop_node, wait_for_startup
+from prechecks import check_redis_is_running
+from tests import run_faucet_integration_tests
+
+logging.basicConfig(
+    stream=sys.stderr,
+    format="%(asctime)s - %(levelname)s - %(message)s",
+    level=logging.INFO,
+)
+
+LOG = logging.getLogger(__name__)
+
+
+def parse_args():
+    # You'll notice there are two argument "prefixes", base and test. These refer to
+    # cases 1 and 2 in the top-level comment.
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        description=__doc__,
+    )
+    parser.add_argument("-d", "--debug", action="store_true")
+    parser.add_argument(
+        "--image-repo-with-project",
+        default="aptoslabs",
+        help=(
+            "What docker image repo (+ project) to use for the local testnet. "
+            "By default we use Docker Hub: %(default)s (so, just aptoslabs for the "
+            "project since Docker Hub is the implied default repo). If you want to "
+            "specify a different repo, it might look like this: "
+            "docker.pkg.github.com/aptoslabs/aptos-core"
+        ),
+    )
+    parser.add_argument(
+        "--base-network",
+        required=True,
+        type=Network,
+        choices=list(Network),
+        help="What branch the Aptos CLI used for the local testnet should be built from",
+    )
+    parser.add_argument(
+        "--base-startup-timeout",
+        type=int,
+        default=30,
+        help="Timeout in seconds for waiting for node and faucet to start up",
+    )
+    parser.add_argument(
+        "--external-test-dir",
+        default="/tmp/testnet",
+        help="Where to mount the test dir that the node is writing to",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def main():
+    args = parse_args()
+
+    if args.debug:
+        logging.getLogger().setLevel(logging.DEBUG)
+        LOG.debug("Debug logging enabled")
+    else:
+        logging.getLogger().setLevel(logging.INFO)
+
+    # Verify that a local Redis instance is running. This is just a basic check that
+    # something is listening at the expected port.
+    check_redis_is_running()
+
+    # Run a node and wait for it to start up.
+    container_name = run_node(
+        args.base_network, args.image_repo_with_project, args.external_test_dir
+    )
+    wait_for_startup(container_name, args.base_startup_timeout)
+
+    # Copy the mint key from the node to where the integration tests expect it to be.
+    copy_mint_key(args.external_test_dir)
+
+    # Build and run the faucet integration tests.
+    run_faucet_integration_tests()
+
+    # Stop the local testnet.
+    stop_node(container_name)
+
+    return True
+
+
+def copy_mint_key(external_test_dir: str):
+    key_name = "mint.key"
+    source_path = os.path.join(external_test_dir, key_name)
+    new_path = os.path.join("/tmp", key_name)
+    try:
+        shutil.copyfile(source_path, new_path)
+    except FileNotFoundError as e:
+        raise RuntimeError(
+            f"Could not find mint key at expected source path: {source_path}"
+        ) from e
+    LOG.info(
+        f"Copied mint key from {source_path} to the path the integration "
+        f"tests expect: {new_path}"
+    )
+
+
+if __name__ == "__main__":
+    if main():
+        sys.exit(0)
+    else:
+        sys.exit(1)