Skip to content

RSDK-13711 Specify test version#239

Merged
EvanDorsky merged 38 commits into
viamrobotics:mainfrom
EvanDorsky:specify-test-version
May 29, 2026
Merged

RSDK-13711 Specify test version#239
EvanDorsky merged 38 commits into
viamrobotics:mainfrom
EvanDorsky:specify-test-version

Conversation

@EvanDorsky
Copy link
Copy Markdown
Member

@EvanDorsky EvanDorsky commented May 15, 2026

This PR re-engineers the test suite to enable testing against a specified version of viam-agent instead of stable.

Interface/infra changes

  • new config value: viam_agent_test to specify the version under test
  • installAgent now uses the local install.sh (just like uninstallAgent) instead of pulling the install script from gcp.
  • installAgent now checks that viam-agent is running at the specified version, instead of just running at all. If the version isn't correct, it installs the specified version.
  • There is a new way to specify the version in test cases, called "the version under test"
  • The after suite hook now pins agent back to the version under test, instead of stable
  • serialclient.EnsureOnline was re-added to the pre-scenario hook (this fixes some instability caused by the provisioning tests leaving the network adapter in a weird state)
  • new code to retrieve specific versions based on the high level version spec, and gcs url based on specific versions (detailed below)

The agent version under test can be specified in a number of ways:

  • pr.xxx will test a PR by number, if it has the dev-release label (so that artifacts are built) e.g. "pr.227"
  • "dev" will test the tip of main
  • "stable" will test the latest stable release
  • "x.y.z" will test any release version e.g. "0.27.3"
  • "file:///path/to/agent" will test an arbitrary binary (that you manually copy onto the pi)

Each feature test now prints the versions under test before running:

=================================================
=================================================
    Testing Agent Version: dev (0.27.3-dev.8)

          Stable Agent Version: 0.27.3
         Stable Server Version: 0.123.0
=================================================
=================================================

Test changes

Tests have been modified to account for the new syntax requirements to specify versions (like stable, the version under test, an old version):

  • viam-agent is installed -> viam-agent is installed at the version under test
  • the viam-agent systemd unit is running -> the viam-agent systemd unit is running with the version under test

The upgrade and downgrade tests have been changed so they make sense with this new flow:

  • upgrade.feature upgrades from stable to the version under test
  • downgrade.feature downgrades from the version under test to stable

Changes beyond tests

This PR makes a small modification to install.sh - it adds $AGENT_CUSTOM_URL, which will be used instead of $URL as the download target for the agent binary if it $AGENT_CUSTOM_URL is passed in. This should be totally transparent if not used, but it enables the test to install any version (instead of relying on agent to update itself via pinning, which isn't really the same thing).

EvanDorsky and others added 30 commits April 21, 2026 13:50
viamrobotics#232)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Member

@danielbotros danielbotros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much needed improvement, thanks Evan! Have not run tests yet but left some initial comments

Comment thread agent_serial_test.go Outdated
Comment thread agent_serial_test.go Outdated
Comment thread agent_serial_test.go
Comment thread features/serial/agent-reject-viam-server-binary.feature Outdated
Comment thread agent_serial_test.go Outdated
Comment thread agent_serial_test.go Outdated
@cheukt
Copy link
Copy Markdown
Member

cheukt commented May 26, 2026

gonna remove myself as a reviewer

@cheukt cheukt removed their request for review May 26, 2026 14:45
@EvanDorsky EvanDorsky requested a review from danielbotros May 27, 2026 17:57
Copy link
Copy Markdown
Member

@jmatth jmatth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The few comments I left are nitpicks and don't need to be addressed to merge.

Comment thread install.sh
@@ -17,6 +17,9 @@ if [ "$OS" = "Darwin" ]; then
BINARY_OS_PREFIX="darwin-"
fi
URL="https://storage.googleapis.com/packages.viam.com/apps/viam-agent/viam-agent-stable-${BINARY_OS_PREFIX}${ARCH}"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to be cool and don't care about readability you can do this conditional assignment in one line:

Suggested change
URL="https://storage.googleapis.com/packages.viam.com/apps/viam-agent/viam-agent-stable-${BINARY_OS_PREFIX}${ARCH}"
URL="${AGENT_CUSTOM_URL}:-https://storage.googleapis.com/packages.viam.com/apps/viam-agent/viam-agent-stable-${BINARY_OS_PREFIX}${ARCH}}"

Comment thread agent_serial_test.go

// latestStableRelease returns the most recently uploaded stable viam-agent
// release version (bare semver, e.g. "0.27.3").
func latestStableRelease(ctx context.Context) (string, error) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be possible to remove the need to specify the stable + dev versions in the config and just replace them whith these helpers.

Comment thread agent_serial_test.go
//
// Only viam-agent specifiers are supported today; viam-server has different
// upload paths and would need its own resolver.
func resolveVersionSpec(ctx context.Context, spec string) (string, error) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gets called by installAgent, which in turn is called multiple times across the test suite. It would be nice to cache the resolved version to cut down on unnecessary GCS queries but not essential.

Copy link
Copy Markdown
Member

@danielbotros danielbotros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, congrats on finishing agent testing 😄!!

@EvanDorsky EvanDorsky merged commit e790087 into viamrobotics:main May 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants