Analyses Generation and Snapshot Testing

Setup

1. Follow the instructions in DEV_SETUP.md
2. cd analyses-snapshot-testing
3. use pyenv to install python 3.13 and set it as the local python version for this directory
4. make setup
5. Have docker installed and ready

Concepts

• If working locally the branch you have checked out is the test code/snapshots you are working with.
  • In CI this is the SNAPSHOT_REF. This is the branch or tag of the test code/snapshots that analyses generated will be compared to.
• The ANALYSIS_REF is the branch or tag that you want analyses generated from.

Build the opentrons-analysis image

This ALWAYS gets the remote code pushed to Opentrons/opentrons for the specified ANALYSIS_REF

• build the base image
  • make build-base-image
• build the opentrons-analysis image
  • make build-opentrons-analysis ANALYSIS_REF=release

Running the tests locally

• Compare the current branch snapshots to analyses generated from the edge branch
  • make build-opentrons-analysis ANALYSIS_REF=edge this builds a docker image named and tagged opentrons-analysis:edge
    • this pulls the latest edge every time it builds!
  • make snapshot-test ANALYSIS_REF=edge
    • This runs the test. The test:
      • Spins up a container from the opentrons-analysis:edge image. ANALYSIS_REF=edge specifies the image to use.
      • Analyses as .json files are generated for all protocols defined in protocols.py and protocols_with_overrides.py
      • the test compares the generated analyses to the snapshots in the ./tests/snapshots/ directory

Updating the snapshots

• Assuming you have already built the opentrons-analysis:edge image
• make snapshot-test-update ANALYSIS_REF=edge
  • This will update the snapshots in the ./tests/snapshots/ directory with the analyses generated from the edge branch

Running the tests against specific protocols

We are omitting ANALYSIS_REF=edge because we can, it is the default in the Makefile

• make snapshot-test PROTOCOL_NAMES=Flex_S_v2_19_Illumina_DNA_PCR_Free OVERRIDE_PROTOCOL_NAMES=none
• make snapshot-test PROTOCOL_NAMES=none OVERRIDE_PROTOCOL_NAMES=Flex_X_v2_18_NO_PIPETTES_Overrides_BadTypesInRTP
• make snapshot-test PROTOCOL_NAMES="Flex_S_v2_19_Illumina_DNA_PCR_Free,OT2_S_v2_18_P300M_P20S_HS_TC_TM_SmokeTestV3" OVERRIDE_PROTOCOL_NAMES=none

Running a Flex just like make -C robot-server dev-flex

This ALWAYS gets the remote code pushed to Opentrons/opentrons for the specified OPENTRONS_VERSION

cd analyses-snapshot-testing \
&& make build-base-image \
&& make build-rs OPENTRONS_VERSION=release \
&& make run-rs OPENTRONS_VERSION=release`

Default OPENTRONS_VERSION=edge in the Makefile so you can omit it if you want latest edge

cd analyses-snapshot-testing \
&& make build-base-image \
&& make build-rs \
&& make run-rs

Running the Analyses Battery against your local code

This copies in your local code to the container and runs the analyses battery against it.

cd PYENV_ROOT && git pull - make sure pyenv is up to date so you may install python 3.13.0 pyenv install 3.13.0 - install python 3.13.0 cd <OPENTRONS_REPO_ROOT>/analyses-snapshot-testing - navigate to the analyses-snapshot-testing directory pyenv local 3.13.0 - set the local python version to 3.13.0 make setup - install the requirements make snapshot-test-local - this target builds the base image, builds the local code into the base image, then runs the analyses battery against the image you just created

You have the option to specify one or many protocols to run the analyses on. This is also described above Running the tests against specific protocols

• make snapshot-test-local PROTOCOL_NAMES=Flex_S_v2_19_Illumina_DNA_PCR_Free OVERRIDE_PROTOCOL_NAMES=none

Updating the snapshots locally

• make snapshot-test-update-local - this target builds the base image, builds the local code into the base image, then runs the analyses battery against the image you just created, updating the snapshots by passing the --update-snapshots flag to the test

Add some protocols to the analyses battery

The below instructions avoid needing docker and executing snapshot tests locally.

1. create new protocol file(s) in the files/protocols directory following the naming convention in files/README.md
2. add the protocol(s) to the protocols.py
3. make format (make sure you have followed setup instructions)
4. commit and push your branch
5. open a PR and add the label gen-analyses-snapshot-pr
6. when the snapshot fails because your new protocols don't have snapshots a PR will be created that heals.
7. merge the healing PR if the snapshots are as expected
8. get a review and merge! 🎉 now your protocols are a part of the test

Add a protocol with overrides to the analyses battery

TODO when we have a more straight forward example