[WIP] Simulation Overhaul #742

amquake · 2023-01-15T10:58:23Z

Putting this (living) draft up for feedback and testing. Some of the videos on this repo's readme show this simulation in action. This is all currently just from my perspective, and there's still lots more that could be added, so I would appreciate opinions on anything and everything.

What does this do?

Deprecates previous sim classes
Has a CameraProperties class for describing a camera's basic/calibration info, and performance values for simulation. Calibration values can be loaded from the config.json in the settings exported by photonvision.
OpenCVHelp provides convenience functions for using opencv methods with wpilib/photonvision classes, mainly to project 3d points to a camera's 2d image and perform solvePnP with the above camera calibration info.
- TargetModels describe the 3d shape of a target, both for projecting into the camera's 2d image and use in solvePnP.
PhotonCameraSim uses camera properties to simulate how 3d targets would appear in its view, and has simulated noise, latency, and FPS. For apriltags, the best/alternate camera-to-target transform is also estimated with solvePnP.
- VideoSimUtil has helper functions for drawing apriltags to a simulated raw and processed MJPEG stream for each camera using the projected tag corners.
VisionSystemSim stores VisionTargetSims and PhotonCameraSims, and is periodically updated with the robot's simulated pose. When updating, camera sims are automatically processed and published with their visible targets from their respective poses with proper latency.

How to test this

Clone this PR
Follow the docs for using photon-lib builds to build and add this PR to your robot project

Possible Additions / Known Issues

Latency is not used correctly for moving vision targets because their poses are not saved over time.
Differentiate camera pipelines
- Only view certain types of vision targets?
- Change camera properties?
Video stream simulation: should on or off by default?
"Raw" video stream apriltag image warping is messy
- Better antialiasing to work-around poor opencv warpPerspective results
- Warp the entire tag image instead of just onto the detected corners
Expand target support
- Get corners(?) / area of spherical targets
- Perform solvePnP on non-apriltag targets
- Draw non-apriltag targets to video stream
Allow processing of simulated video frames directly with wpilib apriltag bindings
- Simulate actual camera image noise instead of random gaussian corner movement
- Simulate motion blur
Draw field wireframe to video stream sim

Things that should be done to finish this PR

Documentation additions/changes
Naming/organization review
More test coverage
API changes based on user testing
C++ :)
Update sim examples

from https://github.com/amquake/AprilTagWorkbench

…into sim-overhaul

photon-lib/src/main/java/org/photonvision/VisionSystemSim.java

photon-lib/src/main/java/org/photonvision/util/VideoSimUtil.java

photon-lib/src/main/java/org/photonvision/PhotonCameraSim.java

photon-lib/src/main/java/org/photonvision/util/RotTrlTransform3d.java

…into sim-overhaul oops

srimanachanta · 2023-06-11T18:07:53Z

All SIM-related content should be moved into a simulation folder following wpilib, also, with deprecated classes, it will be crowded

@mcm001 @amquake Thoughts on this?

srimanachanta · 2023-06-11T19:08:58Z

photon-lib/src/main/java/org/photonvision/simulation/VideoSimUtil.java

+        if (result.empty()) {
+            BufferedImage buf;
+            try {
+                buf = ImageIO.read(resource);


no null check here? Is this assumed to be non-null based on the previous check?

if you mean result, it should always be at least an empty Mat. If you mean buf, I believe it should throw an exception when it can't read the desired image.

I mean resource, you bill check before but not beyond that if statement

If resource is null, ImageIO.read will throw an exception.

photon-lib/src/main/java/org/photonvision/simulation/VideoSimUtil.java

srimanachanta · 2023-06-11T19:12:11Z

photon-lib/src/main/java/org/photonvision/simulation/VisionSystemSim.java

+            addVisionTargets(
+                    "apriltags",
+                    new VisionTargetSim(
+                            tagLayout.getTagPose(tag.ID).get(), // preserve alliance rotation


Check if tag is present, getTagPose returns an optional

We are iterating through the tags that the tag layout reports through getTags() and then redundantly getting their pose from the layout again (for handling different alliance origins), so this should not be empty.

srimanachanta · 2023-06-11T19:13:27Z

photon-lib/src/main/java/org/photonvision/simulation/VisionSystemSim.java

+
+            // use camera pose from the image capture timestamp
+            Pose3d lateRobotPose = getRobotPose(timestampCapture);
+            Pose3d lateCameraPose = lateRobotPose.plus(getRobotToCamera(camSim).get());


getRobotToCamera returns optional, camSim confirmed to be non-empty?

When adding a camera sim to the vision system sim, it is ensured that there is a transform. The only way for the getRobotToCamera method to return empty should be if the given camera sim does not exist in the vision system sim. In this case, we are iterating through the camera sims in this vision system sim.

See github for things I thought warranted further discussion

amquake · 2023-06-11T23:38:48Z

All SIM-related content should be moved into a simulation folder following wpilib, also, with deprecated classes, it will be crowded

@mcm001 @amquake Thoughts on this?

Does moving the deprecated sim classes into the org.photonvision.simulation package break team's code?

mcm001 · 2023-06-18T22:18:24Z

Does moving the deprecated sim classes into the org.photonvision.simulation package break team's code?

Yes, but I think that it's OK as long as we add a note to the docs?

- `PNPResults` can now be empty (`isPresent` = false) - solvePNP methods actually handle errors and return empty `PNPResults` - This reveals an odd error where some inputs to `solvePNP_SQUARE()` resulted in an estimated transform with NaN values, and attempts to handle it - Overwrites java changes from #817 since #742 had duplicate fixes - Minor bugfixes

amquake and others added 18 commits January 7, 2023 01:45

initial migration

d87a241

from https://github.com/amquake/AprilTagWorkbench

hopefully match corner order

17ed37e

gradle deps

3674ca9

gradle test libraries

663d24c

gradle test libraries?

e9d0bfa

Fix JNI loading bugs

b29cae3

Merge branch 'master' of https://github.com/PhotonVision/photonvision …

6235655

…into sim-overhaul

tests

a0ff3d8

Merge branch 'master' of https://github.com/PhotonVision/photonvision …

308c800

…into sim-overhaul

runtime load opencv

562fa66

load images from jar

f831e72

target sort order

ae8f306

better latency timestamp calc

153f1f4

Merge branch 'master' of https://github.com/PhotonVision/photonvision …

b42eec9

…into sim-overhaul

restore previous sim classes as deprecated

6032a7b

class docs

82fba01

minor quality

46af425

update default cam props

2b4dc67

mcm001 reviewed Jan 15, 2023

View reviewed changes

photon-lib/src/main/java/org/photonvision/VisionSystemSim.java Outdated Show resolved Hide resolved

mcm001 reviewed Jan 15, 2023

View reviewed changes

photon-lib/src/main/java/org/photonvision/VisionSystemSim.java Outdated Show resolved Hide resolved

mcm001 reviewed Jan 15, 2023

View reviewed changes

photon-lib/src/main/java/org/photonvision/util/VideoSimUtil.java Outdated Show resolved Hide resolved

mcm001 reviewed Jan 15, 2023

View reviewed changes

photon-lib/src/main/java/org/photonvision/PhotonCameraSim.java Outdated Show resolved Hide resolved

mcm001 added 2 commits January 15, 2023 10:31

Bump to wpilib 2023.2.1 (PhotonVision#741)

2934dbd

Update apriltagexample to new sim classes

6572d41

calcmogul reviewed Jan 15, 2023

View reviewed changes

photon-lib/src/main/java/org/photonvision/util/RotTrlTransform3d.java Outdated Show resolved Hide resolved

amquake and others added 5 commits January 16, 2023 21:19

Update .gitignore

b08c486

set NT publish timestamp instead of overriding

ccbbba2

spotless

4acd7b6

Spotless part 2

8d9f23c

Merge branch 'master' into sim-overhaul

0b16ea3

amquake and others added 4 commits May 22, 2023 04:11

Merge branch 'master' of https://github.com/PhotonVision/photonvision …

7cc29aa

…into sim-overhaul oops

javadoc import

877d31c

Merge branch 'master' into sim-overhaul

adfdd45

Formatting fixes

9f126a6

mcm001 marked this pull request as ready for review June 11, 2023 17:57

mcm001 requested a review from a team as a code owner June 11, 2023 17:57

run format & linting

5146257

mcm001 added 2 commits June 11, 2023 11:08

Fix compile errors

a721d62

Refactor stuff

ad4d300

srimanachanta reviewed Jun 11, 2023

View reviewed changes

photon-lib/src/main/java/org/photonvision/simulation/VideoSimUtil.java Show resolved Hide resolved

srimanachanta reviewed Jun 11, 2023

View reviewed changes

photon-lib/src/main/java/org/photonvision/simulation/VideoSimUtil.java Show resolved Hide resolved

srimanachanta reviewed Jun 11, 2023

View reviewed changes

srimanachanta and others added 4 commits June 11, 2023 15:19

First pass things

fd67a3a

See github for things I thought warranted further discussion

Grammar

56f041c

Formatting fixes

354f2e4

camera transform reset order

4be5c11

mcm001 added 2 commits June 18, 2023 15:22

Merge remote-tracking branch 'upstream/master' into sim-overhaul

8b13990

Fix resource path (on linux at least)

b56ee1b

mcm001 approved these changes Jun 18, 2023

View reviewed changes

Run lint

7d5f7ec

mcm001 merged commit f1cadc1 into PhotonVision:master Jun 18, 2023
19 of 20 checks passed

mcm001 deleted the sim-overhaul branch June 18, 2023 22:54

amquake mentioned this pull request Jun 20, 2023

[photonlib] Simulation robustness #874

Merged

amquake mentioned this pull request Oct 10, 2023

Update C++ Simulation #941

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Simulation Overhaul #742

[WIP] Simulation Overhaul #742

amquake commented Jan 15, 2023 •

edited

srimanachanta commented Jun 11, 2023

srimanachanta Jun 11, 2023

amquake Jun 11, 2023

srimanachanta Jun 11, 2023

amquake Jun 12, 2023

srimanachanta Jun 11, 2023

amquake Jun 11, 2023

srimanachanta Jun 11, 2023

amquake Jun 11, 2023

amquake commented Jun 11, 2023

mcm001 commented Jun 18, 2023

[WIP] Simulation Overhaul #742

[WIP] Simulation Overhaul #742

Conversation

amquake commented Jan 15, 2023 • edited

What does this do?

How to test this

Possible Additions / Known Issues

Things that should be done to finish this PR

srimanachanta commented Jun 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amquake commented Jun 11, 2023

mcm001 commented Jun 18, 2023

amquake commented Jan 15, 2023 •

edited