-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run the votchallenge.net benchmark on mosse-tracker #6
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
fd26cf6
silence linter warning about camelcase
alsuren b412b79
skeleton implementation of vot challenge protocol
alsuren 93b0a4f
also commit generated config.yaml
alsuren 35ddb27
use env_logger in votchallenge example
alsuren a09d7c1
REVERTME: temporarily ignore files needed by vot challenge framework
alsuren b3ae7ef
REVERTME: use debug profile to reduce iteration time for protocol non…
alsuren f3baff1
use anyhow in votchallenge example
alsuren 1e6720b
remove unused imports
alsuren 7973eb1
parse Image and Region from init message
alsuren df720b1
split votchallenge.rs into multiple files
alsuren 7ed8354
rename server.make_hello_message()
alsuren dac02aa
add fixme
alsuren 533c177
impl Debug for MosseTracker
alsuren 120384b
implement process_init()
alsuren 8380d6d
implement 'track' message (Test concluded successfuly)
alsuren ba261d1
debug predictions as images
alsuren 73795a0
get x and y the right way around
alsuren 9bdcc25
get the square in the right place
alsuren dc31f10
compile in release mode; skip debug image writing
alsuren 58e8546
be more desperate for scientific benchmark
alsuren f634202
move the vot nonsense into examples/votchallenge; write README
alsuren e7bdc8d
copy comment updates from my trackers.ini file
alsuren 2534384
run vot analysis and include outputs
alsuren 060ad57
clean up some FIXMEs
alsuren 71b68f1
try min(width, height) as window size
alsuren a226dbb
describe how to re-run the analysis
alsuren be1d571
Use max(width, height) as window size
alsuren File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,15 @@ | ||
target/ | ||
.DS_Store | ||
|
||
# python venv for the `vot` tool | ||
examples/votchallenge/vot_venv | ||
|
||
# folders generated by the `vot` tool | ||
examples/votchallenge/results | ||
examples/votchallenge/sequences | ||
examples/votchallenge/logs | ||
examples/votchallenge/analysis | ||
examples/votchallenge/cache | ||
|
||
# config file for `vot` that needs to be hand-edited per machine | ||
examples/votchallenge/trackers.ini |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
3.9.14 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
# Running the votchallenge.net benchmarks against mosse-tracker | ||
|
||
These instructions are adapted from https://www.votchallenge.net/howto/tutorial_python.html | ||
|
||
All instructions assume that you are in this directory. | ||
|
||
## Installing vot tool in a python virtualenv | ||
|
||
First, you need to install the vot python tool, to run the benchmarks. | ||
|
||
At the time of writing, vot does not work with python 3.10 on macos (import error when starting up), so you may need to use an older version. | ||
|
||
```bash | ||
cd examples/votchallenge # if you are not already here | ||
|
||
python3.9 -m venv vot_venv | ||
source vot_venv/bin/activate | ||
pip install git+https://github.com/votchallenge/vot-toolkit-python | ||
``` | ||
|
||
## Set things up for this project | ||
|
||
```bash | ||
cd examples/votchallenge # if you are not already here | ||
|
||
cargo build --release --example=votchallenge | ||
cp trackers.template.ini trackers.ini | ||
``` | ||
|
||
Then change the last line of your new `trackers.ini`, to point at your | ||
`target/release/examples/votchallenge` executable. This must be an absolute path. | ||
|
||
## Check with a dummy sequence | ||
|
||
```bash | ||
cd examples/votchallenge && source vot_venv/bin/activate # if you are not already here | ||
|
||
vot test MosseRust | ||
``` | ||
|
||
## Run the full benchmark suite | ||
|
||
```bash | ||
cd examples/votchallenge && source vot_venv/bin/activate # if you are not already here | ||
|
||
vot test MosseRust | ||
``` | ||
|
||
This only uses a couple of cores, and take around 30 minutes, so go make yourself a cup of tea. You should see output like this: | ||
|
||
``` | ||
Downloading sequence dataset "VOT2020" with 60 sequences. | ||
Downloading |███████████████████████████████████████████████████████████████████████████| 100% [02:30<00:00] | ||
Download completed | ||
Loading dataset |███████████████████████████████████████████████████████████████████████████| 100% [00:00<00:00] | ||
Loaded workspace in '/Users/alsuren/src/mosse-tracker/examples/votchallenge' | ||
Found data for 1 trackers | ||
Evaluating tracker MosseRust | ||
MosseRust/baseline |███████████████████████████████████████████████████████████████████████████| 100% [13:24<00:00] | ||
MosseRust/realtime |███████████████████████████████████████████████████████████████████████████| 100% [13:11<00:00] | ||
MosseRust/unsupervis |███████████████████████████████████████████████████████████████████████████| 100% [01:34<00:00] | ||
Evaluation concluded successfuly | ||
``` | ||
|
||
## Checking your scores | ||
|
||
```bash | ||
cd examples/votchallenge && source vot_venv/bin/activate # if you are not already here | ||
|
||
vot analysis MosseRust | ||
``` | ||
|
||
This is a bit quicker, and will give you something like: | ||
|
||
``` | ||
Loading dataset |██████████████████████████████████████████████████████████████████████████████████| 100% [00:00<00:00] | ||
Loaded workspace in '/Users/alsuren/src/mosse-tracker/examples/votchallenge' | ||
Found data for 1 trackers | ||
Running analysis |██████████████████████████████████████████████████████████████████████████████████| 100% [00:21<00:00] | ||
Analysis successful, report available as 2022-10-06T22-54-20.997015 | ||
``` | ||
|
||
You can then open ./analysis/2022-10-06T22-54-20.997015/report.html in your web browser, to view the results. | ||
|
||
## Rerunning the analysis after making a change | ||
|
||
```bash | ||
cd examples/votchallenge && source vot_venv/bin/activate # if you are not already here | ||
|
||
cargo build --release --example=votchallenge | ||
rm -rf cache/ results/ | ||
vot evaluate MosseRust && vot analysis MosseRust | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
registry: | ||
- ./trackers.ini | ||
stack: vot2020 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there's no difference between our scores for these two. I think that they represent 1. and 2. in the paper, but I'm not sure. We're given them as bounding boxes because that's all we support. It might be better to use an older benchmark dataset that only deals in terms of bounding boxes, to give ourselves less of a handicap.
ing in RGB images and has been running since VOT2013 with annual up-
dates and modifications. A significant novelty compared to 2019 was that
the target position was encoded by a segmentation mask.
ers as VOT-ST2020, except that the trackers had to process the sequences in
real-time. The challenge was introduced in VOT2017. A significant novelty
compared to 2019 was that the target position was encoded by a segmenta-
tion mask.
ing in RGB images. The challenge was introduced in VOT2018. The target
positions were encoded by bounding boxes.
tracking in RGB+thermal imagery. This challenge was introduced in VOT2019
and can be viewed as evolution of the VOT-TIR challenge introduced in
VOT2015. The target positions were encoded by bounding boxes.
tracking in RGB+depth (RGBD) imagery. This challenge was introduced
in VOT2019. The target positions were encoded by bounding boxes.
I've kind-of ignored the
unsupervis
results, because I'm not sure what they mean.