Speeding up Elasmobranch BRUVS processing by 20x...
Here we provide a guide to learn about SharkTrack. The guide is divided in the following 3 chapters. We suggest you to spend ~15m to read them and understand the model, but feel free to jump to the sections as you like.
| Chapter | Reading Time | Description |
|---|---|---|
| Introduction to SharkTrack | 7m | This page is for users who are just considering the use of AI in their workflow, and aren't even sure yet whether SharkTrack would be useful. It summarizes what we do with the model, how it can help you and how other people use it |
| SharkTrack User Guide | 5m | This is a guide to run SharkTrack on your data. Jump to it if you're already familiar with SharkTrack and you're ready to run it on your data (and you have some familiarity with running Python code). |
| Annotation Pipeline | 6m | This page illustrates the AI-enhanced BRUVS annotation workflow. Jump to it to convert the model's output into MaxN metrics |
Click on the image above to watch a demo
- Overview
- What does SharkTrack do?
- How is it useful?
- How do people run SharkTrack?
- List of people using it
- Contributors
- Collaborations
- Next Steps
Elasmobranch researchers monitor their populations using Baited Remote Underwater Video Systems (BRUVS). This is a time-consuming process, as each video needs to be manually annotated.
SharkTrack is a Machine Learning model that uses computer vision to detect and track Elasmobranchii in BRUVS videos. It does not classify the species. However, by simply detecting video snippets with elasmobranchs, the model drastically speeds up BRUVS annotation. Additionally, this package provides users a workflow to annotate species ID from model detectios.
Therefore SharkTrack is an AI-enhanced workflow to convert raw BRUVS videos to MaxN, which has been tested 20x faster than traditional methods.
SharkTrack is a AI-enhanced BRUVS annotation pipeline.
What this jargon means is that it does the following steps:
- Takes a folder with BRUVS videos, detects where sharks are in the videos.
- Communicates this with an output like this. Read this to learn about our output format and its motivations.
- Provides a workflow for users to quickly clean detections and assign species ID.
Basically, its goal is to allow researchers to spend time only classifying shark species, saving the majority of time spent looking at empty frames.
Example image showing the video and time of the red detection
The main use of SharkTrack is to compute BRUVS MaxN in hours instead of months (or years!). It can do this on a modern but simple laptop, without access to WIFI or GPU.
If not fine tuned, it achieves ~90% MaxN accuracy. We understand that some researchers might require higher accuracy metrics, or might want to double check them. In this case, SharkTrack is very useful in providing a general overview of where sharks are in the BRUVS videos.
Researchers also use SharkTrack during sampling, to process BRUVS overnight and share findings the next day.
We have two main types of users:
They analyse the BRUVS to compute accurate MaxN metrics. High accuracy is crucial to them, and they have less strict time constraints.
They use the default SharkTrack mode, called analyst, to detect Elasmobranch and easily annotate the species and compute MaxN.
They are currently doing BRUVS deployment. They want their daily videos to be processed overnight to share detections with the team and externally and have a precise idea of which sampling site records more sharks.
They can use the peek mode of SharkTrack. This mode extracts interesting frames from the video, without providing annotation or MaxN support, but it is helpful to have a quick overview of their daily videos.
You can check more about the two versions of SharkTrack here
SharkTrack is a publicly-available model, and the SharkTrack User Guide provides instructions for running it using our Python scripts. Many of our users run SharkTrack on their own, either on the cloud or on their local computers.
That said, we know that Python can be a bit daunting to setup. Additionally, SharkTrack requires significant processing power which, despite we have designed it for mobile usage, might still be a challenge. Therefore, some users - particularly high-volume users - send us BRUVS videos (online or physical hard drives), which we run through SharkTrack, then we send back a results file.
If that is of interest for you, please read the Bespoke Deployment section.
Whether you're going to run SharkTrack on your own or work with us, usually the first step with a new user is just running our model on a short BRUVS video and seeing what happens, so if you're interested in trying this on your BRUVS, we can work out a way to transfer a set of example images, just email us.
This software was written and supported by the efforts of
- Filippo Varini
- Sophie Wilday
- Jeremy Jenrette
- Dr Joel Gayford
- Prof Ben Glocker
- Prof Francesco Ferretti
This repository is licensed with the MIT license. If you use SharkTrack in a publication, please cite:
Varini, F. et al (2024). SharkTrack. GitHub. Available at https://github.com/filippovarini/sharktrack
The same citation, in BibTex format:
@article{varini2024sharktrack,
title={SharkTrack},
author={Filippo Varini et al},
year={2024}
}Please submit any issue on GitHub. We aim to respond to it within a week.
This project welcomes contributions, as pull requests, issues, or suggestions by email.
This is the first step of a broader effort to develop generalisable marine species classifiers. We are looking for contributors for this project. If you want to get involved in AI-driven Ocean Conservation please email us.
Sometimes our users have high volume of BRUVS videos they want to be processed or specific requirements for SharkTrack, such for it to classify certain species or work in particularly challenging environments.
In this case, we offer a bespoke deployment service. We understand their requirements, collect their data and train a tailored model ourselves that satisfies their needs. Currently, we require funding for this service.
If this is of interest for your organisation, please reach out via email by answering the following questions:
-
Can you provide a short overview of your project? What ecosystem and locations are you working in? What is the water quality? What are the key species found (besides Elasmobranchii)?
-
About how many hours of BRUVS do you have waiting for processing right now? Are these stereo systems?
-
About how many hours do you expect to collect in the next, e.g., 1 year?
-
What tools do you use to process and annotate images? For example, do you:
- Move images to folders named by species
- Keep an Excel spreadsheet open and fill it with filenames and species IDs
- Use a tool like Eventmeasure
-
About what percentage of your videos is empty?
-
If you are also interested in Species ID, what are the key speceis you would like to classify?
-
Do you have an NVIDIA GPU available (or access to cloud-based NVIDIA GPUs)? "I don't know what a GPU is" is a perfectly good answer.
-
At the place where you plan to do most of your work, how is your bandwidth? If you're able to visit speedtest.net to measure your upload and download speeds, that's helpful.
-
How did you hear about SharkTrack?
-
Do you have any legal or policy constraints that prevent you from using cloud-based tools to manage or review your images?
-
What is your level of fluency in Python?
-
About how many images do you have that you've already annotated, from roughly the same environments? What annotations?
Run the model on your BRUVS by following this guide