Skip to content

Latest commit

 

History

History
178 lines (131 loc) · 13.4 KB

README.md

File metadata and controls

178 lines (131 loc) · 13.4 KB
Raw

clic2021-devkit

Challenge homepage: compression.cc.

Downloading the Video challenge Data

To download all files, run:

bash download.sh path/to/data

It will create a folder path/to/data and extract all frames there, into a structure like:

video1/
    video1_frame1_y.png
    video1_frame1_u.png
    video1_frame1_v.png
    video1_frame2_y.png
    video1_frame2_u.png
    video1_frame2_v.png
    ...
video2/
    video2_frame1_y.png
    video2_frame1_u.png
    video2_frame1_v.png
    ...

For this, one of gsutil, wget, or curl must be available. gsutil is probably the most efficient way.

To download only some videos, use --max_vides: bash download.sh path/to/data --max_videos 10

NOTE: The script first downloads all vidoes as .zip files, resulting in 250GB+ of data. Then all zips are decompressed one by one and subsequently deleted. If you interrupt the script while unpacking, and later re-run it, it will re-download those that were already unpacked. To prevent this at the expense of more hard-drive space used, you can keep the zip files by passing --no_delete_zip.

Perceptual Challenge

This development kit is only provided as an example of what is expected from participants. It is in no way intended to contain data representative of the final challenge simply because that is not possible. The final test set will be created from the files uploaded by the participants in the compression challenge, and as a result it’s simply impossible for us to provide data which will match that distribution in the validation set.

You will first need to download the data from here. We recommend using wget to download this file as it supports resuming the download gets interrupted.

The first step will be to unzip the data:

unzip clic_2021_perceptual_valid.zip

Once unzipped, this should contain 2730 PNG files. These are of the same size (768x768) that will be shown to human raters. The test set will be similarly distributed as PNG files. We don’t yet know how many there will be.

In addition to the PNG files, there are two important CSV files:

validation.csv - this file is an example CSV file which contains the triplet PNG files that are used in the evaluation. The columns in this validation file are <O,A,B> (the file paths to the Original, A and B). The goal of your binary is to take this file as input, and generate a CSV file containing the same triplets, and an additional column which should contain either a 0, or a 1. The last column should have a 0 if the pair <O,A> will be preferred by humans to the pair <O,B>. Otherwise the last column should contain a 1. The output file format is the same as for oracle.csv, meaning we expect the columns to be <O,A,B,T> where T is the predicted value.

oracle.csv - this file contains the “ground truth” and we provide it as a way to evaluate your metric by running “eval_csv.py”. This file will NOT be given to participants as part of the test set. The columns are <O,A,B,T> where T is the target value.

Here is a short description of the scripts related to the pereceptual challenge:

eval_csv.py: this script takes two arguments which are the oracle CSV (see below), and the CSV file generated by your metric.

Example usage:

# You only need to run this command once to install absl-py (a library that's used by eval_csv)
pip install absl-py

# Run this command as many times as you'd like
python3 eval_csv.py --oracle_csv oracle.csv --eval_csv psnr.csv

For convenience (and perhaps as a very naive starting point), we provide psnr.py which is a very simple script which can be used to produce outputs that are compatible with the evaluation script. This takes the validation.csv file (specified as an argument) and produces output compatible with eval_csv.py.

2022 Test Data (Released April 8th, 2022)

We've released the testing image files below. Please download them in order to produce the output from your algorithm.

The CSV file needed to compute binary decisions can be downloaded from:

The cropped images to be downloaded are as follows:

2021 Test Data (Updated on January 20th, 2022, with the release of the oracle)

We released the following files which contain 768x768 (mostly - some files are slightly smaller in dimensions) crops:

You could use the following snippet to download the files in parallel:

wget https://storage.googleapis.com/clic2021_public/perceptual/test/clic_2021_test.zip
for i in 0 1 2 3 4 5 6 7 8 9 a b c d e f; do
 # Note: remove the ampersand if you don't want to have 16 wget processes running at once
 wget https://storage.googleapis.com/clic2021_public/perceptual/test/$i.tar &
done

Once you've downloaded all the files, you'll need to unarchive them.

unzip clic_2021_test.zip
for i in *.tar; do tar -xvf $i; done

This should yield the CSV file that you'll use to produce the results. The file paths are all relative to the directory where you unarchived everything.

2021 Ground Truth (Oracle Released: January 20th, 2022)

The oracle file (i.e., the file you can use to verify the performance of your algorithm/train new algorithms with this data) has been released. Please download it with the link below:

The oracle file is unfiltered, which means that any data cleanup is up to you. We left it like this on purpose, in order to allow participants to better model the human uncerntainty. We would like to highlight the fact that there is noise in this data, and therefore it's possible that a triplet might appear multiple times with both a positive and a negative label.

If there's enough interest, we could be convinced to release an updated oracle which matches the cleanup methods we employed before using the data for the final scoring of CLIC 2021.

Submitting to the validation/test server

Please follow our official submission form at http://compression.cc/submit/. Choose the "Perceptual" task. You will need to provide the CSV file your algorithm has generated.

To view the current leaderboard, please go to http://compression.cc/leaderboard/perceptual/valid/.