DeeperForensics Challenge Submission Example

This repo provides an example Docker image for submission of DeeperForensics Challenge 2020. The submission is in the form of online evaluation.

Important hints

Please refer to the following important hints to ensure a pleasant online evaluation experience:

1. We highly recommend you to test your Docker images locally before submission. Or your code may not be run properly on the online evaluation platform. Please refer to Test the Docker image locally for instructions.

2. In the first week, you will be allocated two extra evaluation chances. For an initial test, we highly recommend you to extract one frame only for each video to test your model, in order to estimate the online evaluation time and avoid timeout. The total runtime (ExecutionTime, in seconds) can be seen when you successfully submit to the CodaLab leaderboard or use download output from scoring step at Submit / View Results.

3. Please carefully read the part regarding the hidden test set at the challenge website (Overview and Terms and Conditions), which may be helpful for successful submission and performance boost.

4. Preprocessing might occupy most of the runtime.

Before you start: request resource provision

1. Create an account on the challenge website (CodaLab), as well as an AWS account (in any region except Beijing and Ningxia);

2. Register DeeperForensics Challenge 2020 using the created CodaLab account;

3. Send your CodaLab user name (i.e., team name), the number of team members, affiliation, CodaLab email address, and AWS account id (12 digits) to the organizers' email address: deeperforensics@gmail.com. We will check and allocate evaluation resources for you. You will receive an email notifying you that the resources have been allocated. If you don't receive the email, it may be recognized as spam.

Install and configure AWS CLI

Then you should install AWS CLI (version 2 recommended).

After installation, you should configure the settings that the AWS Command Line Interface (AWS CLI) uses to interact with AWS:

1. Generate the AWS Access Key ID and AWS Secret Access Key in IAM: AWS Management Console -> Find Services -> Enter 'IAM' -> Choose IAM (Manage access to AWS resources) -> Delete your root access keys -> Manage security credentials -> Access keys (access key ID and secret access key) -> Create New Access Key.

2. Run this command:

aws configure

Then it will require you to input AWS Access Key ID, AWS Secret Access Key, Default region name (please input us-west-2) and Default output format (left it empty). If you still have questions, please refer to here.

Install Docker Engine

In order to build your Docker image, you should install Docker Engine first. Please refer to Install Docker Engine.

Install nvidia-docker

Because GPU is necessary for both the local test and online evaluation, we also need to install nvidia-docker.

Obtain this example

Run the following command to clone this submission example repo:

git clone https://github.com/Drdestiny/DeeperForensicsChallengeSubmissionExample.git

Include your own algorithm & Build Docker image

• Your algorithm codes should be put in model.
• Your detector should inherit DeeperForensicsDetector class in eval_kit/detector.py. For example:

import sys

sys.path.append('..')
from eval_kit.detector import DeeperForensicsDetector

class ToyPredictor(DeeperForensicsDetector): # You can give your detector any name.
    ...

You need to implement the abstract function predict(self, video_frames) in your detector class:

    @abstractmethod
    def predict(self, video_frames):
        """
        Process a list of video frames, the evaluation toolkit will measure the runtime of every call to this method.
        
        params:
            - video_frames (list): may be a list of numpy arrays with dtype=np.uint8 representing frames of **one** video
        return:
            - probability (float)
        """
        pass

• Modify Line 25 in run_evalution.py to import your own detector for evaluation.

########################################################################################################
# please change these lines to include your own face forgery detector extending the eval_kit.detector.DeeperForensicsDetector base class.
from toy_predict import ToyPredictor as DeeperForensicsDetector
########################################################################################################

• Besides, you can implement your own frame extracting method. If so, you should modify eval_kit/extract_frames.py. Make sure that the return type remains the same (<class 'list'>).

• Edit Dockerfile to build your Docker image. If you don't know how to use Dockerfile, please refer to:

  • Best practices for writing Dockerfiles (English)
  • 使用 Dockerfile 定制镜像 (Chinese)
  • Dockerfile 指令详解 (Chinese)

• If you don't know how to build the Docker image, please refer to step 2 in Submit the Docker image.

Test the Docker image locally

The online evaluation for submissions may take several hours. It is slow to debug on the cloud (and the evaluation chances are limited). Hence, we provide the tools for the participants to locally test the correctness of the submission.

Before running, modify Line 28 in local_test.py.

To verify your algorithm can be run properly, run the following command:

docker run -it deeperforensics-challenge-<your_aws_id> python local_test.py

Please refer to step 2 and 3 in Submit the Docker image to learn how to tag your Docker image.

It will run the algorithms in the evaluation workflow on some example videos and print out the results.

The output will look like:

    ================================================================================
    all videos finished, showing verification info below:
    ================================================================================
    
INFO:root:Video ID: 000000.mp4, Runtime: 7.3909759521484375e-06
INFO:root:	gt: 0
INFO:root:	output probability: 0.5
INFO:root:	number of frame: 1
INFO:root:	output time: 7.3909759521484375e-06
INFO:root: 
INFO:root:Video ID: 000001.mp4, Runtime: 2.3603439331054688e-05
INFO:root:	gt: 1
INFO:root:	output probability: 0.5
INFO:root:	number of frame: 1
INFO:root:	output time: 2.3603439331054688e-05
INFO:root: 
INFO:root:Video ID: 000002.mp4, Runtime: 8.106231689453125e-06
INFO:root:	gt: 1
INFO:root:	output probability: 0.5
INFO:root:	number of frame: 1
INFO:root:	output time: 8.106231689453125e-06
INFO:root: 
INFO:root:Video ID: 000003.mp4, Runtime: 2.2649765014648438e-05
INFO:root:	gt: 0
INFO:root:	output probability: 0.5
INFO:root:	number of frame: 1
INFO:root:	output time: 2.2649765014648438e-05
INFO:root: 
INFO:root:Done. Average FPS: 64776.896

Submit the Docker image

1. Retrieve the login command to use to authenticate your Docker client to your registry:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 212923332099.dkr.ecr.us-west-2.amazonaws.com

2. Build your Docker image using the following command. For information on building a Docker file from scratch see the instructions here. You can skip this step if your image has already been built:

cd ../DeeperForensicsChallengeSubmissionExample
docker build -t deeperforensics-challenge-<your_aws_id> .  # . means the current path. Please don't lose it.

For example:

docker build -t deeperforensics-challenge-123412341234 . 

3. After the build is completed, tag your image so you can push the image to the repository:

docker tag deeperforensics-challenge-<your_aws_id>:latest 212923332099.dkr.ecr.us-west-2.amazonaws.com/deeperforensics-challenge-<your_aws_id>:latest

4. Run the following command to push this image to your AWS ECR repository:

docker push 212923332099.dkr.ecr.us-west-2.amazonaws.com/deeperforensics-challenge-<your_aws_id>:latest

After you push to the repo, the evaluation will automatically start and you will receive an email as a notification. In 2.5 hours you should receive another email with the evaluation result. If there is something wrong like timeout or error, you will still receive a reminder email. Please look in the spam if you don't receive any email.

Finally, you can submit the evaluation result (".zip" file) to the challenge website.

PROBLEMS YOU MAY ENCOUNTER

If you encounter problems during evaluation, please search here for the solution FIRST. If your problem still exists, send email to us.

Q1: I submitted the Docker image several hours ago (let's say, 4 hours). But I still receive nothing. There is also no email in the spam. what should I do?

A1: First of all, if there are some bugs in your submission, you'll receive an email informing you that your submission fails, including timeout (time limit is 2.5 hours in the development phase). So not receiving email is not due to failure. The reason is, the command is not python run_evaluation.py when entering your submitted Docker image, which means that it outputs nothing and exit. Maybe you didn't build the Docker image properly. For example, using a container to build another image is not correct (Note: container is different from image. Their relationship is like the one between class and instance). If everything goes well, when you run the Docker image, it will automatically run run_evaluation.py. If you don't know how to properly build the Docker image, please refer to step 2 in Submit the Docker image.