Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions OtherFfmpegArgs.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ TODO - this needs testing, to confirm filter quality.
See: [https://trac.ffmpeg.org/wiki/Scaling](https://trac.ffmpeg.org/wiki/Scaling) for more info.


## Concatination of video files.
## Concatenation of video files.

See: [https://trac.ffmpeg.org/wiki/Concatenate](https://trac.ffmpeg.org/wiki/Concatenate)

Expand Down Expand Up @@ -202,4 +202,3 @@ Convert 360 videos between various formats.
### vectorscope

[vectorscope](https://www.ffmpeg.org/ffmpeg-filters.html#toc-vectorscope)

2 changes: 1 addition & 1 deletion docker/ffmpeg-4.4/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -543,7 +543,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_LIB_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_LIB_DIR}

# rm -rf ${DIR}; \
Expand Down
2 changes: 1 addition & 1 deletion docker/ffmpeg-5.0/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -543,7 +543,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_LIB_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_LIB_DIR}

# rm -rf ${DIR}; \
Expand Down
2 changes: 1 addition & 1 deletion docker/ffmpeg-5.1/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -563,7 +563,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_LIB_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_LIB_DIR}

# rm -rf ${DIR}; \
Expand Down
2 changes: 1 addition & 1 deletion docker/ffmpeg-6.0/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -566,7 +566,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_MODEL_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_MODEL_DIR}

# rm -rf ${DIR}; \
Expand Down
2 changes: 1 addition & 1 deletion docker/ffmpeg-7.1/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -566,7 +566,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_MODEL_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_MODEL_DIR}

# rm -rf ${DIR}; \
Expand Down
2 changes: 1 addition & 1 deletion docker/rocky-ffmpeg-5.1/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r /tmp/vmaf/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_LIB_DIR} && \
echo "Creating VMAF DIRECTGRY:" ${VMAF_LIB_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_LIB_DIR} && \
rm -rf ${DIR}; \
else \
Expand Down
2 changes: 1 addition & 1 deletion docker/rocky-ffmpeg-6.0/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_MODEL_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_MODEL_DIR} && \
rm -rf ${DIR}; \
else \
Expand Down
2 changes: 1 addition & 1 deletion docker/rocky-ffmpeg-6.1/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_MODEL_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_MODEL_DIR} && \
rm -rf ${DIR}; \
else \
Expand Down
2 changes: 1 addition & 1 deletion docker/rocky-ffmpeg-7.0/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_MODEL_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_MODEL_DIR} && \
rm -rf ${DIR}; \
else \
Expand Down
2 changes: 1 addition & 1 deletion docker/rocky-ffmpeg-7.1/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ RUN \
mkdir -p ${PREFIX}/share/model/ && \
cp -r ${DIR}/model/* ${PREFIX}/share/model/ && \
mkdir -p ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTGORY:" ${VMAF_MODEL_DIR} && \
echo "Creating VMAF DIRECTORY:" ${VMAF_MODEL_DIR} && \
cp -vr ${DIR}/model/* ${VMAF_MODEL_DIR} && \
rm -rf ${DIR}; \
else \
Expand Down
2 changes: 1 addition & 1 deletion enctests/HDR_Encoding.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ We explicitly define the X265 parameters (see [x265](https://x265.readthedocs.io
| colormatrix=bt2020nc | UTagging the YcBCr as being encoded using the BT-2020 non-constant luminance. |
| range=limited | Set the source range to be tv range. |
| master-display=G\(13250,34500\)B\(7500,3000\)R\(34000,16000\)WP\(15635,16450\)L\(10000000,1\) | SMPTE ST 2086 mastering display color volume SEI info, specified as a string which is parsed when the stream header Essentially setting the X,Y display primaries for rec2020 along with the Whitepoint, and the Max,min luminance values in units of 0.00001 NITs. See the above docs for more info. |
| max-cll=2000,400 | Set the Maximum content light level (in this case 2000 nits = max content light level and 400 = the MaxFall - the maxiumum frame-average light level) |
| max-cll=2000,400 | Set the Maximum content light level (in this case 2000 nits = max content light level and 400 = the MaxFall - the maximum frame-average light level) |

## HLG

Expand Down
35 changes: 17 additions & 18 deletions enctests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ has_children: true
-*WORK IN PROGRESS*-

## Goals
Create a framework for testing and comparing encoded media through various encoders
Create a framework for testing and comparing encoded media through various encoders
making sure color and quality is preserved.

## Requirements
* FFmpeg with VMAF enabled
* OpenTimelineIO (>=0.15)
* OpenTimelineIO (>=0.15)

## Description

Expand All @@ -40,11 +40,11 @@ optional arguments:
--prep-sources Create *.yml files from media in --source-folder used as sources in encoding tests
--encoded-folder ENCODED_FOLDER
Where to store the encoded files
--output OUTPUT Path to results file including ".otio" extenstion (default: ./encoding-test-results.otio)
--output OUTPUT Path to results file including ".otio" extension (default: ./encoding-test-results.otio)
```

### Prepare your sources
Start by prepping your source files. This is done with the `--prep-sources` flag.
Start by prepping your source files. This is done with the `--prep-sources` flag.
A set of "sourcefile.ext.yml" files get created alongside the source media.
This is done, so you can adjust the desired in point and duration of the media
before running your tests. In-point and duration are set in frames.
Expand All @@ -68,10 +68,10 @@ rate: 25.0

### Prepare your test files
A set of default tests are provided for you in the "test_configs" folder.
The test files are yaml based and require a couple of keys and values to work.
By default, the tests are geared towards encoding with FFmpeg, but you may write
The test files are yaml based and require a couple of keys and values to work.
By default, the tests are geared towards encoding with FFmpeg, but you may write
your own encoder classes and test files geared towards that encoder.
You may provide several tests in the same file separated by "---" so yaml reads
You may provide several tests in the same file separated by "---" so yaml reads
them as separate documents.

#### Example test configuration
Expand Down Expand Up @@ -119,8 +119,8 @@ test_colorspace_yuv420p:
```

#### Additional options in the test config
You may provide a list of sources in a test config. Please note that this will
override the behavior of the `--source-folder` argument. Only the sources
You may provide a list of sources in a test config. Please note that this will
override the behavior of the `--source-folder` argument. Only the sources
provided in the test config will be used in the tests.

Example:
Expand Down Expand Up @@ -149,17 +149,17 @@ python -m testframework.main --source-folder /path/to/sources/ --encoded-folder
```

### Results
The results are stored in an "*.otio" file. Each source clip contains a media reference
for its source plus all additional encoded test files.
Each test is compared against the source with VMAF and the score is stored in
The results are stored in an "*.otio" file. Each source clip contains a media reference
for its source plus all additional encoded test files.
Each test is compared against the source with VMAF and the score is stored in
the media reference's metadata.

## Setup Test Environment
**Please note! We're working on dockerizing this**

In addition to OpenTimelineIO the tests rely on FFmpeg with VMAF support and most
likely OpenImageIO.
The commands below are what's needed to build OTIO for python development.
The commands below are what's needed to build OTIO for python development.
As of the time writing this the multi media-reference feature of OTIO is still in
the main branch of the project.
Also, we're relying on FFmpeg and OIIO being installed on the system. (Guides to come)
Expand Down Expand Up @@ -197,7 +197,7 @@ which you can do in the .venv/Scripts/activate.bat file.

This does not get you the vmaf model though which can be downloaded
https://raw.githubusercontent.com/Netflix/vmaf/master/model/vmaf_v0.6.1.json
Also you need to install an eariler version of kaleido since the current one will hang when generating an image.
Also you need to install an earlier version of kaleido since the current one will hang when generating an image.
```
pip install kaleido==0.1.0post1
```
Expand Down Expand Up @@ -233,7 +233,7 @@ export VMAF_MODEL=/opt/homebrew/Cellar/libvmaf/2.3.1/share/libvmaf/model/
## How to add an encoder
This is still **work in progress** so for now you'll have to add an encoder class
to a file in the "encoders" folder.
Since every encoder has its own set of arguments, a new encoder also requires
Since every encoder has its own set of arguments, a new encoder also requires
its own test configs.

* Create a subclass of the `ABCTestEncoder` found in "encoders/base.py".
Expand All @@ -245,7 +245,7 @@ its own test configs.
* Register your new class in the `encoder_factory` function found in "encoders/\_\_init\_\_.py" (for now)
* The key in the `encoder_map` dictionary needs to match the "app" value in the test configuration file
* Create a test configuration file
* Required keys:
* Required keys:
* "name" - name of test
* "app" - name of application used in mapping mentioned above
* "wedges" - containing list of wedges (which are named freely)
Expand Down Expand Up @@ -295,7 +295,7 @@ reports:
templatefile: basic.html.jinja
```

The configuration uses html [jinja](https://palletsprojects.com/p/jinja/) templates defined in the templates folder. The template combines the parameters from each wedge test with the results of the test, grouped by image name.
The configuration uses html [jinja](https://palletsprojects.com/p/jinja/) templates defined in the templates folder. The template combines the parameters from each wedge test with the results of the test, grouped by image name.

The graphs are generated using the [plotly.express](https://plotly.com/python-api-reference/generated/plotly.express.html#module-plotly.express) library. The parameters in the yaml file are fed directly into the constructor for the graph creation. Currently we are only supporting line graphs and bar graphs, but other graph types should be easy to add.

Expand Down Expand Up @@ -343,4 +343,3 @@ reports:
```

This example creates bar-graphs based on the `-preset` flag.

2 changes: 1 addition & 1 deletion enctests/testframework/encoders/ffmpeg_encoder.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ def run_wedges(self) -> dict:
out_file, testbasename = self.get_output_filename(test_name)

# Remove it, so if the new run fails to create anything
# we are not accidently using the old one.
# we are not accidentally using the old one.
if out_file.exists():
out_file.unlink()

Expand Down
2 changes: 1 addition & 1 deletion enctests/testframework/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ def parse_args():
'--output',
action='store',
default='',
help='Path to results file including ".otio" extenstion '
help='Path to results file including ".otio" extension '
'(default: ./encoding-test-results.otio)'
)

Expand Down
2 changes: 1 addition & 1 deletion enctests/testframework/otio2html.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def parse_args():
'--results',
action='store',
default='',
help='Path to results file including ".otio" extenstion '
help='Path to results file including ".otio" extension '
'(default: ./encoding-test-results.otio)'
)

Expand Down
2 changes: 1 addition & 1 deletion enctests/testframework/templates/index.html.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<TR>
<TH>Test</TH>
<TH>Description</TH>
<TH> Plaform/App version</TH>
<TH> Platform/App version</TH>
<TH>Success</TH>
<TH>Fail</TH>
<TH>Started</TH>
Expand Down
2 changes: 1 addition & 1 deletion index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ You can see the default ffmpeg conversion introduces a dramatic color shift that
5. [Useful Ffmpeg Filters.](OtherFfmpegArgs.html)
6. [HDR Encoding](enctests/HDR_Encoding.html)
7. [Adding Timecode and Editorial Workflow](EditorialWorkflow.html)
6. [Codec Comparsions](Encoding.html#encode)
6. [Codec Comparisons](Encoding.html#encode)
1. [h264](Encodeh264.html)
2. [Prores](EncodeProres.html)
3. [AV1](EncodeAv1.html)
Expand Down
2 changes: 1 addition & 1 deletion tests/codec_web_tests/web_compatibility.html
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
</head>

<body>
<H1>Codec browser compatability</H1>
<H1>Codec browser compatibility</H1>
<H2>8-Bit</H2>
<table>
<tbody>
Expand Down
2 changes: 1 addition & 1 deletion tests/icctest-rev-ps-nuke.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,6 @@

createCompareHtml(outputpath=rootpath+"/compare.html",
listimages=listimages,
introduction="<H1>Color_trc comparison</H1><p> This is trying to reverse out what we think is the gamma for each TRC file, with the hope that if the browser is correctly obaying the flag, that all the ramps would approximately match. The code to generate these files is <a href='../%s'>here</a>. However, the source images were generated in photoshop, by taking the raw.png file, assigning a sRGB profile to it, and then converting to a custom profile, adjusting the gamma but sticking with D65 and HDTV primaries..</p>" % os.path.basename(__file__),
introduction="<H1>Color_trc comparison</H1><p> This is trying to reverse out what we think is the gamma for each TRC file, with the hope that if the browser is correctly obeying the flag, that all the ramps would approximately match. The code to generate these files is <a href='../%s'>here</a>. However, the source images were generated in photoshop, by taking the raw.png file, assigning a sRGB profile to it, and then converting to a custom profile, adjusting the gamma but sticking with D65 and HDTV primaries..</p>" % os.path.basename(__file__),
videohtml = ' ')

2 changes: 1 addition & 1 deletion tests/icctest-rev-ps.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,6 @@

createCompareHtml(outputpath=rootpath+"/compare.html",
listimages=listimages,
introduction="<H1>Color_trc comparison</H1><p> This is trying to reverse out what we think is the gamma for each TRC file, with the hope that if the browser is correctly obaying the flag, that all the ramps would approximately match. The code to generate these files is <a href='../%s'>here</a>. However, the source images were generated in photoshop, by taking the raw.png file, assigning a sRGB profile to it, and then converting to a custom profile, adjusting the gamma but sticking with D65 and HDTV primaries..</p>" % os.path.basename(__file__),
introduction="<H1>Color_trc comparison</H1><p> This is trying to reverse out what we think is the gamma for each TRC file, with the hope that if the browser is correctly obeying the flag, that all the ramps would approximately match. The code to generate these files is <a href='../%s'>here</a>. However, the source images were generated in photoshop, by taking the raw.png file, assigning a sRGB profile to it, and then converting to a custom profile, adjusting the gamma but sticking with D65 and HDTV primaries..</p>" % os.path.basename(__file__),
videohtml = ' ')

2 changes: 1 addition & 1 deletion tests/icctest-rev2.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,6 @@

createCompareHtml(outputpath=rootpath+"/compare.html",
listimages=listimages,
introduction="<H1>Color_trc comparison</H1><p> This is trying to reverse out what we think is the gamma for each TRC file, with the hope that if the browser is correctly obaying the flag, that all the ramps would approximately match. The code to generate these files is <a href='../%s'>here</a>. However, the source images were generated in nuke.</p>" % os.path.basename(__file__),
introduction="<H1>Color_trc comparison</H1><p> This is trying to reverse out what we think is the gamma for each TRC file, with the hope that if the browser is correctly obeying the flag, that all the ramps would approximately match. The code to generate these files is <a href='../%s'>here</a>. However, the source images were generated in nuke.</p>" % os.path.basename(__file__),
videohtml = ' ')

2 changes: 1 addition & 1 deletion tests/python/CompareHtml.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Now we build the web page.

# This is a super basic html template file, trying to keep the number of dependancies down for now.
# This is a super basic html template file, trying to keep the number of dependencies down for now.

header = """
<html>
Expand Down
2 changes: 1 addition & 1 deletion tests/python/CompareOverHtml.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Now we build the web page.

# This is a super basic html template file, trying to keep the number of dependancies down for now.
# This is a super basic html template file, trying to keep the number of dependencies down for now.

header = """
<html>
Expand Down