-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
instructions on how to reproduce the results are missing #2
Comments
also possibly separate instructions without dssim since not every organization has a license to use it |
Software used:
VQMTs:
What I did was:
All the scripts used, start and finish CSVs are made available within the Scripts subdir of this repo. Scripts follow a [dataset].["getScores"/"ROCC"].py naming convention, where getScores reads the starting CSV ("[dataset].MOS.csv"), runs the tools, parses the stdout and adds the result to the final CSV ([dataset].VQMT.csv). The ROCC scripts read the [dataset].VQMT.csv CSV and use pandas+scipy to calculate the correlation coefficients. As a side note, the scripts within the Scripts subdir display some more results: where possible I added some filtering by distortion to measure correlation only for JPEG+JPEG2K and JPEG+JPEG2K+Gaussian blur distortions. Didn't include them in the "frontpage" for the sake of brevity, but thought you might have a use for this. If anything is unclear, I am at your disposal to amend the above so that the repro instructions can reach a presentable state and be included in the README.md file. |
Would you be able to easily rerun the results with https://gitlab.com/wg1/jpeg-xl/blob/master/tools/butteraugli_main.cc instead of https://github.com/google/butteraugli using the p-norm score, not the max butteraugli score? (The jpeg-xl butteraugli prints out both scores.) |
I would and already began doing so for max, 2, 3, 6 and 12-norm scores on the JPEG XR and KADID-10k datasets. I can do them all and post the results here. |
That is just wonderful. I cannot thank you enough. If it doesn't work out, I need to do some serious home work. |
Please note that JPEG XL butteraugli prints out a mixture of p-norms, i.e., when you ask for a 3rd norm, it prints a mix of 3rd, 6th and 12th norm. Just good to know in case if you want to compare against p-norming the other metrics, too. |
Could you please elaborate on that? This is my very trivial patch for butteraugli_main.cc:
Can you confirm or deny that this prints out the 2, 3, 6 and 12-norms before I go full steam ahead with the remaining datasets? |
What you have are not pure p-norms, but always producing a mixture of three p-norms. The ComputeDistanceP function internally doesn't compute a p-norm, but the average of three p-norms: p-norm, (2p)-norm, and (4p)-norm Might be that the mixtures work actually better, so it is ok to start with what you have. ... The lines:
and change
to and remove one more line: |
That makes much more sense seeing the results I'm getting from a few initials runs. Thanks! Maybe I could make a new function, ComputeSimpleDistanceP(...) with the above changes to get pure norms, and include the results along with max and the standard (mixed) 3p-norm. Funky mixes can wait until I have at least the basics covered. |
Perfect! |
please consider adding instructions on how to reproduce the results
The text was updated successfully, but these errors were encountered: