New Image > Compare > Generate color lookup and File > Export > Color lookup tools#415
Merged
tannerhelland merged 13 commits intomainfrom Jun 20, 2022
Merged
New Image > Compare > Generate color lookup and File > Export > Color lookup tools#415tannerhelland merged 13 commits intomainfrom
Image > Compare > Generate color lookup and File > Export > Color lookup tools#415tannerhelland merged 13 commits intomainfrom
Conversation
...ready for testing. This first pass is just a basic implementation, mostly designed to see if I can roughly re-create a LUT after applying it to an image. (This doesn't actually work very well and I'm going to try a different approach, but this gives me a good "backup" implementation.) Only 3dl export is currently implemented, and no UI exists to set export parameters. I'll add that once I see if a new strategy actually works lol
Normally, I use KD-trees for palette matching, specifically for matching a huge set of input colors (a full image) against a small set of output colors (a palette). In this scenario, it is beneficial to take the time to build a perfectly balanced KD-tree, because subsequent tree traversal will go much faster, and since we have millions of pixels to match, we want the fastest tree possible. For auto-constructing 3D LUT files from any arbitrary set of photo edits, the opposite problem arises: I need to match some small sample set of pixels (say an 8x8x8 LUT) against all colors in an image (millions!) to find the best match. In this case, assembling the initial tree becomes the bottleneck for performance, and it's preferable to build the tree as fast as possible, even if that imposes a penalty to later traversal (because we don't have to traverse the tree that many times). I have tested this mode on PD's regular palette code and the trade-off works exactly as predicted. Unfortunately, I'm still not sure it'll be fast enough for e.g. 20-megapixel images... but I need to start somewhere. (If this goes catastrophically badly, I do have some other structures I might try... but I'd rather not write all that code if I can reuse KD-trees mostly as-is.)
I honestly didn't know if this homebrew LUT creation strategy would work, but it does! Yay! Here's the gist: 3D LUTs are used in a number of industries - video editing, game development, photography, etc. LUT files exist in a bunch of different formats, and they're basically just giant tables that map colors from one domain to another. Such tables are extremely helpful for taking complex color transforms with a ton of steps and reducing them into a single table that applies *all* those changes at once - e.g. in a game pipeline you might bump up brightness, reduce yellow tones, increase contrast, improve clarity at high and low ranges of the green spectrum, give everything a slightly violet tint, then tone-map that into a final screen-ready gamut. - but doing all those steps separately takes forever, so instead at development time you use Photoshop (or dedicated color-grading software) to create a LUT that performs all of those steps on every color in the spectrum (or a reasonably representative subset of colors), stores the final mapping of each color in a giant list, and then at run-time you can just apply that LUT to each frame to apply your huge list of edits in a single uniform pass. Whether you're doing 100 adjustments or 1 doesn't matter - a LUT merges all those changes into a single mapping table that does it for you. LUTs are also used extensively for color-grading photos and film, because once you develop a signature "look" you can simply merge the full pipeline of edits into a single LUT, and with one click you get that "look" on any arbitrary photo, video, whatever. LUTs are one of the few photo editing things that works across almost any software and/or platform because at the end of the day, LUTs are pretty much just text files with encoded RGB tables inside. So even if your software doesn't support e.g. a Curves tool, you can let users load a LUT created in software that *does* support curve adjustments and then apply it, because software doesn't care how a LUT was created - it just uses the embedded tables to convert all colors to new values. So applying LUTs is the easy part. PhotoDemon supports a number of LUT formats and lets you apply them to images the same way any other photo editor does. But creating LUTs is a different story, and there was no way to *create* new LUTs inside PD... until this commit. Photoshop limits LUT creation to adjustment layers specifically - you have to set up 1+ adjustment layers on an image, and then you can export the resulting merged adjustment-layer-transform into a new LUT file. This is cool, obviously, and relatively easy to support because the possible range of edits is small. But PhotoDemon doesn't provide adjustment layers (yet) and even Photoshop only supports a subset of its full Adjustment tool library as adjustment layers. Wouldn't it be better if you could just edit a photo using ANY AND EVERY TOOL in the app, and then the app would magically reverse-engineer a LUT for you, encompassing all the changes you'd made? That's what I've attempted to do in PhotoDemon, and by god, it works. Mostly. It's a little slow right now due to huge numbers of classes used in the necessary data structs (so the code is fast, but class teardown is like 60 seconds for the hundreds of thousands of classes that get created), so I'll need to rewrite some data structures either using lightweight classes or by converting them to array-driven methods. But that's easy stuff compared to the work that's already been done! Now for the caveats. LUTs encode 1:1 mapping between colors, so they cannot encode area-driven effects (like blur, distort filters, etc). This means they are best at encoding edits from Adjustment menu tools, but you don't really need to care about that - if you use any Effects, PhotoDemon can still auto-create LUTs for you! But if the same color gets mapped to multiple output colors (due to a blur effect or similar), the quality of the LUT will suffer. For the next caveat, I need to describe PD's LUT creator works. Basically, the algorithm starts by comparing the final, edited image state to its original, unmodified state. A huge tree of all represented colors is constructed, and the algorithm analyzes how each color has changed. From that list of changes, it constructs a full-gamut LUT, directly using relevant color changes where it can and interpolating changes from similar colors for any parts of the color spectrum that the current image doesn't include. This encodes most "normal" adjustment patterns very well, but can produce weird results if your source image has a very limited palette (e.g. it's grayscale, or mostly a single color tone, etc). So for best results, if you intend to export a LUT you'll want to perform your adjustments/effects/etc on a photo with reasonably good color diversity - lots of dark and bright tones of as many different colors as possible. This gives the LUT creator more information to work with, and the resulting LUT file will be more applicable to any type of image. Next up is resolving the damn VB6 class teardown perf issue, then looking at an improved interpolation strategy that provides more accurate coverage of massive state changes (like "invert all colors"). I also want to write a new Render effect for generating color test patterns, which would help immensely for improving gamut coverage. I need this tool available so that I can finally create a default set of LUTs to ship with PhotoDemon. I want to provide similar LUTs to Photoshop's default set, but they copyright their LUTs (which seems silly - can you really copyright a list of numbers? idk). So I can't just ship Photoshop's files outright - but I can certainly make my own set of edits that produce a similar result to theirs, then create my own LUT files and ship *those*. So that's what I'm gonna do. Anyway, I legitimately didn't know if this strategy would work, so I'm pretty stoked to have a workable path forward for this feature.
KD-trees are a cool data structure: https://en.wikipedia.org/wiki/K-d_tree PhotoDemon uses them all over the place. Generally, they are used to match pixels (numbering in the millions) against some subset of colors (numbering in the thousands). Constructing LUTs, however, requires us to solve the opposite problem: matching some subset of LUT colors (numbering in the thousands) against all colors in the image (numbering in the millions). PD's existing KD-tree implementation worked fine for this, but because the generated trees were so large, COM teardown imposed a huge perf penalty (90+% of LUT generation time was spent just freeing the KD-tree!). To work around this, I've now written a separate KD-tree optimized against this special use-case. It uses a backing array to store tree nodes, and is thus basically instantaneous to teardown. It also has a roughly identical construction cost. The traversal of such a tree is unfortunately slower, but the teardown benefits far outweigh this cost. (Roughly speaking, this tree implementation is 2x slower to traverse, but 100000x faster to teardown - so in a real-world example, the overall LUT export process takes some 15 seconds vs 60 seconds before, when accounting for all the different parts of the process.)
Duh, I should have thought this through before I wrote it the way I did - traversal order matters when deciding whether to search sub-trees, and regardless of the backing memory (an array in this case) I still need to traverse it recursively, because a FILO stack like I was using will search sub-trees in a suboptimal order. Stupid mistake on my part. Anyway, the idea here is that Export > Color lookup is now extremely fast, only a few seconds even on images with millions of colors. (In fact, this new class is so fast that I may need to investigate my regular KD-tree to see if I can replace it with this experimental one!)
This is getting closer and closer to being merge-able!
Also, improve debug reporting on 3D LUT generation so I can micro-profile a bit more
Grayscale and invert actions, specifically; these are easily detected via special checks but do not otherwise work correctly using per-channel interpolation.
This means PD can now round-trip any 3D LUT format to any other 3D LUT format! I'm getting closer and closer to finishing this feature, which was a major blocker for a new stable release. (Also, I'm perpetually trying to solve IDE casing issues so apologies for some unrelated case-driven changes here)
Still need to generate localizations for the new UI text, but this tool inches ever closer to completion
This tool is, to my knowledge, the first of its kind in a normal photo editing application. (Other projects exist that attempt something similar, but they are all command-line tools or esoteric research projects.) The idea here is simple: this tool can reverse-engineer color modifications that have been applied to any image. Here's how it works. Load two photos into PhotoDemon: a "before" and an "after". The "after" image should be the same image as the "before" one, but with colors modified in some way - e.g. perhaps an Instagram filter has been applied. Then run this new `Image > Compare > Create color lookup` tool. It will compare the before-and-after images and generate a LUT that describes all color effects applied to the "after" image. This allows you to then apply the same effect on any image you want, *without knowing any technical details about how the effect works*. PhotoDemon handles all of those details for you, and you get a way to implement the effect "for free", in any photo editor of your choosing (PhotoDemon or otherwise). This tool uses the same code as the recently built `File > Export color lookup` tool, but instead of needing to apply the color changes from within PhotoDemon, this one is explicitly designed to work with before-and-after images from any outside source. As a reminder (I swear I *will* actually write documentation for all this someday): 3D LUT files only describe per-pixel color changes. They cannot describe effects, transformations, or anything that involves blending multiple pixels together, like "blur" or "sharpen". This limits them to color transforms, but they can describe *any* color transform (pretty much anything in PD's Adjustments menu). This is the final piece of the puzzle I need to add a default set of LUTs to PD. I want to ship a good set of Instagram-like filters with PD's Color lookup tool (not identical filters, but similar ones), and this gives me a way to do it. I will also be able to ship similar LUTs to the default ones that ship with Photoshop, yay!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Description copy+pasted from recent commits...)
3D LUTs are used in a number of industries - video editing, game development, photography, etc. LUT files exist in a bunch of different formats, and they're basically giant tables that map colors from one domain to another. Such tables are extremely helpful for taking complex color transforms with a ton of steps and reducing them into a single table that applies all those changes at once - e.g. in a game pipeline you might bump up brightness, reduce yellow tones, increase contrast, improve clarity at high and low ranges of the green spectrum, give everything a slightly violet tint, then tone-map that into a final screen-ready gamut - but doing all those steps separately takes forever, so instead at development time you use Photoshop (or dedicated color-grading software) to create a merged LUT that performs all of those steps on every color in the spectrum (or a reasonably representative subset of colors) and stores the final mapping of each color in a giant list. Then, at run-time you can just apply your saved LUT to each frame to apply that huge list of edits in a single very-fast pass. Whether you're doing 100 adjustments or 1 doesn't matter - a LUT merges all those changes into a single mapping table that does everything for you.
LUTs are also used extensively for color-grading photos and film, because once you develop a signature "look" you can simply merge the full pipeline of edits into a single LUT, and with one click you get that "look" on any arbitrary photo, video, whatever. LUTs are one of the few photo editing things that works across almost any software and/or platform because at the end of the day, LUTs are just text files with encoded RGB tables inside. So even if your software doesn't support e.g. a Curves tool, you can let users load a LUT created in software that does support curve adjustments and then apply it, because software doesn't care how a LUT was created - it just uses the embedded tables to convert colors to new values.
So applying LUTs is the easy part. PhotoDemon supports a number of LUT formats and lets you apply them to images the same way any other photo editor does (using the
Adjustments > Color > Color lookuptool).But creating LUTs is a different story, and there was no way to create new LUTs inside PD... until this pull request.
Photoshop limits LUT creation to adjustment layers specifically - you have to set up 1+ adjustment layers on an image, then you can export the resulting merged adjustment-layer-transform into a new LUT file. This is cool, obviously, and relatively easy to support because the possible range of edit operations is small. But PhotoDemon doesn't provide adjustment layers (yet) and even Photoshop only supports a subset of its full Adjustment tool library as adjustment layers. Wouldn't it be better if you could just edit a photo using ANY AND EVERY TOOL in the app, then have the app magically reverse-engineer a LUT for you, encompassing all the changes you'd made?
That's what PhotoDemon now offers. There are two ways to use the new automatic LUT-generator.
First, you can load a photo, apply any adjustments you want, the export the merged adjustments via the
File > Export > Color lookuptool. This will produce a LUT file (in cube, 3dl, or look formats - all formats supported by Photoshop) that you can use in PhotoDemon or any other photo editor.Even better, however, is the ability to auto-generate a LUT from arbitrary image files! Just load two images into PhotoDemon: a "before" image and an "after" image. Then, run the new
Image > Compare > Generate color lookuptool. This tool will automatically analyze both images, solve for a color-mapping that describes all adjustments, then export the result to a generic LUT file of your choosing. This lets you easily copy any adjustments from a film, software (e.g. Instagram filter), or other source - all you need is a "before" and "after" photo so PhotoDemon can figure out how colors have been modified.Pretty neat, eh? This feature was a blocker for the next PD stable release, because I need to generate a good set of LUTs to ship with the "Color lookup" tool, but I lacked a way to generate said LUTs (until now). I thought about just shipping Photoshop's LUTs as-is, but they print copyright details in their LUTs (which seems silly - can you really copyright a list of numbers? idk). Rather than mess with that, I can now just use my own set of edits that produce a similar result to their default LUTs, then export my own LUT files from within PhotoDemon and ship those. Yay for open-source tools!
Now for the caveats.
LUTs encode 1:1 mapping between colors, so they cannot encode area-driven effects (like blur, distort filters, etc). This means they are best at encoding edits from Adjustment menu tools, but you don't really need to care about that - if you use any Effects, PhotoDemon can still auto-create LUTs for you. But if the same color gets mapped to multiple output colors (due to a blur effect or similar), the quality of the LUT will suffer. Try to stick to the Adjustments menu when using this tool.
For the next caveat, I need to describe how PD's LUT creator works. Basically, the algorithm starts by comparing the final, edited image state to its original, unmodified state. A huge tree of all represented colors in the source image is constructed, and the algorithm analyzes how each color has changed. From that list of changes, it constructs a full-gamut LUT, directly using relevant color changes where it can and interpolating changes from similar colors for any parts of the color spectrum that the current image doesn't provide. This encodes most "normal" adjustment patterns very well, but can produce weird results if your source image has a very limited palette (e.g. it's grayscale, or mostly a single color tone, etc).
So for best results, if you intend to export a LUT you'll want to perform your adjustments/effects/etc on a photo with good color diversity - lots of dark and bright tones from as many different colors as possible. This gives the LUT generator more information to work with, and the resulting LUT file will be more applicable to any type of image in the future.