New Edit tools: Content-Aware Fill, Fill, and Stroke #403
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Photoshop has a borderline-magical feature called "content-aware fill": https://helpx.adobe.com/photoshop/using/content-aware-fill.html I want something similar for PhotoDemon, so here we gooooooo! The best reference I've seen on texture synthesis (which is what content-aware fill does - it attempts to "synthesize" a new "texture" that can be used to seamlessly "fill" over the top of an unwanted object) is Paul Harrison's PhD Thesis, "Image Texture Tools: Texture Synthesis, Texture Transfer, and Plausible Restoration": https://www.logarithmic.net/pfh-files/thesis/dissertation.pdf Using the concepts from his thesis, Paul would go on to release the "Resynthesize" plugin that provides content-aware fill in GIMP. Resynthesize is still available as a GIMP plugin, but unfortunately it is under a new maintainer now and it is very challenging to build for Windows (and the source code is GPL anyway, so I can't use it in PhotoDemon even if I wanted to). Fortunately, Paul's thesis is very comprehensive. He lays out pretty much everything you need to consider when writing a texture synthesizer, and he cites many other author's takes on the topic (which are also useful references, with good ideas to be taken from all). From these various references, I have now completed my first draft of a texture synthesizer for PhotoDemon. This first draft is not actually a content-aware fill tool - that will come later - but it *is* a seamless tile generator, which provides a simple test ground for the underlying algorithm. Basically, given an input image, the code tries to construct a novel output image that is capable of seamless tiling, using only pixel data from the original image as its reference. From this, it's easy to study how the algorithm handles complexity and sampling, and whether it's capable of generating representative textures from a variety of inputs. As it stands today, this code is slow, but it works well - very well, even - and I'm now at a stage where I comfortable committing it so I have a good backup reference before I start writing a UI for it. (A UI is needed for further testing, since there are a number of parameters you can toggle to trade-off between quality and performance, and I need to ensure PD's algorithm works well across the entire spectrum.) I don't have an ETA for final merge into PD, but it shouldn't be long - I'm thinking a measurement in days possibly weeks, not months.
Also adds placeholder items for other Fill and Stroke commands, per #401
Holy shit, friends - this actually works. I'm honestly a little (okay a LOT) astonished to see the functional content-aware fill behavior in action, because part of me thought this was never going to work in a reasonably good way. But it works, and it works *well*. Like, this will be a tool that people actually use! There's a ton of clean-up and refinement left to do (and I still need to build a UI) but that's easy-peasy compared to just getting the damn algorithm working in the first place.
(CAF = content-aware fill, I'm tired of typing it lol) - CAF now works on any layer (non-zero offsets, partially off-image, non-destructive transforms, size not matching parent image, etc) - CAF now validates various stages during initialization and returns failure on cases like "no points to synthesize" - CAF now reports progress via progress bar - CAF can now disable extensive debug reporting - CAF now correctly terminates in cases where the selected region does not overlap the active layer
Like pretty much all algorithms, content-aware fill can be designed against a variety of performance vs memory trade-offs (e.g. you can use a lot more memory to gain a lot more speed). My first draft didn't really account for this, so in some places it used minimal memory while in others it used a ton but was very fast. Time to start aligning these goals! In this commit, the in-painter now requires the source, destination, and mask surfaces to all be the same size. (Previously, the mask could be any arbitrary size relative to the src/dst images; the inpainter would handle custom bounds-checking accordingly.) The main benefit of this is that it hugely simplifies boundary checking on pixel accesses. We can perform all boundary checks up-front and then use a single set of compares across *all* subsequent access checks, which provides a ~30% performance improvement. (For example, a fill that used to take 6 seconds now takes 4.) To combat the memory increase required by a larger mask, the parent function now pre-crops out only the relevant region of the full image prior to in-painting. Because various in-painting structs are initialized to the size of the source image, this actually results in a net reduction in memory usage, despite the larger mask space requirement. Unfortunately, all these geometry changes completely break the algorithm for layers that are not the same size as the parent image. I will fix this shortly.
While I expect this feature to primarily be used on single-layer images (e.g. JPEGs), it must still work correctly on multi-layer images, including layers that lie partially off-image. This commit restores that behavior and integrates it with the previous commit's large performance improvements.
Antialiased selections, too; the results of the content-aware fill will now be correctly blended back onto the base layer. Note that this also works with 32-bpp sources!
Initialization consumes a disproportionate amount of processing time. In particular, a list of potential neighbor coordinates needs to be generated and sorted by its Euclidean distance from [0, 0]. This is the kind of operation whose performance can be trivially improved by the usual laundry list of sort algorithm tweaks. This commit knocks out some low-hanging fruit to cut initialization time by ~50%.
This is just another standard optimization of switching to insertion sort when a quicksort sub-list is small. (Plus a few other minor tweaks.) This and previous commits have now cut initialization time down to ~5% of overall effect time, which is good enough for me. Further perf improvements will need to come from within the core algorithm.
I'm all for code reuse, but further perf optimization requires separating these functions into dedicated groups anyway. There are too many places where expensive branches are required on tight inner loops, and it's faster (and ultimately simpler) to remove all "mode" variables and focus on perfecting each task as its own entity. I think enough of the base-level optimization work is finished that any further optimization will be job-specific, anyway, so there won't be any maintenance burden from this. (If anything, it will simplify future maintenance to split these tasks into separate classes.) Splitting the seamless texture code out into a separate class, removing all mode checks from the (new) streamlined pdInpaint class, and in-lining newly svelte functions reduces running time by ~10-12% - an easy win for minimal effort. Next up, reducing cache pressure by reworking the code to operate on a single input/output image.
...instead of separate source/destination images. This is made possible by the previous commit (splitting out the seamless tile code to a separate class). The primary benefit of a single image instead of split source/destination images is reduced cache pressure; instead of two sets of pixels being accessed by the core loop, now there is only one, which means the single image's data is more likely to be cached when we need it. Exact consequences of this are challenging to measure on my laptop (throttling due to heat is a problem on these lengthy back-to-back runs) but looks to be in the neighborhood of ~10-15% speed improvement. All these small gains are adding up to meaningful improvements overall, and I still have some strategies to try - let alone the benefits of adding a UI and letting the user ramp-down extremely high quality settings when they're unnecessary.
More minor perf improvements here (< 10% net running time), but I think I'm nearing the end of "easy" gains. Data's about as localized as I can get it, so further improvements will need to come from changes to the core algorithm, including faster suggested default parameters. To test those will require a UI, so I guess that's the next job!
As part of building a settings dialog for the Content-Aware Fill tool, I have also revisited all default values in the inpainter class. Given PhotoDemon's emphasis on photographs (it's right there in the name!), some defaults needed to be bumped up while others needed to be bumped down. I think the current set of values is an excellent fit for PD's intended use-case, and the resulting performance of the algorithm is better than ever. Speaking of performance, I've tweaked even more code in this version and I think the code is now at a fabulous place on the "quality vs speed" spectrum. Initialization is now so fast that you can't even see it in the UI, and the inpainting process itself is now on-par performance-wise with all major competitors, and significantly better than something like the Content-Aware Fill plugin for Paint.NET (which is based on the same algorithm used here). This is a big win for a VB6 project, and I'll take those wins whenever I can. Note that unlike some competing tools, PhotoDemon's algorithm also works on 32-bpp images (with full transparency support), and it also works along image edges/boundaries with no user input. This allows it to produce excellent results when inpainting something like missing edge chunks in panorama photos taken by your phone. I now consider the core algorithm more or less "complete". GIMP provides some additional options (like filling pixels in outside-in order instead of randomly) and I may add similar options to PhotoDemon pending additional testing.
...without actually modifying the source layer. Many PhotoDemon operations use the concept of a "null-padded layer". This is a layer whose size is temporarily changed to match the parent image, and all new "padding" pixels filled with transparency. This is great because I can work on that copy without regard for boundaries (which is often faster and easier than bounds-checking everything) and when I'm done, PhotoDemon can automatically scan the layer and crop it down to its relevant final size. Historically, I just asked a layer to convert itself to a null-padded layer, then crop itself accordingly I'm done. But for some new Edit menu features, I can reduce memory usage by asking for a temporary null-padded layer copy, then applying or discarding that temporary copy as necessary. (For example, if the user cancels the operation.) I may actually be able to use this to improve memory usage on some other functions... doing that is TBD
Live previews are also working, yay!
Here's how this new tool works. IF A SELECTION IS ACTIVE: 1) The selection area will be filled with your choice of brush (same brushes available for the paint bucket, text, and other tools - gradient, pattern, etc). 2) You can also apply a new opacity, blend mode, and alpha mode to your fill settings. 3) You can choose to keep existing layer size (like GIMP), change layer size to the union of the existing layer and the selection (like Photoshop), or crop the layer to the current selection. A live preview is provided for all of these. IF A SELECTION DOES NOT EXIST, this tool will simply fill the entire active layer with your fill settings. A live preview also exists for this. This allows for all of the same fill features as both Photoshop and GIMP, with excellent performance to boot. Relates to #401
If a selection is active, this will erase the region(s) of the active layer covered by the selection. If a selection is *not *active, "Clear" will erase the current layer. (The layer will still exist at its current size, but all pixels will be set to transparent black.)
Relates to #401 This tool allows you to stroke the outline of the current selection. A bunch of options are provided, and performance looks good! While this first draft works well, I think I can improve stroke quality with some sort of "line refinement" algorithm that performs post-processing clean-up on the marching-squares output (that's how I generate a vector path from the selection mask). I'm not having luck finding an existing algorithm for this, but I've got an idea how to do it myself, so that's my next mission. This would produce noticeably better antialiasing, particularly on ellipse shapes.
The next task I need to tackle for "idealized" selection stroking is a line simplifier that can take the output of a marching squares algorithm (e.g. a path that only moves in strict 90 degree increments) and make it more organic-looking. This would greatly improve stroke quality, particularly in mitered junction mode, and also accelerate drawing performance. This simplifier isn't what I ultimately want, but it works correctly and verifies that I am able to take a GDI+ path, break it down into discrete line segments, simplify those line segments (with a primitive algorithm to test), then reassemble the result into a fully compatible GDI+ path that closely resembles the original. The real pain here is handling subpaths correctly - think three separate, untouching elliptical selections in one composite selection mask - because each subpath needs to be simplified individually. The test simplifier works perfectly with this setup, including complex paths with interior holes and multiple discrete subpaths. I've also designed the "path-to-lines" function in a generalized way that greatly simplifies subpath handling for future simplifiers, so I'm in a good place for the next phase of work: properly simplifying curved paths generated by various selection tools.
All PhotoDemon selections work on a selection mask. This is an image where black indicates unselected pixels, white indicates selected pixels, and gray levels indicates partial selection. When stroking a selection, we have to take that mask, analyze it, try to reconstruct outline paths as best we can from the bitmap data, then stroke *those* paths and hope the result looks OK. It's complicated! This commit adds new vector strategies that try to maximize the quality of the resulting stroke. It works pretty darn well, and I may be able to improve it further pending additional research. For now, however, this feature is "ready to go" and provides a very nice feature that is lacking in similar editors (like Paint.NET).
Includes new Edit menu tools: - Content-aware fill - Fill - Stroke ...and their associated dialogs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Content-aware fill allows you to automatically remove objects from a photo. Here's a nice summary article showing how it works:
https://patdavid.net/2012/08/getting-around-in-gimp-heal-selection/
PhotoDemon's implementation is fast, high-quality, and extremely easy to use. Just select the object you want to remove (any selection tool is compatible), then hit
Edit > Content-aware filland watch the magic happen!In similar news, I've also added
Edit > FillandEdit > Strokecommands, which allow you to either fill the current selection (or layer, if no selection exists) or stroke its outline. Compatible fills include solid colors, patterns, and gradients, and the stroke tool provides a similar assortment of pen attributes (various line joins, dash settings, etc).I will prepare longer articles on these features in the coming days, because there are some fun "extras" built into them, but for now I just want to get them merged so people can start experimenting with them.