Shape completion docs

doczrc.js

@@ -11,5 +11,6 @@ export default {
         'SymCode',
         'Impression',
         'VTracer',
+        'Shape Completion'
     ]
 }

src/components/Figure/index.js

@@ -4,14 +4,15 @@ import { jsx } from 'theme-ui'
 export const Figure = (props) => {
   let imgSrc = props.src;
   let descText = props.text || '';
+  let altText = props.alt ? props.alt : descText;
   let width = props.width || null;
   let height = props.height || null;
   let maxWidth = props.maxWidth || 100 + '%';
   let maxHeight = props.maxHeight || null;
   return (
   <div>
     <img
-      alt={descText}
+      alt={altText}
       style={{
         display: 'block',
         marginLeft: 'auto', marginRight: 'auto',

src/shape-completion.mdx

@@ -0,0 +1,162 @@
+---
+name: Shape Completion
+route: /shape-completion-docs
+---
+
+import { Figure } from './components/Figure'
+
+# Shape Completion
+
+Author: [Sanford Pun](//github.com/shpun817) | Editor: [Chris Tsang](//github.com/tyt2y3) | Published: 2021-12-03
+
+-----
+
+This project aims at recovering (completing) the **structure** of a shape after a portion of it is erased. In other words, given a shape with arbitrarily "discontinued parts", assuming that we have **no prior knowledge** about the shape, how can we **reasonably connect the endpoints** at the discontinuities?
+
+<Figure src='shape-completion/hole_showcase.png' text='A shape with a hole. There are 6 endpoints in this case.' alt='Hole showcase' />
+
+In the demo images shown throughout this documentation, the following coloring scheme is used: Black pixels denote the background. Red pixels denote the rasterized *shape*. White pixels denote the *hole*. The (imaginary) intersection of the shape and the hole is the aforementioned "discontinued parts", which is what we try to recover.
+
+<Figure src='shape-completion/recovered_shape.png' text='An example of completed shape.' alt='Recovered shape showcase.' />
+
+Blue pixels will be used to denote the outline of the recovered parts of the shape.
+
+The whole process of shape completion involves intrapolating the missing outline and then filling the pixels in the hole with appropriate colors.
+
+## Simple Shape Completion
+
+Let's begin the experiment with simple shapes, like an ellipse.
+
+<Figure src='shape-completion/simple/ellipse_groundtruth_and_with_hole.png' text='The first image is the ground truth (for reference only). The second image is the input to the ShapeCompletion pipeline.' alt='An ellipse ground truth and another one with a hole.'/>
+
+### Path Preprocessing
+
+The first stage of the pipeline is to obtain and process the paths (curves) representing the existing outline of the shape.
+
+[Vision Cortex's core library](//github.com/visioncortex/visioncortex) provides the necessary utilities to extract raw paths from an image.
+
+<Figure src='shape-completion/simple/ellipse_preprocessed_path.png' text='Yellow pixels denote the identified outline of the shape after simplification.' alt='Path segments after preprocessing.' />
+
+### Tail Tangent Approximation
+
+The next step is to (divide, if needed, and) extract the two curves from the two endpoints; smoothing is performed to better approximate the tangents near the endpoints (*tails* of the whole curve). After this step, we will obtain two tangents (2-D direction vectors), one at each tail. We will call these tangents *tail tangents*.
+
+<Figure src='shape-completion/simple/tail_tangent_approx.png' text='The naive approach is to simply take A as the tail tangent, but better (more practical/useful) approximations may be obtained by taking more subsequent segments into account (e.g. B, C, D).' alt='Tail tangent approximation results.' />
+
+A number of factors determine the accuracy and robustness of tail tangent approximation. Our implementation supports configurations like how many points to consider from the tails, how long should the segments being considered accumulate to, and how the weights for each segment should change towards the tails.
+
+### Intrapolation
+
+> <strong>Why "<b>Intra</b>polation"?</strong><br />
+If we considered the existing outline of the shape as separate curves at each endpoint, we would be doing <i><b>inter</b>polation</i> <b>between</b> curves. However, in this project, we are focusing on curves (existing + missing) that form an outline of a single shape, so we argue that we are doing <i><b>intra</b>polation</i> <b>within</b> a curve.
+
+With the two endpoints and their corresponding tail tangents, we can calculate for the missing part in different scenarios. The type of curves used in this project is cubic [Bézier curves](//en.wikipedia.org/wiki/B%C3%A9zier_curve). To specify such a curve, four points are required. An important property to note about this type of curves is that *the curve constructed is always fully contained in the quadrilateral defined by the four points*.
+
+The first scenario is when the two tail tangents point to the same side with respect to the line connecting the two endpoints (we call this line the *base*).
+
+<Figure src='shape-completion/simple/intrapolate_same_side.png' text='Both tail tangents at A and B point to the same side of the red, dashed line (the base).' alt='Intrapolation when both tail tangents point to the same side' />
+
+<div>
+    To construct the curve between A and B, we need to identify two *control points* (C<sub>A</sub> and C<sub>B</sub>) between them. In our approach, we started with what we think is intuitive and made tweaks to resolve some issues in practice; this is what we end up with:
+</div>
+
+<br />
+
+<div>
+    First, find the intersection of the two lines starting at A and B along the corresponding tail tangent. The mid-points between A/B and the intersection are then set to be C<sub>A</sub> and C<sub>B</sub>. If the intersection is too far away (i.e. the two lines are close to, if not exactly, parallel), we simply use a point on each line as the control points (e.g. translate A/B along their tail tangent by a factor of base length). Either way, if either C<sub>A</sub> or C<sub>B</sub> end up lying outside the hole region, we *retract* it by pushing it towards the endpoint, until it reaches the hole region.
+</div>
+
+> <strong>What if the intersection was in the other direction?</strong><br />
+<Figure src='shape-completion/simple/intrapolation_bent_outwards.png' text={`If the line originating from A and B intersect in the negative direction (as shown above), we simply correct them by bending them inwards to be perpendicular with the base.`} alt='Tail tangents bent outwards; pulled back to be perpendicular to the base.' />
+
+Another scenario is when the two tail tangents point to different sides of the base, as below.
+
+<Figure src='shape-completion/simple/intrapolate_diff_sides.png' text='A simple dot product operation can be used to detect such a scenario.' alt='Intrapolation with tail tangents pointing to different sides of the base' />
+
+In this case, any intersections detected are meaningless because they must lie outside the hole region. Instead, we divide the curve into two halves and intrapolate two subcurves from each endpoint to the mid-point of the base as shown above.
+
+The last possible scenario is trivial to handle: when the lines are coincident, simply connect the endpoints with a straight line.
+
+<Figure src='shape-completion/simple/intrapolate_coincidence.png' alt='Intrapolation becomes connecting endpoints with a straight line in coincidence' />
+
+The case of our simple ellipse falls into the first scenario. The intrapolated outline is shown as follows:
+
+<Figure src='shape-completion/simple/ellipse_intrapolated.png' alt='Full ellipse outline after intrapolation.' />
+
+### Color filling
+
+To fill the hole with appropriate colors, we define three element types: *Blank*, *Structure*, and *Texture*. Note that in this project, it is restricted that only one shape is processed at once, and actual texture recovery is not performed - we're only interested in knowing which parts of the hole are Blank, Structure, or Texture.
+
+Element     |Description
+:-----------|:----------
+Blank       |Background pixels. (Black in our demo)
+Structure   |Outline of the shape; The intrapolated curve(s) obtained above is rasterized and drawn onto the hole. (Blue in our demo)
+Texture     |Solid part of the shape; To be filled in this section. (Red in our demo)
+
+The Structure elements divide the hole into several subregions. Each of these subregions contains wholly either Blank or Texture elements.
+
+<Figure src='shape-completion/simple/filling_subregions.png' alt='Hole of ellipse divided into two subregions' />
+
+In our example, the hole is divided by the intrapolated curve into two subregions. In order to guess whether to fill the subregions with Blank or Texture elements, we check if the number of Blank elements outside the hole boundary exceeds a certain tolerance.
+
+For the bottom subregion, the pixels right outside the bottom boundary are (almost) all red (Texture), therefore this subregion is classified as Texture and filled with Texture elements.
+
+For the top subregion, the pixels outside the left, top, and right sides of the boundary are considered. All of those pixels are background (Blank), so this subregion is classified as Blank.
+
+After filling, the shape of the ellipse is completed, as follows:
+
+<Figure src='shape-completion/simple/ellipse_complete.png' alt='Complete ellipse' />
+
+If we move the hole around, shape completion yields the following results:
+
+<Figure src='shape-completion/simple/ellipse_diff_holes.png' alt='Complete ellipses with different holes' />
+
+## Complex Shape Completion
+
+The process of shape completion shown above has been rather straightforward because there is a strong assumption - the hole cuts the shape at exactly 2 endpoints only. Consider the following case:
+
+<Figure src='shape-completion/complex/ellipse_hole_across.png' text='At a glance, we can tell how the endpoints should be grouped - A with B, and C with D, but how can we model the problem to match the endpoints such that the result of color filling always makes sense?' alt='Ellipse with a long hole cutting its boundary at 4 endpoints' />
+
+### Endpoint Matching
+
+#### Failed attempt: Local Proximity
+
+An intuitive approach might be by endpoint proximity in a greedy manner. If we simply connect each endpoint to its nearest neighbor, the correct matching is found for the above case. However, this approach ceases to work for the following case:
+
+<Figure src='shape-completion/complex/ellipse_local_proximity_counterexample.png' text='The correct matching seems to be A with B and C with D, but the top two endpoints are the closest.' alt='Tall hole over ellipse.' />
+
+#### Avoiding Intersections
+
+Problematic matchings are the ones that lead to intersecting curves. If intersection occurs, the resulting shape deforms and there may be subregions that are surrounded by others, leading to problems in color filling. Therefore, the key of endpoint matching lies in **avoiding intersections**.
+
+Before intrapolation, some intersecting curves can already be identified by looking at endpoint connections that intersect.
+
+Imagine we have 4 endpoints A, B, C, and D. If the line segment AB intersects with CD, then the curve intrapolated from A to B must intersect with that from C to D.
+
+<Figure src='shape-completion/complex/line_intersect_implies_curve_intersect.png' alt='Line intersection among endpoints implies intersection of intrapolated curves' />
+
+Therefore, the first step to avoiding intersecting curves is to filter out matchings that contain intersecting lines.
+
+This [webpage](//prase.cz/kalva/putnam/psoln/psol794.html) shows that minimizing the total length of endpoint connections is equivalent to finding a matching with no intersecting connections. Hence the problem is reduced to a [Euclidean Bipartite Matching Problem](//core.ac.uk/download/pdf/82212931.pdf), i.e. optimizing the global weights over matchings. The [Hungarian algorithm](//en.wikipedia.org/wiki/Hungarian_algorithm) is used to solve such a problem.
+
+The rest of the intersecting curves have to be caught and filtered out after intrapolation has taken place. Bézier curve intersection can be detected by a recursive method called [De Casteljau's (Bézier Clipping) algorithm](//en.wikipedia.org/wiki/De_Casteljau%27s_algorithm), which is implemented in [flo_curves](//crates.io/crates/flo_curves), the Bézier curve library we use.
+
+## Robustness
+
+Throughout the process of development, we encountered many edge cases where the pipeline would break down when faced with certain geometrical configurations. For example, an endpoint detected at the corner of the hole may or may not be useful, because it may be an edge entering the hole at a corner, or it may just be the hole touching an edge with its corner.
+
+As much as we hope to correctly handle every single situation (if it's possible in the first place), we figured, from an engineering perspective, that we want an effective way to secure maximum overall robustness with a reasonable amount of effort.
+
+From experimentation, we observed that most of the pain-inducing geometrical configurations were deformed once we move the hole by just 1 pixel. Therefore, we decided that once the pipeline breaks down, it should try to **recover by expanding the hole by 1 pixel in each direction**. If it successfully produces something in one of these attempts, that result is used.
+
+<hr />
+
+<Figure src='shape-completion/animated_demo.gif' alt='Animated Demo.' />
+
+As shown above, the performance of our implementation is stable for the most part. Occasionally, the pipeline (arguably) incorrectly handles the cases when tail tangents are (nearly) coincident to the hole boundaries.
+
+<Figure src='shape-completion/incorrect.png' alt='Arguably incorrect case.' />
+
+
+
+

src/vtracer.mdx

@@ -48,7 +48,7 @@ Path simplification consists of 2 steps:
 
 From the previous stage, we have obtained a path whose consecutive edges must have different directions, i.e. the shape is represented by the minimum number of edges with 100% fidelity. However, to represent slant lines and curves in raster graphics, “jaggies” (or pixel staircases) inevitably occur. In this step, we aim at removing these artifacts.
 
-To replace a staircase with a straight line (hence “removing” it), one may adopt an outset (“additive”) or inset (“subtractive”) approach. Both approaches are justifiable in difference contexts, so the key is to maintain consistency in the same shape. In order to determine which points to keep, we make use of the <i>signed area</i>.
+To replace a staircase with a straight line (hence “removing” it), one may adopt an outset (“additive”) or inset (“subtractive”) approach. Both approaches are justifiable in difference contexts, so the key is to maintain consistency in the same shape. In order to determine which points to keep, we make use of the *signed area*.
 
 <Figure src='vtracer/Staircase.svg' />
 
@@ -60,7 +60,7 @@ For each point on the staircase, we calculate the signed area of the triangle fo
 
 ### Simplify by Limiting penalties
 
-The path can be further simplified by evaluating the <i>penalty</i> from replacing a subpath with one long edge from the first to the last point.
+The path can be further simplified by evaluating the *penalty* from replacing a subpath with one long edge from the first to the last point.
 
 <Figure src='vtracer/RemoveWobble.svg' text='Left: Path with wobbles. Right: Path approximated by red line, removing wobbles' />
 
@@ -80,15 +80,15 @@ What we have now is a simplified polygon with high fidelity. However, if we feed
 
 ### 4-Point Scheme
 
-VTracer adapts from the <i>4-Point Scheme</i> subdivision algorithm, which is an interpolating smoothing method. The problem of the 4-Point Scheme is that all vertices would be smoothed into round corners, which, from our point of view, is a loss of fidelity.
+VTracer adapts from the *4-Point Scheme* subdivision algorithm, which is an interpolating smoothing method. The problem of the 4-Point Scheme is that all vertices would be smoothed into round corners, which, from our point of view, is a loss of fidelity.
 
 <Figure src='vtracer/4-PointScheme.svg' text='4-Point Scheme performed on segment A₁A₂' />
 
 <Figure src='vtracer/4-PointSchemeProblem.svg' text='If A2 is a corner, the 4-Point Scheme smooths it out iteratively.' />
 
 ### Finding corners
 
-To preserve corners, we first have to locate them. VTracer finds corners by checking the <i>angle difference</i> at each vertex. If the absolute angle difference exceeds a certain threshold, the vertex is considered to be a corner.
+To preserve corners, we first have to locate them. VTracer finds corners by checking the *angle difference* at each vertex. If the absolute angle difference exceeds a certain threshold, the vertex is considered to be a corner.
 
 <Figure src='vtracer/Corner.svg' />
 
@@ -124,9 +124,9 @@ The path is now populated with nicely positioned and sufficiently dense points t
 
 ### Finding Splice Points
 
-To define a splice point, we make use of the <i>signed angle differences</i>. Interpreting each edge along the path as vectors, we can define the signed angle difference from edge eᵢ to eᵢ₊₁ as the 2D angle of rotation θ ∊ (-π, π] required to rotate eᵢ to the same direction as eᵢ₊₁ (assume positive θ in clockwise direction).
+To define a splice point, we make use of the *signed angle differences*. Interpreting each edge along the path as vectors, we can define the signed angle difference from edge eᵢ to eᵢ₊₁ as the 2D angle of rotation θ ∊ (-π, π] required to rotate eᵢ to the same direction as eᵢ₊₁ (assume positive θ in clockwise direction).
 
-It is sufficient for a vertex to be a splice point if it passes one of two tests: <i>point of inflection test</i> and <i>angle displacement test</i>.
+It is sufficient for a vertex to be a splice point if it passes one of two tests: *point of inflection test*and *angle displacement test*.
 
 Points of inflection can be found by tracking the signs of angle differences along the path. When the concavity of the path changes at a certain point, the sign of the signed angle difference also changes at that point.