Autograd-Compiler-Engine (ACE)

ACE is an automatic differentiation compiler engine with zero external runtime dependencies. Composite scalar formulas written with ordinary Python operators are implicitly intercepted and woven into a directed acyclic graph, and analytic gradients for all input variables are then computed through backpropagation.

Goals

• Support automatic differentiation for arbitrary composite scalar formulas
• Support multivariable formulas, deep nesting, and repeated variable reuse
• Support + - * /, constant powers, exp, log, sin, cos, and tanh
• Support extended operators sinh, cosh, sigmoid, and arctan
• Use analytic automatic differentiation for the actual gradient engine
• Use Richardson Extrapolation for numerical gradient checking

Project Structure

Autograd-Compiler-Engine/
├── ace/
├── docs/
├── examples/
│   ├── evaluate_formula.py
│   ├── export_graph.py
│   ├── export_mega_graph.py
│   ├── export_deep_graph.py
│   ├── export_wide_graph.py
│   ├── export_ultimate_graph.py
│   ├── ace_graph
│   ├── ace_graph.svg
│   ├── ace_graph.png
│   ├── ace_mega_graph
│   ├── ace_mega_graph.svg
│   ├── ace_mega_graph.png
│   ├── ace_deep_graph
│   ├── ace_deep_graph.svg
│   ├── ace_deep_graph.png
│   ├── ace_wide_graph
│   ├── ace_wide_graph.svg
│   ├── ace_wide_graph.png
│   ├── ace_ultimate_graph
│   ├── ace_ultimate_graph.svg
│   └── ace_ultimate_graph.png
├── tests/
├── requirements.txt
└── README.md

Core Mechanisms

ACE depends on the following ideas:

• Operator overloading: turn ordinary mathematical expressions into node operations
• DAG construction: every result node records its direct predecessors
• Reverse-mode automatic differentiation: propagate gradients from output nodes back to input nodes
• DFS-based topological scheduling: ensure dependencies are visited in the correct order during backpropagation
• Analytic derivative rules: every atomic operator provides its own local derivative formula
• Richardson Extrapolation: acts as a high-confidence numerical referee for gradient checking
• Graphviz graph export: visualize the intermediate IR structure

Gradient Checking

ACE does not use numerical differentiation as the main engine, but the tests use Richardson Extrapolation to build a high-confidence numerical reference and verify partial derivatives term by term for complex formulas at multiple points.

Related documents:

• docs/PRD.md
• docs/Richardson_Extrapolation.md
• docs/Algorithms.md
• docs/Graph_Visualization.md

Run Example

PYTHONPATH=. python3 examples/evaluate_formula.py

Export Computation Graphs

Install dependencies:

pip install -r requirements.txt

Export a regular complex formula:

PYTHONPATH=. python3 examples/export_graph.py

Export a mega-complex formula:

PYTHONPATH=. python3 examples/export_mega_graph.py

Export a deeply nested formula:

PYTHONPATH=. python3 examples/export_deep_graph.py

Export a wide multi-branch formula:

PYTHONPATH=. python3 examples/export_wide_graph.py

Export the ultimate large formula:

PYTHONPATH=. python3 examples/export_ultimate_graph.py

If you want bitmap output directly:

dot -Tpng examples/ace_graph -o examples/ace_graph.png
dot -Tpng examples/ace_mega_graph -o examples/ace_mega_graph.png
dot -Tpng examples/ace_deep_graph -o examples/ace_deep_graph.png
dot -Tpng examples/ace_wide_graph -o examples/ace_wide_graph.png
dot -Tpng examples/ace_ultimate_graph -o examples/ace_ultimate_graph.png

These commands generate Graphviz source files together with svg and png outputs under examples/.

Computation Graph Example Formulas

Complex Formula One

The formula used by examples/export_graph.py is:

f(x, y) =
    tanh(x * y)
    + log(x^2 + 1)
    + exp(sin(x + y))
    + sigmoid(arctan(y) + cosh(x - y))

Characteristics:

• It mixes basic algebraic operations with transcendental functions
• It combines trigonometric functions, hyperbolic functions, logarithms, exponentials, and sigmoid
• x and y are reused across multiple paths
• The final graph has both depth and clear branching / merging structure

Exported files:

• examples/ace_graph
• examples/ace_graph.svg
• examples/ace_graph.png

Embedded preview:

Mega-Complex Formula Two

The formula used by examples/export_mega_graph.py is composed of five large blocks:

block_one =
    tanh(
        exp(sin(x * y + arctan(z)) + log(x^2 + y^2 + z^2 + 5))
        / (sigmoid(x - y + z) + 1.5)
    )

block_two =
    sinh(cosh(x - y) * sigmoid(z + x / (y + 3)))

block_three =
    log((x * x + sin(y + z) + exp(arctan(x - z)))^2 + 2)

block_four =
    (x * y * z) / (cosh(z - y) + 2)

block_five =
    tanh(sigmoid(sinh(x + z) - cosh(y - z)) + arctan(x * y))

mega_result =
    block_one + block_two + block_three + block_four + block_five

Characteristics:

• Three variables x, y, and z participate simultaneously
• exp/log/sin/tanh/sinh/cosh/sigmoid/arctan all appear together
• It has both deep nesting and many wide branches
• The same variable is reused across multiple blocks
• The result node receives gradients flowing back from many paths

Exported files:

• examples/ace_mega_graph
• examples/ace_mega_graph.svg
• examples/ace_mega_graph.png

Embedded preview:

Deeply Nested Formula Three

The formula used by examples/export_deep_graph.py is:

deep_result =
    tanh(
        exp(
            log(
                sigmoid(
                    sinh(
                        cosh(
                            arctan(sin(x + y) + x / (y + 2.5))
                        )
                    )
                )
                + x^2 + y^2 + 3
            )
        )
    )

Characteristics:

• It does not have the widest branching, but the chain itself is very deep
• Nearly every layer wraps the expression in another nonlinear function
• It is well suited for observing long-chain rule propagation through the graph

Exported files:

• examples/ace_deep_graph
• examples/ace_deep_graph.svg
• examples/ace_deep_graph.png

Embedded preview:

Wide Multi-Branch Formula Four

The formula used by examples/export_wide_graph.py consists of ten parallel branches:

wide_result =
    sin(x + y)
    + cosh(x - z)
    + exp(arctan(y + z))
    + log(x^2 + z^2 + 4)
    + tanh(x * y)
    + sigmoid(sinh(z + x))
    + (x * y * z) / (y + 2.5)
    + arctan(x - y + z)
    + sinh(y - z)
    + tanh(sigmoid(x + y + z) + log((x - z)^2 + 2))

Characteristics:

• A single variable enters many parallel branches at the same time
• The graph visibly converges in the middle and near the end
• It is useful for checking whether gradients are truly accumulated across multiple paths

Exported files:

• examples/ace_wide_graph
• examples/ace_wide_graph.svg
• examples/ace_wide_graph.png

Embedded preview:

Ultimate Formula Five

The formula used by examples/export_ultimate_graph.py combines two major classes of structures:

deep_a =
    tanh(
        exp(
            log(
                sigmoid(
                    sinh(
                        cosh(
                            arctan(
                                sin(x + y - z)
                                + x / (y + 2.8)
                                + z / (w + 2.2)
                            )
                        )
                    )
                )
                + x^2 + y^2 + z^2 + w^2 + 4
            )
        )
    )

deep_b =
    tanh(
        sigmoid(
            exp(
                sin(
                    log((x - z)^2 + (y + w)^2 + 3)
                    + arctan(x * y - z * w)
                )
            )
        )
    )

wide_1 =
    sin(x + y) + cosh(x - z) + exp(arctan(y + z))

wide_2 =
    log(x^2 + z^2 + w^2 + 5) + tanh(x * y) + sigmoid(sinh(z + x - w))

wide_3 =
    (x * y * z) / (y + 2.7) + arctan(x - y + z - w) + sinh(y - z + w)

wide_4 =
    tanh(sigmoid(x + y + z + w) + log((x - z + w)^2 + 2.5))

wide_5 =
    exp(sin(x * w) + cosh(y - w)) / (sigmoid(z - y + x) + 1.7)

wide_6 =
    log((exp(arctan(x + w)) + sin(y - z) + x^2)^2 + 2)

fusion =
    sigmoid(deep_a + deep_b + wide_1 + wide_2 + wide_3 + wide_4 + wide_5 + wide_6)

tail =
    tanh(
        log(
            fusion^2
            + exp(arctan(x * z))
            + sigmoid(sinh(y + w) - cosh(z - x))
            + 3
        )
    )

ultimate_result =
    deep_a + deep_b + wide_1 + wide_2 + wide_3 + wide_4 + wide_5 + wide_6 + fusion + tail

Characteristics:

• It increases both depth and width at the same time
• The variable count grows from 3 to 4
• Deep nonlinear nesting and many parallel branches coexist
• The same variable flows back repeatedly across distant blocks
• It is very suitable for observing the shape of a large automatic differentiation DAG

Exported files:

• examples/ace_ultimate_graph
• examples/ace_ultimate_graph.svg
• examples/ace_ultimate_graph.png

Embedded preview:

Computation Graph Generation Process

The ACE graph export process can be summarized like this:

1. Wrap input variables as Value nodes first, for example x = Value(0.9, label="x")
2. Compose expressions as if writing normal mathematics; operator overloading intercepts every step
3. Each operation returns a new Value node and stores predecessor nodes, operator labels, and a local backward closure
4. After the expression is complete, the backend has already woven an implicit DAG
5. When result.backward() is called, the system runs DFS topological sorting and then triggers _backward() in reverse order
6. When draw_dot(result) is called, the system backtracks from the root node and collects all nodes and edges
7. Graphviz then renders the nodes, operator labels, edge directions, data, and grad values

Step-by-Step: From Formula to DAG to Backward

The following example uses the complex formula from examples/export_graph.py:

f(x, y) =
    tanh(x * y)
    + log(x^2 + 1)
    + exp(sin(x + y))
    + sigmoid(arctan(y) + cosh(x - y))

Step 1: Break It Into Intermediate Nodes

You can think of the internal decomposition in this order:

t1 = x * y
t2 = tanh(t1)

t3 = x^2
t4 = t3 + 1
t5 = log(t4)

t6 = x + y
t7 = sin(t6)
t8 = exp(t7)

t9 = arctan(y)
t10 = x - y
t11 = cosh(t10)
t12 = t9 + t11
t13 = sigmoid(t12)

result = t2 + t5 + t8 + t13

Each t_i corresponds to a new Value node inside ACE.

Step 2: Build the Graph Implicitly

When these intermediate nodes are created, ACE records:

• the current data of the node
• which operator produced it, via _op
• which predecessor nodes it depends on, via _prev
• the node's local backward closure _backward

For example:

• t1 = x * y creates a * node whose predecessors are x and y
• t5 = log(t4) creates a log node whose predecessor is t4
• t13 = sigmoid(t12) creates a sigmoid node whose predecessor is t12

So the formula is not just a sequence of numeric operations. It is woven into a DAG.

Step 3: Call backward

When executing:

result.backward()

ACE performs these steps:

1. Traverse the whole graph from result using DFS
2. Build a topological order where predecessors come before dependents
3. Reset all node gradients to zero
4. Set result.grad = 1.0
5. Call each node's _backward() in reverse topological order

Step 4: Local Derivatives Flow Backward Layer by Layer

Examples:

• t2 = tanh(t1) multiplies the incoming gradient by 1 - tanh(t1)^2 and sends it back to t1
• t1 = x * y multiplies the gradient by y and x respectively and sends it back to x and y
• t5 = log(t4) multiplies the gradient by 1 / t4 and sends it back to t4
• t13 = sigmoid(t12) multiplies the gradient by sigmoid(t12) * (1 - sigmoid(t12)) and sends it back to t12

Step 5: Accumulate Gradients From Multiple Paths

In this formula, both x and y appear in more than one place:

• x appears in x * y, x^2, x + y, and x - y
• y appears in x * y, x + y, arctan(y), and x - y

So the final x.grad and y.grad are sums of contributions from multiple paths, not the result of a single path. ACE must use:

grad += contribution

If it overwrote the value instead of accumulating, the final gradient would be wrong.

What to Look For in the Graph

The exported graph is not only for aesthetics. It is also a debugging entry point. When reading the graph, check:

• whether input leaf nodes appear correctly
• whether each intermediate expression is broken out into its own node
• whether _op labels match the formula structure
• whether a reused variable really appears across multiple branches
• whether the final result node receives multiple merged paths
• whether the data and grad values of nodes stay in a reasonable range

Generated Graph Files

Regular complex graph:

• examples/ace_graph.svg
• examples/ace_graph.png

Mega-complex graph:

• examples/ace_mega_graph.svg
• examples/ace_mega_graph.png

Deeply nested graph:

• examples/ace_deep_graph.svg
• examples/ace_deep_graph.png

Wide multi-branch graph:

• examples/ace_wide_graph.svg
• examples/ace_wide_graph.png

Ultimate graph:

• examples/ace_ultimate_graph.svg
• examples/ace_ultimate_graph.png

Run Tests

PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v