Significant whitespace changes in Python are ignored

[lazy]

I verified via --dump-syntax that ASTs are significantly different.
Can compare them using pr -m -t -w 160 <(difft --dump-syntax a.py) <(difft --dump-syntax b.py)

Issue duplicated as text:

$ cat a.py
def test(a, b):
    if a:
        print(a)
        if b:
            print (b)
$ cat b.py
def test(a, b):
    if a:
        print(a)
    if b:
        print (b)
$ diff -u a.py b.py
--- a.py        2023-10-25 12:42:54.671789363 +0000
+++ b.py        2023-10-25 12:43:07.762393299 +0000
@@ -1,5 +1,5 @@
 def test(a, b):
     if a:
         print(a)
-        if b:
-            print (b)
+    if b:
+        print (b)
$ difft a.py b.py
b.py --- Python
No syntactic changes.

Using the latest x64 build from releases page in github.

$ difft --version
Difftastic 0.52.0 (b07e519 2023-10-08)

[Wilfred]

Oh my goodness, that's awful. Thanks for the report.

[Wilfred]

This is a display issue: the diffing logic finds changes, but since Python uses indentation for blocks it's not finding anything to display in this case.

$ DFT_LOG=trace difft a.py b.py                                          
 2023-10-25T15:51:54.861Z INFO  difft::options > CLI arguments: ["a.py", "b.py"]
 2023-10-25T15:51:54.875Z INFO  difft::diff::dijkstra > LHS nodes: 16 (1 toplevel), RHS nodes: 16 (1 toplevel)
 2023-10-25T15:51:54.875Z INFO  difft::diff::dijkstra > Saw 58 vertices (a Vertex is 80 bytes), arena consumed 7424 bytes, with 33 vertices left on heap.
 2023-10-25T15:51:54.875Z DEBUG difft::diff::dijkstra > Found a path of 9 with cost 709.
 2023-10-25T15:51:54.875Z DEBUG difft::diff::dijkstra > Initial 5 items on path: [
    "line:2  ...          line:2  ...          --- 100 EnterUnchangedDelimiter { depth_difference: 0 }",
    "line:2 if            line:2  ...          --- 300 EnterNovelDelimiterRHS",

[Wilfred]

The display logic is assuming that you can't have a structural change without having some novel tokens, which isn't true here.

Every token has a matching token on the other side, which is interpreted as no changes. See hunks::sorted_novel_positions.

[Wilfred]

E.g. here's how delimiters are handled in the analogous JS.

[Wilfred]

Should probably mark the first node in the changed subtree as novel when this occurs.

[ztane]

The Python standard library tokenizer module (which you can test with python3 -mtokenize program.py) does produce INDENT and DEDENT tokens; however the dedent tokens it produces have always zero-width lexed content, yes, but the tokenizer marks them found at the new indent depth; however the lexed text for indent token contains the total whitespace from column 1 to the beginning of indented block.