[feature] AST-Based Logical Density Metrics (Cyclomatic Complexity, Block Nesting Depth, LOC Refinement)

[seojeongm]

Summary

Closes #29
Implements per-function and per-file logical density metrics across three dimensions: cyclomatic complexity, block nesting depth, and line-of-code counts. Adds generated/vendored file detection and a structural safety net for zero-function files. Also fixes a build-breaking comment in src/parser/ParseResult.h where a // \ line continuation silently removed avg_cyclomatic_complexity from the struct.

---

Commits

#	Message	Scope
1	feat(parser): compute cyclomatic complexity per function	PythonParser.cpp, ParseResult.h, db.cpp, DbInserter.cpp
2	feat(db): roll up cyclomatic complexity to file level (max/avg)	PythonParser.cpp, ParseResult.h, db.cpp, DbInserter.cpp
3	feat(parser): compute block nesting depth per function	PythonParser.cpp, ParseResult.h, db.cpp, DbInserter.cpp
4	feat(parser): handle comprehension clause depth as sequential nesting	PythonParser.cpp
5	feat(db): roll up block nesting depth to file level (max/avg)	PythonParser.cpp, ParseResult.h, db.cpp, DbInserter.cpp
6	feat(parser): add per-function and per-file LOC	PythonParser.cpp, ParseResult.h, db.cpp, DbInserter.cpp
7	feat(parser): add logical LOC alongside existing raw LOC	PythonParser.cpp, ParseResult.h, db.cpp, DbInserter.cpp, Graph.cpp
8	feat(parser): handle generated/vendored file exclusion or flagging	PythonParser.cpp, ParseResult.h, db.cpp, DbInserter.cpp
8a	fix(parser): use segment split for generated/vendored path detection	PythonParser.cpp
9	refactor(parser): zero-initialize FileEntity rollup fields as structural safety net	ParseResult.h
fix	fix(parser): remove line-continuation backslash from FileEntity comment	ParseResult.h

---

Files Changed

src/parser/ParseResult.h

• FileEntity: renamed loc → raw_loc; added logical_loc, is_generated, and six function-derived rollup fields (max/avg_cyclomatic_complexity, max/avg_block_depth, max/avg_function_loc) with default member initializers
• FunctionEntity: added cyclomatic_complexity, max_block_depth, loc

src/parser/PythonParser.cpp

• countCyclomaticComplexity(TSNode) — new; counts branch-type nodes recursively, stops at function/lambda boundaries
• computeMaxBlockDepthClauses(TSNode, int, int&) — new; handles comprehension clause depth sequentially
• computeMaxBlockDepth(TSNode, int, int&) — new; increments on control-flow constructs, dispatches to clauses helper for comprehensions
• handleFunctionDef() — computes cc, maxBlockDepth, loc per function
• parseFile() — adds logical LOC pass, generated/vendored detection, and file-level rollup block

src/db/db.cpp

• function table: added cyclomatic_complexity, max_block_depth, loc columns
• file table: renamed loc → raw_loc; added logical_loc, is_generated, and six rollup columns

src/db/DbInserter.cpp

• INSERT statements and sqlite3_bind_* calls updated for both function (8 → 11 params) and file (3 → 12 params)

src/algo/Graph.cpp

• build_extra_maps(): SQL updated from SELECT file_id, loc FROM file → SELECT file_id, raw_loc FROM file (required by the loc → raw_loc rename in commit 7)

---

Design Decisions

Cyclomatic Complexity

Lambdas are function boundaries.
countCyclomaticComplexity stops recursion at both function_definition and lambda nodes. A conditional inside f = lambda x: x if x > 0 else -x does not contribute to the outer function's CC. Mirrors the pattern used in collectCalls. To change: remove the lambda stop condition in countCyclomaticComplexity.

Comprehension for/if clauses are not counted.
for_in_clause and if_clause inside list/set/dict comprehensions and generators are not in the branch-type list. The spec covers comprehensions under block nesting depth, not under CC. To change: add for_in_clause and if_clause to the branch-type list in countCyclomaticComplexity.

else is not counted.
else_clause on if, for, while, and try does not add +1. Only if_statement, elif_clause, for_statement, while_statement, case_clause, except_clause, conditional_expression, and boolean_operator contribute. Standard McCabe: else is the implicit fallthrough, not a decision point.

Block Nesting Depth

elif/else/except/finally inherit parent depth.
These are children of their parent construct (if_statement, try_statement, match_statement) and are recursed into at the parent's already-incremented depth. They do not increment depth again.

Comprehension clauses use sequential accumulation (computeMaxBlockDepthClauses).
Tree-sitter Python represents for_in_clause and if_clause as siblings inside a comprehension node — not nested. computeMaxBlockDepthClauses threads depth through them sequentially so [x for x in items if x.valid] correctly reaches depth 2, equivalent to a for containing an if, rather than depth 1.

Nested functions and lambdas reset the counter.
Both computeMaxBlockDepth and computeMaxBlockDepthClauses stop at function_definition and lambda nodes. The inner function gets its own counter starting from 0.

LOC

Function LOC is end_line − start_line + 1 (raw line count).
Includes blank lines and comments within the function body. Lines are 0-indexed from tree-sitter, so a function spanning rows 0–1 has loc = 2.

raw_loc vs logical_loc.
raw_loc is total line count. logical_loc skips blank lines and lines where the first non-whitespace character is #. raw_loc is preserved for the graph scoring pipeline (Graph.cpp); logical_loc is the intended input for complexity density calculations.

Docstrings are not excluded from logical_loc.
The filter only strips #-prefixed lines and blanks. """...""" multi-line strings count as logical lines. A grep across all .py files in this repo found no generation markers in docstrings, so this gap does not affect any current file.

Generated/Vendored Detection

Flagging, not exclusion.
Files are parsed and stored normally with is_generated = 1. The downstream percentile-ranking step (separate issue) filters on WHERE is_generated = 0. Exclusion was rejected because it makes files invisible to the UI and is harder to reverse.

Path-segment matching, not substring.
file_id is split by / and each directory segment (filename excluded) is matched in full against: vendor, _vendor, third_party, node_modules, venv, .venv, site-packages, __pycache__, dist, build, out. Full-segment matching avoids false positives from filenames like vendor_utils.py. build and out carry moderate false-positive risk for projects that use those as source directories.

Content signal: #-prefixed first 5 lines, case-insensitive.
If path detection does not fire, the first 5 lines are checked. Lines starting with # are matched case-insensitively against "generated", "do not edit", and "autogenerated". Docstring-style markers are not detected.

Zero-Function Files

Structural hardening only — not a live bug fix.
The rollup code already produced correct zeros before this commit (max initialized to 0 before the loop; avg guards with n > 0 ? ... : 0.0). Commit 9 adds default member initializers to the six function-derived fields in FileEntity as a safety net against future code changes. raw_loc, logical_loc, and is_generated were intentionally left without defaults — they are unconditionally assigned on every reachable path in parseFile().

---

DB Schema

Important: CREATE TABLE IF NOT EXISTS does not alter existing tables. After checking out this branch, delete any existing DB before scanning:

del C:\tmp\cc_test.db

function table (11 columns)

function_id           TEXT PRIMARY KEY
file_id               TEXT
class_id              TEXT
function_name         TEXT NOT NULL
nesting_depth         INTEGER NOT NULL DEFAULT 0
is_async              INTEGER NOT NULL DEFAULT 0
cyclomatic_complexity INTEGER NOT NULL DEFAULT 1   -- NEW
max_block_depth       INTEGER NOT NULL DEFAULT 0   -- NEW
loc                   INTEGER NOT NULL DEFAULT 0   -- NEW
start_line            INTEGER NOT NULL
end_line              INTEGER NOT NULL

file table (12 columns)

file_id                   TEXT PRIMARY KEY
file_name                 TEXT NOT NULL
language                  TEXT NOT NULL
raw_loc                   INTEGER NOT NULL           -- renamed from loc
logical_loc               INTEGER NOT NULL DEFAULT 0 -- NEW
is_generated              INTEGER NOT NULL DEFAULT 0 -- NEW
max_cyclomatic_complexity INTEGER NOT NULL DEFAULT 0 -- NEW
avg_cyclomatic_complexity REAL    NOT NULL DEFAULT 0 -- NEW
max_block_depth           INTEGER NOT NULL DEFAULT 0 -- NEW
avg_block_depth           REAL    NOT NULL DEFAULT 0 -- NEW
max_function_loc          INTEGER NOT NULL DEFAULT 0 -- NEW
avg_function_loc          REAL    NOT NULL DEFAULT 0 -- NEW

---

How to Test

1. Create test files

Create C:\tmp\cc_test\test_cc.py with the following content:

def simple():
    return 1

def with_if(x):
    if x > 0:
        return x
    return -x

def with_elif(x):
    if x > 0:
        return 1
    elif x < 0:
        return -1
    return 0

def with_loop_and_if(items):
    for item in items:
        if item:
            return item

def with_try():
    try:
        return 1
    except Exception:
        return 0

def with_boolean(x, y, z):
    return x and y or z

def nested_func():
    def inner(x):
        if x:
            return x
        return 0
    return inner

def complex_func(items):
    result = []
    for item in items:
        if item > 0:
            result.append(item)
        elif item == 0:
            pass
        while len(result) > 10:
            result.pop()
    return result

def process(items):
    results = []
    for item in items:
        if item.valid:
            if item.priority == "high":
                results.insert(0, item)
            elif item.priority == "low":
                results.append(item)
            else:
                if item.retry_count > 3:
                    results.append(item)
    return results

def with_comprehension(matrix):
    return [x for row in matrix for x in row if x > 0]

def simple_comp(items):
    return [x for x in items if x.valid]

C:\tmp\cc_test\generated_file.py — single comment line with a generation marker:

# This file was automatically generated. DO NOT EDIT.

C:\tmp\cc_test\imports_only.py — imports only, no functions:

import os
import sys
import json
from pathlib import Path

2. Build and scan

cmake --build build --config Debug --target TelescodeScanner
del C:\tmp\cc_test.db
.\build\Debug\TelescodeScanner.exe C:\tmp\cc_test C:\tmp\cc_test.db

Expected:

[scan] Parsed 3 files.
[scan] Inserted results into C:\tmp\cc_test.db

3. Run verification queries

python -c "
import sqlite3
db = sqlite3.connect(r'C:\tmp\cc_test.db')

print('=== COMMITS 1+2 | Cyclomatic Complexity — per function ===')
for r in db.execute('''
    SELECT function_name, cyclomatic_complexity
    FROM function ORDER BY start_line
''').fetchall(): print(r)

print()
print('=== COMMITS 1+2 | Cyclomatic Complexity — per file ===')
for r in db.execute('''
    SELECT file_name, max_cyclomatic_complexity, avg_cyclomatic_complexity
    FROM file ORDER BY file_name
''').fetchall(): print(r)

print()
print('=== COMMITS 3+4 | Block Nesting Depth — per function ===')
for r in db.execute('''
    SELECT function_name, max_block_depth
    FROM function ORDER BY start_line
''').fetchall(): print(r)

print()
print('=== COMMIT 5 | Block Nesting Depth — per file ===')
for r in db.execute('''
    SELECT file_name, max_block_depth, avg_block_depth
    FROM file ORDER BY file_name
''').fetchall(): print(r)

print()
print('=== COMMIT 6 | Function LOC — per function ===')
for r in db.execute('''
    SELECT function_name, loc
    FROM function ORDER BY start_line
''').fetchall(): print(r)

print()
print('=== COMMIT 6 | Function LOC — per file ===')
for r in db.execute('''
    SELECT file_name, max_function_loc, avg_function_loc
    FROM file ORDER BY file_name
''').fetchall(): print(r)

print()
print('=== COMMIT 7 | raw_loc vs logical_loc ===')
for r in db.execute('''
    SELECT file_name, raw_loc, logical_loc
    FROM file ORDER BY file_name
''').fetchall(): print(r)

print()
print('=== COMMIT 8 | Generated/vendored flagging ===')
for r in db.execute('''
    SELECT file_name, is_generated
    FROM file ORDER BY file_name
''').fetchall(): print(r)

print()
print('=== COMMIT 9 | Zero-function file: all rollup metrics must be 0 ===')
for r in db.execute('''
    SELECT file_name,
           max_cyclomatic_complexity, avg_cyclomatic_complexity,
           max_block_depth, avg_block_depth,
           max_function_loc, avg_function_loc
    FROM file WHERE file_name = 'imports_only.py'
''').fetchall(): print(r)
"

4. Expected output

=== COMMITS 1+2 | Cyclomatic Complexity — per function ===
('simple', 1)
('with_if', 2)
('with_elif', 3)
('with_loop_and_if', 3)
('with_try', 2)
('with_boolean', 3)
('nested_func', 1)
('inner', 2)
('complex_func', 5)
('process', 6)
('with_comprehension', 1)
('simple_comp', 1)

=== COMMITS 1+2 | Cyclomatic Complexity — per file ===
('generated_file.py', 0, 0.0)
('imports_only.py', 0, 0.0)
('test_cc.py', 6, 2.5)

=== COMMITS 3+4 | Block Nesting Depth — per function ===
('simple', 0)
('with_if', 1)
('with_elif', 1)
('with_loop_and_if', 2)
('with_try', 1)
('with_boolean', 0)
('nested_func', 0)
('inner', 1)
('complex_func', 2)
('process', 3)
('with_comprehension', 3)
('simple_comp', 2)

=== COMMIT 5 | Block Nesting Depth — per file ===
('generated_file.py', 0, 0.0)
('imports_only.py', 0, 0.0)
('test_cc.py', 3, 1.3333333333333333)

=== COMMIT 6 | Function LOC — per function ===
('simple', 2)
('with_if', 4)
('with_elif', 6)
('with_loop_and_if', 4)
('with_try', 5)
('with_boolean', 2)
('nested_func', 5)
('inner', 3)
('complex_func', 8)
('process', 11)
('with_comprehension', 2)
('simple_comp', 2)

=== COMMIT 6 | Function LOC — per file ===
('generated_file.py', 0, 0.0)
('imports_only.py', 0, 0.0)
('test_cc.py', 11, 4.5)

=== COMMIT 7 | raw_loc vs logical_loc ===
('generated_file.py', 1, 0)
('imports_only.py', 4, 4)
('test_cc.py', 61, 51)

=== COMMIT 8 | Generated/vendored flagging ===
('generated_file.py', 1)
('imports_only.py', 0)
('test_cc.py', 0)

=== COMMIT 9 | Zero-function file: all rollup metrics must be 0 ===
('imports_only.py', 0, 0.0, 0, 0.0, 0, 0.0)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b6a59ea6-ba9f-4c5e-bb57-2e3bbd1c4986

📥 Commits

Reviewing files that changed from the base of the PR and between 1b9b142 and 59e5a01.

📒 Files selected for processing (1)

• src/parser/ParseResult.h

🚧 Files skipped from review as they are similar to previous changes (1)

• src/parser/ParseResult.h

---

📝 Walkthrough

Walkthrough

Adds AST-derived complexity and LOC metrics to parser entities, persists the new fields in SQLite, and updates graph lookup code to read raw_loc for file nodes.

Changes

Code Metrics Pipeline

Layer / File(s)	Summary
ParseResult field changes src/parser/ParseResult.h	FileEntity now stores raw_loc, logical_loc, is_generated, and aggregated metrics; FunctionEntity now stores cyclomatic_complexity, max_block_depth, and loc.
AST metric helpers src/parser/PythonParser.cpp	New AST-walk helpers compute cyclomatic complexity and maximum control-flow nesting depth while skipping nested function and lambda bodies.
Function and file metric extraction src/parser/PythonParser.cpp	handleFunctionDef assigns per-function metrics; parseFile computes raw and logical LOC, flags generated files, and rolls up per-function aggregates to file-level metrics.
SQLite schema expansion src/db/db.cpp	kInitSQL expands the file and function tables with the new metric columns and LOC variants.
DbInserter binding updates src/db/DbInserter.cpp	prepareStatements, insertFile, and insertFunction are updated to match the widened file and function column sets.
Graph raw_loc lookup src/algo/Graph.cpp	build_extra_maps now reads raw_loc when populating file_loc_map.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• c5ln/Telescode#9: Introduced the initial SQLite schema setup that this PR extends for file and function metrics.
• c5ln/Telescode#13: Updated the same schema-and-binding pipeline that this PR further extends with additional metric columns.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly reflects the main change: AST-based logical density metrics for cyclomatic complexity, block depth, and LOC refinement.
Description check	✅ Passed	The description covers the summary, commit breakdown, changed files, design decisions, schema, and testing, with only minor template gaps.
Linked Issues check	✅ Passed	The PR appears to implement the linked issue's metrics, rollups, raw/logical LOC split, generated-file flagging, and zero-function defaults.
Out of Scope Changes check	✅ Passed	No clearly unrelated changes are present; the Graph.cpp adjustment supports the raw_loc rename required by the feature.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/29-logical-density-metrics

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/db/db.cpp`:
- Around line 12-23: The schema change in db.cpp updates the file table
definition to use raw_loc, but existing databases created by initDb() will still
have the old loc column because CREATE TABLE IF NOT EXISTS does not migrate
tables. Add a migration path alongside the file table setup in db.cpp that
detects the old schema and either ALTER TABLE the existing file table or
recreates/backfills it so loc data is preserved as raw_loc. Make sure the
migration is wired into the same initialization flow that populates the file
table used by the inserter and graph queries, so older databases continue to
work without missing LOC values.

In `@src/parser/ParseResult.h`:
- Around line 10-12: Default-initialize the new ParseResult fields raw_loc,
logical_loc, and is_generated in ParseResult.h so a default-constructed
ParseResult is always safe. Update the ParseResult struct/class definition
alongside the existing rollup fields so PythonParser::parseFile’s early return
path cannot leave these members indeterminate before DbInserter::insertFile
binds them. Keep the change scoped to the ParseResult member declarations and
preserve the existing zero-initialized behavior used by the other loc counters.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c9570037-3fc6-49a2-98ca-ef06b558d8bd

📥 Commits

Reviewing files that changed from the base of the PR and between 34b2346 and 1b9b142.

📒 Files selected for processing (5)

• src/algo/Graph.cpp
• src/db/DbInserter.cpp
• src/db/db.cpp
• src/parser/ParseResult.h
• src/parser/PythonParser.cpp

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Add a migration path for existing databases.

CREATE TABLE IF NOT EXISTS will leave existing file tables unchanged, so databases created with the old loc column will still lack raw_loc. The updated inserter and graph query then fail or silently skip file LOC data. Add an ALTER TABLE/schema-version migration or recreate/backfill strategy for loc → raw_loc.

Possible migration sketch

+    // After creating/opening schema, migrate older DBs that still have file.loc.
+    // Prefer a schema_version/user_version based migration if this project already has one.
+    // Example only:
+    // ALTER TABLE file RENAME COLUMN loc TO raw_loc;
+    // ALTER TABLE file ADD COLUMN logical_loc INTEGER NOT NULL DEFAULT 0;
+    // ...

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/db/db.cpp` around lines 12 - 23, The schema change in db.cpp updates the
file table definition to use raw_loc, but existing databases created by initDb()
will still have the old loc column because CREATE TABLE IF NOT EXISTS does not
migrate tables. Add a migration path alongside the file table setup in db.cpp
that detects the old schema and either ALTER TABLE the existing file table or
recreates/backfills it so loc data is preserved as raw_loc. Make sure the
migration is wired into the same initialization flow that populates the file
table used by the inserter and graph queries, so older databases continue to
work without missing LOC values.

[coderabbitai]

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Default-initialize raw_loc, logical_loc, and is_generated.

These three new int members are left uninitialized, unlike the six rollup fields below that you defaulted to 0. PythonParser::parseFile returns a default-constructed ParseResult on the file-open-failure early-return (return result; at line 753) before any of these are assigned. That result then flows to DbInserter::insertFile, which binds the garbage values into raw_loc/logical_loc/is_generated (all NOT NULL columns), persisting indeterminate data.

🛡️ Proposed fix

-    int raw_loc;           // total line count (including blanks and comments)
-    int logical_loc;       // non-blank, non-comment line count
-    int is_generated;      // 1 if vendored/generated, 0 otherwise
+    int raw_loc = 0;       // total line count (including blanks and comments)
+    int logical_loc = 0;   // non-blank, non-comment line count
+    int is_generated = 0;  // 1 if vendored/generated, 0 otherwise

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	int raw_loc; // total line count (including blanks and comments)
	int logical_loc; // non-blank, non-comment line count
	int is_generated; // 1 if vendored/generated, 0 otherwise
	int raw_loc = 0; // total line count (including blanks and comments)
	int logical_loc = 0; // non-blank, non-comment line count
	int is_generated = 0; // 1 if vendored/generated, 0 otherwise

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/parser/ParseResult.h` around lines 10 - 12, Default-initialize the new
ParseResult fields raw_loc, logical_loc, and is_generated in ParseResult.h so a
default-constructed ParseResult is always safe. Update the ParseResult
struct/class definition alongside the existing rollup fields so
PythonParser::parseFile’s early return path cannot leave these members
indeterminate before DbInserter::insertFile binds them. Keep the change scoped
to the ParseResult member declarations and preserve the existing
zero-initialized behavior used by the other loc counters.