feat(engine): near-duplicate-blocks engine and layered architecture

[aspala]

Summary

• Layered engine: Analyzer / Collector / Pipeline / Registry / Parallel / FileContext
• AST subsystem with lexing, parsing, signal-based classification, and enriched compound nodes
• 40+ language definitions organized by category (native, scripting, vm, web, markup, config, data)
• Per-run OTP supervisors for behavior configs, file context, and file metrics
• Near-duplicate-block detection with bigram-based similarity scoring
• Block impact analysis with codebase/file impact and refactoring potentials
• Combined metrics system with scalar applier and YAML behavior configs
• Reorganized metrics modules: flat lib/codeqa/metrics/*.ex → lib/codeqa/metrics/file/ and lib/codeqa/metrics/codebase/

Dependencies

• Depends on chore/samples (shared sample fixtures duplicated here for independence)
• The tools/scalar_tuner UI is split into a separate PR

Merge notes

This branch includes the 791 sample fixtures from chore/samples for independence. After chore/samples merges:

• Rebase this branch onto main
• Drop the chore(samples): include sample fixtures for independent PR commit
• Or resolve the conflicts on those files with accept-incoming (contents are identical)

Extra work done during split

Two extra cleanup commits were needed because the pr-b-files.txt only listed files on the original branch — files deleted from main by the original branch were not in the list and had to be removed separately:

• refactor(engine): remove superseded modules from pre-engine architecture (15 files)
• refactor(engine): remove flat metrics modules moved into file/ and engine/ subdirs (23 files)

Test plan

• mix test passes (884 tests, 0 failures — verified locally)
• mix dialyzer clean (ignored warnings listed in .dialyzer_ignore.exs)
• mix codeqa.health runs on this repo

[github-actions]

🟠 Code Health: C (61/100)

186 files · codeqa-action · 2026-06-01

Combined metric scores use cosine similarity: +1 = metric profile perfectly matches healthy pattern for this behavior, 0 = no signal, −1 = anti-pattern detected. Mapped to 0–100 using breakpoints (approx: ≥0.5→A, ≥0.2→B, ≥0.0→C, ≥−0.3→D, <−0.3→F); actual letter grades use the full 15-step scale.

%%{init: {'theme': 'neutral'}}%%
xychart-beta
    title "Code Health Scores"
    x-axis ["Readability", "Complexity", "Structure", "Duplication", "Naming", "Magic Numbers", "Combined Metrics"]
    y-axis "Score" 0 --> 100
    bar [95, 31, 86, 48, 96, 100, 63]

Loading

Readability       ███████████████████░   95  🟢 A
Complexity        ██████░░░░░░░░░░░░░░   31  🔴 D-
Structure         █████████████████░░░   86  🟢 A-
Duplication       ██████████░░░░░░░░░░   48  🟠 C-
Naming            ███████████████████░   96  🟢 A
Magic Numbers     ████████████████████  100  🟢 A
Combined Metrics  █████████████░░░░░░░   63  🔴 D

[github-actions]

🔍 Top Likely Issues (cosine similarity)

Most negative cosine = file's metric profile best matches this anti-pattern.

Behavior	Cosine	Score
code_smells.no_debug_print_statements	-0.51	-9.41
code_smells.no_dead_code_after_return	-0.45	-25.06
file_structure.single_responsibility	-0.44	-8.65
file_structure.line_count_under_300	-0.44	-8.50
scope_and_assignment.used_only_once	-0.41	-13.52
dependencies.low_coupling	-0.37	-6.30
file_structure.uses_standard_indentation_width	-0.35	-0.05
scope_and_assignment.shadowed_by_inner_scope	-0.34	-4.65
scope_and_assignment.declared_close_to_use	-0.30	1.87
variable_naming.loop_var_is_single_letter	-0.24	3.69

🟢 Readability — A (95/100)

Codebase averages: flesch_adapted=97.99, fog_adapted=4.72, avg_tokens_per_line=9.31, avg_line_length=35.15

Metric	Value	Score
readability.flesch_adapted	97.99	100
readability.fog_adapted	4.72	100
readability.avg_tokens_per_line	9.31	73
readability.avg_line_length	35.15	100

🔴 Complexity — D- (31/100)

Codebase averages: difficulty=41.09, effort=233830.77, volume=4030.66, estimated_bugs=1.34

Metric	Value	Score
halstead.difficulty	41.09	42
halstead.effort	233830.77	0
halstead.volume	4030.66	46
halstead.estimated_bugs	1.34	47

🟢 Structure — A- (86/100)

Codebase averages: branching_density=0.14, mean_depth=3.92, avg_function_lines=8.68, max_depth=10.06, max_function_lines=20.73, variance=7.34, avg_param_count=1.13, max_param_count=1.98

Metric	Value	Score
branching.branching_density	0.14	76
indentation.mean_depth	3.92	88
function_metrics.avg_function_lines	8.68	88
indentation.max_depth	10.06	85
function_metrics.max_function_lines	20.73	90
indentation.variance	7.34	89
function_metrics.avg_param_count	1.13	100
function_metrics.max_param_count	1.98	100

🟠 Duplication — C- (48/100)

Codebase averages: redundancy=0.59, bigram_repetition_rate=0.55, trigram_repetition_rate=0.37

Metric	Value	Score
compression.redundancy	0.59	57
ngram.bigram_repetition_rate	0.55	37
ngram.trigram_repetition_rate	0.37	40

🟢 Naming — A (96/100)

Codebase averages: entropy=0.89, mean=6.66, variance=18.87, avg_sub_words_per_id=1.17

Metric	Value	Score
casing_entropy.entropy	0.89	100
identifier_length_variance.mean	6.66	100
identifier_length_variance.variance	18.87	85
readability.avg_sub_words_per_id	1.17	100

🟢 Magic Numbers — A (100/100)

Codebase averages: density=0.00

Metric	Value	Score
magic_number_density.density	0.00	100

🔴 Combined Metrics — D (63/100)

Category	Score	Grade
Code Smells	40	🔴 D
Consistency	50	🟠 C-
Dependencies	27	🔴 D-
Documentation	83	🟡 B+
Error Handling	92	🟢 A-
File Structure	42	🔴 D+
Function Design	81	🟡 B+
Naming Conventions	80	🟡 B+
Scope And Assignment	28	🔴 D-
Testing	83	🟡 B+
Type And Value	89	🟢 A-
Variable Naming	74	🟡 B

🔴 Code Smells — D (40/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
no_debug_print_statements	-0.51	21	E+
no_dead_code_after_return	-0.45	24	E+
no_fixme_comments	0.27	75	B

Worst offender (lib/codeqa/ast/classification/node_protocol.ex:1-7):

defprotocol CodeQA.AST.Classification.NodeProtocol do
  @moduledoc """
  Common interface for all typed AST node structs.

  All node struct types (CodeNode, DocNode, FunctionNode, etc.) implement this
  protocol, allowing downstream code to work with any node type uniformly.
  """

🟠 Consistency — C- (50/100)

Cosine similarity scores for 0 behaviors.

Behavior	Cosine	Score	Grade

Worst offender (lib/codeqa/languages/code/scripting/lua.ex:29-31):

  def delimiters, do: ~w[
    ( ) { } , . : ;
  ] ++ ~w( [ ] )

🔴 Dependencies — D- (27/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
low_coupling	-0.37	27	D-

Worst offender (lib/codeqa/health_report/delta.ex:32-40):

    |> Enum.reduce(%{}, fn key, acc ->
      case {Map.get(base, key), Map.get(head, key)} do
        {b, h} when is_number(b) and is_number(h) ->
          Map.put(acc, key, Float.round((h - b) * 1.0, 4))

        _ ->
          acc
      end
    end)

🟡 Documentation — B+ (83/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
file_has_module_docstring	0.29	76	B
function_has_docstring	0.43	86	A-
docstring_is_nonempty	0.44	86	A-

Worst offender (lib/codeqa/metrics/file/comment_structure.ex:1-14):

defmodule CodeQA.Metrics.File.CommentStructure do
  @moduledoc """
  Measures comment density and annotation patterns.

  Counts lines that begin with a comment marker (language-agnostic: `#`, `//`,
  `/*`, ` *`) relative to non-blank lines. Also counts TODO/FIXME/HACK/XXX
  markers which indicate deferred work or known issues.

  ## Output keys

  - `"comment_line_ratio"` — comment lines / non-blank lines
  - `"comment_line_count"` — raw count of comment lines
  - `"todo_fixme_count"` — occurrences of TODO, FIXME, HACK, or XXX
  """

🟢 Error Handling — A- (92/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
error_message_is_descriptive	0.51	90	A-
does_not_swallow_errors	0.60	92	A-
returns_typed_error	0.70	94	A

Worst offender (lib/codeqa/languages/code/scripting/php.ex:26-28):

  def operators, do: ~w[
    == === != !== <= >= + - * / % ** << >> & | ^ ~ && || ?? = += -= *= /= %= -> :: =>
  ]

🔴 File Structure — D+ (42/100)

Cosine similarity scores for 4 behaviors.

Behavior	Cosine	Score	Grade
single_responsibility	-0.44	24	E+
line_count_under_300	-0.44	24	E+
uses_standard_indentation_width	-0.35	28	D-
no_magic_numbers	0.56	91	A-

Worst offender: lib/codeqa/health_report/formatter/github.ex:54-73 (20 lines)

🟡 Function Design — B+ (81/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
has_verb_in_name	0.35	80	B+
is_less_than_20_lines	0.35	80	B+
no_magic_numbers	0.37	82	B+

Worst offender (lib/codeqa/ast/classification/node_classifier.ex:69-77):

  @type_modules %{
    doc: DocNode,
    attribute: AttributeNode,
    function: FunctionNode,
    module: ModuleNode,
    import: ImportNode,
    test: TestNode,
    code: CodeNode
  }

🟡 Naming Conventions — B+ (80/100)

Cosine similarity scores for 2 behaviors.

Behavior	Cosine	Score	Grade
function_name_matches_return_type	0.25	73	B
function_name_is_not_single_word	0.45	87	A-

Worst offender (lib/codeqa/engine/file_context.ex:16-28):

  @type t :: %__MODULE__{
          content: String.t(),
          tokens: [CodeQA.Engine.Pipeline.Token.t()],
          token_counts: map(),
          words: list(),
          identifiers: list(),
          lines: list(),
          encoded: String.t(),
          byte_count: non_neg_integer(),
          line_count: non_neg_integer(),
          path: String.t() | nil,
          blocks: [CodeQA.AST.Enrichment.Node.t()] | nil
        }

🔴 Scope And Assignment — D- (28/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
used_only_once	-0.41	25	D-
shadowed_by_inner_scope	-0.34	28	D-
declared_close_to_use	-0.30	30	D-

Worst offender: lib/codeqa/ast/signals/structural/colon_indent_signal.ex:1-17 (17 lines)

🟡 Testing — B+ (83/100)

Cosine similarity scores for 2 behaviors.

Behavior	Cosine	Score	Grade
test_single_concept	0.26	74	B
test_name_describes_behavior	0.56	91	A-

Worst offender (lib/codeqa/languages/code/vm/clojure.ex:30-32):

  def delimiters, do: ~w[
    ( ) { } , . : ; # @ ^
  ] ++ ~w( [ ] )

🟢 Type And Value — A- (89/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
hardcoded_url_or_path	0.48	89	A-

Worst offender (lib/codeqa/metrics/file/inflector.ex:32-45):

  """
  @spec detect_casing(String.t()) ::
          :pascal_case | :camel_case | :snake_case | :macro_case | :kebab_case | :other
  def detect_casing(identifier) do
    cond do
      identifier =~ ~r/^[A-Z][a-zA-Z0-9]*$/ -> :pascal_case
      identifier =~ ~r/^[a-z][a-z0-9]*(?:[A-Z][a-zA-Z0-9]*)+$/ -> :camel_case
      identifier =~ ~r/^[a-z]+(_[a-z0-9]+)*$/ -> :snake_case
      identifier =~ ~r/^[A-Z]+(_[A-Z0-9]+)*$/ -> :macro_case
      identifier =~ ~r/^[a-z]+(-[a-z0-9]+)*$/ -> :kebab_case
      true -> :other
    end
  end
end

🟡 Variable Naming — B (74/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
name_is_generic	0.26	74	B

Worst offender (lib/codeqa/cli/health_report.ex:1-14):

defmodule CodeQA.CLI.HealthReport do
  @moduledoc false

  @behaviour CodeQA.CLI.Command

  alias CodeQA.CLI.Options
  alias CodeQA.Config
  alias CodeQA.Engine.Analyzer
  alias CodeQA.Engine.Collector
  alias CodeQA.Git
  alias CodeQA.HealthReport

  @impl CodeQA.CLI.Command
  def usage do

[github-actions]

🔴 1 critical block — review required before merge

1 critical · 9 high

What	Where	Action
🔴 No Implicit Null Initial	lib/codeqa/metrics/file/inflector.ex:32-45	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:54-73	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:14-25	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:39-47	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:61-65	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:98-105	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:128-133	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:137-140	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:147-157	Review this code block
🟠 Single Responsibility	lib/codeqa/health_report/formatter/github.ex:162-164	Review this code block

🔴 lib/codeqa/metrics/file/inflector.ex:32-45 — No Implicit Null Initial

Issues:

• 🔴 CRITICAL type_and_value/no_implicit_null_initial (Δ 0.74)
• 🟠 HIGH scope_and_assignment/used_only_once (Δ 0.47)
• 🟠 HIGH function_design/cyclomatic_complexity_under_10 (Δ 0.45)

  32 │   """
  33 │   @spec detect_casing(String.t()) ::
  34 │           :pascal_case | :camel_case | :snake_case | :macro_case | :kebab_case | :other
  35 │   def detect_casing(identifier) do
  36 │     cond do
  37 │       identifier =~ ~r/^[A-Z][a-zA-Z0-9]*$/ -> :pascal_case
  38 │       identifier =~ ~r/^[a-z][a-z0-9]*(?:[A-Z][a-zA-Z0-9]*)+$/ -> :camel_case
  39 │       identifier =~ ~r/^[a-z]+(_[a-z0-9]+)*$/ -> :snake_case
  40 │       identifier =~ ~r/^[A-Z]+(_[A-Z0-9]+)*$/ -> :macro_case
  41 │       identifier =~ ~r/^[a-z]+(-[a-z0-9]+)*$/ -> :kebab_case
  42 │       true -> :other
  43 │     end
  44 │   end
  45 │ end

🟠 lib/codeqa/health_report/formatter/github.ex:54-73 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

  54 │   """
  55 │   @spec render_part_2(map(), keyword()) :: String.t()
  56 │   def render_part_2(report, opts \\ []) do
  57 │     detail = Keyword.get(opts, :detail, :default)
  58 │     display_categories = merge_cosine_categories(report.categories)
  59 │     worst_blocks = Map.get(report, :worst_blocks_by_category, %{})
  60 │ 
  61 │     [
  62 │       top_issues_section(Map.get(report, :top_issues, []), detail),
  63 │       category_sections(display_categories, detail, worst_blocks),
  64 │       sentinel(2)
  65 │     ]
  66 │     |> List.flatten()
  67 │     |> Enum.join("\n")
  68 │   end
  69 │ 
  70 │   @doc """
  71 │   Renders Part 3: blocks section (top 10 blocks with code).
  72 │   Returns a list with a single part since blocks are now limited to top 10.
  73 │   """

🟠 lib/codeqa/health_report/formatter/github.ex:14-25 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

  14 │     [
  15 │       pr_summary_section(Map.get(report, :pr_summary)),
  16 │       header(report),
  17 │       cosine_legend(),
  18 │       delta_section(Map.get(report, :codebase_delta)),
  19 │       if(chart?, do: mermaid_chart(display_categories), else: []),
  20 │       progress_bars(display_categories),
  21 │       top_issues_section(Map.get(report, :top_issues, []), detail),
  22 │       blocks_section(Map.get(report, :top_blocks, [])),
  23 │       category_sections(display_categories, detail, worst_blocks),
  24 │       footer()
  25 │     ]

🟠 lib/codeqa/health_report/formatter/github.ex:39-47 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

  39 │     [
  40 │       pr_summary_section(Map.get(report, :pr_summary)),
  41 │       header(report),
  42 │       cosine_legend(),
  43 │       delta_section(Map.get(report, :codebase_delta)),
  44 │       if(chart?, do: mermaid_chart(display_categories), else: []),
  45 │       progress_bars(display_categories),
  46 │       sentinel(1)
  47 │     ]

🟠 lib/codeqa/health_report/formatter/github.ex:61-65 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

  61 │     [
  62 │       top_issues_section(Map.get(report, :top_issues, []), detail),
  63 │       category_sections(display_categories, detail, worst_blocks),
  64 │       sentinel(2)
  65 │     ]

🟠 lib/codeqa/health_report/formatter/github.ex:98-105 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

  98 │         combined = %{
  99 │           type: :cosine_group,
 100 │           key: "combined_metrics",
 101 │           name: "Combined Metrics",
 102 │           score: combined_score,
 103 │           grade: grade_letter_from_score(combined_score),
 104 │           categories: cosine
 105 │         }

🟠 lib/codeqa/health_report/formatter/github.ex:128-133 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

 128 │     [
 129 │       "## #{emoji} Code Health: #{report.overall_grade} (#{report.overall_score}/100)",
 130 │       "",
 131 │       "> #{report.metadata.total_files} files · #{extract_project_name(report.metadata.path)} · #{format_date(report.metadata.timestamp)}",
 132 │       ""
 133 │     ]

🟠 lib/codeqa/health_report/formatter/github.ex:137-140 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

 137 │     [
 138 │       "> *Combined metric scores use cosine similarity: +1 = metric profile perfectly matches healthy pattern for this behavior, 0 = no signal, −1 = anti-pattern detected. Mapped to 0–100 using breakpoints (approx: ≥0.5→A, ≥0.2→B, ≥0.0→C, ≥−0.3→D, <−0.3→F); actual letter grades use the full 15-step scale.*",
 139 │       ""
 140 │     ]

🟠 lib/codeqa/health_report/formatter/github.ex:147-157 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

 147 │     [
 148 │       "```mermaid",
 149 │       "%%{init: {'theme': 'neutral'}}%%",
 150 │       "xychart-beta",
 151 │       "    title \"Code Health Scores\"",
 152 │       "    x-axis [#{names}]",
 153 │       "    y-axis \"Score\" 0 --> 100",
 154 │       "    bar [#{scores}]",
 155 │       "```",
 156 │       ""
 157 │     ]

🟠 lib/codeqa/health_report/formatter/github.ex:162-164 — Single Responsibility

Issues:

• 🟠 HIGH file_structure/single_responsibility (Δ 0.67)
• 🔴 CRITICAL variable_naming/name_is_abbreviation (Δ 0.56)
• 🔴 CRITICAL variable_naming/name_is_too_short (Δ 0.53)

 162 │       Enum.reduce(categories, 0, fn cat, acc ->
 163 │         max(acc, String.length(cat.name))
 164 │       end)