RFC: Future Architecture Direction

[TFM-HITESH]

I and @shreyaGupta1202 feel that vaar is at the point where a refactor is needed and worth it. Wanted to discuss some of the important details here. We want to follow the following general direction and separation of vaar into the following modules :

Scope = decide boundaries and which areas to be searched through for extracting info. (eg- envfile search, repo search)

Sources = extract source-specific facts (eg- sources/dotenv for all parsing of .env files, sources/githubactions for all parsing of actions)

Analysis = combine and index facts and generate snapshots (and update/generate artifacts in the future). Will have the required structs of processable info. (Thinking of creating 2 snapshots. 1 with keys + values to support lint and diff etc., 1 with keys only - to support Query)

Engines = apply command-specific meaning to analyses to get meaningful results

Results = represent outcomes to be consumed by /output

Output = adapt outcomes for consumers such as file artifacts, JSON/SARIF, terminal outputs, API/web payloads, GUI outputs and so on

Mutations = perform controlled changes such as --fix. Both destructive and non destructive changes

FS = low-level shared filesystem primitives like walking directories, reading files, symink handling, writing files etc.

CLI = orchestrate the complete workflow. Root from which Cobra can operate.

flowchart LR
    Scope --> Sources --> Analysis --> Engines --> Results --> Output
    Engines -. approved changes .-> Mutations
    Mutations -. rebuild .-> Scope

Loading

After this is done, we will be able to safely release v0.1.0 officially

[TFM-HITESH]

Proposed architecture :

flowchart TB
    CLI["CLI / Application Boundary<br/>Commands, flags, orchestration, exit codes"]

    Scope["Scope<br/>Decides where and what to analyse"]

    subgraph Sources["Sources — extract source-specific information"]
        Dotenv["Dotenv source<br/>Lexer, parser, model"]
        Code["Codebase source<br/>Language-specific usage extraction"]
        Contract["Contract source<br/>Schema and requirements"]
        Infra["Infrastructure sources<br/>Docker, CI, frameworks"]
        External["External providers<br/>Optional integrations"]
    end

    Analysis["Analysis<br/>Normalises facts, builds indexes,<br/>provenance and analysis snapshot"]

    subgraph Engines["Command engines — apply meaning"]
        Lint["Lint"]
        Diff["Diff"]
        Query["Query"]
    end

    subgraph Results["Typed results"]
        LintResult["lint.Result"]
        DiffResult["diff.Result"]
        QueryResult["query.Result"]
    end

    subgraph Output["Output adapters"]
        Text["Terminal text"]
        JSON["JSON"]
        SARIF["Future: SARIF / CI annotations"]
        API["Future: API / GUI payload"]
    end

    subgraph Consumers["Consumers"]
        Terminal["CLI users"]
        Automation["Agents / automation"]
        CI["CI systems"]
        GUI["Future GUI"]
    end

    Mutations["Mutations<br/>Validate plans, backup,<br/>atomic write, rollback"]

    FS["Filesystem utilities<br/>Walking, paths, safe reads,<br/>permissions, atomic operations"]

    CLI --> Scope
    Scope --> Dotenv
    Scope --> Code
    Scope --> Contract
    Scope --> Infra
    Scope --> External

    Dotenv --> Analysis
    Code --> Analysis
    Contract --> Analysis
    Infra --> Analysis
    External --> Analysis

    Analysis --> Lint
    Analysis --> Diff
    Analysis --> Query

    Lint --> LintResult
    Diff --> DiffResult
    Query --> QueryResult

    LintResult --> Text
    LintResult --> JSON
    LintResult --> SARIF

    DiffResult --> Text
    DiffResult --> JSON

    QueryResult --> Text
    QueryResult --> JSON
    QueryResult --> API

    Text --> Terminal
    JSON --> Automation
    SARIF --> CI
    API --> GUI

    Lint -. optional fix plan .-> Mutations
    Mutations -. request re-analysis .-> CLI

    Scope --> FS
    Dotenv --> FS
    Code --> FS
    Contract --> FS
    Infra --> FS
    Mutations --> FS

Loading

[TFM-HITESH]

Please note that some of the stuff discussed here (such as analysis of code, dockerfiles etc) are planned in our future roadmap and not scoped out yet. This refactor is just to make the codebase ready to handle such patterns and requirements eventually without turning into a messy slop

[TFM-HITESH]

A visual representation of the mermaid architecture we generated using eraser.io AI because we wanted something pretty and colourful to look at 😺

[TFM-HITESH]

I will be opening a new discussion soon about Query. It is a new feature I and @shreyaGupta1202 have been planning to add to vaar to increase its capabilities and utilities, esp for agentic dev workflows

[TFM-HITESH]

@Nitjsefnie what are your thoughts on this ?

[Nitjsefnie]

Thanks for looping me in — the direction reads well. Splitting Scope → Sources → Analysis → Engines → Results → Output, with Mutations as a side-channel, cleanly separates "what we know" from "what a command means" from "what we change." A few thoughts from having just spent time in the lint/fix internals:

Mutations as a first-class module is the right call — and it structurally kills a bug class we just hit. The fix-halves work (#45) decomposed the whole-file Normalize into per-rule transforms, and the --only/--skip scoping (#54) exposed that those transforms carried an implicit "line-ending normalization runs first" dependency: a scoped --only=ending-blank-line on a CRLF or lone-CR file could leave its own finding unrepaired, or emit a mixed line ending. Giving Mutations its own explicit, rule-addressable fix order and a delimiter/line model it owns — rather than relying on composition order — makes that whole class impossible by construction instead of tested-against.

The line model belongs in Sources/dotenv, and should be lossless. Today the lexer records a lone \r as LineEndingNone (there's no CR constant), so a lone-CR file can't round-trip byte-for-byte. If Analysis is going to carry provenance into Output (SARIF/CI annotations) and Mutations, it's worth firming that delimiter model in the dotenv source so everything downstream can rely on exact source spans and original bytes.

On the two snapshots (keys+values for lint/diff, keys-only for Query): sensible. The thing I'd guard is that provenance — line/col plus original byte offsets — survives into both, since Output adapters (SARIF, API) and Mutations both need to map a result back to an exact span.

Query for agentic workflows sounds genuinely useful — happy to dig in when you open that discussion. And glad to help chip at the refactor where it's useful; the Mutations extraction in particular is close to what #45/#54 have been circling.

[TFM-HITESH]

Agreed. Lets continue with the slightly longer way of doing things for now in #45/#54. When the refactor is eventually done properly, our codebase would be much more mature in handling these kinds of problems/bugs.

lll go ahead with opening issues for this effort and will loop you in on some of them.

[TFM-HITESH]

Also, what are your thoughts @Nitjsefnie about how to go about this refactor ? I dont want to just issue dump too many linked and dependent issue chains. That will stop new contributors and others from being able to freely work without constant rebasing and handling merge conflicts.

Ideally, the refactor occurs in small atomic sections acc to the plan, which doesnt disrupt other coders due to maintaining simple interfaces and patterns.