Feat: granular allowlist

[TheDZhon]

Summary

Implements granular allowed_diffs allowlists for both source and bytecode comparisons (closes #136).

Instead of blanket --allow-source-diff/--allow-bytecode-diff CLI flags that silently suppress all diffs for an address, diffs can now be declared precisely in config with mandatory reason fields and composable facets.

Bytecode facets

• immutables — pin expected values at exact byte offsets
• cbor_metadata — allow Solidity CBOR metadata hash differences
• byte_ranges — allow arbitrary byte ranges
• constructor_args / constructor_calldata — override constructor parameters per-rule
• any — blanket allow (also suppresses RPC simulation errors)

Source facets

• line_ranges — allow diffs in specific file regions (github/explorer line counts)
• files — allow diffs in specific files
• any — blanket allow

Auto-suggestions

When diffs are detected but not covered by rules, the tool prints a ready-to-paste JSON allowed_diffs snippet with the exact facets needed. This makes adopting granular rules trivial — run once, copy the suggestion, fill in the reason.

Key changes

Area	What changed
diffyscan/utils/allowed_diffs.py (new, 660 lines)	Validation, evaluation, normalization, and suggestion engine
diffyscan/diffyscan.py	Integrates allowlist evaluation into source/bytecode pipelines; prints suggestions in final summary; "any" bytecode rules now suppress RPC simulation errors
diffyscan/utils/common.py	Config loading validates allowed_diffs structure, boolean fields, and YAML hex quoting for allowlist addresses
diffyscan/utils/custom_types.py	Config TypedDict includes allowed_diffs and missing optional fields
pyproject.toml	Added mypy + type stubs to dev deps; [tool.mypy] config for incremental type checking
Regression configs (6 files)	Replaced all CLI --allow-*-diff flags with precise in-config allowed_diffs entries discovered via real runs against mainnet and hoodi RPCs
.github/workflows/regression.yml	Removed flags matrix, DIFFYSCAN_FLAGS env var, and bash flag-parsing; run command is now just uv run diffyscan -Y -E -G "$DIFFYSCAN_CONFIG"

Config example

{
  "allowed_diffs": {
    "bytecode": {
      "0xd6A6...9426": [{
        "reason": "Escrow stores own deployed address as immutable",
        "immutables": [
          {"offset": 1093, "value": "0x000...9426"},
          {"offset": 6887, "value": "0x000...9426"}
        ]
      }]
    },
    "source": {
      "0x8aa3...9F3": [{
        "reason": "Import paths differ between GitHub and explorer-verified source",
        "line_ranges": [{
          "file": "contracts/CuratedSubmitExitRequestHashes.sol",
          "github": {"start": 6, "count": 7},
          "explorer": {"start": 6, "count": 7}
        }]
      }]
    }
  }
}

Test plan

• 130 unit tests pass (pytest -q) — covers validation, evaluation, edge cases, suggestions, and config loading
• mypy diffyscan/ — 0 errors across 16 source files
• All 12 regression matrix jobs pass (4 mainnet clean, 4 mainnet with allowed_diffs, 2 hoodi with allowed_diffs, 2 mainnet clean governance/voting)
• Verify suggestion output: run any config without its allowed_diffs section and confirm the printed snippet can be pasted back to make it pass

[Copilot]

Pull request overview

This PR implements granular allowed_diffs allowlists for both source and bytecode comparisons, replacing the blanket --allow-source-diff/--allow-bytecode-diff CLI flags. Diffs can now be declared precisely in config files with mandatory reason fields and composable facets (immutables, cbor_metadata, byte_ranges, line_ranges, files, etc.). When uncovered diffs are detected, the tool prints ready-to-paste config snippets.

Changes:

• New allowed_diffs validation, evaluation, normalization, and suggestion engine in diffyscan/utils/allowed_diffs.py
• Refactored bytecode analysis into structured analyze_bytecode_diff() returning detailed mismatch information, with deep_match_bytecode() preserved as backward-compatible wrapper
• Integrated allowlist evaluation into the main source/bytecode pipelines, with suggestion output in final summary

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
diffyscan/utils/allowed_diffs.py	New module: validation, evaluation, normalization, suggestion engine for granular allowlists
diffyscan/utils/binary_verifier.py	Refactored to expose analyze_bytecode_diff() and log_bytecode_diff_analysis(); normalized 0x prefix
diffyscan/diffyscan.py	Integrated allowlist evaluation into source/bytecode pipelines; structured result dicts; suggestion printing
diffyscan/utils/common.py	Config validation for allowed_diffs, boolean fields, YAML hex quoting in allowlist entries
diffyscan/utils/custom_types.py	TypedDict definitions for allowlist rules and missing optional Config fields
pyproject.toml	Added mypy + type stubs to dev deps; mypy and black config
tests/test_allowed_diffs.py	New comprehensive tests for allowlist validation, evaluation, and suggestions
tests/test_config_loading.py	Tests for allowed_diffs config validation edge cases and boolean field validation
tests/test_binary_verifier.py	Tests for new analyze_bytecode_diff function
tests/*.py (other)	Type annotation fixes for mypy compliance
tests/fixtures/full_config.*	Added allowed_diffs sections to test fixtures
config_samples/**/*.json	Replaced CLI flags with in-config allowed_diffs entries
.github/workflows/regression.yml	Removed flags matrix and bash flag-parsing; simplified run command
.pre-commit-config.yaml	Added mypy pre-commit hook
diffyscan/utils/explorer.py, compiler.py, etc.	Type annotation fixes for mypy compliance
uv.lock	Updated lock file with new dev dependencies
README.md	Documentation for granular allowlists feature

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[TheDZhon]

Self-reminder: need to rebase on the latest main and update skills ⚠️

[Copilot]

Pull request overview

This PR introduces a granular allowed_diffs allowlist system for both source and bytecode comparisons, replacing blanket per-address CLI suppression with composable, reasoned, config-driven facets and auto-suggested snippets when diffs aren’t covered.

Changes:

• Added diffyscan/utils/allowed_diffs.py with allowlist validation, rule evaluation, normalization, and suggestion rendering.
• Integrated allowlist evaluation + suggestion reporting into the source and bytecode diff pipelines (including constructor override facets and deprecating --allow-*-diff as any).
• Hardened config loading/type-checking (mypy/dev deps, boolean validation, YAML hex quoting validation) and updated regression configs + workflow accordingly.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
uv.lock	Updates dev dependency lock to include mypy and type stubs; bumps black.
pyproject.toml	Adds mypy + stubs to dev deps and configures [tool.mypy].
.pre-commit-config.yaml	Adds a mypy pre-commit hook.
.github/workflows/regression.yml	Removes flag matrix/flag parsing; runs diffyscan directly and adds mypy to CI.
.env.example	Removes LOCAL_RPC_URL example entry.
README.md	Documents the new allowed_diffs schema and behavior.
diffyscan/utils/allowed_diffs.py	New: allowlist schema validation, evaluation engine, and suggestion snippet rendering.
diffyscan/diffyscan.py	Integrates allowlists into source/bytecode workflows; prints suggestion snippets; deprecates CLI allow flags.
diffyscan/utils/common.py	Validates boolean fields and allowed_diffs; expands YAML hex-int coercion checks to allowed_diffs.
diffyscan/utils/custom_types.py	Extends config TypedDicts to include allowed_diffs structures and optional fields.
diffyscan/utils/binary_verifier.py	Adds structured analyze_bytecode_diff + logging; keeps deep_match_bytecode wrapper.
diffyscan/utils/explorer.py	Tightens typing in a few helpers; minor refactors/returns.
diffyscan/utils/github.py	Adds explicit casts/type ignores for cached content return typing.
diffyscan/utils/node_handler.py	Casts RPC result to str on return.
diffyscan/utils/compiler.py	Adds explicit dict casts/typing around JSON decoding and returns.
diffyscan/utils/calldata.py	Returns a list copy for constructor ABI inputs.
diffyscan/utils/encoder.py	Adjusts tuple encoding typing and joins to satisfy mypy.
tests/test_allowed_diffs.py	New: unit tests for allowlist validation/evaluation/suggestions.
tests/test_binary_verifier.py	Adds tests for new bytecode diff analysis helper.
tests/test_config_loading.py	Adds tests for allowed_diffs validation + boolean field validation; refactors YAML string literals.
tests/test_utils.py	Adds typing and mypy ignores for intentional type mismatch tests.
tests/test_refactored.py	Adds mypy ignore and typing tweaks.
tests/test_github_utils.py	Adds assertion to satisfy non-optional repo usage.
tests/test_bytecode_metadata.py	Adds typing annotations for mypy.
tests/fixtures/full_config.yaml	Adds fixture allowed_diffs entries.
tests/fixtures/full_config.json	Adds fixture allowed_diffs entries.
config_samples/ethereum/mainnet/vaults/vaults_easy_track_config.json	Migrates prior CLI allow flags into granular allowed_diffs rules.
config_samples/ethereum/mainnet/vaults-wrapper/vaults_wrapper_factory_config.json	Adds bytecode immutable allowlist rules.
config_samples/ethereum/mainnet/tw/tw_easy_track_config.json	Adds source+bytecode allowed_diffs for known import-path/metadata diffs.
config_samples/ethereum/mainnet/tw/tw_config.json	Adds bytecode metadata allowlist rules.
config_samples/ethereum/mainnet/gualGovernance/dual_governance_mainnet_config.json	Adds immutable allowlist rules; formatting updates.
config_samples/ethereum/hoodi/vaults/hoodi_vaults_testnet_config.json	Adds allowed_diffs for source/bytecode in testnet configs; formatting updates.
config_samples/ethereum/hoodi/vaults/hoodi_vaults_easy_track_config.json	Adds allowed_diffs for hoodi easy track configs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

evaluate_bytecode_rules() calls analysis_provider(rule) unconditionally inside the loop. For constructor override rules, analysis_provider can raise (e.g., RPC simulation NodeError / calldata errors), which will abort evaluation and prevent later rules from being considered—even if another rule (like cbor_metadata or byte_ranges) would have allowed the diff. Consider catching expected exceptions around analysis_provider(rule) per-rule (logging and treating that rule as non-matching) so evaluation can continue and only fail if no rule matches.

[Copilot]

_range_is_fully_immutable() checks immutability by iterating every byte in the mismatch range and scanning all immutable regions (_offset_in_immutable), which is O(mismatch_bytes × immutable_regions). For large mismatch ranges this can get expensive and may slow down bytecode comparisons. Consider rewriting this as an interval-coverage check (e.g., treat immutables as sorted [start,end) ranges and verify each mismatch range is fully covered without per-byte iteration).

Suggested change

	return all(
	_offset_in_immutable(byte_offset, immutables)
	for byte_offset in range(offset, offset + length)
	)
	"""
	Return True if the byte range [offset, offset + length) is fully covered by
	immutable regions, using an interval-coverage check instead of per-byte scans.
	"""
	# Preserve previous behavior: empty ranges are considered fully immutable.
	if length == 0:
	return True
	if not immutables:
	return False
	range_start = offset
	range_end = offset + length
	# Build and sort immutable intervals as [start, end) pairs.
	intervals = sorted(
	(start, start + imm_len) for start, imm_len in immutables.items()
	)
	current = range_start
	for start, end in intervals:
	# Skip intervals that end before or exactly at the current coverage point.
	if end <= current:
	continue
	# If there's a gap between the current coverage and the next interval start,
	# the range is not fully immutable.
	if start > current:
	break
	# Extend coverage.
	if end > current:
	current = end
	if current >= range_end:
	return True
	return False