Simcheck

Overview

This repository contains SimCheck.ipynb, a small notebook that demonstrates multiple ways to compute similarity between Python code snippets (syntactic, semantic, and structural). The notebook generates simple variants of a function and computes several metrics to compare each variant to a baseline implementation.

Quick Summary of Results

• Baseline: the first generated function in generated_codes.
• Variants: four other implementations generated by the notebook.
• Key observation: Variant code_0_vs_4 (the factors(y) variant) scored highest on both syntactic and semantic (BERT) similarity in the sample run.

Results (sample run):

comparison	syntactic_score	bert_score
code_0_vs_1	0.519	0.920
code_0_vs_2	0.442	0.897
code_0_vs_3	0.498	0.903
code_0_vs_4	0.627	0.946

• Correlation between syntactic and BERT (semantic) similarity in this run: ~0.97 (strong positive correlation).

Files

• SimCheck.ipynb: The main notebook. Run this to reproduce everything. See the "Sequence of the work" section for which cells produce which outputs.

Dependencies

• Python packages installed in the notebook cell: nltk, scikit-learn, scipy, pycode-similar, bert-score, seaborn, plotly, pandas, autopep8.
• The notebook downloads NLTK resources (punkt and punkt_tab) at runtime to ensure tokenizers work across environments.

How to run (recommended)

1. From a terminal in this repository (recommended inside a virtualenv or container):

# run the notebook interactively using Jupyter
jupyter notebook SimCheck.ipynb

2. Execute the cells top-to-bottom. The first cell installs required packages and downloads NLTK tokenizer resources.

3. Cells to check for outputs (by cell number in the notebook):

• Cell 1: installs packages and imports libraries (also downloads NLTK models).
• Cell 4: defines compute_syntactic_score and helper functions.
• Cell 5: generates generated_codes (code variants).
• Cell 8: displays formatted code variants.
• Cell 9: computes syntactic and BERT scores and creates results_df (DataFrame of results).
• Cell 10: static Matplotlib/Seaborn plots (bar chart and scatter).
• Cell 11: heatmap of syntactic sub-metrics.
• Cell 12: summary statistics and correlation.
• Cell 13: interactive Plotly scatter.

Note: cell numbers above are relative to the included SimCheck.ipynb file. If you add/remove cells, re-check the cell positions.

Snapshots (how to save them)

The notebook renders the following useful visualizations. To save them as files, add the plt.savefig(...) commands after each plotting cell or run the following helper steps.

1. Create a directory for plots:

mkdir -p plots

2. Save Matplotlib figures inside the notebook (example):

# After the barplot/scatter plotting code cell
plt.savefig('plots/syntactic_scores_bar.png', bbox_inches='tight')

3. Save the heatmap similarly:

plt.savefig('plots/syntactic_metrics_heatmap.png', bbox_inches='tight')

4. Save Plotly figure:

fig.write_image('plots/interactive_scatter.png')

You can then include these snapshots in reports or documentation. The notebook outputs already show the sample visuals (rendered when running interactively).

Deep Dive: Code Comparison Logic

The notebook implements three complementary approaches to code similarity:

1. Syntactic Similarity (text-based metrics)

Syntactic similarity measures how alike two code snippets are at the surface level (character and token sequences). It aggregates seven distinct metrics:

a) Sequence Similarity (SequenceMatcher ratio)

• Uses Python's difflib.SequenceMatcher to find the longest contiguous matching subsequence.
• Normalized to [0,1] where 1.0 means identical sequences.
• Captures overall structural alignment but ignores rearrangement.
• Example: def f(n): return [i for i in range(1, n)] vs def g(n): return [x for x in range(1, n)] → high similarity (only variable names differ).

b) Longest Common Subsequence (LCS)

• Uses SequenceMatcher.find_longest_match() to find the longest substring present in both code samples.
• Normalized by max(len(code1), len(code2)).
• Captures the longest contiguous block of identical code.
• Example: the actual loop logic [i for i in range(1, n+1) if n % i == 0] can match across variants.

c) Edit Distance Score (Levenshtein distance)

• Minimum number of single-character edits (insert, delete, substitute) needed to transform one string into another.
• Formula: 1 - (edit_distance / max_length) to normalize to [0,1].
• Lower distance → higher similarity.
• Example: changing variable names requires fewer edits than changing loop structure.

d) Jaccard Similarity (token-level)

• Tokenizes both code strings using NLTK's word_tokenize().
• Computes Jaccard distance on token sets: 1 - jaccard_distance(set(tokens1), set(tokens2)).
• Captures vocabulary overlap (keywords, variable names, operators).
• Example: two variants using the same operators (if, range, append) will have high Jaccard.

e) N-Gram TF-IDF Cosine Similarity

• Builds n-grams (default n=2: bigrams) from token sequences.
• Computes TF-IDF vectors and calculates cosine similarity between them.
• Captures phrase-level overlap while accounting for term importance.
• Example: the bigram [i for appears in multiple variants and contributes to high similarity.

f) Hamming Distance Score

• Aligns both strings to the same length (pad shorter one) and compares character-by-character.
• Formula: 1 - (hamming_distance / max_length).
• Two modes: regular (positional) or sorted (order-insensitive).
• Example: character-level insertion/deletion impact is captured.

g) Sorensen–Dice Coefficient

• Approximates Dice using character-presence binary vectors.
• For each code snippet, creates a binary vector where b[i] = 1 if char exists in other snippet else 0.
• Computes F1 score on these vectors (captures shared character set).
• Example: two variants using the same characters (even in different order) have similar Dice scores.

Aggregate Syntactic Score: Simple arithmetic mean of all seven metrics.

• Rationale: Equal weighting assumes all metrics are equally informative; you can modify to use weighted combinations.

---

2. Semantic Similarity (BERT-based)

Semantic similarity measures whether two code snippets express the same intent/functionality, regardless of surface form.

BERTScore

• Uses a pretrained RoBERTa Transformer model to embed both code snippets into contextual vector representations.
• Compares embeddings via cosine similarity (precision, recall, F1).
• We use the F1 score as the final metric (bert_score).
• Key characteristic: BERTScore captures meaning and intent, not literal text matching.
• Example:

  # Variant 1: list comprehension
  def f(n): return [i for i in range(1, n+1) if n % i == 0]
  
  # Variant 2: explicit loop
  def f(n): 
      result = []
      for j in range(1, n+1):
          if n % j == 0: result.append(j)
      return result

  Despite different syntax, both compute divisors → high BERT similarity (~0.92).

Why Both Syntactic and Semantic?

• Syntactic catches literal text differences (useful for detecting code clones, plagiarism).
• Semantic catches functional equivalence (useful for code review, refactoring detection).
• A strong correlation (as observed ~0.97 in sample) suggests syntax and semantics align well for simple code variants.

---

3. Structural Similarity (AST-based) — Optional

Structural similarity analyzes the Abstract Syntax Tree (AST) of Python code.

How it works (using pycode_similar):

• Parses each code snippet into an AST (removes syntactic sugar, represents structure).
• Compares ASTs using either UnifiedDiff or TreeDiff modes.
• Returns a plagiarism ratio (0..1) indicating how much AST structure is shared.
• Example: a while-loop and for-loop with the same logic have different ASTs but may share high node counts.

Note: Structural similarity is commented out in the default scoring pipeline for performance. Uncomment if you need AST-level analysis.

---

Highlights / Interpretation

• The syntactic score captures literal and structural text similarity but not runtime behavior. Useful for detecting copy-paste code with minor edits.
• The BERT score captures functional equivalence and intent. Useful for identifying semantically similar code written in different styles.
• In the sample run, code_0_vs_4 (the factors(y) variant) scored highest on both syntactic (0.627) and semantic (0.946) metrics because:
  • Syntactically: it uses the same list-comprehension pattern as the baseline.
  • Semantically: it expresses the identical divisor-finding logic with minimal vocabulary change.
• Variants using while-loops or nested sets (like code_0_vs_3) were less similar syntactically (0.498) but still semantically close (0.903) because the fundamental algorithm remained the same.
• The near-perfect correlation (~0.97) between syntactic and BERT scores suggests that for simple algorithmic code, surface structure often mirrors intent.

Performance / Practical notes

• bert-score uses a pretrained Transformer (RoBERTa/RoBERTa-large by default) — this causes downloads and can be slow and memory intensive. To speed up runs, change the model used by bert-score:

# faster / smaller model
_, _, F1 = score([code], [baseline], lang='en', model_type='distilroberta-base')

• If you prefer to avoid runtime NLTK downloads entirely, replace word_tokenize with a simple regex tokenizer (I can patch this for a lighter dependency).

Reproducible execution (non-interactive) To execute the notebook headlessly and save outputs (including updated cells you may add to save figures):

jupyter nbconvert --to notebook --execute SimCheck.ipynb --inplace

Or export to HTML for sharing:

jupyter nbconvert --to html SimCheck.ipynb

---

Workflow & Data Flow

Step-by-step execution flow:

1. Setup (Cell 1)

   • Install dependencies: nltk, scikit-learn, scipy, pycode-similar, bert-score, seaborn, plotly, pandas.
   • Download NLTK tokenizer models (punkt, punkt_tab).
   • Import all required libraries.

2. Define Similarity Functions (Cells 3–4)

   • compute_similarity(): AST-based helper (converts pycode_similar output to scores).
   • structural_similarity(): Compares code ASTs using UnifiedDiff or TreeDiff modes.
   • compute_syntactic_score(): Aggregates 7 text-based metrics (sequence, LCS, edit distance, Jaccard, TF-IDF, Hamming, Dice).
   • syntactic_similarity_driver(): Wrapper to compare all variants against baseline.

3. Generate Code Variants (Cell 5)

   • Creates 5 implementations of a "find divisors" function:
     • Variant 0 (baseline): List comprehension.
     • Variant 1: Explicit for-loop with append.
     • Variant 2: While-loop with counter.
     • Variant 3: Set-based with sqrt optimization.
     • Variant 4: Alternate function name.

4. Display Formatted Code (Cells 6–8)

   • Install autopep8 for code formatting.
   • Display all variants with syntax highlighting for visual inspection.

5. Compute Similarity Scores (Cell 9)

   • For each variant (1–4), compute:
     • Syntactic score: aggregate of 7 metrics (outputs 7 sub-metrics + aggregate).
     • BERTScore (F1): semantic similarity via Transformer embeddings.
   • Store results in results_df DataFrame.
   • Save to plots/results.csv.

6. Generate Visualizations (Cells 10–13)

   • Cell 10: Bar chart of aggregate syntactic scores + scatter (syntactic vs. BERT).
   • Cell 11: Heatmap of all 7 syntactic sub-metrics across variants.
   • Cell 12: Summary statistics (mean scores, top variant, correlation).
   • Cell 13: Interactive Plotly scatter (saved as HTML).

Data structures:

results_df (main output):

Column	Type	Description
comparison	str	Label (e.g., "code_0_vs_1")
syntactic_score	float	Aggregate of 7 metrics
bert_score	float	Semantic similarity (F1)
sequence_similarity	float	SequenceMatcher ratio
edit_distance_score	float	Normalized Levenshtein distance
jaccard_similarity	float	Token set overlap
cosine_similarity_score	float	TF-IDF cosine
hamming_distance_score	float	Character-level alignment
sorensen_dice_coefficient	float	Dice coefficient
longest_common_subsequence	float	LCS ratio

Output files (in plots/ directory):

• results.csv — Full results table.
• syntactic_scores_bar.png — Bar chart of aggregate scores.
• syntactic_vs_bert_scatter.png — 2D scatter of syntactic vs. semantic.
• syntactic_metrics_heatmap.png — Heatmap of all 7 sub-metrics.
• interactive_scatter.html — Interactive Plotly visualization (fully zoomable/draggable).

---

Interpreting Results

Reading the outputs:

1. High Syntactic + High BERT (e.g., code_0_vs_4):

   • Code is textually similar and functionally equivalent.
   • Likely refactored with minor variable/style changes.

2. Low Syntactic + High BERT (e.g., code_0_vs_3):

   • Code is structurally different but semantically equivalent.
   • Different algorithm (e.g., while-loop vs. comprehension) achieving same goal.
   • Example: the sqrt-optimization variant reduces search space differently but finds the same divisors.

3. High Syntactic + Low BERT (rare):

   • Code is textually similar but semantically different.
   • Likely indicates token-level similarity masking behavioral difference.

4. Low Syntactic + Low BERT (minimal overlap):

   • Completely different implementation.

Correlation analysis:

• Strong positive correlation (~0.97 in sample) indicates syntax and semantics align.
• This is typical for simple, deterministic code.
• For complex code with multiple implementations, correlation may be weaker.

---

Generated from the current run of SimCheck.ipynb — run interactively to reproduce the plots and CSV results.