Skip to content

ReviewPulse v3.0.0-rc.1 — ABSA Release Candidate

Pre-release
Pre-release

Choose a tag to compare

View all tags
@lfariabr lfariabr released this 22 Jul 18:26
v3.0.0-rc.1
f980720

ReviewPulse v3.0.0 — Aspect-Based Sentiment Analysis

Status: release candidate
Track: DLE602 Assessment 3
Last updated: 2026-07-23

Summary

ReviewPulse v3.0.0 is the DLE602 evolution of the product delivered for ISY503. Versions 1.x and 2.x classify the overall sentiment of an Amazon review. Version 3.0 adds an isolated aspect-based sentiment analysis (ABSA) workflow that predicts negative, neutral or positive for each restaurant aspect supplied by the user.

The legacy v2.3.0 workflow remains available and its binary API and artifacts are not reused by v3. The new implementation lives under src/absa/, with separate data contracts, model artifacts, evaluation outputs and application entry points.

This document is the consolidated delivery record for v3. Detailed experiment contracts remain in docs/dle602-a3/, and GitHub issue #72 remains the implementation tracker.

Delivery Status

Area Delivered in the candidate Remaining release gate
Data and task contract Restaurants parser, offset audit, fixed labels, grouped splits and mixed-polarity subset None for core scope
Four-model ladder TF-IDF, target-agnostic LSTM, ATAE-LSTM and DistilBERT sentence pair Regenerate the final four verified artifacts together
Training controls Seeded runs, regularisation, early stopping, best-checkpoint restoration and run metadata Review the final overfitting diagnostics
Common evaluation Reproducible runner, shared predictions, metrics, efficiency evidence, confusion matrices and error candidates Complete issue #84 from reviewed code and final artifacts
Application Separate v2/v3 pages, manual multi-aspect input, sample generation and controlled missing-artifact errors Final clean-process smoke test
Token evidence ATAE-LSTM attention payload and indicative-evidence contract Complete the user-facing ATAE/DistilBERT evidence work in issue #85
Academic package Protocols and traceable machine-readable outputs Write the evidence-backed report in issue #88
Release package Release notes and documented commands Clean install, package, tag and archive in issue #89

1. Data and Task Contract

The v3 core dataset is the SemEval-2014 Task 4 Restaurants domain. Laptops, automatic aspect extraction and topic modelling are outside the core release.

  • Training and evaluation use the SemEval gold aspect terms.
  • The application asks the user to enter one or more aspects manually.
  • Labels have the fixed order negative, neutral, positive.
  • The original SemEval conflict label is counted during audit and excluded from three-class modelling.
  • The mixed-polarity multi-aspect subset contains sentences with at least two retained aspects assigned different polarities.
  • Development splits are grouped by sentence so instances from the same sentence cannot leak across partitions.
  • Raw sentence text and character offsets remain authoritative; v2 text cleaning is not applied to ABSA alignment.

Acquisition, licensing constraints and checksum recording are documented in docs/dle602-a3/semeval-restaurants.md.

2. Four-Model Comparison Ladder

All four models use the same retained examples, label mapping and official Restaurants test partition.

Model Input contract Purpose
TF-IDF + Logistic Regression Review only Lightweight classical baseline
Target-agnostic LSTM Review only Controlled neural baseline without aspect conditioning
ATAE-LSTM Review and aspect Lightweight aspect-conditioned model with attention weights
DistilBERT sentence pair Review and aspect Pretrained contextual aspect-conditioned model

The review-only models intentionally do not receive the target aspect. Their behaviour on mixed-polarity sentences is part of the comparison, rather than an accidental input mismatch. No v2 binary checkpoint or metric is accepted as v3 evidence.

3. Reproducible Training

Issue #91 aligned the implementation with the submitted A2 methodology:

  • Python, NumPy and PyTorch are seeded before split, model and data-loader creation.
  • CUDA, Apple MPS and CPU device selection are supported where applicable.
  • Neural optimisers use recorded weight decay.
  • Development macro-F1 selects checkpoints; the best state is restored before final metrics are produced.
  • Early stopping, complete epoch history, best epoch and overfitting diagnostics are persisted.
  • Training duration, configuration, dataset checksums, seed and source commit are stored with the artifacts.
  • Loss functions are instantiated outside batch loops.

The exact defaults and the fixed-seed versus multi-seed decision rule are defined in docs/dle602-a3/training-protocol.md.

4. Common Evaluation and Error Analysis

Issue #84 provides one command that loads the four verified model families and evaluates them over one shared official-test prediction set.

The evaluation records:

  • accuracy, macro-F1, per-class metrics and confusion matrices;
  • full-test and mixed-polarity multi-aspect results;
  • training time, artifact size, cold start, first-prediction time, warm latency and throughput;
  • representative conditioned-model wins, review-only wins, disagreements and common errors;
  • source commit, seed, device, dataset checksums and the SHA-256 digest of the shared predictions.

Generated files below outputs/absa/evaluation/ are:

File Release evidence
results.json Metrics, environment, provenance and efficiency measurements
predictions.csv Shared gold labels and four-model predictions
comparison.md Report-ready comparison table
confusion_matrices.png Four official-test confusion matrices
error_analysis.json Ordered error categories and candidate examples

The default runner rejects older artifacts that do not contain the reproducibility metadata introduced by issue #91. The diagnostic override for legacy local files must not be used for report claims. Final numerical results will be added only after all four artifacts are regenerated from the same reviewed commit and verified inputs.

5. Application Experience

The Streamlit landing page presents two deliberately separate product phases:

  • v2.3.0 / ISY503: binary sentiment for one complete Amazon review.
  • v3.0.0 / DLE602: three-class sentiment for one or more manually entered restaurant aspects.

The v3 page supports comma-separated aspects, ordered per-aspect results, model selection and sample generation for repeatable demonstrations. TF-IDF and the target-agnostic LSTM receive only the review; ATAE-LSTM and DistilBERT receive the review-and-aspect pair. Missing data, dependencies or model artifacts produce controlled application errors and never fall back silently to a v2 model.

6. Indicative Token Evidence

Interpretability claims are intentionally limited.

  • ATAE-LSTM can expose aligned attention weights.
  • DistilBERT token attribution or attention visualisation is tracked in issue #85.
  • TF-IDF and the target-agnostic LSTM are not required to produce a heatmap.
  • Any displayed weights are indicative token-level evidence, not model reasoning or a causal explanation.

The final v3 release requires the supported evidence views, legends, alignment checks and caveats from issue #85. Until that issue closes, the evidence presentation is incomplete even though aspect predictions are functional.

7. Reliability and Safety

The candidate includes focused protection for the new pipeline:

  • external SemEval XML is parsed with defusedxml;
  • parser tests fail when available source XML unexpectedly produces no examples;
  • split tests assert deterministic sentence-level separation;
  • checkpoint tests protect label mappings, configuration and clean loading;
  • legacy v2 and isolated v3 workflows have non-regression coverage;
  • the current full-suite result on this candidate branch is 230 passed, 8 skipped.

The eight skips apply when the optional, gitignored legacy Amazon dataset is absent.

Canonical Commands

From the repository root in the recorded Python environment:

# Verify the isolated v3 path.
.venv/bin/python -m pytest tests/absa -q
.venv/bin/python scripts/smoke_absa.py

# Regenerate all four verified artifacts.
.venv/bin/python -m src.absa.training.runner --device auto

# Produce the common comparison package.
.venv/bin/python -m src.absa.evaluation.runner --device auto

# Run the full product test suite and application.
.venv/bin/python -m pytest tests/ -q
.venv/bin/streamlit run app.py

SemEval source files and generated model/evaluation artifacts remain local and gitignored. The release package must preserve their checksums and reproducibility metadata without committing restricted data or large binaries.

Known Limitations

  • Core scope covers Restaurants only.
  • Users must provide aspects; v3 does not extract them automatically.
  • The SemEval benchmark is small, so model comparisons must retain the recorded overfitting and variance caveats.
  • Attention or attribution does not establish a faithful causal explanation.
  • Final performance and efficiency claims depend on a common verified run and must identify the recorded device.
  • Large model artifacts are local rather than hosted.

Issue Coverage and Final Gates

Completed foundation: issues #73–#83, #86, #87 and #91.

Open release path:

  1. #84 — merge the reviewed four-model comparison and regenerate final evidence.
  2. #85 — complete indicative token-evidence views.
  3. #88 — write the A3 implementation report from verified outputs.
  4. #89 — verify a clean install, assemble the package and create the v3.0.0 tag.

This file becomes the final release record when those gates are closed and the status is changed from release candidate to released.

Assets 2
Loading