SemanticScatter

SemanticScatter is a Python library for building an interactive semantic scatter plot from textual data. It ingests your dataset into DuckDB, computes sentence embeddings, reduces them to 2D with t-SNE, and opens an interactive JScatter/Solara dashboard for exploration.

Features

• Load data from a file path, a list of dictionaries, or a tabular object such as a pandas DataFrame or Arrow table.
• Map your dataset columns to the library concepts with a simple ColumnMapping object.
• Compute sentence embeddings with sentence-transformers.
• Reduce embeddings to 2D with openTSNE.
• Persist data, embeddings, and reductions in DuckDB.
• Visualize and filter results through an interactive Solara + JScatter web app.

Requirements

SemanticScatter is designed for Python 3.10+ and depends on:

• duckdb
• jupyter-scatter
• numpy
• openTSNE
• pynndescent
• pyarrow
• sentence-transformers
• solara
• joblib

Notebook and JupyterLab support packages are also included in requirements.txt.

Installation

To avoid conflicts with other packages on your system, it is highly recommended to install SemanticScatter inside a virtual environment.

1. Create a virtual environment in the project folder:

   python -m venv venv

2. Activate the virtual environment:

   • On Windows:

     venv\Scripts\activate

   • On macOS/Linux:

     source venv/bin/activate

3. Install the required packages:

   pip install -r requirements.txt

Quick start

from semantic_scatter.core.api import SemanticScatter
from semantic_scatter.core.interfaces import ColumnMapping
from semantic_scatter.engines.database_engine import DuckDBEngine
from semantic_scatter.engines.embedding_engine import SentenceEmbeddingEngine
from semantic_scatter.engines.ingestion_engine import IngestionEngine
from semantic_scatter.engines.reduction_engine import TsneReductionEngine
from semantic_scatter.engines.jscatter_visual_engine import JScatterVisualizationEngine

app = SemanticScatter(
    db_engine=DuckDBEngine("semantic_scatter_database.db"),
    ingestion_engine=IngestionEngine(),
    embedding_engine=SentenceEmbeddingEngine(),
    reduction_engine=TsneReductionEngine(),
    visualization_engine=JScatterVisualizationEngine(),
)

mapping = ColumnMapping(
    id_col="id",
    text_col="abstract",
    color_col="category",
    label_col="title",
    tooltip_cols=["authors", "year"],
    extra_cols=[]
)

# Remember to customize the file path and column names to match your local dataset
app.load_data("path/to/your_dataset.csv", mapping, overwrite=False)
app.embed(batch_size=1000)
app.reduce()
app.show(min_label_freq=0)

Data mapping

ColumnMapping tells SemanticScatter how your dataset should be interpreted:

• id_col: unique identifier column from your dataset.
• text_col: column containing the text to embed.
• color_col: column used to color points in the scatter plot.
• label_col: column used for label placement.
• tooltip_cols: optional columns shown in the tooltip.
• extra_cols: optional extra columns stored in the database.

load_data() accepts:

• a file path as a string,
• a list of dictionaries,
• a pandas DataFrame,
• an Arrow table, or any similar tabular object supported by DuckDB/Arrow.

Typical workflow

1. Load the data with app.load_data(...).
2. Compute embeddings with app.embed(...).
3. Run dimensionality reduction with app.reduce().
4. Open the interactive dashboard with app.show(...).

If you call load_data(..., overwrite=True), the existing documents, embeddings, reductions, and saved reduction model are cleared before importing the new dataset.

Example data

The repository includes a test dataset in sample_data/ and evaluation notebooks in notebooks/. This is useful for testing the pipeline and exploring the visual interface.

Project structure

• semantic_scatter/core/ — public API and abstract interfaces.
• semantic_scatter/engines/ — DuckDB, ingestion, embedding, reduction, and visualization engines.
• semantic_scatter/app.py — Solara entrypoint used by the web UI.

Notes

• The default embedding model is all-MiniLM-L6-v2.
• The default database file is semantic_scatter_database.db.
• On the first run, model weights may be downloaded automatically by sentence-transformers.

License

This project is licensed under the MIT License - see the LICENSE file for details.