WebMiner v0.0.8

📦 Assets

Asset	Description
scent-miner.jar	Self-contained JAR (98MB)
webminer.ps1	PowerShell launcher

⚡ Quick Start

Or directly with Java:

🔧 Requirements

• Java 17 or later

Full Changelog: v0.0.7...v0.0.8

WebMiner v0.0.7

📦 Assets

Asset	Description
scent-miner.jar	Self-contained JAR (98MB)
webminer.ps1	PowerShell launcher

⚡ Quick Start

Or directly with Java:

🔧 Requirements

• Java 17 or later

Full Changelog: v0.0.6...v0.0.7

WebMiner v0.0.6

📦 Assets

Asset	Description
scent-miner.jar	Self-contained JAR (98MB)
webminer.ps1	PowerShell launcher

⚡ Quick Start

Or directly with Java:

🔧 Requirements

• Java 17 or later

Full Changelog: v0.0.5...v0.0.6

WebMiner v0.0.5

📦 Assets

Asset	Description
scent-miner.jar	Self-contained JAR (98MB)
webminer.ps1	PowerShell launcher

⚡ Quick Start

Or directly with Java:

🔧 Requirements

• Java 17 or later

Full Changelog: v0.0.4...v0.0.5

WebMiner v0.0.4

📦 Assets

Asset	Description
scent-miner.jar	Self-contained JAR (98MB)
webminer.ps1	PowerShell launcher

⚡ Quick Start

Or directly with Java:

🔧 Requirements

• Java 17 or later

What's Changed

• fix: correct JAR path to lib/scent-miner.jar in release workflow by @galaxyeye in #1

New Contributors

• @galaxyeye made their first contribution in #1

Full Changelog: v0.0.3...v0.0.4

v0.0.3 — WebMiner Subcommand CLI

Breaking Changes

The CLI has been redesigned from --input flag-based to subcommand-based:

Old (legacy):

WebMiner --input <html-dir> [options]

New:

WebMiner <command> [args...] [options...]

Commands

• all <html-dir> — Full pipeline: encode → cluster → views
• encode <html-dir> — Encode HTML files to CSV feature vectors
• cluster <csv-path> — Run KMeans on an encoded CSV
• views <result-dir> — Build HTML/XLSX views from clustering results

Examples

WebMiner all /tmp/amazon-pages
WebMiner all /tmp/amazon-pages --k 12 --max-files 50
WebMiner all /tmp/amazon-pages --resume
WebMiner encode /tmp/amazon-pages
WebMiner cluster /tmp/encoded.csv

Fixes

• run-example now works correctly with the installed JAR
• The PowerShell launcher no longer needs the -cp workaround

0.0.1

WebMiner

Extract structured data from local HTML files. WebMiner analyzes a directory of
web pages, identifies recurring patterns and data tables, and exports the
results as HTML reports and Excel spreadsheets.

Everything runs locally — no network calls, no data leaves your machine.

Latest release: v0.0.1 | Download: scent-miner.jar

Quick Start

1. Install JDK 17+ from Adoptium or
   Microsoft OpenJDK.

2. Download scent-miner.jar and webminer.ps1 from the
   latest release.

3. Run:

powershell .\webminer.ps1 --input C:\path\to\html\files

Or directly with Java:

�ash java -jar scent-miner.jar --input /path/to/html/files

The miner scans all *.html / *.htm files, extracts structured tables, and
writes the results to -views/.

Usage

WebMiner --input <html-dir> [options]

Options

Flag	Default	Purpose
--input, -i	required	Directory containing *.html / *.htm files
--component-selector, -c	(none)	CSS selector for the main content area
--require-size	500000	Minimum page size in bytes
--limit, -l	(no limit)	Load at most N pages
--no-trust-samples	off	Validate and clean samples
--help, -h	—	Print usage

Examples

`�ash

Mine product pages from a local dump

java -jar scent-miner.jar --input /data/amazon-pages

Custom CSS selector for a different site layout

java -jar scent-miner.jar
--input /data/ebay-pages
--component-selector "#mainContent"

Limit to the first 50 pages for a quick test

java -jar scent-miner.jar --input /data/pages --limit 50

Validate samples for stricter extraction

java -jar scent-miner.jar --input /data/pages --no-trust-samples

Skip small stub pages

java -jar scent-miner.jar --input /data/pages --require-size 1000000
`

PowerShell Launcher

The webminer.ps1 script auto-detects Java 17 and applies the required JVM
flags:

powershell .\webminer.ps1 --input C:\data\html-pages .\webminer.ps1 --input C:\data\html-pages -c "#mainContent" -l 50 .\webminer.ps1 --input C:\data\html-pages --no-trust-samples .\webminer.ps1 -JavaHome "D:\jdk-17" --input C:\data\html-pages

Output

<input-dir>-views/ └── views/ ├── index.html # HTML report of extracted tables ├── *.xlsx # Excel export of tabulated data └── ...

Open index.html in a browser to browse extracted data, or load .xlsx files
in Excel for further analysis.

Requirements

Requirement	Version	Notes
JDK	17+	Adoptium recommended
OS	Windows x86_64	Native libraries are Windows-only in this release

How It Works

WebMiner loads each HTML file, parses its structure and visual layout, detects
recurring content patterns across the document collection, and extracts
structured data tables from those patterns.

Built with:

• Kotlin 2.3.x / Java 17
• ProGuard 7.9 — bytecode obfuscation
• Spring Boot 4.1 — dependency injection
• Apache Spark ML — clustering & feature extraction
• Pulsar DOM — HTML parsing & visual analysis

License

Proprietary. All rights reserved.