Skip to content

0.0.1

Choose a tag to compare

View all tags
@galaxyeye galaxyeye released this 20 Jul 17:08
· 39 commits to main since this release
v0.0.1
06bdd6f

WebMiner

Extract structured data from local HTML files. WebMiner analyzes a directory of
web pages, identifies recurring patterns and data tables, and exports the
results as HTML reports and Excel spreadsheets.

Everything runs locally — no network calls, no data leaves your machine.

Latest release: v0.0.1 | Download: scent-miner.jar

Quick Start

  1. Install JDK 17+ from Adoptium or
    Microsoft OpenJDK.

  2. Download scent-miner.jar and webminer.ps1 from the
    latest release.

  3. Run:

powershell .\webminer.ps1 --input C:\path\to\html\files

Or directly with Java:

�ash java -jar scent-miner.jar --input /path/to/html/files

The miner scans all *.html / *.htm files, extracts structured tables, and
writes the results to -views/.

Usage

WebMiner --input <html-dir> [options]

Options

Flag Default Purpose
--input, -i required Directory containing *.html / *.htm files
--component-selector, -c (none) CSS selector for the main content area
--require-size 500000 Minimum page size in bytes
--limit, -l (no limit) Load at most N pages
--no-trust-samples off Validate and clean samples
--help, -h — Print usage

Examples

`�ash

Mine product pages from a local dump

java -jar scent-miner.jar --input /data/amazon-pages

Custom CSS selector for a different site layout

java -jar scent-miner.jar
--input /data/ebay-pages
--component-selector "#mainContent"

Limit to the first 50 pages for a quick test

java -jar scent-miner.jar --input /data/pages --limit 50

Validate samples for stricter extraction

java -jar scent-miner.jar --input /data/pages --no-trust-samples

Skip small stub pages

java -jar scent-miner.jar --input /data/pages --require-size 1000000
`

PowerShell Launcher

The webminer.ps1 script auto-detects Java 17 and applies the required JVM
flags:

powershell .\webminer.ps1 --input C:\data\html-pages .\webminer.ps1 --input C:\data\html-pages -c "#mainContent" -l 50 .\webminer.ps1 --input C:\data\html-pages --no-trust-samples .\webminer.ps1 -JavaHome "D:\jdk-17" --input C:\data\html-pages

Output

<input-dir>-views/ └── views/ ├── index.html # HTML report of extracted tables ├── *.xlsx # Excel export of tabulated data └── ...

Open index.html in a browser to browse extracted data, or load .xlsx files
in Excel for further analysis.

Requirements

Requirement Version Notes
JDK 17+ Adoptium recommended
OS Windows x86_64 Native libraries are Windows-only in this release

How It Works

WebMiner loads each HTML file, parses its structure and visual layout, detects
recurring content patterns across the document collection, and extracts
structured data tables from those patterns.

Built with:

  • Kotlin 2.3.x / Java 17
  • ProGuard 7.9 — bytecode obfuscation
  • Spring Boot 4.1 — dependency injection
  • Apache Spark ML — clustering & feature extraction
  • Pulsar DOM — HTML parsing & visual analysis

License

Proprietary. All rights reserved.

Assets 3
Loading