Website Analyzer

Website Analyzer is a FastAPI service for basic website quality checks. It opens a page in Chrome, saves a screenshot, classifies the screenshot with a Keras model, and then runs Selenium checks for links, forms, and buttons when the page is classified as normal.

This repository is positioned as an ML inference and deployment project. The original training dataset was lost, so the current focus is serving, testing, documenting, and operating an existing model artifact.

Architecture

Client
  -> FastAPI
  -> Selenium screenshot capture
  -> CNN/Keras model inference
  -> score + threshold + label
  -> optional Selenium DOM checks
  -> JSON response

Project Structure

.
├── src/website_analyzer/     # Application package
├── docs/                     # Model card and recovery plan
├── data/screenshots/         # Runtime screenshots
├── logs/                     # Runtime logs
├── models/                   # Downloaded Keras model
├── requirements.txt          # Python dependencies
├── Dockerfile                # Container image
├── docker-compose.yml        # Docker Compose service
├── Makefile                  # Common commands
└── .env.example              # Environment variable template

Setup

Recommended Python version: 3.11.

Create an environment file:

cp .env.example .env

Install dependencies:

make install

Usage

Start the API:

make api

Open the interactive API docs:

http://localhost:8000/docs

Analyze a website:

curl -X POST http://localhost:8000/analyze-url \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Example response:

{
  "url": "https://example.com/",
  "status": "completed",
  "model_version": "legacy-defect-model-v1",
  "classification": "normal",
  "score": 0.23,
  "threshold": 0.5,
  "screenshot_path": "data/screenshots/screenshot_https___example_com.png",
  "selenium_checks_run": true
}

Run inference on an uploaded screenshot:

curl -X POST http://localhost:8000/predict-image \
  -F "file=@data/screenshots/example.png"

Health check:

curl http://localhost:8000/health

The CLI entry point is still available for local debugging:

make run URL=https://example.com

Format code:

make format

Run tests:

make test

Docker

Build the image:

make docker-build

Run the API in Docker:

make docker-run

The API will be available at http://localhost:8000.

The compose service mounts data/, logs/, and models/ so screenshots, logs, and the downloaded model persist between runs. Docker Compose reads .env automatically when it exists and falls back to the defaults from docker-compose.yml.

Configuration

Configuration is read from .env.

Variable	Description
MODEL_VERSION	Human-readable model artifact version.
MODEL_FILE_ID	Google Drive file ID for the trained Keras model.
MODEL_PATH	Local path where the model is stored.
SCREENSHOTS_DIR	Directory for generated screenshots.
LOG_FILE	Log file path.
SELENIUM_HEADLESS	Run Chrome in headless mode.
PAGE_LOAD_DELAY	Seconds to wait after opening a page.
PREDICTION_THRESHOLD	Score threshold for normal vs defective.
CHROME_BINARY_PATH	Optional path to Chrome or Chromium binary.
CHROME_DRIVER_PATH	Optional path to ChromeDriver.

ML Notes

The model returns a raw score. Scores greater than or equal to PREDICTION_THRESHOLD are labeled defective; lower scores are labeled normal.

Training Summary

The trained model artifact is supported by a historical diploma report that documented the original experiments. The dataset itself is no longer available, so the metrics below should be treated as historical results rather than currently reproducible benchmarks.

Original dataset and preprocessing:

• more than 500 website screenshots
• two classes: normal and defective
• approximately balanced class distribution
• image resize to 224x224
• pixel normalization to [0, 1]
• stratified train/test split: 80/20, random_state=42

Augmentation techniques used during training:

• random rotations
• horizontal and vertical flips
• width and height shifts
• zoom and shear transforms
• brightness and contrast changes
• random noise and blur
• ImageDataGenerator and albumentations

Model experiments:

• baseline custom CNN trained from scratch
• baseline result: training accuracy above 90%, validation accuracy around 50%
• conclusion from baseline: overfitting and weak generalization
• final model: transfer learning with VGG16 pretrained on ImageNet
• fine-tuning: last 4 VGG16 layers unfrozen
• classification head: global average pooling, dense layer, dropout, binary output
• optimizer: Adam with learning rate 0.0001
• callbacks: EarlyStopping and ReduceLROnPlateau
• class balancing: class_weight

Reported final result:

• training accuracy: above 90%
• validation accuracy: approximately 85%
• validation loss stabilized without strong signs of overfitting

The original training script also included confusion_matrix, classification_report, roc_curve, and optimal threshold search. Since the original screenshots and labels were lost, those values cannot be regenerated from this repository today.

Because the original dataset was lost, the repository includes documentation that makes this limitation explicit:

• Model Card
• Dataset Recovery Plan
• API Examples

Current limitations:

• training metrics are not reproducible
• the original label policy is unknown
• the model should be treated as a legacy artifact
• the strongest part of this project is the ML inference API and deployment pipeline