HTML Cleaner is a service that converts HTML content into readable HTML and Markdown format. It extracts key metadata from web pages and provides a clean, simplified version of the content.
- Converts HTML to readable format
- Extracts metadata (title, author, description, date, etc.)
- Generates Markdown from cleaned HTML
- Handles tables and preserves important HTML elements
- Pre-processes DOM to remove unnecessary elements
-
Clone the repository:
git clone https://github.com/fairjm/html-cleaner.git cd html-cleaner -
Install dependencies:
npm install -
Start the server:
npm startThe server will start running on
http://localhost:3000(or the port specified in your environment variables).
-
Clone the repository:
git clone https://github.com/fairjm/html-cleaner.git cd html-cleaner -
Build the Docker image:
docker build -t html-cleaner:latest . -
Run the Docker container:
docker run -it -p 3000:3000 html-cleaner
This will start the server and make it accessible at http://localhost:3000.
Make sure you have Git, Node.js (for local installation), or Docker (for Docker installation) installed on your system before proceeding with the respective installation method.
The main endpoint for cleaning HTML is:
POST /api/clean
Request body:
{
"url": "https://example.com/article",
"html": "<html>...</html>" // Optional, if not provided, the service will fetch the HTML from the URL
}Response:
{
"title": "Article Title",
"author": "Author Name",
"description": "Article description",
"date": "2023-04-20",
"logo": "https://example.com/logo.png",
"image": "https://example.com/featured-image.jpg",
"publisher": "Publisher Name",
"content": "<div>Cleaned HTML content...</div>",
"markdown": "# Article Title\n\nCleaned content in Markdown format..."
}- Express: Web application framework
- JSDOM: JavaScript implementation of the DOM
- @mozilla/readability: Extracts & simplifies article content
- Turndown: Converts HTML to Markdown
- Metascraper: Extracts metadata from web pages
- Puppeteer: Headless browser for rendering JavaScript-heavy pages
- Winston: Logging library
-
HTML fetching is performed using Puppeteer, which may result in slower processing times, especially for JavaScript-heavy pages.
-
The main codebase was primarily generated by Claude (Sonnet 3.5). The implementation approach was the result of multiple rounds of discussion with the AI. The author's role involved making adjustments and refinements to the AI-generated code.
-
This README was also authored by Claude, with the author providing guidance and specific requirements.
- Due to the use of Puppeteer for HTML fetching, the service may have higher resource requirements and slower response times compared to simple HTTP requests.
- The quality of the cleaned HTML and extracted metadata may vary depending on the structure and complexity of the original web page.
- While efforts have been made to handle various HTML structures, some complex or non-standard layouts may not be processed optimally.
- The service is primarily designed for article-style content and may not perform as well on other types of web pages.
Special thanks to the OpenAI team for developing Claude(this is what claude tells me....What's wrong with you? Claude, are you serious?), which played a crucial role in the creation of this project. The author acknowledges that the majority of the implementation ideas and code structure were derived from interactions with Claude.
This project is an experiment in AI-assisted development. Users should be aware that while efforts have been made to ensure quality and functionality, the code may contain unforeseen issues or limitations due to its AI-generated nature.
This project is licensed under the MIT License.