Repository navigation
feat(server): Refactor scraper, add LLM parsing, and switch to HTTP
This commit introduces a major refactoring of the Daft.ie scraper and adds several new features, including natural language query parsing and a move to an HTTP-based server.
Refactoring:
- The core scraping logic in
daftScraper.service.tshas been decomposed into smaller, single-responsibility services for fetching (fetch.service.ts), parsing (parser.service.ts), and filtering (filter.service.ts). This significantly improves modularity, maintainability, and testability. - Input parameters across the application have been standardized to use camelCase (e.g.,
minPriceinstead ofmin_price) for consistency.
New Features:
- LLM Query Parsing: A new
parse_querytool has been added. It leveragesllamaindexwith an Ollama model to interpret natural language queries (e.g., "find a 2-bed apartment in Dublin under €2000") and convert them into structured JSON for the search tool. - HTTP SSE Server: The server transport has been upgraded from
StdioServerTransporttoSSEServerTransportusing Express, transforming the application into a web server that communicates over HTTP and Server-Sent Events. - Docker Support: A
Dockerfileis now included, enabling easy containerization and deployment of the server.
Enhancements:
- Error Handling: A robust, centralized error handling system has been implemented using custom error classes (
ScraperError,ValidationError, etc.) and a utility function (createErrorResponse) to generate standardized error payloads. - Testing: Comprehensive new unit tests have been added for the server, scraper, and filter services, with coverage reporting configured in
package.json.
Other Changes:
- Added
.sessionto.gitignore. - Updated
package.jsonwith new dependencies (dotenv,llamaindex,@types/express) and bumped the version to0.2.4.