Simeru Scraper

Simeru Scraper is a web scraper that scrapes schedules from the official website and caches the data in Redis. The API is built with Go and Fiber.

This repository has moved

The Simeru Scraper repository has been moved to https://github.com/savioruz/smrv2-api. This repository is now archived and no longer maintained.

Table of Contents

• Features
• Deployment
• Requirements
• Installation
• Usage
  • Running the API
  • API Documentation
• Project Structure
• Contributing
• License
• Acknowledgements

Features

• Scrapes schedules from the official website
• Caches data in Redis
• Cron job to update data
• API Documentation with Swagger
• Docker support

Deployment

• Koyeb

Requirements

• Go 1.23+
• Docker
• Redis
• Make

Installation

1. Clone the repository:

   git clone https://github.com/savioruz/simeru-scraper.git
   cd simeru-scraper

2. Environment Variables:

   Create a .env file in the root directory and add the following:

   cp .env.example .env

Usage

Running the API

You can run the API using Docker or directly with Make.

Docker (Recommended)

1. Run redis:

   make docker.redis

2. Run the application:

   make docker.run

For production, you need to secure redis on Makefile with a password.

Make

1. Run the application:

   make run

You need to have Redis running on your machine.

API Documentation

Swagger documentation is available at: http://localhost:3000/swagger.

Project Structure

.
├── config/                 # Configuration files
│── docs/                   # Project documentation
├── internal/
│   ├── adapters/           # Adapters for external services
│   │   ├── cache/          # Cache layer
│   │   ├── handlers/       # Handlers layer
│   │   └── repositories/   # Storage layer integration
│   └── cores/              # Core business layer
│       ├── entities/       # Business entities
│       ├── ports/          # Adapter implementations
│       │   └── ports.go
│       └── services/       # Use cases layer
├── pkg/
│   ├── constant/
│   ├── middleware/
│   ├── routes/
│   ├── server/            # Server configuration
│   └── utils/             # Utility functions
├── main.go
├── .env
└── Dockerfile

Contributing

Feel free to open issues or submit pull requests with improvements.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgements

• Fiber