MailMine

The Ultimate Web Scraping & Lead Generation Platform
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Project Structure
4. Usage
5. Roadmap
6. Contributing
7. License
8. Contact

🚀 About The Project

MailMine is a sophisticated web scraping and lead generation solution designed to streamline the process of gathering business intelligence. Whether you are a marketer looking for leads, a recruiter finding candidates, or a data analyst gathering insights, MailMine provides the tools to automate the extraction of emails, links, and contact information from the web.

Key Features

• 🕷️ Advanced Web Scraping: Powered by Jsoup and Selenium to handle complex websites.
• 🎯 Smart Query Generation: Automatically generate search queries tailored to specific industries and roles.
• 📊 Interactive Dashboard: Visualize your scraping jobs, success rates, and data collection in real-time.
• 🔐 Secure Authentication: Enterprise-grade security with JWT tokens and role-based access control.
• ⚡ High Performance: Asynchronous processing for handling multiple scraping jobs simultaneously.
• 📱 Responsive Design: Fully responsive admin panel built with the latest Tailwind CSS.

(back to top)

Built With

This project is built using a modern, robust tech stack ensuring scalability and performance.

• 
• 
• 
• 
• 
• 

(back to top)

🏁 Getting Started

To get a local copy up and running, follow these simple steps.

Prerequisites

Ensure you have the following installed on your system:

• Java JDK 21+ (Backend requirement)
• Node.js 18+ (Frontend requirement)
• PostgreSQL 15+ (Database)
• Maven (Optional, wrapper provided)
• npm or yarn (Package manager for frontend)

Installation

1. Clone the repo

   git clone https://github.com/HKH-soft/Netpick.MailMine.git

2. Backend Setup

   cd Backend
   # Configure database in src/main/resources/application.yml
   # Update PostgreSQL connection settings
   ./mvnw spring-boot:run

   The backend API will be available at http://localhost:8080

3. Frontend Setup

   cd Frontend
   npm install
   # Configure API endpoint if needed
   npm run dev

   The frontend will be available at http://localhost:3000

4. Database Setup

   • Create a PostgreSQL database named mailmine
   • Update Backend/src/main/resources/application.yml with your database credentials
   • Flyway migrations will run automatically on startup

5. Access the App Open your browser and navigate to http://localhost:3000.

(back to top)

📁 Project Structure

Netpick.MailMine/
├── Backend/                 # Spring Boot backend application
│   ├── src/
│   │   ├── main/
│   │   │   ├── java/        # Java source code
│   │   │   │   └── ir/netpick/mailmine/
│   │   │   │       ├── auth/           # Authentication & JWT
│   │   │   │       ├── scrape/         # Web scraping logic
│   │   │   │       ├── email/          # Email handling
│   │   │   │       └── ai/             # AI integration (Gemini)
│   │   │   └── resources/
│   │   │       ├── application.yml     # Configuration
│   │   │       └── db/migration/       # Flyway migrations
│   │   └── test/             # Unit and integration tests
│   └── pom.xml              # Maven dependencies
├── Frontend/                # Next.js frontend application
│   ├── src/
│   │   ├── app/             # Next.js 15 app directory
│   │   ├── components/      # React components
│   │   └── types/           # TypeScript type definitions
│   ├── public/              # Static assets
│   └── package.json         # npm dependencies
└── README.md               # This file

(back to top)

💻 Usage

Secure Sign In

Access your dashboard securely. The system supports persistent login sessions.

Managing Scrape Pipelines

1. Navigate to the Scrape section.
2. Then in Control select what part of the scraping you want to start
3. Monitor the progress in real-time on the dashboard.

(back to top)

🗺️ Roadmap

• Basic Web Scraping
• User Authentication
• Proxy Support
• Grafana Dashboard Analytics
• API Access for 3rd Party Apps
• Cloud Deployment Scripts (Docker)

See the open issues for a full list of proposed features (and known issues).

(back to top)

🤝 Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

📝 License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

📧 Contact

Hossein - hkh.soft.eng@gmail.com

Project Link: https://github.com/HKH-soft/Netpick.MailMine

(back to top)