🚀 dataengineering - Easily Analyze NYC Taxi Data

Welcome to the dataengineering project! This application provides a user-friendly way to analyze NYC Yellow Taxi Trip Data. It combines several powerful tools, including Spark for big data processing, Airflow for managing workflows, MinIO as a cloud storage solution, and DuckDB for SQL queries.

Before you start, make sure your computer meets these requirements:

• Operating System: Windows, macOS, or Linux
• RAM: At least 8 GB recommended
• Disk Space: Minimum of 1 GB available
• Docker: This project uses Docker, so you need to install it. You can find installation instructions on the official Docker website.

1. Download the Software

   • Visit this page to download the latest version of the application.

2. Install Docker

   • Go to the official Docker website and choose the version that matches your operating system.
   • Follow the instructions to install Docker on your computer.

3. Run the Application

   • Open your terminal or command prompt.
   • Use the command to pull the Docker image:

     docker pull codex56799/dataengineering

   • Start the Docker container with the command:

     docker run -p 8080:8080 codex56799/dataengineering

After you run the above command, the application will start, and you can access it in your web browser at http://localhost:8080.

• Data Analysis: Access and visualize NYC Yellow Taxi Trip Data.
• Workflow Management: Use Apache Airflow to automate data workflows.
• Storage Solutions: Leverage MinIO for S3-compatible cloud storage.
• SQL Queries: Utilize DuckDB for quick and efficient SQL querying.

1. Accessing the Interface

   • Open your web browser and go to http://localhost:8080.
   • You will find an easy-to-use interface to start your analysis.

2. Loading Data

   • You can upload your CSV files or connect to the NYC Yellow Taxi Trip Data available online.

3. Running Workflows

   • Use pre-defined DAGs in Airflow to schedule and automate data processing tasks.

4. Querying Your Data

   • Write SQL queries in the DuckDB interface to analyze your data efficiently.

For detailed documentation, including how to create custom workflows, manage data, and optimize performance, consult our official docs.

No, this application is designed for users with little to no programming knowledge.

Yes, as long as your laptop meets the basic system requirements, you can easily run the application.

You can open an issue on our GitHub Issues page if you encounter any problems.

We welcome contributions! If you want to contribute, please read our contribution guidelines in the repository to ensure a smooth collaboration.

Thanks to the open-source communities of Spark, Airflow, MinIO, and DuckDB for providing essential tools that make this project possible.

Remember to visit this page to download your copy of the software. Follow the instructions provided above to get started. Enjoy analyzing data with ease!