PySpark Data Processing

This repository contains small PySpark applications demonstrating common data engineering tasks such as data validation, cleansing, and enrichment.
It is designed as a simple reference project for anyone learning PySpark or exploring how to build ETL-style pipelines using Spark DataFrames.

---

📁 Project Structure

PySpark/
│
├── main.py # Entry point to run PySpark jobs
├── learn_main.py # Additional learning/demo scripts
│
├── product_enrichment.py # Cleans & enriches product dataset
├── ace_valid.py # Validates and separates valid/error data
│
├── data/
│ ├── product_source.csv # Sample product source data
│ └── telecom_details.csv # Sample telecom details
│
├── valid_table/ # Output: validated records
├── error_table/ # Output: rejected/error records
└── product_enriched/ # Output: enriched dataset

---

⚙ Requirements

• Python 3.8+
• Apache Spark 3.x
• PySpark (pip install pyspark)
• (Optional) Jupyter Notebook for exploration

---

🚀 How to Run

1. Clone the repository

   git clone https://github.com/Rohithshan/PySpark.git cd PySpark

2. Install dependencies

   pip install pyspark

3. Run the main script

   python main.py

   Or execute any module directly, for example:

   python product_enrichment.py

4. Check outputs

   • ✅ Validated data → valid_table/
   • ❌ Errors → error_table/
   • ✨ Enriched dataset → product_enriched/

---

🧠 Concepts Demonstrated

• Reading CSV data using Spark DataFrames
• Performing validation checks (nulls, data types, schema consistency)
• Splitting valid vs. invalid records
• Joining and enriching datasets
• Writing processed data to new folders
• Understanding transformations vs. actions in PySpark

---

📚 Learning Use Cases

This project can be used to:

• Practice PySpark syntax and DataFrame transformations
• Understand ETL pipeline structure
• Prototype validation/enrichment logic before scaling to production

---

🧑‍💻 Author

Rohith Shan
Learning & exploring PySpark for scalable data engineering.
Feel free to fork and extend this repo!

---

📝 License

This project is open-source and available under the MIT License.

Would you like the README revised to have a recruiter-focused or portfolio-style tone?

Sources