Extract, transform, and load (ETL) LCH (London Clearing House) CDS clearing essentials pricing data into an MSSQL database, leveraging Docker for seamless deployment and scalability.
- Efficient Scraping: Designed to scrape CDS clearing essentials pricing data directly from the LCH's official website.
- Data Transformation: Tailored data transformation for easy database insertion.
- MSSQL Support: Built-in support to insert data into a Microsoft SQL Server database.
- Dockerized: Simplified deployment and setup using Docker.
- Robust Error Handling: Multi-retry mechanisms and comprehensive logging.
- Clone the repository:
git clone git@github.com:alimghmi/lch-client.git cd lch-client
- Create an
.env
file in the project root and configure the following:Replace the placeholders (URL="https://www.lch.com/services/cdsclear/essentials/pricing-data" LOG_LEVEL="INFO" OUTPUT_TABLE=<name_of_the_output_table> INSERTER_MAX_RETRIES=2 REQUEST_MAX_RETRIES=3 REQUEST_BACKOFF_FACTOR=2 MSSQL_SERVER=<mssql_server> MSSQL_DATABASE=<mssql_database> MSSQL_USERNAME=<mssql_username> MSSQL_PASSWORD=<mssql_password>
<...>
) with the appropriate values.
- Install the required packages:
pip install -r requirements.txt
- Run the
main.py
script:python main.py
- Build the Docker image:
docker build -t lch-data-scraper .
- Run the Docker container:
docker run --env-file .env lch-data-scraper
- Ali Moghimi (alimghmi)
- Clemens Struck (clemstruck)
Contributions are welcome! Fork the repository, apply your changes, and submit a pull request.