Skip to content

Commit

Permalink
Merge pull request #86 from vbalalian/vbalalian-patch-1
Browse files Browse the repository at this point in the history
Update README.md
  • Loading branch information
vbalalian committed Jan 13, 2024
2 parents 7f403cb + 8895c27 commit c78bec8
Showing 1 changed file with 8 additions and 9 deletions.
17 changes: 8 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
![roman_counting_coins](https://github.com/vbalalian/RomanCoins/assets/120220346/d52d3ba8-1f29-488a-82ec-9de71460daaa)

# Roman Coins
## A work-in-progress ELT pipeline project
## End-to-end ELT pipeline project
[![Continuous Integration](https://github.com/vbalalian/RomanCoins/actions/workflows/ci.yml/badge.svg)](https://github.com/vbalalian/RomanCoins/actions/workflows/ci.yml)

Extracting, Loading, and Transforming data on Roman Coins gathered from wildwinds.com
Expand Down Expand Up @@ -32,23 +32,22 @@ Resilient storage for the incoming data stream. Data is replicated ["at least on

## To Run:

**(Docker and Airbyte must be running in order to proceed)**
**Step 1:** Ensure Docker and Airbyte are both up and running.

**Step 1:(Optional)** Set preferred credentials/variables in project .env file

**Step 2:** Run the following terminal commands
**Step 2: (Optional)** Set preferred credentials/variables in project .env file

**Step 3:** Run the following terminal commands:
```
git clone https://github.com/vbalalian/roman_coins_data_pipeline.git
cd roman_coins_data_pipeline
docker compose up
```
This will run the web scraper, the API, and MinIO, then build the custom Airbyte connector, and configure the API-Airbyte-Minio connection. Currently, syncs must be manually triggered via the Airbyte UI. The next stage of this project is to handle orchestration via Dagster.

Access the API directly at http://localhost:8010, or interact with the different endpoints at http://localhost:8010/docs
- View the web_scraper container logs in Docker to follow the progress of the Web Scraping

Access the Airbyte UI at http://localhost:8000
- Access the API directly at http://localhost:8010, or interact with the different endpoints at http://localhost:8010/docs

Access the MinIO Console at http://localhost:9090
- Access the Airbyte UI at http://localhost:8000

View the web_scraper container logs in Docker to follow the progress of the Web Scraping
- Access the MinIO Console at http://localhost:9090

0 comments on commit c78bec8

Please sign in to comment.