Add stack overflow ingest

[Lee-W]

refactor 47933b5 and make it consistent with existing archive logic

closes: #126

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	8e5e26d
Status:	✅  Deploy successful!
Preview URL:	https://a2f2bd4e.ask-astro.pages.dev
Branch Preview URL:	https://add-stack-overflow-ingest.ask-astro.pages.dev

View logs

[sunank200]

How did you test the ingestion?

[pankajastro]

LGTM

[sunank200]

We need to try the following before merging this change to the main:

1. Ingest the data from StackOverflow APIs and make sure the data from the archive is ingested as well from the APIs(as it is the dump of the Slackoverflow data.)
2. Make sure that the historical data from Stackoverflow is ingested. ( very old relevant post or comment)
3. Deploy this change after running ingestion DAG to the dev backend and request @vatsrahul1001 to test this. The goal is that the performance should not degrade

[Lee-W]

Ingest the data from StackOverflow APIs and make sure the data from the archive is ingested as well from the APIs(as it is the dump of the Slackoverflow data.)

If that's the case, I'm unsure whether we should remove the archive. As we already have the archive data, shouldn't we just ingest that data, and use the API to parse data after that?

[sunank200]

Ingest the data from StackOverflow APIs and make sure the data from the archive is ingested as well from the APIs(as it is the dump of the Slackoverflow data.)

If that's the case, I'm unsure whether we should remove the archive. As we already have the archive data, shouldn't we just ingest that data, and use the API to parse data after that?

We will create a new Weaviate class and do a fresh ingest. We don't need to remove archive data in old database

[Lee-W]

@sunank200 As #194 has been done, should we re-review this one?