Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Astro forum docs #212

Merged
merged 12 commits into from
Dec 20, 2023
Merged

Add Astro forum docs #212

merged 12 commits into from
Dec 20, 2023

Conversation

pankajastro
Copy link
Collaborator

@pankajastro pankajastro commented Dec 15, 2023

closes: #120

Copy link

cloudflare-pages bot commented Dec 15, 2023

Deploying with  Cloudflare Pages  Cloudflare Pages

Latest commit: 491d8d9
Status: ✅  Deploy successful!
Preview URL: https://bfbd202f.ask-astro.pages.dev
Branch Preview URL: https://astro-forum-docs.ask-astro.pages.dev

View logs

@pankajastro pankajastro marked this pull request as ready for review December 17, 2023 06:44
@pankajastro
Copy link
Collaborator Author

@vatsrahul1001 We need to test this PR too. please let me know when you are planning to test I'll keep ready db accordingly

Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets ingest data to a separate weaviate index and test this out. cc: @pankajastro @vatsrahul1001

airflow/dags/ingestion/ask-astro-forum-load.py Outdated Show resolved Hide resolved
default_args=default_args,
)
def ask_astro_load_astro_forum():
_import_data = (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should ideally be in three tasks right?

extract_forum_docs >> split_docs >> ingest_data

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ingestion for this docs take around 1min so splitting won't add much value

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or this is a requirement?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a reference implementation. It's good to keep different tasks broken down.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

splited it. PTAL

airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
@vatsrahul1001
Copy link
Collaborator

Lets ingest data to a separate weaviate index and test this out. cc: @pankajastro @vatsrahul1001

@pankajastro let me know when index is pointed to dev

airflow/dags/ingestion/ask-astro-forum-load.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
airflow/include/tasks/extract/astro_forum_docs.py Outdated Show resolved Hide resolved
@pankajastro pankajastro mentioned this pull request Dec 19, 2023
@pankajastro pankajastro merged commit 441def6 into main Dec 20, 2023
7 checks passed
@pankajastro pankajastro deleted the astro-forum-docs branch December 20, 2023 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Astro Forum docs
4 participants