Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api)(frontend): add task scheduling feature #27

Merged
merged 14 commits into from
Apr 19, 2024

Conversation

mawandm
Copy link
Contributor

@mawandm mawandm commented Apr 16, 2024

Description
This PR introduces the concept of tasks within the API module. A use case for task is running an ingestion pipeline from a datasource.

The result of this PR is that we will not need to current uniform scheduling mechanism. Rather, users are able to pick when to schedule the ingestion of a specific datasource.

Testing this PR
To test this PR,

  1. Run the RAG Microservice (Set the env variable NESIS_RAG_EMBEDDING_DIMENSIONS = 384 in order to use HuggingFace's embeddings models)
  2. Run the API Microservice
  3. Create a session with
curl -X POST -H "Content-Type: application/json"  -k http://localhost:6000/v1/sessions -d '{"email":"test.user@domain.com","password":"password"}'
  1. Create a datasource with (Use token from step 3. above
curl -d '{"type": "minio", "name": "cloud-documents", "connection": {"user": "your_username","password": "your_password","endpoint": "http://localhost:59000","dataobjects": "boc"}}' -H "Content-Type: application/json"  -H "Authorization: Bearer <token>" -X POST http://127.0.0.1:6000/v1/datasources
  1. Create a task to run immediately with replace parent_id and datasource.id with the datasource id from step 4.
curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer <token>" -k http://localhost:6000/v1/tasks -d '{"type":"ingest_datasource","parent_id":"<datasource-id>",  "definition":{"datasource":{"id":"<datasource-id>"}}}'

Ingestion should commence shortly

Closes #8

@mawandm mawandm marked this pull request as draft April 16, 2024 18:28
@mawandm mawandm requested review from akizito and removed request for akizito April 18, 2024 01:00
@mawandm mawandm marked this pull request as ready for review April 18, 2024 09:39
@mawandm mawandm requested a review from akizito April 18, 2024 09:39
@mawandm mawandm changed the title feat(api): add task scheduling feature feat(api)(frontend): add task scheduling feature Apr 19, 2024
@mawandm mawandm changed the title feat(api)(frontend): add task scheduling feature feat(api,frontend): add task scheduling feature Apr 19, 2024
@mawandm mawandm changed the title feat(api,frontend): add task scheduling feature feat(api)(frontend): add task scheduling feature Apr 19, 2024
@mawandm mawandm self-assigned this Apr 19, 2024
@mawandm mawandm merged commit 5099ffb into main Apr 19, 2024
6 checks passed
@mawandm mawandm deleted the 8-introduce-task-scheduling branch April 19, 2024 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Add ability to schedule a datasource ingestion job
2 participants