Add cosmosdb management from api_input#14
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR integrates Azure Cosmos DB for data persistence and refactors multiple indexer modules to use API endpoints for status updates and content storage. Key changes include:
- Replacing direct Cosmos DB writes with API calls for saving statuses and content.
- Removing unused imports and outdated in-memory caching mechanisms.
- Standardizing code across website, image, note, and PDF indexers for improved consistency and maintainability.
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/website_indexer/website_indexer.py | Updates to use API status updates and content sending instead of local storage. |
| src/api_input/api_input.py | Integrates full Cosmos DB client setup and endpoints; removes deprecated code. |
| src/image_indexer/image_indexer.py | Refactors image indexing to match other indexers and uses API for status updates. |
| src/note_indexer/note_indexer.py | Adjusts note indexing to send content and status updates via API calls. |
| src/pdf_indexer/pdf_indexer.py | Updates PDF processing to require request_id and uses API updates. |
Comments suppressed due to low confidence (1)
src/image_indexer/image_indexer.py:87
- The variable 'container_name' is used here while a different name ('blob_container_name') is used in another module; ensure consistent naming if they refer to the same container.
blob_client = blob_service_client.get_blob_client( container=container_name, blob=image_location)
|
|
||
| # Use replace_item instead of patch_item for better compatibility | ||
| item['status'] = new_status | ||
| item['last_updated'] = str(datetime.datetime.now()) |
There was a problem hiding this comment.
The usage of datetime.datetime.now() requires importing the datetime module, which is missing.
| asyncio.sleep(5) | ||
|
|
There was a problem hiding this comment.
In an asynchronous function, asyncio.sleep(5) is not awaited; use await asyncio.sleep(5) to ensure proper coroutine suspension.
| asyncio.sleep(5) | |
| await asyncio.sleep(5) |
| asyncio.sleep(5) | ||
|
|
There was a problem hiding this comment.
The coroutine asyncio.sleep(5) is not awaited in a synchronous context; use time.sleep(5) for a blocking delay in non-async code.
| asyncio.sleep(5) | |
| await asyncio.sleep(5) |
This pull request includes significant updates to the
src/api_input/api_input.pyandsrc/pdf_indexer/pdf_indexer.pyfiles, focusing on integrating Azure Cosmos DB for data storage and improving the handling of PDF processing.Integration with Azure Cosmos DB:
src/api_input/api_input.py: Replaced in-memory caching with Azure Cosmos DB for storing and retrieving input data. Added methods for saving, updating, and querying inputs in Cosmos DB. [1] [2]src/api_input/api_input.py: Updated API endpoints to interact with Cosmos DB, including endpoints for getting input by ID, getting all inputs, and updating input status. [1] [2]PDF Processing and Status Updates:
src/pdf_indexer/pdf_indexer.py: Modified theindex_pdffunction to process PDFs and return metadata, which is then sent to theapi_inputservice. [1] [2]src/pdf_indexer/pdf_indexer.py: Added functions to update the status and send processed content to theapi_inputservice via API calls. [1] [2]Code Cleanup and Refactoring:
src/api_input/api_input.py: Removed unused imports and in-memory caching mechanisms. [1] [2]src/pdf_indexer/pdf_indexer.py: Removed unused imports and refactored the code for better readability and maintainability. [1] [2]These changes enhance the system's reliability and scalability by leveraging Azure Cosmos DB for data persistence and improving the integration between the PDF processing service and the API input service.