feat: add backend api url support using docker build args #170
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
akashg3627
approved these changes
May 28, 2024
S1LV3RJ1NX
added a commit
that referenced
this pull request
Jun 26, 2024
* Added doc for using the UI * Updated ReadMe * Updated Readme * Updated Readme * Added support for singlestore vectordb and some refactoring * Added support for cloning qdrant db across different/same qdrant instance (#132) * Added support for cloning qdrant db * Update backend/migration/qdrant_migration.py Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Update backend/migration/qdrant_migration.py Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Update backend/migration/qdrant_migration.py Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Addressed comments, added condn for dest collection exists in separate qdrant --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Added client * Exception handling * Updated Readme (#134) * Added banner images * Update README.md Added Readme Banner * Rename Github Readme Banner.png to readme-banner.png * Rename Github Webinar Banner.png to webinar-banner.png * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Updated ReadMe * feat: add drawer for details (#136) * Updated ReadMe with intro * Updated ReadMe * Updated ReadMe * Create LICENSE * update: cognita logo (#138) * Doctr parser (#139) * Added doctr parser * Updated docker file * Updated docker file with poppler * Updated tfy ignore * Updated Yaml * Added dockerignore * Fixed linting --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> * fixed warnings * Updated env and Readme * Updated table parser * fix: retriever selection (#140) * update: change prompt template based on retriever example (#142) * update: re-ordering controllers, collections (#144) * Updated requirements * Bump pydantic from 1.10.12 to 1.10.13 in /backend Bumps [pydantic](https://github.com/pydantic/pydantic) from 1.10.12 to 1.10.13. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](pydantic/pydantic@v1.10.12...v1.10.13) --- updated-dependencies: - dependency-name: pydantic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Removed tfy-api key from modules * Updated req * Udaed embedd upload order * Updated transformers * Updated embedding arrangement * Updated git and tfy ignore * Add functionality to delete files after chunking * pre-commit config * Updated requirements * Tfparser (#148) * Some changes for table parsing * Allowed local embedding models * Added examples * Added summary controller * update: change prompt template based on retriever example (#141) * Updated table parser * fix: retriever selection (#140) * update: change prompt template based on retriever example --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * update: re-ordering controllers, collections (#143) * Updated table parser * fix: retriever selection (#140) * update: change prompt template based on retriever example (#142) * update: re-ordering controllers, collections --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Added summary * Updated docker file * Added tf parser * Updated local metadata and added new controller * Removed tfy-api key from modules * Fixed linting * Fixing parser * Integrated tf-parser to indexing --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: sayan-truefoundry <136362719+sayan-truefoundry@users.noreply.github.com> Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Updated requirements * Updated requirements * Bump gunicorn from 21.2.0 to 22.0.0 in /backend Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 21.2.0 to 22.0.0. - [Release notes](https://github.com/benoitc/gunicorn/releases) - [Commits](benoitc/gunicorn@21.2.0...22.0.0) --- updated-dependencies: - dependency-name: gunicorn dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Cleanup * Cleanup * Added TFParser that efficiently parses pdf, added auto parser registry during indexing when parser is not specified, fixed embedding bugs * Added multimodal parser and intelligent response controller * changed model download from hf to multi-modal parser * updated local metadata yaml * Vision model routed th' llm-gateway * Updated Dockerfile * Updated docker file for system package updates * Refactored multi-modal parser, added extra config to support it * Added exception handling in docs stream * Changed summary from async to sync * Fixed parser (#150) * Removed summarization for now * removed multi-query retriever for now * Updated streaming in summary * Updated query controller * fix: overflow of collection name in collections page (#151) * feat: show page number in docs source (#152) Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Multimodal Controller (#153) * Modified metadata sending * Modified metadata sending * Added multi-modal query controller and modified indexing to make parallel api calls and summarize only main image * Removed old code * Added batching over api * Updted prompt for controller, added timeout to qdrant * Fixed typo * Cleaned multi-modal parser, and some adhoc refactoring * lazy loading of deepdoctection due to logger issues * Added list collections api (#154) * Update requirements.txt * updated internal apis * updated req for langchain core * docs: update README.md (#162) trigerred -> triggered * Embedding class for embedding service deployment (#160) * Added Embedding service class that can consume embedding apis from hf deploying using tf * Added Embedding service class that can consume embedding apis from hf deploying using tf * Added kwargs to embedder svc * Added doc for embedding svc * update: FE changes for collections list, data source list API (#155) * Added exception handling for datasources list * feat: support path based routing (#165) * updates to pdf table parser (#168) * Updated embedding svc (#169) * updates to pdf table parser * Updated embedding svc class * Updated ReadMe * Added support for Infinity API, cleaned requirements, and dockerfile * Cleaned and reduced the main code package * updated requirements * updated requirements * Bump requests from 2.31.0 to 2.32.2 in /backend Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.31.0...v2.32.2) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * feat: add backend api url support using docker build args (#170) * Update ReadMe * Update ReadMe * Update ReadMe * added feature to add upload name to data dir upload (#167) * added feature to add upload name to data dir upload * feat: add upload name field in UI for local upload * Modifed regex for data-dir, and added error handling --------- Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Added embedding models list from embedding service of infinity * Explicitly pick cuda or cpu in reranker (#176) * Updated local metadata yaml * Fixed data source visiblity on FE (#177) * Fix datasource visiblity (#178) * Fixed data source visiblity on FE * Fixed data source return list in local * Uncomment torch from embedder.requirements.txt * Bump pillow from 10.2.0 to 10.3.0 in /backend Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.2.0 to 10.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.2.0...10.3.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Bump fastapi from 0.104.1 to 0.109.1 in /backend Bumps [fastapi](https://github.com/tiangolo/fastapi) from 0.104.1 to 0.109.1. - [Release notes](https://github.com/tiangolo/fastapi/releases) - [Commits](fastapi/fastapi@0.104.1...0.109.1) --- updated-dependencies: - dependency-name: fastapi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Refactored multimodal parser * Fixed multimodal parser import * Update truefoundry to 0.2.4 * Changed names of query controllers * Remove unused code * fix: FE dockerfile api url null safety (#196) * Docker compose and fixes (#195) * Added docker compose for all services * Fixed docker compose files - health checks and port expose * Removed mixedbread embedding class, updated dockerfile to take build arg and export as env, updated reranker class to usie only small reranker * testing for reranker * disable Override env vars * Enabled rerankers * Enabled overide for env * Modified docker file for exp-1 * Modified docker file for exp-2 * Modified docker file for exp-3 * Modified docker file for exp-4 * Modified docker file for exp-5 * Modified docker file for exp-6 * Removed reranker locally should now use infinity service * Added embedding svc url to docker file * Updated FE docker compose * Fixed Frontend docker compose * updated readme for docker compose * Updated ReadMe * Updated Readme * Updated Readme * Fixed indexer to index entire collection when FQN is not supplied (#199) * Fixed indexer to index entire collection when FQN is not supplied * Added run as job arg * Updated readme with warnings * Added get single collection api (#201) * Bump qdrant-client from 1.7.3 to 1.9.0 in /backend (#204) Bumps [qdrant-client](https://github.com/qdrant/qdrant-client) from 1.7.3 to 1.9.0. - [Release notes](https://github.com/qdrant/qdrant-client/releases) - [Commits](qdrant/qdrant-client@v1.7.3...v1.9.0) --- updated-dependencies: - dependency-name: qdrant-client dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fixed qdrant migration on update of qdrant-client (#206) * Fixed qdrant migration on update of qdrant-client * modified args parse * modifed logging statements * update: optimize collections page with new api (#205) * update: optimize collections page with new api * cleanup --------- Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * cleanup: remove sidebar (#208) * Fixed argparse migration * Made API routes consistent in terms of trailing slashes (#212) * Made API routes consistent in terms of trailing slashes * Added additional routes to support the existing FE until then * Update compose.env * Update docker-compose.yaml * Update README.md * Update docker-compose.yaml * Bump @grpc/grpc-js from 1.8.21 to 1.8.22 in /frontend Bumps [@grpc/grpc-js](https://github.com/grpc/grpc-node) from 1.8.21 to 1.8.22. - [Release notes](https://github.com/grpc/grpc-node/releases) - [Commits](https://github.com/grpc/grpc-node/compare/@grpc/grpc-js@1.8.21...@grpc/grpc-js@1.8.22) --- updated-dependencies: - dependency-name: "@grpc/grpc-js" dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Added dependancy conditions (#213) * Added dependancy conditions * Changed ollama healthcheck * Update docker-compose.yaml Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Update README.md * Update compose.env * Bump braces from 3.0.2 to 3.0.3 in /frontend (#217) Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3. - [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md) - [Commits](micromatch/braces@3.0.2...3.0.3) --- updated-dependencies: - dependency-name: braces dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor qdrant client config parsing (#221) * Refactor qdrant url parsing * Fix NoneType parsing * Set prefer grpc explicitly False * fix fix: Correct environment variable handling in Frontend Dockerfile (#225) (#226) - Adjusted Dockerfile to properly set environment variables using ARG and ENV - Ensured VITE_USE_RELATIVE_BASE_URL, VITE_DOCS_QA_STANDALONE_PATH, and VITE_QA_FOUNDRY_URL are correctly passed and set during the build process Co-authored-by: HanSeulJo <hsjo@surromind.ai> * [WIP] Added models_config and gateway to make it easy to switch between models * [WIP] Started work of streamlining different models * Cleaned changes * Minor changes in model gateway * Minor fixes * Minor fixes and removed unused code * Started using the api_key env variable * fix: emvedding models response structure update * Remove unused code * Relational Metdata store with Prisma (#203) * Added prisma schema * Fixed data sources API, modified METADATA_STORE client, TODOs: fix clients in all the code, fix all collections APIs * Completed datasources and collections api * Added data ingestion apis * Added delete ingestion runs in delete collection * Modified query controllers * Wrapped TF metadata store in agent executor loop to support for sync function * Added docker compose for prisma and postgress * Tfloader is optional available only when TFY_API_KEY is present, modified regex for collection and datasource in FE * import of PrismaStore only for Local environment * Added indexer support via UI * Symlinks and API, for localdir upload, WIP * Added feature to upload data to docker vol * Added delete datasource API for prisma * Added empty function for delete_data_source in Truefoundry * feat: upload to local directory integration in FE (#228) * feat: upload to local directory integration in FE * feat: add data source unlink button * feat: integrate delete data source api * fix: add VITE_USE_LOCAL in dockerfile * Fixed docker-compose for FE * Removed console logs * Updated compose.env * Remove Local MetadataStore and Update ReadMe (#231) * Removed local metadata store * Updated ReadMe * Update README.md * fix: refetch data sources on delete, update endpoints, view all apis link (#233) * chore: remove trailing slash from API (#234) * feat: optional gtag manager support (#235) * fix: create collection new params * remove provider * fix: url based data source deletion (#237) * Update README.md (#238) * Minor fixes in models config (#239) * Small changes * Minor fixes * Streamline Docker compose installation and updated ReadMe (#240) * Streamlined docker compose * Updated ReadME * Updated ReadME * Added sample models.config.json * Added sample models.config.json * updated model types * Fixed Multimodal query controller for VLM calling, minor fixes (#241) * Fixed Multimodal query controller for VLM calling, minor fixes * Fixed streaming for controllers * Refactor and make all metadata store calls async (#243) * Fix models_config.json file issue (#242) * Ignore registring models if API key is not present, and put out a warning. Also create a config.json file at runtime, helpful for template deployments * increase stream timeout * Raise error when config file is not correct, acces file using paths, and computations during init * Added to_dict for models * Reduce env vars in compose * Misc improvements and fixes for local setup (#245) * Misc improvements and fixes for local setup * Fix ports for infinity and length check when embedding * Update workflow conditions * Update workflow * Revert removing reranker and embedding service url as build args * Fix backend url for FE in compose.env (#246) * Make models_config YAML for easy editing (#247) * Make models_config YAML for easy editing * Make models_config configurable via env * Remove redundant code * Add a comment * Pass on MODELS_CONFIG_PATH in env (#248) * Fix typos `contexual` -> `contextual` + remove unsued imports (#249) * Fix typos `contexual` -> `contextual` * Remove unused imports * Add calls to super() (#250) * Fix potentially undefined local variables (#251) * Added reranking via model_gateway class (#253) * Added reranking via model_gateway class * Added Auth for infinity API --------- Co-authored-by: Abhishek Choudhary <abhichoudhary06@gmail.com> * Fixed INFINITY_API_KEY issue in local deployment (#254) * Added INFINITY_API_KEY in config.sample * Optional auth for infnity service based on API key * Updated docker compose ---------
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.