Release v1.12.0 #594
Merged
Release v1.12.0 #594
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* update * update tests
* update OpenAI version to 1.2.3 * Fix dictionary convert
* add dream_emotion dist with annotators * add component cards * fix codestyle flake8 * fix codestyle * another codestyle fix * PR fixes * fixed batches * move preprocessing to annotator * fix codestyle black * minor fixes * removed unnecessary api_configs * changed type of annotator to bot utts annotator * fixed component file * fixed service.yml files * fixed formatters * fixed emotional response server and dist files * moved prompt into separate file * fixed annotators * fixed annotators again * fixed emotion response selector * fixed emotion dist and components * fixed codestyle * fixed codestyle again * fixed codestyle black * fixed naming of emotion response selector * fixed emotional_bot_response component * fixed emotion response selector server * added prev_services in annotator in pipeline_conf * fixed result formatting of bot_emo_cls * fixed codestyle * fixed codestyle again * changed description of emo_bot_response * added necessary component cards * fix: timeout * fix: timeout --------- Co-authored-by: Dilyara Zharikova (Baymurzina) <dilyara.rimovna@gmail.com>
* feat: update openai * fix: revert version of dockerfile
* custom entity linking * codestyle * update * feat: Add terminusdb service * chore: Add terminusdb service volume to dev * feat: Add user knowledge graph annotator * feat: Add KG skills * user_kg_skill * travel_italy_skill * confidence adjusted, pattern added * fix custom el * refactor: Replace env vars * In user-knowledge-graph and in travel_italy_skill * refactor * update pipeline_conf * update: extract multiple triplets * fix typo * remove bot uttr for cuxtom-el * use generic relations * fix url * switch to t5 lite * chore: Update user-kg to CRUD batch operations * fix: Separate storing entities and relationships This let us avoid assertion error about lists lengths in deeppavlov-kg * fix: Make all entities lowercase before storing This fixes the problem of storing the same entity many times * remove repetitive triplets * minor fix * fix: Check if entity exists in kg before adding it This prevents creating the same entity many times in KG * fix: Delete invalid condition * fix ckpt name * created kg_prompted distribution * fix: Update name_scenario func after DP-kg changes * chore: Modify logs * generative skill and prop_ext update * update lite model * fix: custom-el ports * user-kg: prompt key to write into ctx * remove repeating triplets with low score * fix: Update entity-detection file location * chore: Fix 'added' to return only added triplets * test: Update test.py for user-kg annotator * test: Fix according to prompted dist * test: Correct path of index sqlite db * llm update: gpt-j replaced by gpt-jt, oasst added * fix: removed extra services, added property-extraction to pipeline * fix proxy * feat: Turn KG triplets to NL & insert into prompt * update custom_el * chore: Add component & container cards * style: optimize Dockerfile for custom_el * style: flake and black up * style: flake and black up properly * feat: exrtact multiple triplets * codestyle * codestyle * fix * fix: Add env vars & correct ports * test: Add test.py * codestyle * fix * update: kg_prompt processing in generative skill * update: kg part added to template-prompted-skill * removed badlisted classifier * doc: Add readme to terminusdb * doc: update terminusdb_service readme * refactor: assistant_dists for dream_kg_prompted * update download path * container and component cards for kg-prompted-skill * update: pipeline for dream_kg_prompted dist * fix * typo * delete file * update: knowledge_prompted_skill moved from template_prompted_skill * chore: container cards for knowledge_prompted_skill * in prompt_selector * in agent_services * fix: dream_kg dist optimized * fix: pipeline changed for dream_kg_prompted dist * fix: prop-ex ports, terminusdb server db * fix: knowledge prompt added, pipeline optimized * refactor: KG code deleted from template-prompted-skill * fix: user-kg added to dp_formatters * reverse changes * chore: user-knowledge-graph component deleted * chore: vars moved & user-km added to distribution * vars moved to docker-compose * user-kg replaced by user-km in dream_kg_prompted * pipeline fixes * vars added in Dockerfiles for user-km & dff-knowledge-prompted-skill * added container cards for dream_kg_prompted agent * chore: user-kg formatter removed * revert: change in dream_kg * chore: del component card for last_chance_service * chore: removed USER_KG vars from template-skill * chore: del template_prompted skill formatter * revert: change in template-prompted-skill * chore: args update for knowledge-prompted-skill * style: black and flake8 up * style: update * feat: convert triplets to NL * changed args for user-km * added variable to use knowledge about user * drafted functionality for user-km to convert triplets from KG into natural language * changed config for terminusdb-server * chore: container cards update and style is up * style: flake8 up * feat: func to retrieve knowledge from user's graph * added functionality to extract triplets from user's graph * timeout increased for user-km * feat: relativity filter in user-km * added sentence-ranker port to user-km args in docker-compose and Dockerfile * added func to filter knowledge related to current context * renamed func and variables in user-km * timeout increase for user-km * style: black is up * fix: user-km annotations retrieval from context * user-kg replaced with user-km * kg_promt processing before adding to final prompt * chore: rename variables * fix: data in component card for knowledge skill * revert: change in convers evaluation selector * chore: proxy2 deleted * revert: change in rule_based_selector * chore: variables changed in user-km and kg-skill * generative service and sentence-ranker passed in as urls, not just ports * generative service configs passed in as variable from docker-compose * USE_KG_DATA variable added to kg-skill args in docker-compose * USE_KG_DATA variable converted into integer when uploaded from environment * service_configs cards and Dockerfiles updated for user-km and kg-skill * chore: chatgpt request into try-except in user-km * chore: improvements in user-km * prompt impoved for triplets conversion into NL * specified input and output data type for functions * chore: reserved port for knowledge_prompted_skill * chore: variable rename in user-km * chore: sentence-ranker request into try-except * chore: knowledge_prompted removed from agent_servs * update: dff_knowledge_prompted_skill * deleted condition.py from skill * removed two nodes from main.py * style: flake8 reformatting * remove unnecessary file * chore: minor changes in distribution * terminusdb-server added to components.tsv * timeout is moved to build args for services * service cards and Dockerfile for user-km updated * timeout changed to float variable * unnecessary file deleted from the skill * chore: user-km config variable detailed * config.json renamed * build args changes reflected in service cards * related_knowledge default value is added in case len(uttrs) is 0 * chore: fixed service_configs and component cards * for prompt selector and skill selector * changes reflected in pipeline * chore: spacy-nounphrases deleted from distribution * chore: variables renamed * chore: minor changes * fixed port in custom-el Dockerfile * RELEVANT_KNOWLEDGE_THRESHOLD renamed and moved to build args * changes in user-km service cards * fix: last_chace_service for dream_kg_prompted * chore: envvars for user-km * refactor: removed unused loops and vars in skill * fix: envvars_to_send as a List * revert: rule-based-selector component card * refactor: unused blocks of code del from skill * fix: last_chance_service component card * fix: reserve new port for knowledge-prompted-skill * refactor: del unused blocks of code from skill * refactor: last_human_uttr extraction from ctx * refactor: generative request in user-km * feat: variable to access LLM via proxy * revert: env variables * fix: user-km params for LLM request * fix: skilss test and gpus to be used --------- Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> Co-authored-by: annakorz <anna.korzanova12@gmail.com> Co-authored-by: Raushan <raushan.turganbay@alumni.nu.edu.kz> Co-authored-by: kpetyxova <kpetyxova@mail.ru> Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com>
* feat: add mistral 7b 128k * fix: model name * fix: model name * fix: configs * fix: config names and cards * fix: more cards * fix: mem * fix: no proxy available
* default values for unused variables in user-km for dream_kg distribution * missing tests for travel-italy-skill in dream_kg
* feat: add system requirements * feat: info about new components to README_ru.md * feat: readme update for Code AI
* Add configs for gpt4-turbo * fixed comments * fix: table style --------- Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com>
Add configs to envvars_to_send
* chore: user-km-prompted for dream_kg_prompted * reserve port for base version of user-km * component card and service_configs updated * updates in dream_kg_prompted distribution * updates in knowledge-prompted-skill * chore: user-km for dream_kg * component card created and service_configs updated * changed port in dream_kg dist * style: flake8 and black up * fix: user-km component and service name * style: black up * fix: container names for user-km-prompted
* moving files from a broken fork * adding tg token * changing the name of an attribute to be in line with dp-agent * an attempt to fix []'s and a fix in rule based skill selector * fixing attributes * fixing validation * small actor.py fix * testing * removing redundant string operations * a minor fix that should enable .oga -> .wav * using subprocess ffmpeg to convert audio * adding missing files * a facepalm fix * another facepalm fix * keyerror fix * model path fix * minor refactor * adding support for mp3 and MP3 * dcase-2020 -> dcase-2022 * adding experiments.zip to project * apt install gdown to download model files * pip install gdown to download model files * correct download links from gdrive * gdown -> wget * gdown -> wget 2 * fixing dockerfile * another dockerfile fix * adding fire to reqs * fixing new audio-captioning * mkdir -> makedirs * small error fix * debugging commit 1 * debugging commit 2 * debugging commit 3 * debugging commit 4 * debugging commit 5 * debugging commit 6 * debugging commit 7 * debugging commit 8 * debugging commit 9 * first working demo commit * added flac and ogg formats, minor refactoring * purging audio files dir before inference * minor fix * adding video notes support * removed hardcoded tg_token * model, service and docker files upgraded * captioning model fixed * image captioning restored * agent commit changed, image_captioning temporary excluded * Old agent * Fixed all but pipeline_conf.json * pipeline_conf.json fixed according to dev * actor.py fixed according to dev * Frommage removed * dream_voice distributive created * dream_voice distributive restored * dream_multimodal updated to dev * dream_multimodal updated to dev 2.0 * dream_multimodal updated to dev 3.0 * dream_multimodal updated to dev 4.0 * formatters fix in progress * dp_formatters updates * codestyle fixed, skill messages changed * badlisted words removed from dream_voice * pr fix * added tests to voice skill * voice_service fixed, tests added * another pr fix * pr fix * voice formatter fixed * another pr fix * captions -> caption, small pr fix * codestyle fix * flake8 fix * typo fixed * codestyle fix * tests fixed for voice skill * another typo fixed * reverted voice_service dockerfile * minor pr fix * voice service video fix * voice service video fix 2 * git repo commit hardcoded --------- Co-authored-by: Alexandra Dolidze <dolidze@gpu4.ipavlov.mipt.ru>
* first batch of readmes to the key services, TBC * readmes for dream multilingual dist * main distribution readmes upd, TBC * first batch of readmes to the key services, TBC * readmes for dream multilingual dist * main distribution readmes upd, TBC * readmes upd * fixed readmes, those old ones with wrong format * some more readmes * added readme templates for the components that hadn't have them yet. Also added info about I/O to the existing readmes. TBA * readme upd for skills -- templates added * readme upd for skills -- templates added * fixed readmes * upd readmes in skill selectors and response selectors * upd in dependencies and I/O * readmes for dream multilingual dist * readme upd - rebasing the branch after pulling the fresh dev * readme upd - rebasing the branch after pulling the fresh dev * fixes during rebase * fixes during rebase * fixes during rebase * fixes during rebase * fixes during rebase * missing readme added * fixes during rebase * fixed merge conflicts, re-updated files * ancient components readmes upd: added more content and examples * readme fixes: custom_el + personal_info_skill
* fix: prop-ex ports in dream_kg_prompted * fix: prop-ex component card * changes reflected in pipelines in dream_kg and dream_kg_prompted
* update pipeline_conf * fix: Use dream_kg pipeline for that dist * update: extract multiple triplets * fix typo * remove bot uttr for cuxtom-el * use generic relations * fix url * switch to t5 lite * chore: Update user-kg to CRUD batch operations * fix: Separate storing entities and relationships This let us avoid assertion error about lists lengths in deeppavlov-kg * fix: Make all entities lowercase before storing This fixes the problem of storing the same entity many times * remove repetitive triplets * minor fix * fix: Check if entity exists in kg before adding it This prevents creating the same entity many times in KG * fix: Delete invalid condition * fix ckpt name * created kg_prompted distribution * fix: Update name_scenario func after DP-kg changes * chore: Modify logs * generative skill and prop_ext update * update lite model * fix: custom-el ports * user-kg: prompt key to write into ctx * remove repeating triplets with low score * fix: Update entity-detection file location * chore: Fix 'added' to return only added triplets * test: Update test.py for user-kg annotator * test: Fix according to prompted dist * test: Correct path of index sqlite db * llm update: gpt-j replaced by gpt-jt, oasst added * fix: removed extra services, added property-extraction to pipeline * fix proxy * feat: Turn KG triplets to NL & insert into prompt * update custom_el * chore: Add component & container cards * style: optimize Dockerfile for custom_el * style: flake and black up * style: flake and black up properly * feat: exrtact multiple triplets * codestyle * codestyle * fix * fix: Add env vars & correct ports * test: Add test.py * codestyle * update user-knowledge-graph Bring updates from feat/new_prop_ex * chore: Add component card & occupy a port * chore: Delete add entities from custom_el * The user-kg does the work now utilizing index module of deeppavlov-kg * chore: Change app port & print test results * test: Fix test_el.py after deleting add_entities * fix * refactor: Clear user-kg formatter up * chore: Get rel_kind_dict out to a file * update: kg_prompt processing in generative skill * update: dff_travel_italy_skill modified * fix: bug in dff_travel_italy_skill * update: container cards for dff_travel_italy_skill * style: flake8 and black reformatting * update: Dockerfiles refactoring * dff_travel_italy_skill * dff_user_kg_skill * update: container cards for user-kg-skill * style: flake8 and black reformat for user-kg-skill * refactor: Divide into more comprehensible functions * update: source data added for KG skills * update: del repeated lines, add .env to skill * update: edit description, ram_usage changed * update: add .env_secret to travel-italy-skill * refactor: Make it work * doc: Add docstring to functions * style: Black up * update: kg part added to template-prompted-skill * removed badlisted classifier * doc: Add readme to terminusdb * doc: update terminusdb_service readme * refactor: assistant_dists for dream_kg_prompted * update download path * fix: Correct user_id, PE input & rel_list * container and component cards for kg-prompted-skill * update: pipeline for dream_kg_prompted dist * fix * test: Delete '_' from relationship in test.py * refactor: Rename to user-knowledge-memorizer * typo * delete file * update: knowledge_prompted_skill moved from template_prompted_skill * refactor: Add abstract kind scenario * doc: Complete check_entities_in_kg docstring * fix: skill tallied with refactored user-km * chore: container cards for knowledge_prompted_skill * in prompt_selector * in agent_services * fix: dream_kg dist optimized * fix: pipeline changed for dream_kg_prompted dist * fix: prop-ex ports, terminusdb server db * refactor: Add the user properties scenario * fix: knowledge prompt added, pipeline optimized * refactor: KG code deleted from template-prompted-skill * Merge branch 'dev' into feat/kg_annotator * chore: Replace env vars into args of container * chore: Replace env vars to skill container args * fix: travel_italy_skill test * Revert "chore: Replace env vars into args of container" This reverts commit 3dbaccd. * Revert "Merge branch 'dev' into feat/kg_annotator" This reverts commit c70dc5a. * chore: Replace env vars into args of container * chore: Delete proxy2 * remove unnecessary services * style: add line to end * fix: terminusdb bug * chore: optimized services for dream_kg dist * use prop extraction in dff skills * fix: bug in wiki_skill * feat: pipeline change for bot uttr in prop-ex * chore: PE formatter change in kg_prompted dist * chore: user-kg changed to user-km in prompted dist * user-kg deleted from annotators * dream_kg_prompted dist changed to user-km * bot property extraction * chore: prop-ex formatter changed in dream_kg * Revert "chore: prop-ex formatter changed in dream_kg" This reverts commit ed895cd. * chore: prop-ex formatter changed in dream_kg * fix: prop_ex_bot_formatter fix for segmented uttrs * chore: user-km formatters changed * fix: formatter for user-km last bot uttr * prop-ex input as segments * analyze user&bot utterance * chore: prop-ex bot deleted * fix: changes for user-km in dream_kg_prompted * ports changed in docker-compose, dev, pipeline and component card * terminusdb variables added to user-km in docker-compose * source added for user-km in pipeline * fix: pipeline_conf in dream_kg_prompted * for prop-ex and user-km * fix: user_knowledge_memorizer * if-statement added to accept error if no bot_utterances * user variables changed to bot * fix: User to Bot in user-km * fix: function name revert * feat: custom-el changes to annotate bot_utterances * custom-el added to pipeline to response_annotators * custom-el formatter for last_bot_utt * drafts for some other funcs to process last_bot_utt * fix: custom_el_last_bot formatter bug * feat: changes for entity detection * entity-detection formatter changed for last_bot_utt * entity-detection moved to response_annotators in pipeline * database changed in docker-compose * fix: entity-detection formatter for last_bot_utt * feat: entity_linking to response_annot in pipeline * feat: entity-linking changed for last_bot_utt * updates for entity-linking formatter for last_bot_utt * formatter corrected for entity-linking in pipeline * fix: EL error on an empty uttr * update: prop-ex last_bot_uttr formatter * added get_entities func * fix: bugs in bot-km and custom-el * bot_id changed in bot-km to avoid mismatch with index * bot_id is passed to custom-el via formatter as user_id * fix: Decode symbols in custom_el * when getting user_id from index * fix: check existance of a dict key * fix: Let terminusdb run locally * fix: log an error instead of exiting the annotator * fix: Iterate over reversed indices ** to avoid iterating over incorrect elements * chore: bot-km as a separate service * bot-km moved to its own folder and renamed * container cards for bot-km created * component card created for bot-km * bot-km under its own name in dream_kg_prompted distribution * fix: bot-km building along the whole platform * fix: bot-km error when adding entity to index * chore: one log info is off in bot-km * fix: decode symbols in custom_el * chore: timeout added to response_annotators in ppl * revert: user-km to its original view * INDEX_LOAD_PATH moved to CONFIG * chore: minor changes for bot-km * INDEX_LOAD_PATH moved to config * timeout increased for bot-km * fix: index path upload from config in bot-km * chore: bot_km updates * BOT_ID is moved to global variables * tests edited for bot-km * config fix in Dockerfile * remove: spacy-nounphrases from proxy * update: dist renamed into dream_bot_kg_prompted * fix: removed unused variable from the skill * fix: wrong utterance annotation * fix: Delete duplicates from to-create kinds * fix: del duplicates from to-create kinds in bot-km * feat: bot-km part to convert triplets into NL * generate_prompt function to bot-km and a test example * env variables in docker-compose and Dockerfile * variable to .env to use bot's knowledge * fix: pipeline for skill to wait for bot-km info * feat: skill adds info about bot to prompt * fix: bug in custom-el formatter * fix: timeout increase for bot-km & entity-linking * fix: iterate over reversed indices * to avoid iterating over incorrect elements * fix: accept and return a batch of utterances * chore: delete unneccessary index from bot-km test * chore : bot-km func to convert triplets to NL * added function to retrieve knowledge about bot * added functionality to convert triplets to natural language * some variables changed and added in Dockerfile * USE_BOT_KG_DATA moved from .env into docker-compose * bot-km service cards updated * chore: updates for knowledge-prompted-skill * USE_BOT_KG_DATA added to docker-compose and Dockerfile * Dockerfile refactored * service cards updated for the skill * USE_BOT_KG_DATA converted into integer when uploaded * feat: added functionality to bot-km * added func to filter related knowledge * added function to create prompt for the skill * added last_human_utterance for bot-km in dp_formatters * args added for bot-km to docker-compose, Dockerfile * servie_configs updated for bot-km * chore: triplet preprocessing in skill * feat: bot-kg-skill use knowledge-skill as template * skill renamed to dff_knowledge_prompted_skill * service cards added * bot_knowledge prompt added * bot_knowledge prompt added to prompt_selector * chore: triplet processing * triplet format changed in bot-km * triplet processing updated in skill * chore: component cards and service configs * for dream_bot_kg_prompted dist created/modified * changes reflected in pipeline * fix: Check in kg ONLY triplets that ARE IN index * chore: ports reservation for bot-km and the skill * ports reserved in components.tsv * changes reflected in distribution and components cards * chore: spacy-nounphrases deleted from distribution * chore: variables added to build args for bot-km * changes reflected in Dockerfile and service cards * chore: variable added to bot-km build args * changes reflected in Dockerfile and service cards * refactor: unused vars and loops deleted from skill * refactor: generative func in bot-km * chore: change of ports for bot-km and bot-skill * changes reflected in distribution and service cards * chore: formatters update * fix: custom-el, knowledge-skill * custom-el service_port fixed * knowledge skill refactored * style format * fix: prop-ex and prompt-selector * blocks of code returned to pro-ex to extract triplets from segments * sentence-ranker url added to .env for prompt-selector to work * fix: triplets conversion in bot-km and style * fix: variable upload to use bot knowledge in skill * fix: GPUs in dream_bot_kg_prompted * fix: bot-km port in tests * fix: prop-ex * fix: tests for bot-km * revert: merge errors * fix: removed old files and merge error * fix: package versions, component card * component card changed in pipeline_conf * sentence-ranker url removed from .env * fix: bot_id retrieved from context * changes in bot-km server file * custom-el formatter changed * property extraction formatter update * revert: changes in dream_kg * fix: prop-ex port in dream_bot_kg_prompted dist * fix: added default values to skill Dockerfile * USE_KG_DATA and USE_BOT_KG_DATA * fix: entity-detection formatter * fix: environemnet for bot-km * fixes propr ex * fix if bot uttr * fix: entity-linking and custom-el formatters * changes reflected in services server-files * style format * fix: Send unique list of kinds to be created in db * fix: default output for bot-km * returns the output in the usual form and prevents skill from error * fix: component cards for dream_bot_kg_prompted * property-extraction * entity-detection * entity-linking * custom-entity-linking * ner * response_selector * bot-knowledge-memorizer * dff-bot-knowledge-prompted-skill * fix: Send unique list of kinds to be created in db * fix: sentence-ranker in args for prompt-selector * changes reflected in service_configs for dream_kg_prompted and dream_bot_kg_prompted * fix: ner component card * fix: pipeline and component cards * bot-knowledge-prompted-skill * response_selector --------- Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> Co-authored-by: Ramimashkouk <rami.n.mashkouk@gmail.com> Co-authored-by: Raushan <raushan.turganbay@alumni.nu.edu.kz> Co-authored-by: kpetyxova <kpetyxova@mail.ru> Co-authored-by: Rami <54779216+Ramimashkouk@users.noreply.github.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Федор Игнатов <ignatov@gpu2.ipavlov.mipt.ru> Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com>
* fromage image skill fixed * codestyle fixed for pr
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.