Feat/ru intent catcher transformers #171

dilyararimovna · 2022-06-16T12:13:34Z

No description provided.

…rmers

…_intent_catcher_transformers

…rmers

assistant_dists/dream_russian/docker-compose.override.yml

…ers' into feat/ru_intent_catcher_transformers

* fix: itnent catcher ru transformers * fix: ru itnent catcher * fix: intent catcher updated * fix: INTENT_PHRASES_PATH as a main variable * fix: dockerfile updates * fix: test gpu * fix: black style * fix: add tests files * fix: tests * fix: rights on file * fix: regiths on file * fix: regiths * fix: numb hyps * fix: remove without threads

* feat: docker compose with main components * Feat/tests russian (#90) * feat: runtests russian * fix: test file and elements * feat: tests russian in jenkinsfile * feat: files for tests * fix; program-y name * fix: change to dff-intent-responder-skill * fix: sync with agent updates * fix: cleanup for both runtests * fix: fix path to pipeline conf * fix: remove ner from tests * fix: unbuild english bot before russian tests * fix: codestyle Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> * fix: no sentrewrite needed * Feat/ru program y (#88) * feat: ru program-y version * fix: variable name * fix: russian tests * feat: test files * fix: dff program-y skill for russian * fix: dff program-y skill for russian docker compose * fix: dff program-y skip eng tests * fix: logs * fix: add variable env * fix: revert dangerous skil * fix: type * fix: imports types * fix: line buffering * fix: default value * fix: tests * fix: program-y patterns * Feat/spellchecker levenstein ru (#89) * feat: add files with correct rights * feat: spell check ru * fix: add dockerfile path * fix: add commit * feat: new files * fix: config name * fix: config address * fix: config as a file * fix: config title * fix: consider list not sample * fix: test fix * fix: test codestyle * fix: levenstein tests * fix: levenstein limit memroy * fix: levenshtein spelling * fix: mapping for spelling * feat: batch processing * Feat/ru badwords (#93) * feat: russian obscene words * fix: badlist ru named as en * fix: badlist language * fix: badlist tests passing * Feat/dummy skill ru (#94) * fix: russian dummy responses for russian letters in human utterance * fix: codestyle * fix: black * Feat/ner russian (#92) * feat: ner config * feat: files for ner ru * feat: ner model * feat: ner integration * fix: format yml config * fix: format dockerfile * fix: path to data * fix: tests for ner russian * fix: codestyle * fix: update ner version * add russian entity detection * add russian entity linking * Update requirements.txt * Update ner_chunker.py * fix: rus entity detection tests (#96) * fix: rus entity detection tests * black codestyle * fix codestyle * fix codestyle * fix bug * codestyle * codestyle * codestyle Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> * Feat/intent catcher Ru based on multi lingual USE (#98) * fix: intent catcher params and paths * fix: paths in dockerfile * fix: pintent ru phrases without random ones * fix: random intent phrases * fix: intent training params * fix: intent requirements * fix: intent requirements * fix: download model * fix: model which to download * fix: imports for correct work * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: ccorrect path to save json threshold * feat: intent data ru json * fix: ccorrect path to save tests * fix: existing var * fix: regular phrases * fix: next test * fix: trianing logs and new threshold * fix: trianing logs and new thre change prhases * fix: change regexps * fix: change thresholds * fix: new template for itnent phrases * fix: etsts ru * feat: upd model * fix: upd logs of training, upd conf value * fix: punctuation * fix: punctuation * est: upd model * fix: training logs * fix: tests * fix: phrases for opinion * feat: upd model * feat: training logs * feat: upd model * fix: tests * fix: remove opinion request intent * feat: upd model * feat: upd model * fix: new train logs * fix: new phrases * fix: min prcision for intent * fix: lower boundary * fix: usage of lib * fix: codestyle * feat: add itsdangerous requirements * fix: spelling preproc endpoint * Feat/dialogpt ru and dff-generative-skill (#97) * Fix requirements.txt (#84) * feat: initialize dialogpt_RU * feat: files init * feat: basic integration of dialogpt_RU * fix: rename dialogpt * fix: dialogpt to device * fix: dialogpt final version * fix: dialogpt test * fix: dialogpt test * fix: dialogpt resources consumption * fix: dialogpt to tests * feat: dff generative skill * feat: dff generative skill * fix: remove extra files * fix: input to dialogpt * fix: input to dialogpt * fix: logging * fix: turn on tests * fix: get dialog from context * fix: get uttrs from context * fix: geempty uttrs * fix: return empty resp * fix: test file * fix: tests * fix: test ratio * add speech_function_* dist * add speech_function_* dist readme * added sf_functions * fix ports * fix:m codestyle * fix deployment config * fix: tests for generative skill * fix: codestyle * add formatters, fix pipeline * update speech function * sources * fix: check if dialogpt is ready * fix: wait services * rename book skill * remove old book skill, update usages * fix readme * fix codestyle * fix codestyle * fix codestyle * fix codestyle line length * move res_cor.json to shared files * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix cpu.yml, dockerfiles and test for sfc, sfp * fix codestyle issues * blacked with -l 120 * following Dilya's holy orders * following Dilya's not so holy orders * fix formatters * fix pipeline * fix pipeline and formatters * Adding timeouts + mapping of book skill * removed old & irrelevant tests * we've set confidence to super level * feat: midas cls sent tokenize only if needed (#101) * feat: midas cls sent tokenize only if needed * feat: take into account tokenized uttrs by bot * fix: codestyle * fix: itsdangerous reqs * fix: docker reqs * fix: check another container * fix: rights for file * fix: coestyle * fix: return tests for intent responder * fix: revert intent responder * fix: review fixes * fix: codestyle Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: remove convert and sentseg for now * Feat/dff-intent-responder-skill ru (#99) * feat: prepare new intent responder * fix: responses for intent responder ru * fix: test based on language * fix: path to intent response phrases * fix: remove convert and sentseg * fix: another gpus * fix: file path and logs * fix: env and logs for intent responder * fix: exit response * fix: choose_topic to low prioity intents * feat: tests for ru * fix: tests for exit ru * fix: black codestyle * fix: tests for itnent catcher en * fix: куьщму сщтмуке фтв ыутеыуп акщь еуыеы * feat: turn on generative skill * Feat/wiki parser RU (#114) * update * codestyle * add language parameter * fix: language arg * fix: language arg and revert generative in dockercompose * fix tests * codestyle * fix: tests for ru * fix: language value * fix: ru test results * fix: test pipe * fix: sort types_2hop * fix: black codestyle * fix: tests for en wiki * fix: quotes * fix: codestyle * fix: sort objects * fix: test for wiki parser * fix: codestyle Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com> * Feat/ru friendship skill (#120) * feat: add language parameters * fix: black codestyle * fix: codestyle * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru уокротила реплики * fix: dff friendship tests * fix: dff friendship tests * fix: languAGE FOR WIKI * fix: language default value * fix: language default value * fix: language env var * fix: use templates by language * fix: ru templates * fix: no lang env var in common * fix: lang to ackn * fizx: codestyle * feat: default lang value * fix: dummy for russian * fix: no en acks * fix: how are you ru * fix: logs for response functions * fix: logs for condition functions * fix: ru version if what to talk about * feat: ru tests * fix: codestyle * fix: ru condition to resp selector * fix: ru condition to resp selector * fix: logging level and configuration * fix: ascii in tests * fix: add 'user' to dff input * fix: add language env variable everywhere * Feat/dialogrpt ru (#121) * fix: file drafts * feat: files for dialogrpt * feat: dialogrpt pipeline and scores * feat: dialogrpt pipeline and scores * feat: dialogrpt readme * fix: small readme * fix: sno healthcheck * feat: add dialogrpt to pipeline * fix: codestyle * fix: test files * feat: upd packages in dockerfile * fix: path to file * fix: shared file * fix: codestyle * fix: imports * fix: option consider * fix: option consider * fix: codestyle * fix: vars * fix: test file * fix: convert to list predictions * fix: tests * fix: codestyle * fix: codestyle * fix: codestyle * fix: readme * fix: dialogrpt to tests * feat: no extra files, add tokenizer as parameter * fix: codestyle * fix: var name * fix: batch prediction * fix: batch prediction parameter * fix: test choice * fix: format values * fix: codestyle * fix: upd deeppavlov download * fix: dialogrpt container name * fix: dialogrpt as hyp annotator * fix: dialogrpt test * Feat/ru personal info (#125) * fix: ignorecase and no text in code * fix: russian in patterns * fix: language env var * fix: russian patterns and responses * fix: russian patterns and responses * fix: path to file * fix: test for new version * fix: test for en * fix: codestyle * fix: f palceholders * fix: format usage * fix: codestyle * fix: logs * fix: my name is not * fix: homeland pattern fixes * fix: me name is not function * fix: more logs * fix: fix my name is not function * fix: my name is not * fix: do you know my name * fix: test format * fix: test format * fix: test format and mroe tests * fix: test format * fix: more tests * fix: more tests * fix: mtest format prints * fix: black * fix: en tests * fix: en tests * fix: en tests * fix: en tests * fix entity detection (#127) * Feat/spacy lemmatizer (#129) * fix: add spacy annotator * fix: usage of spacy attributes * fix: test spacy annotator * fix: add params * fix: add params * fix: fix test * fix: rights on file * fix: codestyle * fix: extra f string * Feat/russian sentseg (#128) * feat: basic config (with no changes) * feat: data preproc * feat: data processing * fix: codestyle * fix: sentseg ru like dp.ner_rus config * fix: rename config * fix: fpath * fix: readme * fix: custom sentseg reader * fix: custom sentseg config * feat: sent segmentation * feat: sent segmentation tests * fix: rights on file * fix: codestyle * fix: data preproc in sentseg_ru too * fix: metric values for sentseg trained on ru subtitiles * fix: path to sentseg to download * fix: usae sentseg ru model * fux: rights for file * fix: newer spacy version * fix: newer depepavlov version * fix: reqs * fix: server * feat: new config for bert model * fix: upd sentseg config * fix: upd sentseg config * fix: remove old config * fix: config path * fix: deeppavlov 17 2 * fix: remove extra import * fix: new docker image base * fix: reinstall spacy * fix: resentseg tests * fix: codestyle * fix: docs * fix: add sentseg to tests * fix: dockerfile * fix: model path * fix: add dialogrpt o wait hosts * fix: more complicated test for badwords annotator * Fix/upd badlisted words (#130) * fix: more complicated test for badwords annotator * fix: revert badlisted en words * fix: russian badlisted words * fix: give tokenized sents after spacy * fix: ru badlisted words * fix: ru badlisted words folder * fix: ru badlisted words get data * fix: test file * fix: ru badlisted words tokenized sent * fix: ru badlisted words tokens * fix: codestyle * fix: revert badlisted to dev * fix: pipeline conf post_skill_selector_annotators * fix: sleep before re try to connect to dialogpt * fix: formatter format * fix: more russian badwords * fix: correct endpoint for spacy annotator * Feat/ru random questions (#131) * feat: random russian questions * feat: dummy provides random russian questions * fix: refactor questions * fix: add pre-dummy phrase * fix: add pre-dummy phrase * fix: codestyle * fix: path to file * fix: strip russian questions * fix: last chance response * fix: documentation * fix: more confident generative skill * fix: dummy response always available * fix: intent responder check if exist * fix: most dummy responses language based * fix: remove punctuation if present * fix: documentation * fix: documentation * fix: new limits for russian baseline * fix: dialogrpt scores as conveval * fix: sentseg ru remove commas * fix: no wiki-skill yet * fix: ner no threads * fix: can add prompt * fix: prompt with conf * fix: remove bad questions * fix: add punctuation to generated hyp * fix: remove quotes * fix: re-choose hyp only for en version * fix: dff-generative is aka script * fix: increase intent conf thresholds * fix: store only tokens for hyps * fix: consider only special intents * fix: codestyle * fix: final fixes, resp selectiorna and thresholds intent * fix: more obscene words * fix: Russian documentation * fix: image in docs * fix: questions * fix: bad words * feat: ru toxic classifier * fix: toxic check batch hypotheses too * fix: intent responder uses lang * fix * fix: correct usage of human bot utterances * fix: return 5 hypotheses * fix: more hyps, fix reqs * fix: black codestyle * fix: codestyle * fix: codestyle * feat: response selector uses params * fix: requirements * fix: requirements * fix: revmoe dialogpt prev ru * fix: requirements * fix: add dialogrpt again * fix: add dialogrpt * fix: add dialogpt ru * fix: requirements for dialogpt and dialogrpt * fix: return pymorphy to reqs * Feat/ru intent catcher transformers (#171) * fix: itnent catcher ru transformers * fix: ru itnent catcher * fix: intent catcher updated * fix: INTENT_PHRASES_PATH as a main variable * fix: dockerfile updates * fix: test gpu * fix: black style * fix: add tests files * fix: tests * fix: rights on file * fix: regiths on file * fix: regiths * fix: numb hyps * fix: remove without threads * fix: документация * fix: add LET_ME_ASK_YOU_PHRASES * fix: black style * fix: revert extra files * fix: dream mini uses the same params * fix: generative default response * fix: incase of no gpu * fix: resources and gpus consumption * fix: new image * fix: add prompt ones * fix: ru and en version sentsegs Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com>

* fix/slot extr conf (deeppavlov#156) * Fix requirements.txt (deeppavlov#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix slot extraction confidence Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Fix/simplier skill selection (deeppavlov#159) * feat: simplier skill selector * fix: refactor skill selector * fix: codestyle * fix: get detected_topics * fixes (deeppavlov#160) * Feat/transformers intent catcher (deeppavlov#135) * feat: train intent catcher * feat: intrnt_catcher transformers train and use * feat: intrnt_catcher transformers train and use * feat: intent_catcher server and test * fix: model to cuda * fix: cuda is available * fix: paths * fix: ignore index * fix: black style * fix: paths * fix: paths * fix:model modes * fix: load dataset * fix: no extra info * fix: no extra info * fix: params * fix: paths * fix: paths * fix: paths * fix: dockerfile and downloading * fix: black style * fix: extra im ports * fix: dockerfile * fix: paths and reqs * fix: paths * fix: lineterminator * fix: paths to save model * fix: paths to save model * fix: paths * fix: sentences * fix: test file * fix: working version * fix: working * fix info * files * fix: gpu for tests * fix: gpu distr * fix: codestyle * fix: codestyle * fix: friendship fallback * fix: codestyle * fix: book skill start if lets chat about books * fix: no repeat * fix book turn on * fix: remove extra comments * fix: some changes * fix: use dp based model * fix: random prhases tto * fix: dockerfile * fix: after review * fix: codestyle * fix: yml configs * fix: add files for intent catcher * fix: codestyle * fix: nvidia error fix * fix: upd config * fix nvidia keyring using wget * fix: working version * fix: add files * file path in dockerfile * fix: order of init * fix: codestyle * fix: upd intent catcher version v2 * fix: revert to prev version * fix: tests for upd version * correct scores Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> * fix: proxy dialogpt (deeppavlov#164) * Dream mini (deeppavlov#161) * Fix requirements.txt (deeppavlov#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * Add mini version of Dream * Update cpu.yml * Update pipeline_conf.json * Update proxy.yml * fix: cpu only for existing components * fix: cpu only itnent catcher Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/infilling: not used yet (deeppavlov#163) * Fix requirements.txt (deeppavlov#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * infilling added (not tested) * files moved, some paths fixed * [DGM-49] path to model fixed, test added, seems working * takes a batch, bigger test added * assert added to test * assert added to test * minor changes * fix: codestyle * fix: proxy pass * fix: yml configs * fix: refactor infilling and usage * fix: paths * fix: dockerfile * fix: upd files * fix: working version * fix: codestyle * fix: codestyle * fix: works on gpu * gix: readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/update dialogpt (deeppavlov#170) * feat: update dialogpt * fix: codestyle * fix: book skill false start * Feat/parametrize response selector (deeppavlov#169) * feat: parameterization in progress * fix: some progress * fix: parametrised * fix: parametrised resp selector * feat: confs for intent responder * working version * fix: black style * fix: codestyle * fix: resp selector * fix: dialogpt params * fix: one more param * fix: codestyle * fix: params * fix: revert dialogpt * feat: change params of dialogpt (deeppavlov#172) * Fix/protobuf version (deeppavlov#173) * fix: fix protobuf version for sentseg * fix: ner protobuf * feat: first russian dream (deeppavlov#176) * feat: docker compose with main components * Feat/tests russian (deeppavlov#90) * feat: runtests russian * fix: test file and elements * feat: tests russian in jenkinsfile * feat: files for tests * fix; program-y name * fix: change to dff-intent-responder-skill * fix: sync with agent updates * fix: cleanup for both runtests * fix: fix path to pipeline conf * fix: remove ner from tests * fix: unbuild english bot before russian tests * fix: codestyle Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> * fix: no sentrewrite needed * Feat/ru program y (deeppavlov#88) * feat: ru program-y version * fix: variable name * fix: russian tests * feat: test files * fix: dff program-y skill for russian * fix: dff program-y skill for russian docker compose * fix: dff program-y skip eng tests * fix: logs * fix: add variable env * fix: revert dangerous skil * fix: type * fix: imports types * fix: line buffering * fix: default value * fix: tests * fix: program-y patterns * Feat/spellchecker levenstein ru (deeppavlov#89) * feat: add files with correct rights * feat: spell check ru * fix: add dockerfile path * fix: add commit * feat: new files * fix: config name * fix: config address * fix: config as a file * fix: config title * fix: consider list not sample * fix: test fix * fix: test codestyle * fix: levenstein tests * fix: levenstein limit memroy * fix: levenshtein spelling * fix: mapping for spelling * feat: batch processing * Feat/ru badwords (deeppavlov#93) * feat: russian obscene words * fix: badlist ru named as en * fix: badlist language * fix: badlist tests passing * Feat/dummy skill ru (deeppavlov#94) * fix: russian dummy responses for russian letters in human utterance * fix: codestyle * fix: black * Feat/ner russian (deeppavlov#92) * feat: ner config * feat: files for ner ru * feat: ner model * feat: ner integration * fix: format yml config * fix: format dockerfile * fix: path to data * fix: tests for ner russian * fix: codestyle * fix: update ner version * add russian entity detection * add russian entity linking * Update requirements.txt * Update ner_chunker.py * fix: rus entity detection tests (deeppavlov#96) * fix: rus entity detection tests * black codestyle * fix codestyle * fix codestyle * fix bug * codestyle * codestyle * codestyle Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> * Feat/intent catcher Ru based on multi lingual USE (deeppavlov#98) * fix: intent catcher params and paths * fix: paths in dockerfile * fix: pintent ru phrases without random ones * fix: random intent phrases * fix: intent training params * fix: intent requirements * fix: intent requirements * fix: download model * fix: model which to download * fix: imports for correct work * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: ccorrect path to save json threshold * feat: intent data ru json * fix: ccorrect path to save tests * fix: existing var * fix: regular phrases * fix: next test * fix: trianing logs and new threshold * fix: trianing logs and new thre change prhases * fix: change regexps * fix: change thresholds * fix: new template for itnent phrases * fix: etsts ru * feat: upd model * fix: upd logs of training, upd conf value * fix: punctuation * fix: punctuation * est: upd model * fix: training logs * fix: tests * fix: phrases for opinion * feat: upd model * feat: training logs * feat: upd model * fix: tests * fix: remove opinion request intent * feat: upd model * feat: upd model * fix: new train logs * fix: new phrases * fix: min prcision for intent * fix: lower boundary * fix: usage of lib * fix: codestyle * feat: add itsdangerous requirements * fix: spelling preproc endpoint * Feat/dialogpt ru and dff-generative-skill (deeppavlov#97) * Fix requirements.txt (deeppavlov#84) * feat: initialize dialogpt_RU * feat: files init * feat: basic integration of dialogpt_RU * fix: rename dialogpt * fix: dialogpt to device * fix: dialogpt final version * fix: dialogpt test * fix: dialogpt test * fix: dialogpt resources consumption * fix: dialogpt to tests * feat: dff generative skill * feat: dff generative skill * fix: remove extra files * fix: input to dialogpt * fix: input to dialogpt * fix: logging * fix: turn on tests * fix: get dialog from context * fix: get uttrs from context * fix: geempty uttrs * fix: return empty resp * fix: test file * fix: tests * fix: test ratio * add speech_function_* dist * add speech_function_* dist readme * added sf_functions * fix ports * fix:m codestyle * fix deployment config * fix: tests for generative skill * fix: codestyle * add formatters, fix pipeline * update speech function * sources * fix: check if dialogpt is ready * fix: wait services * rename book skill * remove old book skill, update usages * fix readme * fix codestyle * fix codestyle * fix codestyle * fix codestyle line length * move res_cor.json to shared files * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix cpu.yml, dockerfiles and test for sfc, sfp * fix codestyle issues * blacked with -l 120 * following Dilya's holy orders * following Dilya's not so holy orders * fix formatters * fix pipeline * fix pipeline and formatters * Adding timeouts + mapping of book skill * removed old & irrelevant tests * we've set confidence to super level * feat: midas cls sent tokenize only if needed (deeppavlov#101) * feat: midas cls sent tokenize only if needed * feat: take into account tokenized uttrs by bot * fix: codestyle * fix: itsdangerous reqs * fix: docker reqs * fix: check another container * fix: rights for file * fix: coestyle * fix: return tests for intent responder * fix: revert intent responder * fix: review fixes * fix: codestyle Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: remove convert and sentseg for now * Feat/dff-intent-responder-skill ru (deeppavlov#99) * feat: prepare new intent responder * fix: responses for intent responder ru * fix: test based on language * fix: path to intent response phrases * fix: remove convert and sentseg * fix: another gpus * fix: file path and logs * fix: env and logs for intent responder * fix: exit response * fix: choose_topic to low prioity intents * feat: tests for ru * fix: tests for exit ru * fix: black codestyle * fix: tests for itnent catcher en * fix: куьщму сщтмуке фтв ыутеыуп акщь еуыеы * feat: turn on generative skill * Feat/wiki parser RU (deeppavlov#114) * update * codestyle * add language parameter * fix: language arg * fix: language arg and revert generative in dockercompose * fix tests * codestyle * fix: tests for ru * fix: language value * fix: ru test results * fix: test pipe * fix: sort types_2hop * fix: black codestyle * fix: tests for en wiki * fix: quotes * fix: codestyle * fix: sort objects * fix: test for wiki parser * fix: codestyle Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com> * Feat/ru friendship skill (deeppavlov#120) * feat: add language parameters * fix: black codestyle * fix: codestyle * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru уокротила реплики * fix: dff friendship tests * fix: dff friendship tests * fix: languAGE FOR WIKI * fix: language default value * fix: language default value * fix: language env var * fix: use templates by language * fix: ru templates * fix: no lang env var in common * fix: lang to ackn * fizx: codestyle * feat: default lang value * fix: dummy for russian * fix: no en acks * fix: how are you ru * fix: logs for response functions * fix: logs for condition functions * fix: ru version if what to talk about * feat: ru tests * fix: codestyle * fix: ru condition to resp selector * fix: ru condition to resp selector * fix: logging level and configuration * fix: ascii in tests * fix: add 'user' to dff input * fix: add language env variable everywhere * Feat/dialogrpt ru (deeppavlov#121) * fix: file drafts * feat: files for dialogrpt * feat: dialogrpt pipeline and scores * feat: dialogrpt pipeline and scores * feat: dialogrpt readme * fix: small readme * fix: sno healthcheck * feat: add dialogrpt to pipeline * fix: codestyle * fix: test files * feat: upd packages in dockerfile * fix: path to file * fix: shared file * fix: codestyle * fix: imports * fix: option consider * fix: option consider * fix: codestyle * fix: vars * fix: test file * fix: convert to list predictions * fix: tests * fix: codestyle * fix: codestyle * fix: codestyle * fix: readme * fix: dialogrpt to tests * feat: no extra files, add tokenizer as parameter * fix: codestyle * fix: var name * fix: batch prediction * fix: batch prediction parameter * fix: test choice * fix: format values * fix: codestyle * fix: upd deeppavlov download * fix: dialogrpt container name * fix: dialogrpt as hyp annotator * fix: dialogrpt test * Feat/ru personal info (deeppavlov#125) * fix: ignorecase and no text in code * fix: russian in patterns * fix: language env var * fix: russian patterns and responses * fix: russian patterns and responses * fix: path to file * fix: test for new version * fix: test for en * fix: codestyle * fix: f palceholders * fix: format usage * fix: codestyle * fix: logs * fix: my name is not * fix: homeland pattern fixes * fix: me name is not function * fix: more logs * fix: fix my name is not function * fix: my name is not * fix: do you know my name * fix: test format * fix: test format * fix: test format and mroe tests * fix: test format * fix: more tests * fix: more tests * fix: mtest format prints * fix: black * fix: en tests * fix: en tests * fix: en tests * fix: en tests * fix entity detection (deeppavlov#127) * Feat/spacy lemmatizer (deeppavlov#129) * fix: add spacy annotator * fix: usage of spacy attributes * fix: test spacy annotator * fix: add params * fix: add params * fix: fix test * fix: rights on file * fix: codestyle * fix: extra f string * Feat/russian sentseg (deeppavlov#128) * feat: basic config (with no changes) * feat: data preproc * feat: data processing * fix: codestyle * fix: sentseg ru like dp.ner_rus config * fix: rename config * fix: fpath * fix: readme * fix: custom sentseg reader * fix: custom sentseg config * feat: sent segmentation * feat: sent segmentation tests * fix: rights on file * fix: codestyle * fix: data preproc in sentseg_ru too * fix: metric values for sentseg trained on ru subtitiles * fix: path to sentseg to download * fix: usae sentseg ru model * fux: rights for file * fix: newer spacy version * fix: newer depepavlov version * fix: reqs * fix: server * feat: new config for bert model * fix: upd sentseg config * fix: upd sentseg config * fix: remove old config * fix: config path * fix: deeppavlov 17 2 * fix: remove extra import * fix: new docker image base * fix: reinstall spacy * fix: resentseg tests * fix: codestyle * fix: docs * fix: add sentseg to tests * fix: dockerfile * fix: model path * fix: add dialogrpt o wait hosts * fix: more complicated test for badwords annotator * Fix/upd badlisted words (deeppavlov#130) * fix: more complicated test for badwords annotator * fix: revert badlisted en words * fix: russian badlisted words * fix: give tokenized sents after spacy * fix: ru badlisted words * fix: ru badlisted words folder * fix: ru badlisted words get data * fix: test file * fix: ru badlisted words tokenized sent * fix: ru badlisted words tokens * fix: codestyle * fix: revert badlisted to dev * fix: pipeline conf post_skill_selector_annotators * fix: sleep before re try to connect to dialogpt * fix: formatter format * fix: more russian badwords * fix: correct endpoint for spacy annotator * Feat/ru random questions (deeppavlov#131) * feat: random russian questions * feat: dummy provides random russian questions * fix: refactor questions * fix: add pre-dummy phrase * fix: add pre-dummy phrase * fix: codestyle * fix: path to file * fix: strip russian questions * fix: last chance response * fix: documentation * fix: more confident generative skill * fix: dummy response always available * fix: intent responder check if exist * fix: most dummy responses language based * fix: remove punctuation if present * fix: documentation * fix: documentation * fix: new limits for russian baseline * fix: dialogrpt scores as conveval * fix: sentseg ru remove commas * fix: no wiki-skill yet * fix: ner no threads * fix: can add prompt * fix: prompt with conf * fix: remove bad questions * fix: add punctuation to generated hyp * fix: remove quotes * fix: re-choose hyp only for en version * fix: dff-generative is aka script * fix: increase intent conf thresholds * fix: store only tokens for hyps * fix: consider only special intents * fix: codestyle * fix: final fixes, resp selectiorna and thresholds intent * fix: more obscene words * fix: Russian documentation * fix: image in docs * fix: questions * fix: bad words * feat: ru toxic classifier * fix: toxic check batch hypotheses too * fix: intent responder uses lang * fix * fix: correct usage of human bot utterances * fix: return 5 hypotheses * fix: more hyps, fix reqs * fix: black codestyle * fix: codestyle * fix: codestyle * feat: response selector uses params * fix: requirements * fix: requirements * fix: revmoe dialogpt prev ru * fix: requirements * fix: add dialogrpt again * fix: add dialogrpt * fix: add dialogpt ru * fix: requirements for dialogpt and dialogrpt * fix: return pymorphy to reqs * Feat/ru intent catcher transformers (deeppavlov#171) * fix: itnent catcher ru transformers * fix: ru itnent catcher * fix: intent catcher updated * fix: INTENT_PHRASES_PATH as a main variable * fix: dockerfile updates * fix: test gpu * fix: black style * fix: add tests files * fix: tests * fix: rights on file * fix: regiths on file * fix: regiths * fix: numb hyps * fix: remove without threads * fix: документация * fix: add LET_ME_ASK_YOU_PHRASES * fix: black style * fix: revert extra files * fix: dream mini uses the same params * fix: generative default response * fix: incase of no gpu * fix: resources and gpus consumption * fix: new image * fix: add prompt ones * fix: ru and en version sentsegs Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: proxy usage command (deeppavlov#183) * Feat/multilingual ner (deeppavlov#186) * feat: ner multilingual case_agnostic * fix: ner config * fix: ner dockerfile * fix: upd config * fix: config for ner multilingual * feat: updated config * feat: working ner multilingual * fix: codfestyle * feat: upd spellcheck * fix: add cuda visible devices * fix: cuda visible devices * update fact-retrieval and text-qa (deeppavlov#168) * update fact retrieval * update squad * add answer sentence * update * fixes * update formatter * fixes * add logit ranker * codestyle * codestyle * fixes * codestyle * fix tests Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> * feat: upd dp-ner with extended version (deeppavlov#189) * feat: upd dp-ner with extended version * fix: upd tests * fix: working for tags * fix: codestyle * fix: user new model * feat: woking * fix: config * fix: upd ner dockerfile * fix: revert format list * fix: change ner for all dists * fix: upd dialogpt en params (deeppavlov#190) * fix: upd dialogpt en params * fix: black style * fix: upd params * fix: context format * fix: context format * fix: codestyle * docker fixes for hydra configuration poc (deeppavlov#34) * docker fixes hydra configuration poc * fix agent installation * fix dp-agent commit in dockerfile_agent * Fix requirements.txt (deeppavlov#84) * update pr against the new main branch * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * minimal reproducible example for new dream * add pem files to gitignore, small agent docker fix * change commit, remove copy settings * fix agent command in base compose file * fix agent installation * fix agent command in other dists * fix commands in readme, add telegram section * update en and ru readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> Co-authored-by: dmitrijeuseew <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> Co-authored-by: Olga Sofronova <60696748+olkaso@users.noreply.github.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> Co-authored-by: zucchini-nlp <100715397+zucchini-nlp@users.noreply.github.com>

* fix/slot extr conf (deeppavlov#156) * Fix requirements.txt (deeppavlov#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix slot extraction confidence Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Fix/simplier skill selection (deeppavlov#159) * feat: simplier skill selector * fix: refactor skill selector * fix: codestyle * fix: get detected_topics * fixes (deeppavlov#160) * Feat/transformers intent catcher (deeppavlov#135) * feat: train intent catcher * feat: intrnt_catcher transformers train and use * feat: intrnt_catcher transformers train and use * feat: intent_catcher server and test * fix: model to cuda * fix: cuda is available * fix: paths * fix: ignore index * fix: black style * fix: paths * fix: paths * fix:model modes * fix: load dataset * fix: no extra info * fix: no extra info * fix: params * fix: paths * fix: paths * fix: paths * fix: dockerfile and downloading * fix: black style * fix: extra im ports * fix: dockerfile * fix: paths and reqs * fix: paths * fix: lineterminator * fix: paths to save model * fix: paths to save model * fix: paths * fix: sentences * fix: test file * fix: working version * fix: working * fix info * files * fix: gpu for tests * fix: gpu distr * fix: codestyle * fix: codestyle * fix: friendship fallback * fix: codestyle * fix: book skill start if lets chat about books * fix: no repeat * fix book turn on * fix: remove extra comments * fix: some changes * fix: use dp based model * fix: random prhases tto * fix: dockerfile * fix: after review * fix: codestyle * fix: yml configs * fix: add files for intent catcher * fix: codestyle * fix: nvidia error fix * fix: upd config * fix nvidia keyring using wget * fix: working version * fix: add files * file path in dockerfile * fix: order of init * fix: codestyle * fix: upd intent catcher version v2 * fix: revert to prev version * fix: tests for upd version * correct scores Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> * fix: proxy dialogpt (deeppavlov#164) * Dream mini (deeppavlov#161) * Fix requirements.txt (deeppavlov#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * Add mini version of Dream * Update cpu.yml * Update pipeline_conf.json * Update proxy.yml * fix: cpu only for existing components * fix: cpu only itnent catcher Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/infilling: not used yet (deeppavlov#163) * Fix requirements.txt (deeppavlov#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * infilling added (not tested) * files moved, some paths fixed * [DGM-49] path to model fixed, test added, seems working * takes a batch, bigger test added * assert added to test * assert added to test * minor changes * fix: codestyle * fix: proxy pass * fix: yml configs * fix: refactor infilling and usage * fix: paths * fix: dockerfile * fix: upd files * fix: working version * fix: codestyle * fix: codestyle * fix: works on gpu * gix: readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/update dialogpt (deeppavlov#170) * feat: update dialogpt * fix: codestyle * fix: book skill false start * Feat/parametrize response selector (deeppavlov#169) * feat: parameterization in progress * fix: some progress * fix: parametrised * fix: parametrised resp selector * feat: confs for intent responder * working version * fix: black style * fix: codestyle * fix: resp selector * fix: dialogpt params * fix: one more param * fix: codestyle * fix: params * fix: revert dialogpt * feat: change params of dialogpt (deeppavlov#172) * Fix/protobuf version (deeppavlov#173) * fix: fix protobuf version for sentseg * fix: ner protobuf * feat: first russian dream (deeppavlov#176) * feat: docker compose with main components * Feat/tests russian (deeppavlov#90) * feat: runtests russian * fix: test file and elements * feat: tests russian in jenkinsfile * feat: files for tests * fix; program-y name * fix: change to dff-intent-responder-skill * fix: sync with agent updates * fix: cleanup for both runtests * fix: fix path to pipeline conf * fix: remove ner from tests * fix: unbuild english bot before russian tests * fix: codestyle Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> * fix: no sentrewrite needed * Feat/ru program y (deeppavlov#88) * feat: ru program-y version * fix: variable name * fix: russian tests * feat: test files * fix: dff program-y skill for russian * fix: dff program-y skill for russian docker compose * fix: dff program-y skip eng tests * fix: logs * fix: add variable env * fix: revert dangerous skil * fix: type * fix: imports types * fix: line buffering * fix: default value * fix: tests * fix: program-y patterns * Feat/spellchecker levenstein ru (deeppavlov#89) * feat: add files with correct rights * feat: spell check ru * fix: add dockerfile path * fix: add commit * feat: new files * fix: config name * fix: config address * fix: config as a file * fix: config title * fix: consider list not sample * fix: test fix * fix: test codestyle * fix: levenstein tests * fix: levenstein limit memroy * fix: levenshtein spelling * fix: mapping for spelling * feat: batch processing * Feat/ru badwords (deeppavlov#93) * feat: russian obscene words * fix: badlist ru named as en * fix: badlist language * fix: badlist tests passing * Feat/dummy skill ru (deeppavlov#94) * fix: russian dummy responses for russian letters in human utterance * fix: codestyle * fix: black * Feat/ner russian (deeppavlov#92) * feat: ner config * feat: files for ner ru * feat: ner model * feat: ner integration * fix: format yml config * fix: format dockerfile * fix: path to data * fix: tests for ner russian * fix: codestyle * fix: update ner version * add russian entity detection * add russian entity linking * Update requirements.txt * Update ner_chunker.py * fix: rus entity detection tests (deeppavlov#96) * fix: rus entity detection tests * black codestyle * fix codestyle * fix codestyle * fix bug * codestyle * codestyle * codestyle Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> * Feat/intent catcher Ru based on multi lingual USE (deeppavlov#98) * fix: intent catcher params and paths * fix: paths in dockerfile * fix: pintent ru phrases without random ones * fix: random intent phrases * fix: intent training params * fix: intent requirements * fix: intent requirements * fix: download model * fix: model which to download * fix: imports for correct work * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: ccorrect path to save json threshold * feat: intent data ru json * fix: ccorrect path to save tests * fix: existing var * fix: regular phrases * fix: next test * fix: trianing logs and new threshold * fix: trianing logs and new thre change prhases * fix: change regexps * fix: change thresholds * fix: new template for itnent phrases * fix: etsts ru * feat: upd model * fix: upd logs of training, upd conf value * fix: punctuation * fix: punctuation * est: upd model * fix: training logs * fix: tests * fix: phrases for opinion * feat: upd model * feat: training logs * feat: upd model * fix: tests * fix: remove opinion request intent * feat: upd model * feat: upd model * fix: new train logs * fix: new phrases * fix: min prcision for intent * fix: lower boundary * fix: usage of lib * fix: codestyle * feat: add itsdangerous requirements * fix: spelling preproc endpoint * Feat/dialogpt ru and dff-generative-skill (deeppavlov#97) * Fix requirements.txt (deeppavlov#84) * feat: initialize dialogpt_RU * feat: files init * feat: basic integration of dialogpt_RU * fix: rename dialogpt * fix: dialogpt to device * fix: dialogpt final version * fix: dialogpt test * fix: dialogpt test * fix: dialogpt resources consumption * fix: dialogpt to tests * feat: dff generative skill * feat: dff generative skill * fix: remove extra files * fix: input to dialogpt * fix: input to dialogpt * fix: logging * fix: turn on tests * fix: get dialog from context * fix: get uttrs from context * fix: geempty uttrs * fix: return empty resp * fix: test file * fix: tests * fix: test ratio * add speech_function_* dist * add speech_function_* dist readme * added sf_functions * fix ports * fix:m codestyle * fix deployment config * fix: tests for generative skill * fix: codestyle * add formatters, fix pipeline * update speech function * sources * fix: check if dialogpt is ready * fix: wait services * rename book skill * remove old book skill, update usages * fix readme * fix codestyle * fix codestyle * fix codestyle * fix codestyle line length * move res_cor.json to shared files * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix cpu.yml, dockerfiles and test for sfc, sfp * fix codestyle issues * blacked with -l 120 * following Dilya's holy orders * following Dilya's not so holy orders * fix formatters * fix pipeline * fix pipeline and formatters * Adding timeouts + mapping of book skill * removed old & irrelevant tests * we've set confidence to super level * feat: midas cls sent tokenize only if needed (deeppavlov#101) * feat: midas cls sent tokenize only if needed * feat: take into account tokenized uttrs by bot * fix: codestyle * fix: itsdangerous reqs * fix: docker reqs * fix: check another container * fix: rights for file * fix: coestyle * fix: return tests for intent responder * fix: revert intent responder * fix: review fixes * fix: codestyle Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: remove convert and sentseg for now * Feat/dff-intent-responder-skill ru (deeppavlov#99) * feat: prepare new intent responder * fix: responses for intent responder ru * fix: test based on language * fix: path to intent response phrases * fix: remove convert and sentseg * fix: another gpus * fix: file path and logs * fix: env and logs for intent responder * fix: exit response * fix: choose_topic to low prioity intents * feat: tests for ru * fix: tests for exit ru * fix: black codestyle * fix: tests for itnent catcher en * fix: куьщму сщтмуке фтв ыутеыуп акщь еуыеы * feat: turn on generative skill * Feat/wiki parser RU (deeppavlov#114) * update * codestyle * add language parameter * fix: language arg * fix: language arg and revert generative in dockercompose * fix tests * codestyle * fix: tests for ru * fix: language value * fix: ru test results * fix: test pipe * fix: sort types_2hop * fix: black codestyle * fix: tests for en wiki * fix: quotes * fix: codestyle * fix: sort objects * fix: test for wiki parser * fix: codestyle Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com> * Feat/ru friendship skill (deeppavlov#120) * feat: add language parameters * fix: black codestyle * fix: codestyle * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru уокротила реплики * fix: dff friendship tests * fix: dff friendship tests * fix: languAGE FOR WIKI * fix: language default value * fix: language default value * fix: language env var * fix: use templates by language * fix: ru templates * fix: no lang env var in common * fix: lang to ackn * fizx: codestyle * feat: default lang value * fix: dummy for russian * fix: no en acks * fix: how are you ru * fix: logs for response functions * fix: logs for condition functions * fix: ru version if what to talk about * feat: ru tests * fix: codestyle * fix: ru condition to resp selector * fix: ru condition to resp selector * fix: logging level and configuration * fix: ascii in tests * fix: add 'user' to dff input * fix: add language env variable everywhere * Feat/dialogrpt ru (deeppavlov#121) * fix: file drafts * feat: files for dialogrpt * feat: dialogrpt pipeline and scores * feat: dialogrpt pipeline and scores * feat: dialogrpt readme * fix: small readme * fix: sno healthcheck * feat: add dialogrpt to pipeline * fix: codestyle * fix: test files * feat: upd packages in dockerfile * fix: path to file * fix: shared file * fix: codestyle * fix: imports * fix: option consider * fix: option consider * fix: codestyle * fix: vars * fix: test file * fix: convert to list predictions * fix: tests * fix: codestyle * fix: codestyle * fix: codestyle * fix: readme * fix: dialogrpt to tests * feat: no extra files, add tokenizer as parameter * fix: codestyle * fix: var name * fix: batch prediction * fix: batch prediction parameter * fix: test choice * fix: format values * fix: codestyle * fix: upd deeppavlov download * fix: dialogrpt container name * fix: dialogrpt as hyp annotator * fix: dialogrpt test * Feat/ru personal info (deeppavlov#125) * fix: ignorecase and no text in code * fix: russian in patterns * fix: language env var * fix: russian patterns and responses * fix: russian patterns and responses * fix: path to file * fix: test for new version * fix: test for en * fix: codestyle * fix: f palceholders * fix: format usage * fix: codestyle * fix: logs * fix: my name is not * fix: homeland pattern fixes * fix: me name is not function * fix: more logs * fix: fix my name is not function * fix: my name is not * fix: do you know my name * fix: test format * fix: test format * fix: test format and mroe tests * fix: test format * fix: more tests * fix: more tests * fix: mtest format prints * fix: black * fix: en tests * fix: en tests * fix: en tests * fix: en tests * fix entity detection (deeppavlov#127) * Feat/spacy lemmatizer (deeppavlov#129) * fix: add spacy annotator * fix: usage of spacy attributes * fix: test spacy annotator * fix: add params * fix: add params * fix: fix test * fix: rights on file * fix: codestyle * fix: extra f string * Feat/russian sentseg (deeppavlov#128) * feat: basic config (with no changes) * feat: data preproc * feat: data processing * fix: codestyle * fix: sentseg ru like dp.ner_rus config * fix: rename config * fix: fpath * fix: readme * fix: custom sentseg reader * fix: custom sentseg config * feat: sent segmentation * feat: sent segmentation tests * fix: rights on file * fix: codestyle * fix: data preproc in sentseg_ru too * fix: metric values for sentseg trained on ru subtitiles * fix: path to sentseg to download * fix: usae sentseg ru model * fux: rights for file * fix: newer spacy version * fix: newer depepavlov version * fix: reqs * fix: server * feat: new config for bert model * fix: upd sentseg config * fix: upd sentseg config * fix: remove old config * fix: config path * fix: deeppavlov 17 2 * fix: remove extra import * fix: new docker image base * fix: reinstall spacy * fix: resentseg tests * fix: codestyle * fix: docs * fix: add sentseg to tests * fix: dockerfile * fix: model path * fix: add dialogrpt o wait hosts * fix: more complicated test for badwords annotator * Fix/upd badlisted words (deeppavlov#130) * fix: more complicated test for badwords annotator * fix: revert badlisted en words * fix: russian badlisted words * fix: give tokenized sents after spacy * fix: ru badlisted words * fix: ru badlisted words folder * fix: ru badlisted words get data * fix: test file * fix: ru badlisted words tokenized sent * fix: ru badlisted words tokens * fix: codestyle * fix: revert badlisted to dev * fix: pipeline conf post_skill_selector_annotators * fix: sleep before re try to connect to dialogpt * fix: formatter format * fix: more russian badwords * fix: correct endpoint for spacy annotator * Feat/ru random questions (deeppavlov#131) * feat: random russian questions * feat: dummy provides random russian questions * fix: refactor questions * fix: add pre-dummy phrase * fix: add pre-dummy phrase * fix: codestyle * fix: path to file * fix: strip russian questions * fix: last chance response * fix: documentation * fix: more confident generative skill * fix: dummy response always available * fix: intent responder check if exist * fix: most dummy responses language based * fix: remove punctuation if present * fix: documentation * fix: documentation * fix: new limits for russian baseline * fix: dialogrpt scores as conveval * fix: sentseg ru remove commas * fix: no wiki-skill yet * fix: ner no threads * fix: can add prompt * fix: prompt with conf * fix: remove bad questions * fix: add punctuation to generated hyp * fix: remove quotes * fix: re-choose hyp only for en version * fix: dff-generative is aka script * fix: increase intent conf thresholds * fix: store only tokens for hyps * fix: consider only special intents * fix: codestyle * fix: final fixes, resp selectiorna and thresholds intent * fix: more obscene words * fix: Russian documentation * fix: image in docs * fix: questions * fix: bad words * feat: ru toxic classifier * fix: toxic check batch hypotheses too * fix: intent responder uses lang * fix * fix: correct usage of human bot utterances * fix: return 5 hypotheses * fix: more hyps, fix reqs * fix: black codestyle * fix: codestyle * fix: codestyle * feat: response selector uses params * fix: requirements * fix: requirements * fix: revmoe dialogpt prev ru * fix: requirements * fix: add dialogrpt again * fix: add dialogrpt * fix: add dialogpt ru * fix: requirements for dialogpt and dialogrpt * fix: return pymorphy to reqs * Feat/ru intent catcher transformers (deeppavlov#171) * fix: itnent catcher ru transformers * fix: ru itnent catcher * fix: intent catcher updated * fix: INTENT_PHRASES_PATH as a main variable * fix: dockerfile updates * fix: test gpu * fix: black style * fix: add tests files * fix: tests * fix: rights on file * fix: regiths on file * fix: regiths * fix: numb hyps * fix: remove without threads * fix: документация * fix: add LET_ME_ASK_YOU_PHRASES * fix: black style * fix: revert extra files * fix: dream mini uses the same params * fix: generative default response * fix: incase of no gpu * fix: resources and gpus consumption * fix: new image * fix: add prompt ones * fix: ru and en version sentsegs Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: proxy usage command (deeppavlov#183) * Feat/multilingual ner (deeppavlov#186) * feat: ner multilingual case_agnostic * fix: ner config * fix: ner dockerfile * fix: upd config * fix: config for ner multilingual * feat: updated config * feat: working ner multilingual * fix: codfestyle * feat: upd spellcheck * fix: add cuda visible devices * fix: cuda visible devices * update fact-retrieval and text-qa (deeppavlov#168) * update fact retrieval * update squad * add answer sentence * update * fixes * update formatter * fixes * add logit ranker * codestyle * codestyle * fixes * codestyle * fix tests Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> * feat: upd dp-ner with extended version (deeppavlov#189) * feat: upd dp-ner with extended version * fix: upd tests * fix: working for tags * fix: codestyle * fix: user new model * feat: woking * fix: config * fix: upd ner dockerfile * fix: revert format list * fix: change ner for all dists * fix: upd dialogpt en params (deeppavlov#190) * fix: upd dialogpt en params * fix: black style * fix: upd params * fix: context format * fix: context format * fix: codestyle * docker fixes for hydra configuration poc (deeppavlov#34) * docker fixes hydra configuration poc * fix agent installation * fix dp-agent commit in dockerfile_agent * Fix requirements.txt (deeppavlov#84) * update pr against the new main branch * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * minimal reproducible example for new dream * add pem files to gitignore, small agent docker fix * change commit, remove copy settings * fix agent command in base compose file * fix agent installation * fix agent command in other dists * fix commands in readme, add telegram section * update en and ru readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * fix: prompts from dummy skill (deeppavlov#193) * Feat/sentence ranker as a service (deeppavlov#191) * feat: sentence ranker almost * feat: sentence ranker * fix: tests * fix: get scores * fix: codestyle * fix: reqs * fix: flask jsonify * fix: flask jsonify * fix: json types * fix: logs * fix: usage of single scores calcution * fix: codestyle * fix: codestyle * fix: refactor * fix: scores and curr_single_scores * fix: codestyle * feat: log * fix: model and test * fix: upd reqs for kg service (deeppavlov#195) * image captioning * update image captioning server.py * update image captioning server.py * add dream_multimodal * updates * updated pipeline Co-authored-by: dmitrijeuseew <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> Co-authored-by: Olga Sofronova <60696748+olkaso@users.noreply.github.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> Co-authored-by: zucchini-nlp <100715397+zucchini-nlp@users.noreply.github.com>

* Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * update (#2) * fix/slot extr conf (#156) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix slot extraction confidence Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Fix/simplier skill selection (#159) * feat: simplier skill selector * fix: refactor skill selector * fix: codestyle * fix: get detected_topics * fixes (#160) * Feat/transformers intent catcher (#135) * feat: train intent catcher * feat: intrnt_catcher transformers train and use * feat: intrnt_catcher transformers train and use * feat: intent_catcher server and test * fix: model to cuda * fix: cuda is available * fix: paths * fix: ignore index * fix: black style * fix: paths * fix: paths * fix:model modes * fix: load dataset * fix: no extra info * fix: no extra info * fix: params * fix: paths * fix: paths * fix: paths * fix: dockerfile and downloading * fix: black style * fix: extra im ports * fix: dockerfile * fix: paths and reqs * fix: paths * fix: lineterminator * fix: paths to save model * fix: paths to save model * fix: paths * fix: sentences * fix: test file * fix: working version * fix: working * fix info * files * fix: gpu for tests * fix: gpu distr * fix: codestyle * fix: codestyle * fix: friendship fallback * fix: codestyle * fix: book skill start if lets chat about books * fix: no repeat * fix book turn on * fix: remove extra comments * fix: some changes * fix: use dp based model * fix: random prhases tto * fix: dockerfile * fix: after review * fix: codestyle * fix: yml configs * fix: add files for intent catcher * fix: codestyle * fix: nvidia error fix * fix: upd config * fix nvidia keyring using wget * fix: working version * fix: add files * file path in dockerfile * fix: order of init * fix: codestyle * fix: upd intent catcher version v2 * fix: revert to prev version * fix: tests for upd version * correct scores Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> * fix: proxy dialogpt (#164) * Dream mini (#161) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * Add mini version of Dream * Update cpu.yml * Update pipeline_conf.json * Update proxy.yml * fix: cpu only for existing components * fix: cpu only itnent catcher Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/infilling: not used yet (#163) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * infilling added (not tested) * files moved, some paths fixed * [DGM-49] path to model fixed, test added, seems working * takes a batch, bigger test added * assert added to test * assert added to test * minor changes * fix: codestyle * fix: proxy pass * fix: yml configs * fix: refactor infilling and usage * fix: paths * fix: dockerfile * fix: upd files * fix: working version * fix: codestyle * fix: codestyle * fix: works on gpu * gix: readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/update dialogpt (#170) * feat: update dialogpt * fix: codestyle * fix: book skill false start * Feat/parametrize response selector (#169) * feat: parameterization in progress * fix: some progress * fix: parametrised * fix: parametrised resp selector * feat: confs for intent responder * working version * fix: black style * fix: codestyle * fix: resp selector * fix: dialogpt params * fix: one more param * fix: codestyle * fix: params * fix: revert dialogpt * feat: change params of dialogpt (#172) * Fix/protobuf version (#173) * fix: fix protobuf version for sentseg * fix: ner protobuf * feat: first russian dream (#176) * feat: docker compose with main components * Feat/tests russian (#90) * feat: runtests russian * fix: test file and elements * feat: tests russian in jenkinsfile * feat: files for tests * fix; program-y name * fix: change to dff-intent-responder-skill * fix: sync with agent updates * fix: cleanup for both runtests * fix: fix path to pipeline conf * fix: remove ner from tests * fix: unbuild english bot before russian tests * fix: codestyle Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> * fix: no sentrewrite needed * Feat/ru program y (#88) * feat: ru program-y version * fix: variable name * fix: russian tests * feat: test files * fix: dff program-y skill for russian * fix: dff program-y skill for russian docker compose * fix: dff program-y skip eng tests * fix: logs * fix: add variable env * fix: revert dangerous skil * fix: type * fix: imports types * fix: line buffering * fix: default value * fix: tests * fix: program-y patterns * Feat/spellchecker levenstein ru (#89) * feat: add files with correct rights * feat: spell check ru * fix: add dockerfile path * fix: add commit * feat: new files * fix: config name * fix: config address * fix: config as a file * fix: config title * fix: consider list not sample * fix: test fix * fix: test codestyle * fix: levenstein tests * fix: levenstein limit memroy * fix: levenshtein spelling * fix: mapping for spelling * feat: batch processing * Feat/ru badwords (#93) * feat: russian obscene words * fix: badlist ru named as en * fix: badlist language * fix: badlist tests passing * Feat/dummy skill ru (#94) * fix: russian dummy responses for russian letters in human utterance * fix: codestyle * fix: black * Feat/ner russian (#92) * feat: ner config * feat: files for ner ru * feat: ner model * feat: ner integration * fix: format yml config * fix: format dockerfile * fix: path to data * fix: tests for ner russian * fix: codestyle * fix: update ner version * add russian entity detection * add russian entity linking * Update requirements.txt * Update ner_chunker.py * fix: rus entity detection tests (#96) * fix: rus entity detection tests * black codestyle * fix codestyle * fix codestyle * fix bug * codestyle * codestyle * codestyle Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> * Feat/intent catcher Ru based on multi lingual USE (#98) * fix: intent catcher params and paths * fix: paths in dockerfile * fix: pintent ru phrases without random ones * fix: random intent phrases * fix: intent training params * fix: intent requirements * fix: intent requirements * fix: download model * fix: model which to download * fix: imports for correct work * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: ccorrect path to save json threshold * feat: intent data ru json * fix: ccorrect path to save tests * fix: existing var * fix: regular phrases * fix: next test * fix: trianing logs and new threshold * fix: trianing logs and new thre change prhases * fix: change regexps * fix: change thresholds * fix: new template for itnent phrases * fix: etsts ru * feat: upd model * fix: upd logs of training, upd conf value * fix: punctuation * fix: punctuation * est: upd model * fix: training logs * fix: tests * fix: phrases for opinion * feat: upd model * feat: training logs * feat: upd model * fix: tests * fix: remove opinion request intent * feat: upd model * feat: upd model * fix: new train logs * fix: new phrases * fix: min prcision for intent * fix: lower boundary * fix: usage of lib * fix: codestyle * feat: add itsdangerous requirements * fix: spelling preproc endpoint * Feat/dialogpt ru and dff-generative-skill (#97) * Fix requirements.txt (#84) * feat: initialize dialogpt_RU * feat: files init * feat: basic integration of dialogpt_RU * fix: rename dialogpt * fix: dialogpt to device * fix: dialogpt final version * fix: dialogpt test * fix: dialogpt test * fix: dialogpt resources consumption * fix: dialogpt to tests * feat: dff generative skill * feat: dff generative skill * fix: remove extra files * fix: input to dialogpt * fix: input to dialogpt * fix: logging * fix: turn on tests * fix: get dialog from context * fix: get uttrs from context * fix: geempty uttrs * fix: return empty resp * fix: test file * fix: tests * fix: test ratio * add speech_function_* dist * add speech_function_* dist readme * added sf_functions * fix ports * fix:m codestyle * fix deployment config * fix: tests for generative skill * fix: codestyle * add formatters, fix pipeline * update speech function * sources * fix: check if dialogpt is ready * fix: wait services * rename book skill * remove old book skill, update usages * fix readme * fix codestyle * fix codestyle * fix codestyle * fix codestyle line length * move res_cor.json to shared files * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix cpu.yml, dockerfiles and test for sfc, sfp * fix codestyle issues * blacked with -l 120 * following Dilya's holy orders * following Dilya's not so holy orders * fix formatters * fix pipeline * fix pipeline and formatters * Adding timeouts + mapping of book skill * removed old & irrelevant tests * we've set confidence to super level * feat: midas cls sent tokenize only if needed (#101) * feat: midas cls sent tokenize only if needed * feat: take into account tokenized uttrs by bot * fix: codestyle * fix: itsdangerous reqs * fix: docker reqs * fix: check another container * fix: rights for file * fix: coestyle * fix: return tests for intent responder * fix: revert intent responder * fix: review fixes * fix: codestyle Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: remove convert and sentseg for now * Feat/dff-intent-responder-skill ru (#99) * feat: prepare new intent responder * fix: responses for intent responder ru * fix: test based on language * fix: path to intent response phrases * fix: remove convert and sentseg * fix: another gpus * fix: file path and logs * fix: env and logs for intent responder * fix: exit response * fix: choose_topic to low prioity intents * feat: tests for ru * fix: tests for exit ru * fix: black codestyle * fix: tests for itnent catcher en * fix: куьщму сщтмуке фтв ыутеыуп акщь еуыеы * feat: turn on generative skill * Feat/wiki parser RU (#114) * update * codestyle * add language parameter * fix: language arg * fix: language arg and revert generative in dockercompose * fix tests * codestyle * fix: tests for ru * fix: language value * fix: ru test results * fix: test pipe * fix: sort types_2hop * fix: black codestyle * fix: tests for en wiki * fix: quotes * fix: codestyle * fix: sort objects * fix: test for wiki parser * fix: codestyle Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com> * Feat/ru friendship skill (#120) * feat: add language parameters * fix: black codestyle * fix: codestyle * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru уокротила реплики * fix: dff friendship tests * fix: dff friendship tests * fix: languAGE FOR WIKI * fix: language default value * fix: language default value * fix: language env var * fix: use templates by language * fix: ru templates * fix: no lang env var in common * fix: lang to ackn * fizx: codestyle * feat: default lang value * fix: dummy for russian * fix: no en acks * fix: how are you ru * fix: logs for response functions * fix: logs for condition functions * fix: ru version if what to talk about * feat: ru tests * fix: codestyle * fix: ru condition to resp selector * fix: ru condition to resp selector * fix: logging level and configuration * fix: ascii in tests * fix: add 'user' to dff input * fix: add language env variable everywhere * Feat/dialogrpt ru (#121) * fix: file drafts * feat: files for dialogrpt * feat: dialogrpt pipeline and scores * feat: dialogrpt pipeline and scores * feat: dialogrpt readme * fix: small readme * fix: sno healthcheck * feat: add dialogrpt to pipeline * fix: codestyle * fix: test files * feat: upd packages in dockerfile * fix: path to file * fix: shared file * fix: codestyle * fix: imports * fix: option consider * fix: option consider * fix: codestyle * fix: vars * fix: test file * fix: convert to list predictions * fix: tests * fix: codestyle * fix: codestyle * fix: codestyle * fix: readme * fix: dialogrpt to tests * feat: no extra files, add tokenizer as parameter * fix: codestyle * fix: var name * fix: batch prediction * fix: batch prediction parameter * fix: test choice * fix: format values * fix: codestyle * fix: upd deeppavlov download * fix: dialogrpt container name * fix: dialogrpt as hyp annotator * fix: dialogrpt test * Feat/ru personal info (#125) * fix: ignorecase and no text in code * fix: russian in patterns * fix: language env var * fix: russian patterns and responses * fix: russian patterns and responses * fix: path to file * fix: test for new version * fix: test for en * fix: codestyle * fix: f palceholders * fix: format usage * fix: codestyle * fix: logs * fix: my name is not * fix: homeland pattern fixes * fix: me name is not function * fix: more logs * fix: fix my name is not function * fix: my name is not * fix: do you know my name * fix: test format * fix: test format * fix: test format and mroe tests * fix: test format * fix: more tests * fix: more tests * fix: mtest format prints * fix: black * fix: en tests * fix: en tests * fix: en tests * fix: en tests * fix entity detection (#127) * Feat/spacy lemmatizer (#129) * fix: add spacy annotator * fix: usage of spacy attributes * fix: test spacy annotator * fix: add params * fix: add params * fix: fix test * fix: rights on file * fix: codestyle * fix: extra f string * Feat/russian sentseg (#128) * feat: basic config (with no changes) * feat: data preproc * feat: data processing * fix: codestyle * fix: sentseg ru like dp.ner_rus config * fix: rename config * fix: fpath * fix: readme * fix: custom sentseg reader * fix: custom sentseg config * feat: sent segmentation * feat: sent segmentation tests * fix: rights on file * fix: codestyle * fix: data preproc in sentseg_ru too * fix: metric values for sentseg trained on ru subtitiles * fix: path to sentseg to download * fix: usae sentseg ru model * fux: rights for file * fix: newer spacy version * fix: newer depepavlov version * fix: reqs * fix: server * feat: new config for bert model * fix: upd sentseg config * fix: upd sentseg config * fix: remove old config * fix: config path * fix: deeppavlov 17 2 * fix: remove extra import * fix: new docker image base * fix: reinstall spacy * fix: resentseg tests * fix: codestyle * fix: docs * fix: add sentseg to tests * fix: dockerfile * fix: model path * fix: add dialogrpt o wait hosts * fix: more complicated test for badwords annotator * Fix/upd badlisted words (#130) * fix: more complicated test for badwords annotator * fix: revert badlisted en words * fix: russian badlisted words * fix: give tokenized sents after spacy * fix: ru badlisted words * fix: ru badlisted words folder * fix: ru badlisted words get data * fix: test file * fix: ru badlisted words tokenized sent * fix: ru badlisted words tokens * fix: codestyle * fix: revert badlisted to dev * fix: pipeline conf post_skill_selector_annotators * fix: sleep before re try to connect to dialogpt * fix: formatter format * fix: more russian badwords * fix: correct endpoint for spacy annotator * Feat/ru random questions (#131) * feat: random russian questions * feat: dummy provides random russian questions * fix: refactor questions * fix: add pre-dummy phrase * fix: add pre-dummy phrase * fix: codestyle * fix: path to file * fix: strip russian questions * fix: last chance response * fix: documentation * fix: more confident generative skill * fix: dummy response always available * fix: intent responder check if exist * fix: most dummy responses language based * fix: remove punctuation if present * fix: documentation * fix: documentation * fix: new limits for russian baseline * fix: dialogrpt scores as conveval * fix: sentseg ru remove commas * fix: no wiki-skill yet * fix: ner no threads * fix: can add prompt * fix: prompt with conf * fix: remove bad questions * fix: add punctuation to generated hyp * fix: remove quotes * fix: re-choose hyp only for en version * fix: dff-generative is aka script * fix: increase intent conf thresholds * fix: store only tokens for hyps * fix: consider only special intents * fix: codestyle * fix: final fixes, resp selectiorna and thresholds intent * fix: more obscene words * fix: Russian documentation * fix: image in docs * fix: questions * fix: bad words * feat: ru toxic classifier * fix: toxic check batch hypotheses too * fix: intent responder uses lang * fix * fix: correct usage of human bot utterances * fix: return 5 hypotheses * fix: more hyps, fix reqs * fix: black codestyle * fix: codestyle * fix: codestyle * feat: response selector uses params * fix: requirements * fix: requirements * fix: revmoe dialogpt prev ru * fix: requirements * fix: add dialogrpt again * fix: add dialogrpt * fix: add dialogpt ru * fix: requirements for dialogpt and dialogrpt * fix: return pymorphy to reqs * Feat/ru intent catcher transformers (#171) * fix: itnent catcher ru transformers * fix: ru itnent catcher * fix: intent catcher updated * fix: INTENT_PHRASES_PATH as a main variable * fix: dockerfile updates * fix: test gpu * fix: black style * fix: add tests files * fix: tests * fix: rights on file * fix: regiths on file * fix: regiths * fix: numb hyps * fix: remove without threads * fix: документация * fix: add LET_ME_ASK_YOU_PHRASES * fix: black style * fix: revert extra files * fix: dream mini uses the same params * fix: generative default response * fix: incase of no gpu * fix: resources and gpus consumption * fix: new image * fix: add prompt ones * fix: ru and en version sentsegs Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: proxy usage command (#183) * Feat/multilingual ner (#186) * feat: ner multilingual case_agnostic * fix: ner config * fix: ner dockerfile * fix: upd config * fix: config for ner multilingual * feat: updated config * feat: working ner multilingual * fix: codfestyle * feat: upd spellcheck * fix: add cuda visible devices * fix: cuda visible devices * update fact-retrieval and text-qa (#168) * update fact retrieval * update squad * add answer sentence * update * fixes * update formatter * fixes * add logit ranker * codestyle * codestyle * fixes * codestyle * fix tests Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> * feat: upd dp-ner with extended version (#189) * feat: upd dp-ner with extended version * fix: upd tests * fix: working for tags * fix: codestyle * fix: user new model * feat: woking * fix: config * fix: upd ner dockerfile * fix: revert format list * fix: change ner for all dists * fix: upd dialogpt en params (#190) * fix: upd dialogpt en params * fix: black style * fix: upd params * fix: context format * fix: context format * fix: codestyle * docker fixes for hydra configuration poc (#34) * docker fixes hydra configuration poc * fix agent installation * fix dp-agent commit in dockerfile_agent * Fix requirements.txt (#84) * update pr against the new main branch * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * minimal reproducible example for new dream * add pem files to gitignore, small agent docker fix * change commit, remove copy settings * fix agent command in base compose file * fix agent installation * fix agent command in other dists * fix commands in readme, add telegram section * update en and ru readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> Co-authored-by: dmitrijeuseew <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> Co-authored-by: Olga Sofronova <60696748+olkaso@users.noreply.github.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> Co-authored-by: zucchini-nlp <100715397+zucchini-nlp@users.noreply.github.com> * image captioning * image captioning * Image captioning (#4) * fix/slot extr conf (#156) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix slot extraction confidence Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Fix/simplier skill selection (#159) * feat: simplier skill selector * fix: refactor skill selector * fix: codestyle * fix: get detected_topics * fixes (#160) * Feat/transformers intent catcher (#135) * feat: train intent catcher * feat: intrnt_catcher transformers train and use * feat: intrnt_catcher transformers train and use * feat: intent_catcher server and test * fix: model to cuda * fix: cuda is available * fix: paths * fix: ignore index * fix: black style * fix: paths * fix: paths * fix:model modes * fix: load dataset * fix: no extra info * fix: no extra info * fix: params * fix: paths * fix: paths * fix: paths * fix: dockerfile and downloading * fix: black style * fix: extra im ports * fix: dockerfile * fix: paths and reqs * fix: paths * fix: lineterminator * fix: paths to save model * fix: paths to save model * fix: paths * fix: sentences * fix: test file * fix: working version * fix: working * fix info * files * fix: gpu for tests * fix: gpu distr * fix: codestyle * fix: codestyle * fix: friendship fallback * fix: codestyle * fix: book skill start if lets chat about books * fix: no repeat * fix book turn on * fix: remove extra comments * fix: some changes * fix: use dp based model * fix: random prhases tto * fix: dockerfile * fix: after review * fix: codestyle * fix: yml configs * fix: add files for intent catcher * fix: codestyle * fix: nvidia error fix * fix: upd config * fix nvidia keyring using wget * fix: working version * fix: add files * file path in dockerfile * fix: order of init * fix: codestyle * fix: upd intent catcher version v2 * fix: revert to prev version * fix: tests for upd version * correct scores Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> * fix: proxy dialogpt (#164) * Dream mini (#161) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * Add mini version of Dream * Update cpu.yml * Update pipeline_conf.json * Update proxy.yml * fix: cpu only for existing components * fix: cpu only itnent catcher Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/infilling: not used yet (#163) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * infilling added (not tested) * files moved, some paths fixed * [DGM-49] path to model fixed, test added, seems working * takes a batch, bigger test added * assert added to test * assert added to test * minor changes * fix: codestyle * fix: proxy pass * fix: yml configs * fix: refactor infilling and usage * fix: paths * fix: dockerfile * fix: upd files * fix: working version * fix: codestyle * fix: codestyle * fix: works on gpu * gix: readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/update dialogpt (#170) * feat: update dialogpt * fix: codestyle * fix: book skill false start * Feat/parametrize response selector (#169) * feat: parameterization in progress * fix: some progress * fix: parametrised * fix: parametrised resp selector * feat: confs for intent responder * working version * fix: black style * fix: codestyle * fix: resp selector * fix: dialogpt params * fix: one more param * fix: codestyle * fix: params * fix: revert dialogpt * feat: change params of dialogpt (#172) * Fix/protobuf version (#173) * fix: fix protobuf version for sentseg * fix: ner protobuf * feat: first russian dream (#176) * feat: docker compose with main components * Feat/tests russian (#90) * feat: runtests russian * fix: test file and elements * feat: tests russian in jenkinsfile * feat: files for tests * fix; program-y name * fix: change to dff-intent-responder-skill * fix: sync with agent updates * fix: cleanup for both runtests * fix: fix path to pipeline conf * fix: remove ner from tests * fix: unbuild english bot before russian tests * fix: codestyle Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> * fix: no sentrewrite needed * Feat/ru program y (#88) * feat: ru program-y version * fix: variable name * fix: russian tests * feat: test files * fix: dff program-y skill for russian * fix: dff program-y skill for russian docker compose * fix: dff program-y skip eng tests * fix: logs * fix: add variable env * fix: revert dangerous skil * fix: type * fix: imports types * fix: line buffering * fix: default value * fix: tests * fix: program-y patterns * Feat/spellchecker levenstein ru (#89) * feat: add files with correct rights * feat: spell check ru * fix: add dockerfile path * fix: add commit * feat: new files * fix: config name * fix: config address * fix: config as a file * fix: config title * fix: consider list not sample * fix: test fix * fix: test codestyle * fix: levenstein tests * fix: levenstein limit memroy * fix: levenshtein spelling * fix: mapping for spelling * feat: batch processing * Feat/ru badwords (#93) * feat: russian obscene words * fix: badlist ru named as en * fix: badlist language * fix: badlist tests passing * Feat/dummy skill ru (#94) * fix: russian dummy responses for russian letters in human utterance * fix: codestyle * fix: black * Feat/ner russian (#92) * feat: ner config * feat: files for ner ru * feat: ner model * feat: ner integration * fix: format yml config * fix: format dockerfile * fix: path to data * fix: tests for ner russian * fix: codestyle * fix: update ner version * add russian entity detection * add russian entity linking * Update requirements.txt * Update ner_chunker.py * fix: rus entity detection tests (#96) * fix: rus entity detection tests * black codestyle * fix codestyle * fix codestyle * fix bug * codestyle * codestyle * codestyle Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> * Feat/intent catcher Ru based on multi lingual USE (#98) * fix: intent catcher params and paths * fix: paths in dockerfile * fix: pintent ru phrases without random ones * fix: random intent phrases * fix: intent training params * fix: intent requirements * fix: intent requirements * fix: download model * fix: model which to download * fix: imports for correct work * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: ccorrect path to save json threshold * feat: intent data ru json * fix: ccorrect path to save tests * fix: existing var * fix: regular phrases * fix: next test * fix: trianing logs and new threshold * fix: trianing logs and new thre change prhases * fix: change regexps * fix: change thresholds * fix: new template for itnent phrases * fix: etsts ru * feat: upd model * fix: upd logs of training, upd conf value * fix: punctuation * fix: punctuation * est: upd model * fix: training logs * fix: tests * fix: phrases for opinion * feat: upd model * feat: training logs * feat: upd model * fix: tests * fix: remove opinion request intent * feat: upd model * feat: upd model * fix: new train logs * fix: new phrases * fix: min prcision for intent * fix: lower boundary * fix: usage of lib * fix: codestyle * feat: add itsdangerous requirements * fix: spelling preproc endpoint * Feat/dialogpt ru and dff-generative-skill (#97) * Fix requirements.txt (#84) * feat: initialize dialogpt_RU * feat: files init * feat: basic integration of dialogpt_RU * fix: rename dialogpt * fix: dialogpt to device * fix: dialogpt final version * fix: dialogpt test * fix: dialogpt test * fix: dialogpt resources consumption * fix: dialogpt to tests * feat: dff generative skill * feat: dff generative skill * fix: remove extra files * fix: input to dialogpt * fix: input to dialogpt * fix: logging * fix: turn on tests * fix: get dialog from context * fix: get uttrs from context * fix: geempty uttrs * fix: return empty resp * fix: test file * fix: tests * fix: test ratio * add speech_function_* dist * add speech_function_* dist readme * added sf_functions * fix ports * fix:m codestyle * fix deployment config * fix: tests for generative skill * fix: codestyle * add formatters, fix pipeline * update speech function * sources * fix: check if dialogpt is ready * fix: wait services * rename book skill * remove old book skill, update usages * fix readme * fix codestyle * fix codestyle * fix codestyle * fix codestyle line length * move res_cor.json to shared files * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix cpu.yml, dockerfiles and test for sfc, sfp * fix codestyle issues * blacked with -l 120 * following Dilya's holy orders * following Dilya's not so holy orders * fix formatters * fix pipeline * fix pipeline and formatters * Adding timeouts + mapping of book skill * removed old & irrelevant tests * we've set confidence to super level * feat: midas cls sent tokenize only if needed (#101) * feat: midas cls sent tokenize only if needed * feat: take into account tokenized uttrs by bot * fix: codestyle * fix: itsdangerous reqs * fix: docker reqs * fix: check another container * fix: rights for file * fix: coestyle * fix: return tests for intent responder * fix: revert intent responder * fix: review fixes * fix: codestyle Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: remove convert and sentseg for now * Feat/dff-intent-responder-skill ru (#99) * feat: prepare new intent responder * fix: responses for intent responder ru * fix: test based on language * fix: path to intent response phrases * fix: remove convert and sentseg * fix: another gpus * fix: file path and logs * fix: env and logs for intent responder * fix: exit response * fix: choose_topic to low prioity intents * feat: tests for ru * fix: tests for exit ru * fix: black codestyle * fix: tests for itnent catcher en * fix: куьщму сщтмуке фтв ыутеыуп акщь еуыеы * feat: turn on generative skill * Feat/wiki parser RU (#114) * update * codestyle * add language parameter * fix: language arg * fix: language arg and revert generative in dockercompose * fix tests * codestyle * fix: tests for ru * fix: language value * fix: ru test results * fix: test pipe * fix: sort types_2hop * fix: black codestyle * fix: tests for en wiki * fix: quotes * fix: codestyle * fix: sort objects * fix: test for wiki parser * fix: codestyle Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com> * Feat/ru friendship skill (#120) * feat: add language parameters * fix: black codestyle * fix: codestyle * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru уокротила реплики * fix: dff friendship tests * fix: dff friendship tests * fix: languAGE FOR WIKI * fix: language default value * fix: language default value * fix: language env var * fix: use templates by language * fix: ru templates * fix: no lang env var in common * fix: lang to ackn * fizx: codestyle * feat: default lang value * fix: dummy for russian * fix: no en acks * fix: how are you ru * fix: logs for response functions * fix: logs for condition functions * fix: ru version if what to talk about * feat: ru tests * fix: codestyle * fix: ru condition to resp selector * fix: ru condition to resp selector * fix: logging level and configuration * fix: ascii in tests * fix: add 'user' to dff input * fix: add language env variable everywhere * Feat/dialogrpt ru (#121) * fix: file drafts * feat: files for dialogrpt * feat: dialogrpt pipeline and scores * feat: dialogrpt pipeline and scores * feat: dialogrpt readme * fix: small readme * fix: sno healthcheck * feat: add dialogrpt to pipeline * fix: codestyle * fix: test files * feat: upd packages in dockerfile * fix: path to file * fix: shared file * fix: codestyle * fix: imports * fix: option consider * fix: option consider * fix: codestyle * fix: vars * fix: test file * fix: convert to list predictions * fix: tests * fix: codestyle * fix: codestyle * fix: codestyle * fix: readme * fix: dialogrpt to tests * feat: no extra files, add tokenizer as parameter * fix: codestyle * fix: var name * fix: batch prediction * fix: batch prediction parameter * fix: test choice * fix: format values * fix: codestyle * fix: upd deeppavlov download * fix: dialogrpt container name * fix: dialogrpt as hyp annotator * fix: dialogrpt test * Feat/ru personal info (#125) * fix: ignorecase and no text in code * fix: russian in patterns * fix: language env var * fix: russian patterns and responses * fix: russian patterns and responses * fix: path to file * fix: test for new version * fix: test for en * fix: codestyle * fix: f palceholders * fix: format usage * fix: codestyle * fix: logs * fix: my name is not * fix: homeland pattern fixes * fix: me name is not function * fix: more logs * fix: fix my name is not function * fix: my name is not * fix: do you know my name * fix: test format * fix: test format * fix: test format and mroe tests * fix: test format * fix: more tests * fix: more tests * fix: mtest format prints * fix: black * fix: en tests * fix: en tests * fix: en tests * fix: en tests * fix entity detection (#127) * Feat/spacy lemmatizer (#129) * fix: add spacy annotator * fix: usage of spacy attributes * fix: test spacy annotator * fix: add params * fix: add params * fix: fix test * fix: rights on file * fix: codestyle * fix: extra f string * Feat/russian sentseg (#128) * feat: basic config (with no changes) * feat: data preproc * feat: data processing * fix: codestyle * fix: sentseg ru like dp.ner_rus config * fix: rename config * fix: fpath * fix: readme * fix: custom sentseg reader * fix: custom sentseg config * feat: sent segmentation * feat: sent segmentation tests * fix: rights on file * fix: codestyle * fix: data preproc in sentseg_ru too * fix: metric values for sentseg trained on ru subtitiles * fix: path to sentseg to download * fix: usae sentseg ru model * fux: rights for file * fix: newer spacy version * fix: newer depepavlov version * fix: reqs * fix: server * feat: new config for bert model * fix: upd sentseg config * fix: upd sentseg config * fix: remove old config * fix: config path * fix: deeppavlov 17 2 * fix: remove extra import * fix: new docker image base * fix: reinstall spacy * fix: resentseg tests * fix: codestyle * fix: docs * fix: add sentseg to tests * fix: dockerfile * fix: model path * fix: add dialogrpt o wait hosts * fix: more complicated test for badwords annotator * Fix/upd badlisted words (#130) * fix: more complicated test for badwords annotator * fix: revert badlisted en words * fix: russian badlisted words * fix: give tokenized sents after spacy * fix: ru badlisted words * fix: ru badlisted words folder * fix: ru badlisted words get data * fix: test file * fix: ru badlisted words tokenized sent * fix: ru badlisted words tokens * fix: codestyle * fix: revert badlisted to dev * fix: pipeline conf post_skill_selector_annotators * fix: sleep before re try to connect to dialogpt * fix: formatter format * fix: more russian badwords * fix: correct endpoint for spacy annotator * Feat/ru random questions (#131) * feat: random russian questions * feat: dummy provides random russian questions * fix: refactor questions * fix: add pre-dummy phrase * fix: add pre-dummy phrase * fix: codestyle * fix: path to file * fix: strip russian questions * fix: last chance response * fix: documentation * fix: more confident generative skill * fix: dummy response always available * fix: intent responder check if exist * fix: most dummy responses language based * fix: remove punctuation if present * fix: documentation * fix: documentation * fix: new limits for russian baseline * fix: dialogrpt scores as conveval * fix: sentseg ru remove commas * fix: no wiki-skill yet * fix: ner no threads * fix: can add prompt * fix: prompt with conf * fix: remove bad questions * fix: add punctuation to generated hyp * fix: remove quotes * fix: re-choose hyp only for en version * fix: dff-generative is aka script * fix: increase intent conf thresholds * fix: store only tokens for hyps * fix: consider only special intents * fix: codestyle * fix: final fixes, resp selectiorna and thresholds intent * fix: more obscene words * fix: Russian documentation * fix: image in docs * fix: questions * fix: bad words * feat: ru toxic classifier * fix: toxic check batch hypotheses too * fix: intent responder uses lang * fix * fix: correct usage of human bot utterances * fix: return 5 hypotheses * fix: more hyps, fix reqs * fix: black codestyle * fix: codestyle * fix: codestyle * feat: response selector uses params * fix: requirements * fix: requirements * fix: revmoe dialogpt prev ru * fix: requirements * fix: add dialogrpt again * fix: add dialogrpt * fix: add dialogpt ru * fix: requirements for dialogpt and dialogrpt * fix: return pymorphy to reqs * Feat/ru intent catcher transformers (#171) * fix: itnent catcher ru transformers * fix: ru itnent catcher * fix: intent catcher updated * fix: INTENT_PHRASES_PATH as a main variable * fix: dockerfile updates * fix: test gpu * fix: black style * fix: add tests files * fix: tests * fix: rights on file * fix: regiths on file * fix: regiths * fix: numb hyps * fix: remove without threads * fix: документация * fix: add LET_ME_ASK_YOU_PHRASES * fix: black style * fix: revert extra files * fix: dream mini uses the same params * fix: generative default response * fix: incase of no gpu * fix: resources and gpus consumption * fix: new image * fix: add prompt ones * fix: ru and en version sentsegs Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: proxy usage command (#183) * Feat/multilingual ner (#186) * feat: ner multilingual case_agnostic * fix: ner config * fix: ner dockerfile * fix: upd config * fix: config for ner multilingual * feat: updated config * feat: working ner multilingual * fix: codfestyle * feat: upd spellcheck * fix: add cuda visible devices * fix: cuda visible devices * update fact-retrieval and text-qa (#168) * update fact retrieval * update squad * add answer sentence * update * fixes * update formatter * fixes * add logit ranker * codestyle * codestyle * fixes * codestyle * fix tests Co-authored-by: Дмитрий Евсеев <dmitrij.euseew@yandex.ru> * feat: upd dp-ner with extended version (#189) * feat: upd dp-ner with extended version * fix: upd tests * fix: working for tags * fix: codestyle * fix: user new model * feat: woking * fix: config * fix: upd ner dockerfile * fix: revert format list * fix: change ner for all dists * fix: upd dialogpt en params (#190) * fix: upd dialogpt en params * fix: black style * fix: upd params * fix: context format * fix: context format * fix: codestyle * docker fixes for hydra configuration poc (#34) * docker fixes hydra configuration poc * fix agent installation * fix dp-agent commit in dockerfile_agent * Fix requirements.txt (#84) * update pr against the new main branch * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * minimal reproducible example for new dream * add pem files to gitignore, small agent docker fix * change commit, remove copy settings * fix agent command in base compose file * fix agent installation * fix agent command in other dists * fix commands in readme, add telegram section * update en and ru readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * fix: prompts from dummy skill (#193) * Feat/sentence ranker as a service (#191) * feat: sentence ranker almost * feat: sentence ranker * fix: tests * fix: get scores * fix: codestyle * fix: reqs * fix: flask jsonify * fix: flask jsonify * fix: json types * fix: logs * fix: usage of single scores calcution * fix: codestyle * fix: codestyle * fix: refactor * fix: scores and curr_single_scores * fix: codestyle * feat: log * fix: model and test * fix: upd reqs for kg service (#195) * image captioning * update image captioning server.py * update image captioning server.py * add dream_multimodal * updates * updated pipeline Co-authored-by: dmitrijeuseew <dmitrij.euseew@yandex.ru> Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> Co-authored-by: Olga Sofronova <60696748+olkaso@users.noreply.github.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> Co-authored-by: zucchini-nlp <100715397+zucchini-nlp@users.noreply.github.com> * add: files * fix: docs line * fix: codestyule * fix: run command * fix: run command * fix itsdangerous requirements * image captioning * image captioning * Image captioning (#4) * fix/slot extr conf (#156) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix slot extraction confidence Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Fix/simplier skill selection (#159) * feat: simplier skill selector * fix: refactor skill selector * fix: codestyle * fix: get detected_topics * fixes (#160) * Feat/transformers intent catcher (#135) * feat: train intent catcher * feat: intrnt_catcher transformers train and use * feat: intrnt_catcher transformers train and use * feat: intent_catcher server and test * fix: model to cuda * fix: cuda is available * fix: paths * fix: ignore index * fix: black style * fix: paths * fix: paths * fix:model modes * fix: load dataset * fix: no extra info * fix: no extra info * fix: params * fix: paths * fix: paths * fix: paths * fix: dockerfile and downloading * fix: black style * fix: extra im ports * fix: dockerfile * fix: paths and reqs * fix: paths * fix: lineterminator * fix: paths to save model * fix: paths to save model * fix: paths * fix: sentences * fix: test file * fix: working version * fix: working * fix info * files * fix: gpu for tests * fix: gpu distr * fix: codestyle * fix: codestyle * fix: friendship fallback * fix: codestyle * fix: book skill start if lets chat about books * fix: no repeat * fix book turn on * fix: remove extra comments * fix: some changes * fix: use dp based model * fix: random prhases tto * fix: dockerfile * fix: after review * fix: codestyle * fix: yml configs * fix: add files for intent catcher * fix: codestyle * fix: nvidia error fix * fix: upd config * fix nvidia keyring using wget * fix: working version * fix: add files * file path in dockerfile * fix: order of init * fix: codestyle * fix: upd intent catcher version v2 * fix: revert to prev version * fix: tests for upd version * correct scores Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> * fix: proxy dialogpt (#164) * Dream mini (#161) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * Add mini version of Dream * Update cpu.yml * Update pipeline_conf.json * Update proxy.yml * fix: cpu only for existing components * fix: cpu only itnent catcher Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/infilling: not used yet (#163) * Fix requirements.txt (#84) * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * infilling added (not tested) * files moved, some paths fixed * [DGM-49] path to model fixed, test added, seems working * takes a batch, bigger test added * assert added to test * assert added to test * minor changes * fix: codestyle * fix: proxy pass * fix: yml configs * fix: refactor infilling and usage * fix: paths * fix: dockerfile * fix: upd files * fix: working version * fix: codestyle * fix: codestyle * fix: works on gpu * gix: readme Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Dilyara Baymurzina <dilyara.rimovna@gmail.com> * Feat/update dialogpt (#170) * feat: update dialogpt * fix: codestyle * fix: book skill false start * Feat/parametrize response selector (#169) * feat: parameterization in progress * fix: some progress * fix: parametrised * fix: parametrised resp selector * feat: confs for intent responder * working version * fix: black style * fix: codestyle * fix: resp selector * fix: dialogpt params * fix: one more param * fix: codestyle * fix: params * fix: revert dialogpt * feat: change params of dialogpt (#172) * Fix/protobuf version (#173) * fix: fix protobuf version for sentseg * fix: ner protobuf * feat: first russian dream (#176) * feat: docker compose with main components * Feat/tests russian (#90) * feat: runtests russian * fix: test file and elements * feat: tests russian in jenkinsfile * feat: files for tests * fix; program-y name * fix: change to dff-intent-responder-skill * fix: sync with agent updates * fix: cleanup for both runtests * fix: fix path to pipeline conf * fix: remove ner from tests * fix: unbuild english bot before russian tests * fix: codestyle Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> Co-authored-by: Fedor Ignatov <ignatov.fedor@gmail.com> * fix: no sentrewrite needed * Feat/ru program y (#88) * feat: ru program-y version * fix: variable name * fix: russian tests * feat: test files * fix: dff program-y skill for russian * fix: dff program-y skill for russian docker compose * fix: dff program-y skip eng tests * fix: logs * fix: add variable env * fix: revert dangerous skil * fix: type * fix: imports types * fix: line buffering * fix: default value * fix: tests * fix: program-y patterns * Feat/spellchecker levenstein ru (#89) * feat: add files with correct rights * feat: spell check ru * fix: add dockerfile path * fix: add commit * feat: new files * fix: config name * fix: config address * fix: config as a file * fix: config title * fix: consider list not sample * fix: test fix * fix: test codestyle * fix: levenstein tests * fix: levenstein limit memroy * fix: levenshtein spelling * fix: mapping for spelling * feat: batch processing * Feat/ru badwords (#93) * feat: russian obscene words * fix: badlist ru named as en * fix: badlist language * fix: badlist tests passing * Feat/dummy skill ru (#94) * fix: russian dummy responses for russian letters in human utterance * fix: codestyle * fix: black * Feat/ner russian (#92) * feat: ner config * feat: files for ner ru * feat: ner model * feat: ner integration * fix: format yml config * fix: format dockerfile * fix: path to data * fix: tests for ner russian * fix: codestyle * fix: update ner version * add russian entity detection * add russian entity linking * Update requirements.txt * Update ner_chunker.py * fix: rus entity detection tests (#96) * fix: rus entity detection tests * black codestyle * fix codestyle * fix codestyle * fix bug * codestyle * codestyle * codestyle Co-authored-by: dmitry <dmitrij.euseew@yandex.ru> * Feat/intent catcher Ru based on multi lingual USE (#98) * fix: intent catcher params and paths * fix: paths in dockerfile * fix: pintent ru phrases without random ones * fix: random intent phrases * fix: intent training params * fix: intent requirements * fix: intent requirements * fix: download model * fix: model which to download * fix: imports for correct work * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: corrected phrases * fix: ccorrect path to save json threshold * feat: intent data ru json * fix: ccorrect path to save tests * fix: existing var * fix: regular phrases * fix: next test * fix: trianing logs and new threshold * fix: trianing logs and new thre change prhases * fix: change regexps * fix: change thresholds * fix: new template for itnent phrases * fix: etsts ru * feat: upd model * fix: upd logs of training, upd conf value * fix: punctuation * fix: punctuation * est: upd model * fix: training logs * fix: tests * fix: phrases for opinion * feat: upd model * feat: training logs * feat: upd model * fix: tests * fix: remove opinion request intent * feat: upd model * feat: upd model * fix: new train logs * fix: new phrases * fix: min prcision for intent * fix: lower boundary * fix: usage of lib * fix: codestyle * feat: add itsdangerous requirements * fix: spelling preproc endpoint * Feat/dialogpt ru and dff-generative-skill (#97) * Fix requirements.txt (#84) * feat: initialize dialogpt_RU * feat: files init * feat: basic integration of dialogpt_RU * fix: rename dialogpt * fix: dialogpt to device * fix: dialogpt final version * fix: dialogpt test * fix: dialogpt test * fix: dialogpt resources consumption * fix: dialogpt to tests * feat: dff generative skill * feat: dff generative skill * fix: remove extra files * fix: input to dialogpt * fix: input to dialogpt * fix: logging * fix: turn on tests * fix: get dialog from context * fix: get uttrs from context * fix: geempty uttrs * fix: return empty resp * fix: test file * fix: tests * fix: test ratio * add speech_function_* dist * add speech_function_* dist readme * added sf_functions * fix ports * fix:m codestyle * fix deployment config * fix: tests for generative skill * fix: codestyle * add formatters, fix pipeline * update speech function * sources * fix: check if dialogpt is ready * fix: wait services * rename book skill * remove old book skill, update usages * fix readme * fix codestyle * fix codestyle * fix codestyle * fix codestyle line length * move res_cor.json to shared files * fix itsdangerous requirements * pin itsdangerous requirements for all flask==1.1.1 servers * fix cpu.yml, dockerfiles and test for sfc, sfp * fix codestyle issues * blacked with -l 120 * following Dilya's holy orders * following Dilya's not so holy orders * fix formatters * fix pipeline * fix pipeline and formatters * Adding timeouts + mapping of book skill * removed old & irrelevant tests * we've set confidence to super level * feat: midas cls sent tokenize only if needed (#101) * feat: midas cls sent tokenize only if needed * feat: take into account tokenized uttrs by bot * fix: codestyle * fix: itsdangerous reqs * fix: docker reqs * fix: check another container * fix: rights for file * fix: coestyle * fix: return tests for intent responder * fix: revert intent responder * fix: review fixes * fix: codestyle Co-authored-by: Andrii.Hura <54397922+AndriiHura@users.noreply.github.com> Co-authored-by: mtalimanchuk <mtalimanchuk@gmail.com> Co-authored-by: Daniel Kornev <daniel@zetuniverse.com> * fix: remove convert and sentseg for now * Feat/dff-intent-responder-skill ru (#99) * feat: prepare new intent responder * fix: responses for intent responder ru * fix: test based on language * fix: path to intent response phrases * fix: remove convert and sentseg * fix: another gpus * fix: file path and logs * fix: env and logs for intent responder * fix: exit response * fix: choose_topic to low prioity intents * feat: tests for ru * fix: tests for exit ru * fix: black codestyle * fix: tests for itnent catcher en * fix: куьщму сщтмуке фтв ыутеыуп акщь еуыеы * feat: turn on generative skill * Feat/wiki parser RU (#114) * update * codestyle * add language parameter * fix: language arg * fix: language arg and revert generative in dockercompose * fix tests * codestyle * fix: tests for ru * fix: language value * fix: ru test results * fix: test pipe * fix: sort types_2hop * fix: black codestyle * fix: tests for en wiki * fix: quotes * fix: codestyle * fix: sort objects * fix: test for wiki parser * fix: codestyle Co-authored-by: dilyararimovna <dilyara.rimovna@gmail.com> * Feat/ru friendship skill (#120) * feat: add language parameters * fix: black codestyle * fix: codestyle * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru * fix: dff friendship ru уокротила реплики * fix: dff friendship tests * fix: dff friendship tests * fix: languAGE FOR WIKI * fix: language default value * fix: language default value * fix: language env var * fix: use templates by language * fix: ru templates * fix: no lang env var in common * fix: lang to ackn * fizx: codestyle * feat: default lang value * fix: dummy for russian * fix: no en acks * fix: how are you ru * fix: logs for response functions * fix: logs for condition functions * fix: ru version if what to talk about * feat: ru tests * fix: codestyle * fix: ru condition to resp selector * fix: ru condition to resp selector * fix: logging level and configuration * fix: ascii in tests * fix: add 'user' to dff input * fix: add language env variable everywhere * Feat/dialogrpt ru (#121) * fix: file drafts * feat: files for dialogrpt * feat: dialogrpt pipeline and scores * feat: dialogrpt pipeline and scores * feat: dialogrpt readme * fix: small readme * fix: sno healthcheck * feat: add dialogrpt to pipeline * fix: codestyle * fix: test files * feat: upd packages in dockerfile * fix: path to file * fix: shared file * fix: codestyle * fix: imports * fix: option consider * fix: option consider * fix: codestyle * fix: vars * fix: test file * fix: convert to list predictions * fix: tests * fix: codestyle * fix: codestyle * fix: codestyle * fix: readme * fix: dialogrpt to tests * feat: no extra files, add tokenizer as parameter * fix: codestyle * fix: var name * fix: batch prediction * fix: batch prediction parameter * fix: test choice * fix: format values * fix: codestyle * fix: upd deeppavlov download * fix: dialogrpt container name * fix: dialogrpt as hyp annotator * fix: dialogrpt test * Feat/ru personal info (#125) * fix: ignorecase and no text in code * fix: russian in patterns * fix: language env var * fix: russian patterns and responses * fix: russian patterns and responses * fix: path to file * fix: test for new version * fix: test for en * fix: codestyle * fix: f palceholders * fix: format usage * fix: codestyle * fix: logs * fix: my name is not * fix: homeland pattern fixes * fix: me name is not function * fix: more logs * fix: fix my name is not function * fix: my name is not * fix: do you know my name * fix: test format * fix: test format * fix: test format and mroe tests * fix: test format * fix: more tests * fix: more tests * fix: mtest format prints * fix: black * fix: en tests * fix: en tests * fix: en tests * fix: en tests * fix entity detection (#127) * Feat/spacy lemmatizer (#129) * fix: add spacy annotator * fix: usage of spacy attributes * fix: test spacy annotator * fix: add params * fix: add params * fix: fix test * fix: rights on file * fix: codestyle * fix: extra f string * Feat/russian sentseg (#128) * feat: basic config (with no changes) * feat: data preproc * feat: data processing * fix: codestyle * fix: sentseg ru like dp.ner_rus config * fix: rename config * fix: fpath * fix: readme * fix: custom sentseg reader * fix: custom sentseg config * feat: sent segmentation * feat: sent segmentation tests * fix: rights on file * fix: codestyle * fix: data preproc in sentseg_ru too * fix: metric values for sentseg trained on ru subtitiles * fix: path to sentseg to download * fix: usae sentseg ru model * fux: rights for file * fix: newer spacy version * fix: newer depepavlov version * fix: reqs * fix: server * feat: new config for bert model * fix: upd sentseg config * fix: upd sentseg config * fix: remove old config * fix: config path * fix: deeppavlov 17 2 * fix: remove extra import * fix: new docker image base * fix: reinstall spacy * fix: resentseg tests * fix: codestyle * fix: docs * fix: add sentseg to tests * fix: dockerfile * fix: model path * fix: add dialogrpt o wait hosts * fix: more complicated test for badwords annotator * Fix/upd badlisted words (#130) * fix: more complicated test for badwords annotator * fix: revert badlisted en words * fix: russian badlisted words * fix: give tokenized sents after spacy * fix: ru badlisted words * fix: ru badlisted words folder * fix: ru badlisted words get data * fix: test file * fix: ru badlisted words tokenized sent * fix: ru badlisted words tokens * fix: codestyle * fix: revert badlisted to dev * fix: pipeline conf post_skill_selector_annotators * fix: sleep before re try to connect to dialogpt * fix: formatter format * fix: more russian badwords * fix: correct endpoint for spacy annotator * Feat/ru random questions (#131) * feat: random russian questions * feat: dummy provides random russian questions * fix: refactor questions * fix: add pre-dummy phrase * fix: add pre-dummy phrase * fix: codestyle * fix: path to file * fix: strip russian questions * fix: last chance response * fix: documentation * fix: more confident generative skill * fix: dummy response always available * fix: intent responder check if exist * fix: most dummy responses language based * fix: remove punctuation if present * fix: documentation * fix: documentation * fix: new limits for russian baseline * fix: dialogrpt scores as conveval * fix: sentseg ru remove commas * fix: no wiki-skill yet * fix: ner no threads * fix: can add prompt * fix: prompt with conf * fix: remove bad questions * fix: add punctuation to generated hyp * fix: remove quotes * fix: re-choose hyp only for en version * fix: dff-generative is aka script * fix: increase intent conf thresholds * fix: store only tokens for hyps * fix: consider only special intents * fix: codestyle * fix: final fixes, resp selectiorna …

* feat: transcript id instead of random for meeting files in teams * remove dialog id

dilyararimovna added 19 commits May 24, 2022 12:03

fix: itnent catcher ru transformers

3d246d9

fix: ru itnent catcher

f337afa

fix: intent catcher updated

e209bfb

Merge branch 'fix/intent_catcher' into feat/ru_intent_catcher_transfo…

6c335a6

…rmers

fix: INTENT_PHRASES_PATH as a main variable

758e509

Merge remote-tracking branch 'origin/fix/intent_catcher' into feat/ru…

069abbb

…_intent_catcher_transformers

fix: dockerfile updates

0c81a7c

Merge remote-tracking branch 'origin/fix/intent_catcher' into feat/ru…

0261084

…_intent_catcher_transformers

Merge remote-tracking branch 'origin/fix/intent_catcher' into feat/ru…

fd0dc3e

…_intent_catcher_transformers

Merge remote-tracking branch 'origin/fix/intent_catcher' into feat/ru…

d00ee0f

…_intent_catcher_transformers

Merge branch 'fix/intent_catcher' into feat/ru_intent_catcher_transfo…

4a583f9

…rmers

fix: test gpu

6263445

fix: black style

e04f0d5

fix: add tests files

a50fb6a

fix: tests

559b84a

fix: rights on file

9490462

fix: regiths on file

4a950b6

fix: regiths

b3fa1e5

fix: numb hyps

c034a1b

dilyararimovna requested review from kudep and mtalimanchuk June 26, 2022 13:30

mtalimanchuk reviewed Jun 27, 2022

View reviewed changes

assistant_dists/dream_russian/docker-compose.override.yml Outdated Show resolved Hide resolved

dilyararimovna added 2 commits June 27, 2022 12:57

fix: remove without threads

fe4e81f

Merge remote-tracking branch 'origin/feat/ru_intent_catcher_transform…

3ebc5b0

…ers' into feat/ru_intent_catcher_transformers

mtalimanchuk approved these changes Jun 27, 2022

View reviewed changes

dilyararimovna merged commit ae48eb4 into fix/intent_catcher Jun 27, 2022

dilyararimovna deleted the feat/ru_intent_catcher_transformers branch June 27, 2022 11:14

IgnatovFedor pushed a commit that referenced this pull request Jan 24, 2024

feat: transcript id instead of random for meeting files in teams (#171)

62fb573

* feat: transcript id instead of random for meeting files in teams * remove dialog id

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/ru intent catcher transformers #171

Feat/ru intent catcher transformers #171

dilyararimovna commented Jun 16, 2022

Feat/ru intent catcher transformers #171

Feat/ru intent catcher transformers #171

Conversation

dilyararimovna commented Jun 16, 2022