Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Maintenance] improve agenda police #31

Closed
zqwerty opened this issue Jun 24, 2020 · 2 comments
Closed

[Maintenance] improve agenda police #31

zqwerty opened this issue Jun 24, 2020 · 2 comments
Assignees
@zqwerty

Comments

@zqwerty
Copy link
Member

zqwerty commented Jun 24, 2020

Describe the feature
Improve user simulator, mostly agenda policy

Expected behavior

  1. add dialogue act which indicates the current domain (such as ['Inform', 'Hotel', 'none, 'none']) when the simulator talks about a new domain.
  2. add dontcare response for those system requests that are not in the user goal.
  3. user policy should take ['Internet', 'none'] system act as ['Internet', 'yes'], and ['Parking', 'none'] as ['Parking', 'yes']. The system acts only have 'none' as value for 'Internet' and 'Parking' slots, while user acts have 'yes', 'no', 'none' for these slots.
  4. When the system recommends one or more choices, the user policy could say "ok" or "just randomly pick one".
  5. When the system offers to book but the user does not need to book, the user could say "I 'm not looking to make a booking at the moment."
  6. Improve the order of dialogue acts for templateNLG, such as say the name first, then other slots.

Additional context
Have look a few simulated dialogues.
@zqwerty zqwerty self-assigned this Jun 24, 2020
@zqwerty zqwerty mentioned this issue Jun 29, 2020
Open
zqwerty added a commit to zqwerty/ConvLab-2 that referenced this issue Jul 10, 2020
@zqwerty
improve agenda police thu-coai#31, the order of NLG could be more det…
80d00d6 
…ailed in TemplateNLG:sorted_dialog_act
@zqwerty zqwerty mentioned this issue Jul 10, 2020
Merged
zqwerty added a commit that referenced this issue Jul 10, 2020
@zqwerty
improve agenda police #31, the order of NLG could be more detailed in…
37f2df8 
… TemplateNLG:sorted_dialog_act
zqwerty added a commit that referenced this issue Jul 15, 2020
@zqwerty
Improve agenda policy (#52)
c6372b1 
* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police #31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy
@zqwerty
Copy link
Member Author

zqwerty commented Jul 15, 2020

#52 improve goal sampling strategy. Origin: sample each slot independently. Current: can choose to sample slots independently or sample the slot combination from the train set. Add two arguments:

sample_info_from_trainset: if True, sample info slots combination from train set, else sample each slot independently
sample_reqt_from_trainset: if True, sample reqt slots combination from train set, else sample each slot independently

@zqwerty zqwerty mentioned this issue Jul 16, 2020
Open
zqwerty added a commit that referenced this issue Jul 22, 2020
@zqwerty
Improve agenda policy (#60)
89dc730 
* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police #31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy

* fix self.cur_domain=None when system offer book
zqwerty added a commit that referenced this issue Jul 24, 2020
@zqwerty
Improve agenda policy (#62)
acfef62 
* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police #31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy

* fix self.cur_domain=None when system offer book

* fix agenda for 0 choice
ruleGreen pushed a commit to ruleGreen/ConvLab-2 that referenced this issue Oct 20, 2020
@zqwerty
improve agenda police thu-coai#31, the order of NLG could be more det…
5b2257b 
…ailed in TemplateNLG:sorted_dialog_act
ruleGreen pushed a commit to ruleGreen/ConvLab-2 that referenced this issue Oct 20, 2020
@zqwerty
Improve agenda policy (thu-coai#52)
ffeb01e 
* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police thu-coai#31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy
ruleGreen pushed a commit to ruleGreen/ConvLab-2 that referenced this issue Oct 20, 2020
@zqwerty
Improve agenda policy (thu-coai#60)
a074dc8 
* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police thu-coai#31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy

* fix self.cur_domain=None when system offer book
ruleGreen pushed a commit to ruleGreen/ConvLab-2 that referenced this issue Oct 20, 2020
@zqwerty
Improve agenda policy (thu-coai#62)
13d048e 
* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police thu-coai#31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy

* fix self.cur_domain=None when system offer book

* fix agenda for 0 choice
zqwerty added a commit that referenced this issue Oct 22, 2020
@ruleGreen @zqwerty
@truthless11 @liangrz15 @Mehrad0711 @pengbaolin @jincli @msft-shahins @aaa123git @brunonishimoto @vojtsek @zz-jacob @Jiaxin-Wen @zimozhou
Add warmup for DQN and fix minor bugs (#150)
e368dee 
* Initial commit

* Update README.md

* sync with commit aa1af0ee81ba591d1cf3c222c9d71963ed1dca98

* add gitignore

* update tutorial

* update mdrg, not use dbPointer

* update mdrg, not use dbPointer

* update mdrg, download before use dbPointer

* update analyzer

* update README

* update tutorial

* Fix dbquery when matching name

* move change to dev branch

* move dbquery change from master to dev branch

* disable travis for now

* do nothing in travis

* do nothing in travis for now

* not deploy now

* add docs

* update .travis.yml

* update .travis.yml

* update .travis.yml

* update rst files

* add alias center for centre in dbquery

* fix the policy training bug

* fix the bug of nan gradient

* add cross-lingual dst data

* Update README.md

* Update README.md

* Add CrossWoz Web support and some minor bug fix (#19)

* Initial commit

* first commit

* add build

* add build

* add build

* add recommend

* add crosswoz config in deploy

* add crosswoz at html

* debug chinese vision

* fix system bug according to convlab2

* master change

* modify .gitignore

* delete svm_camrest_usr.pickle

Co-authored-by: kflab_2018 <kflab_2018@kflab-2018s-MacBook-Air.local>
Co-authored-by: CarreyWong <carreywong@CarreyWongs-MacBook-Pro.local>

* modify xdst data name

* Translation train on MultiWOZ (Chinese) nad CrossWOZ (English) of SUMBT (#17)

* multiwoz_zh

* crosswoz_en

* translation train

* test translation train

* update evaluation code

* update evaluation code for crosswoz

* evaluate human val set

* update readme

* evaluate machine val

* extract all ontology, bad result

* update evalutate

* update evalutation result on crosswoz-en

* updata xdst baseline

* Update README.md

* fix allennlp==0.9.0

* Update README.md

* modify build message function for goal generation

* Fix goal generator and dbquery for multiwoz (#32)

* move dbquery change from master to dev branch

* add alias center for centre in dbquery

* replace attraction type 'mutliple sports' to 'multiple sports', involving only one entity

* add depart and destination constraints for searching db (ignore=False), modify goal generator to draw the values of these two slots from database

* fix bug (#35)

* multiwoz_zh

* crosswoz_en

* translation train

* test translation train

* update evaluation code

* update evaluation code for crosswoz

* evaluate human val set

* update readme

* evaluate machine val

* extract all ontology, bad result

* update evalutate

* update evalutation result on crosswoz-en

* fix bug #34

* revert changes

* update demo video link

* Update README.md

* some changes in #36 (#37)

* multiwoz_zh

* crosswoz_en

* translation train

* test translation train

* update evaluation code

* update evaluation code for crosswoz

* evaluate human val set

* update readme

* evaluate machine val

* extract all ontology, bad result

* update evalutate

* update evalutation result on crosswoz-en

* fix bug #34

* revert changes

* revert changes

* some changes of #36

* fix analyzer example.py

* dst/evaluate.py: Use utf-8 encoding

* use transformers library to automate model caching

* Update README.md

* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police #31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* fix nlu max len

* update travis

* Update run_agent.py

* Create README.md

* Update README.md

* modify human_eval README

* fix sclstm crosswoz import issues

* update travis.yml

* try to fix deploy

* Update README.md

* Update README.md

* Improve agenda policy (#52)

* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police #31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy

* Update README.md

#53

* Update README.md (#57)

* Improve agenda policy (#60)

* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police #31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy

* fix self.cur_domain=None when system offer book

* Improve agenda policy (#62)

* cut sentences that exceed 512 tokens in jointBERT

* Notice: The results are for commits before bdc9dba (inclusive). We will update the results after improving user policy.

* improve agenda police #31, the order of NLG could be more detailed in TemplateNLG:sorted_dialog_act

* improve goal sample strategy

* fix self.cur_domain=None when system offer book

* fix agenda for 0 choice

* fix sequicityy

* fix sequicityy

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* update setup.py:add tokenizers requirement

* fix typo

* update user nlg template

* Update README.md

* remove fail book in multiwoz goal generator

* fix taxi dontcare problem

* can manually set user goal in agenda now

* test goal overlap between generator and trainset

* change default taxi depart and destination from address to name/'the hotel/restaurant'

* change initiative from 4 to randint(2,4)

* agenda pop more da when only answer dontcare

* add 'the same area/pricerange/people/day' in agenda with 0.3 probability

* remove unnecessary thank you

* add domain for postcode and Phone in user templateNLG

* add **kwargs in init_session for self-defined goal; remove request for nooffer-slot in rule-sys-policy

* add template for interent-no, parking-no in templatenlg

* update Evaluator: check whether final goal satisfies constraints

* update evaluator: check booked entity

* output goal analysis to file

* update goal analysis

* update

* Update analyzer.py

* Fix simulator (#83)

* remove fail book in multiwoz goal generator

* fix taxi dontcare problem

* can manually set user goal in agenda now

* test goal overlap between generator and trainset

* change default taxi depart and destination from address to name/'the hotel/restaurant'

* change initiative from 4 to randint(2,4)

* agenda pop more da when only answer dontcare

* add 'the same area/pricerange/people/day' in agenda with 0.3 probability

* remove unnecessary thank you

* add domain for postcode and Phone in user templateNLG

* add **kwargs in init_session for self-defined goal; remove request for nooffer-slot in rule-sys-policy

* add template for interent-no, parking-no in templatenlg

* remove police and hospital domain in goal generator

* update multiwoz evaluator: adding 'internet/parking-none, 24:** to valid value

* fix nlg template (#88)

* add new_goal_model without police and hospital domain (#89)

* Normalize string comparisons in multiwoz template nlg to be case insensitive (#87)

* normalize template nlg keys to be lower case

* fix slot comparison in multiwoz nlg to be case insensitive

* use value_lower instead of calling .lower() on each comparison

* Add police n hospital (#95)

* add back police and hospital goal

* update police db:add postcode; update hospital db:add address and postcode; update dbquery: query hospital with department, deepcopy query result

* update dbquery and session (#99)

* update dbquery: ? matches all; fix bug in init_session

* update multiwoz_eval, check Ref of booked

* filter domain in final_goal_analyze

Co-authored-by: newRuntieException <wdz15@mails.tsinghua.edu.cn>

* Add dockerfile (#98)

* fix nlg template

* add dockerfile

* include missing packages at setup.py (#102)

* multiwoz dbquery doesnt require mutable constraints (#106)

* Add police n hospital (#107)

* add back police and hospital goal

* update police db:add postcode; update hospital db:add address and postcode; update dbquery: query hospital with department, deepcopy query result

* update user templatenlg

* add test set example for dstc9 (multiwoz_zh, crosswoz_en) (#108)

* Add dockerfile (#110)

* fix nlg template

* add dockerfile

* add package for dockerfile

* update versions

* Update README.md

* Update versions in setup (#111)

* move dbquery change from master to dev branch

* add alias center for centre in dbquery

* fix sequicityy

* update versions

Co-authored-by: zqwerty <zhuq96@hotmail.com>
Co-authored-by: zhuqi <zqwerty@users.noreply.github.com>

* Update README.md

* Update README.md

* Update README.md

* fix system nlg template bug (#117)

* add 'book' in DST evaluation. (#85)

* Maintenance (#119)

* add test set example for dstc9 (multiwoz_zh, crosswoz_en)

* update new_goal_model.pkl

* update crosswoz auto_sys_template_nlg

* add postcode as special case for NLU tokenization

* dstc9 eval

* dstc9 xldst evaluation

* Nlg template fix (#121)

* fix nlg template

* fix user nlg template issue

* modify example

* add .gitignore

* remove precision, recall, f1

* release 250 test data

* dstc9 xldst evaluation (#122)

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* dstc9 eval

* dstc9 xldst evaluation

* modify example

* add .gitignore

* remove precision, recall, f1

* release 250 test data

* revise evaluation

* fix file submission example

* revise xldst evaluation (#124)

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* dstc9 eval

* dstc9 xldst evaluation

* modify example

* add .gitignore

* remove precision, recall, f1

* release 250 test data

* revise evaluation

* fix file submission example

* Update dst.py

* update precision, recall, f1 calculation

* minor change

* fix policy evaluation

* Nlg template fix (#127)

* fix nlg template

* fix user nlg template issue

* fix system NLG template

* nlu update and bugfix (#118)

* jointBERT_new avaliable && fix milu dataset_reader && fix jointBERT/tag2id

* remove jointBERT_new

* update milu/multiwoz/nlu.py model_file path

* add metrics in XLDST evaluation (#126)

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* dstc9 eval

* dstc9 xldst evaluation

* modify example

* add .gitignore

* remove precision, recall, f1

* release 250 test data

* revise evaluation

* fix file submission example

* update precision, recall, f1 calculation

* minor change

* add input reqt vals in human eval (#128)

* Maintenance (#129)

* add test set example for dstc9 (multiwoz_zh, crosswoz_en)

* update new_goal_model.pkl

* update crosswoz auto_sys_template_nlg

* add postcode as special case for NLU tokenization

* fix lower case for int value in nlg.py

* Human (#131)

* change task config

* add final goal logging

* encapsule PipelineAgent internal state interface for return and
replacement

* Maintenance (#132)

* add test set example for dstc9 (multiwoz_zh, crosswoz_en)

* update new_goal_model.pkl

* update crosswoz auto_sys_template_nlg

* add postcode as special case for NLU tokenization

* fix lower case for int value in nlg.py

* fix empty user utterance problem in multiwoz simulator, issue #130

* remove debug output

* fix a database typo

* Maintenance (#134)

* add test set example for dstc9 (multiwoz_zh, crosswoz_en)

* update new_goal_model.pkl

* update crosswoz auto_sys_template_nlg

* add postcode as special case for NLU tokenization

* fix lower case for int value in nlg.py

* fix empty user utterance problem in multiwoz simulator, issue #130

* remove debug output

* fix goal generator for police domain message

* fix a minor typo in crosswoz database (#133)

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* dstc9 eval

* dstc9 xldst evaluation

* modify example

* add .gitignore

* remove precision, recall, f1

* release 250 test data

* revise evaluation

* fix file submission example

* update precision, recall, f1 calculation

* minor change

* fix a database typo

* use selectedResults for missing name

* remove low performance baselines (#136)

* Human2 (#137)

* change task config

* add final goal logging

* encapsule PipelineAgent internal state interface for return and
replacement

* fix bug associted with the issue of strange user input

* Fix a bug in TRADE CrossWOZ training (#138)

* add 'book' in DST evaluation.

* Fix TRADE crosswoz training evaluation bug

Co-authored-by: zheng <zheng@zhangzheng-PC.lan>

* Maintenance (#140)

* add test set example for dstc9 (multiwoz_zh, crosswoz_en)

* update new_goal_model.pkl

* update crosswoz auto_sys_template_nlg

* add postcode as special case for NLU tokenization

* fix lower case for int value in nlg.py

* fix empty user utterance problem in multiwoz simulator, issue #130

* remove debug output

* fix goal generator for police domain message

* update template NLG

* Add note for deploy web service (#139)

* add 'book' in DST evaluation.

* Fix TRADE crosswoz training evaluation bug

* Add note on deploy

Co-authored-by: zheng <zheng@zhangzheng-PC.lan>

* add value unification

* fix XLDST evaluation (#141)

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* dstc9 eval

* dstc9 xldst evaluation

* modify example

* add .gitignore

* remove precision, recall, f1

* release 250 test data

* revise evaluation

* fix file submission example

* update precision, recall, f1 calculation

* minor change

* fix a database typo

* use selectedResults for missing name

* add value unification

* fix user Nlg template (#142)

* fix system nlg template bug

* fix user nlg issue

* fix white character issue #144

* deal with white charater in XLDST evaluation (#145)

* update sumbt translation train result with evaluation mode set

* update extract values

* automatically download sumbt model

* dstc9 eval

* dstc9 xldst evaluation

* modify example

* add .gitignore

* remove precision, recall, f1

* release 250 test data

* revise evaluation

* fix file submission example

* update precision, recall, f1 calculation

* minor change

* fix a database typo

* use selectedResults for missing name

* add value unification

* fix white character issue #144

* DQN (#113)

* implemented script to extract all the statistics for all dialogue_act in data

* changed script for actions be compatible to sys_da_voc.txt actions

* multiwoz vector now supports composite actions

* implemented ReplayMemory and EpsilongGreedyPolicy

* implemented a basic version of dqn

* included some comments

* Add DQN Test and Change file structure  (#146)

* Initial commit

* first commit

* add build

* add build

* add build

* add recommend

* add crosswoz config in deploy

* add crosswoz at html

* debug chinese vision

* fix system bug according to convlab2

* master change

* modify .gitignore

* delete svm_camrest_usr.pickle

* Update server.py

* add test for DQN

* change server

Co-authored-by: Carrey Wang <cwhongru@cuc.edu.cn>
Co-authored-by: kflab_2018 <kflab_2018@kflab-2018s-MacBook-Air.local>
Co-authored-by: CarreyWong <carreywong@CarreyWongs-MacBook-Pro.local>
Co-authored-by: zimozhou <47972969+zimozhou@users.noreply.github.com>
Co-authored-by: MR. WANG <hrwang@kfsrv03.se.cuhk.edu.hk>

* update eval

* dump dst eval results

* make value lower

* add progress bar

* fix bug in last commit

* Update policy_agenda_multiwoz.py

* remove unnecessary mapping (#147)

* udpate dstc9 eval

* make value lower

* add warm up for dqn and fix bugs

* rm unrelated files

Co-authored-by: zhuqi <zqwerty@users.noreply.github.com>
Co-authored-by: zqwerty <zhuq96@hotmail.com>
Co-authored-by: Ryuichi Takanobu <truthless11@gmail.com>
Co-authored-by: newRuntieException <wdz15@mails.tsinghua.edu.cn>
Co-authored-by: liangrz <liangrz15@mails.tsinghua.edu.cn>
Co-authored-by: Carrey Wang <cwhongru@cuc.edu.cn>
Co-authored-by: kflab_2018 <kflab_2018@kflab-2018s-MacBook-Air.local>
Co-authored-by: CarreyWong <carreywong@CarreyWongs-MacBook-Pro.local>
Co-authored-by: 罗崚骁 <function2@qq.com>
Co-authored-by: mehrad <mehrad@stanford.edu>
Co-authored-by: pengbaolin <39398162+pengbaolin@users.noreply.github.com>
Co-authored-by: Jinchao Li <38700695+jincli@users.noreply.github.com>
Co-authored-by: Shahin Shayandeh <shahins@microsoft.com>
Co-authored-by: aaa123git <43716234+aaa123git@users.noreply.github.com>
Co-authored-by: Bruno Eidi Nishimoto <bruno_nishimoto@hotmail.com>
Co-authored-by: Vojtěch Hudeček <vojta.hudecek@gmail.com>
Co-authored-by: zhangzthu <zhangz.goal@gmail.com>
Co-authored-by: xw <48146603+xwwwwww@users.noreply.github.com>
Co-authored-by: zheng <zheng@zhangzheng-PC.lan>
Co-authored-by: zimozhou <47972969+zimozhou@users.noreply.github.com>
Co-authored-by: MR. WANG <hrwang@kfsrv03.se.cuhk.edu.hk>
@zqwerty
Copy link
Member Author

zqwerty commented Nov 21, 2020

agenda policy for dstc9 evaluation & 1.0 version release

@zqwerty zqwerty closed this as completed Nov 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zqwerty zqwerty

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@zqwerty