Skip to content
This repository has been archived by the owner on Jan 15, 2024. It is now read-only.

sequence sampling #272

Merged
merged 7 commits into from
Aug 16, 2018
Merged

sequence sampling #272

merged 7 commits into from
Aug 16, 2018

Conversation

hhexiy
Copy link
Contributor

@hhexiy hhexiy commented Aug 13, 2018

Description

sampling feature for beam search.

Checklist

Essentials

  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

Changes

  • added _SamplingStepUpdate to sample from each beam
  • updated scripts/beam_search/beam_search_generator.py to include sampling option and changed LMDecoder to give the sampler raw output scores

@hhexiy hhexiy requested a review from szha as a code owner August 13, 2018 04:15
@szha szha requested a review from sxjscience August 13, 2018 04:17
@mli
Copy link
Member

mli commented Aug 15, 2018

Job PR-272/3 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/3/index.html

@codecov
Copy link

codecov bot commented Aug 15, 2018 •
edited
Loading

Codecov Report

Merging #272 into master will increase coverage by 0.03%.
The diff coverage is 91.66%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #272      +/-   ##
=========================================
+ Coverage   76.66%   76.7%   +0.03%     
=========================================
  Files          74      74              
  Lines        6429    6443      +14     
  Branches     1004    1008       +4     
=========================================
+ Hits         4929    4942      +13     
+ Misses       1266    1264       -2     
- Partials      234     237       +3
Impacted Files Coverage Δ
gluonnlp/model/__init__.py 89.47% <100%> (ø) ⬆️
gluonnlp/model/sequence_sampler.py 86.41% <91.35%> (ø)
gluonnlp/model/sampled_block.py 87% <0%> (-0.5%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c6fbb9...6191ade. Read the comment docs.

sxjscience
sxjscience reviewed Aug 15, 2018
View reviewed changes
gluonnlp/model/sequence_sampler.py Outdated
chosen_word_ids = F.sample_multinomial(smoothed_probs).astype('float32')
chosen_word_log_probs = log_probs[mx.nd.arange(log_probs.shape[0]),
chosen_word_ids.reshape(-1)] \
.reshape_like(chosen_word_ids)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd better use int32 for indexing. Float32 may overflow if there are a large number of words.

sxjscience
sxjscience reviewed Aug 15, 2018
View reviewed changes
gluonnlp/model/sequence_sampler.py
new_valid_length = valid_length + beam_alive_mask

# Update the samples and vaild_length
new_samples = F.concat(samples, chosen_word_ids.expand_dims(2), dim=2)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can append -1 instead of the chosen words if the beam is not alive.

@mli
Copy link
Member

mli commented Aug 15, 2018

Job PR-272/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/4/index.html

@mli
Copy link
Member

mli commented Aug 15, 2018

Job PR-272/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/5/index.html

@mli
Copy link
Member

mli commented Aug 15, 2018

Job PR-272/6 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/6/index.html

@mli
Copy link
Member

mli commented Aug 15, 2018

Job PR-272/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/7/index.html

@mli
Copy link
Member

mli commented Aug 16, 2018

Job PR-272/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/8/index.html

@mli
Copy link
Member

mli commented Aug 16, 2018

Job PR-272/9 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/9/index.html

@mli
Copy link
Member

mli commented Aug 16, 2018

Job PR-272/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-272/10/index.html

@szha szha merged commit 022929a into dmlc:master Aug 16, 2018
@szha szha mentioned this pull request Aug 17, 2018
Open
paperplanet pushed a commit to paperplanet/gluon-nlp that referenced this pull request Jun 9, 2019
@hhexiy @szha
sequence sampling (dmlc#272)
d4d5fe4 
* beam search sampling

* refactor

* move tests

* update readme

* use int32

* append -1 for dead beams

* add sampling test
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@szha szha szha approved these changes

@sxjscience sxjscience sxjscience approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hhexiy @mli @szha @sxjscience