Apply Transformer model for the word language problem in pytorch/examples #555

zhangguanheng66 · 2019-05-03T18:36:57Z

Apply recently developed PyTorch Transformer model for the word language problem. Any reviews and suggestions are appreciated. Please don't land this PR until the transformer model is fully supported by PyTorch. @cpuhrsch @soumith

Encoder/decoder embeddings normalized by sqrt(d_model). test loss 3.84 lr=5.0 Encoder/decoder embeddings normalized by sqrt(d_model). test loss 4.68 lr=20.0 Remove print out. Revise main.py file. Load the best training model through epochs. Update README.md file to include the transformer model. Update the README.md file. Use PositionalEncoding in transformer. test loss 0.30 lr=5.0

Update generate.py to generate text with transformer.pt model. Add CUDA function to generate.py when running transformer model. Add generate_subsequent_mask() in Transformer Generate transformer model in main.py. Revise generate.py working for both RNN and Transformer models. Remove decoder_data Add some changes because of transformer.py.

Change d_ff to dim_feedforward. Remove Embeddings and PositionalEncoder out of transformer.py.

zhangguanheng66 · 2019-05-29T17:49:53Z

A few minor updates. Not using superclass for the transformer model. Check transformer module exists before using. @cpuhrsch

word_language_model/README.md

cpuhrsch · 2019-05-30T17:56:37Z

word_language_model/generate.py

+        with torch.no_grad():  # no tracking history
+            for i in range(args.words):
+                seq_mask = model.generate_square_subsequent_mask(input.size(0)).to(device)
+                output = model(input, input, src_mask=seq_mask, tgt_mask=seq_mask)


Since seq_mask is generated only to be fed into the model, maybe we can do this within the model forward?

Generate seq_mask and tgt_mask is doable inside the forward function. Currently those are generated at the beginning of each epoch so they are re-used through the time steps. I thought this saves efforts.

word_language_model/generate.py

cpuhrsch · 2019-05-30T17:58:33Z

word_language_model/main.py

+    try:
+        from torch.nn import Transformer
+    except:
+        raise ImportError('Transformer module exists in PyTorch 1.1 and above.')


I think it might be better to say "Transformer module does not exist in PyTorch 1.1 or lower".

Move this into model.py

cpuhrsch · 2019-05-30T17:59:42Z

word_language_model/main.py


 ntokens = len(corpus.dictionary)
-model = model.RNNModel(args.model, ntokens, args.emsize, args.nhid, args.nlayers, args.dropout, args.tied).to(device)
+if args.model == 'Transformer':


Can we do this within Model? That is, you extend RNNModel to contain both and pass a flag to decide which one to use. This is in line with RNN settings GRU, LSTM, etc.

I was thinking about merging TransformerSeq2Seq and RNNModel, but realized they are very different in the architecture.

Let's talk about this in person.

cpuhrsch · 2019-05-30T18:00:12Z

word_language_model/main.py

 ntokens = len(corpus.dictionary)
-model = model.RNNModel(args.model, ntokens, args.emsize, args.nhid, args.nlayers, args.dropout, args.tied).to(device)
+if args.model == 'Transformer':
+	model = model.TransformerSeq2Seq(ntokens, ntokens, nhead=args.transformer_head, 


RNNModel isn't using keyword arguments to pass ntokens, dropout etc. - We could be consistent so it's easier to read.

cpuhrsch · 2019-05-30T18:01:18Z

word_language_model/main.py

+
+            if args.model == 'Transformer':
+                if args.bptt > len(data_source) - 1 - i:
+                    tgt_mask = model.transformer.generate_square_subsequent_mask(len(data_source) - 1 - i).to(device)


Is there any way to do this implicitly within model and cache it via say self.tgt_mask?

Should this be len(data) instead of len(data_source)?

When we initiate the transformer model, we have no idea the length of tgt and src, unless we pass it through as an argument. The masks will change depending on the incoming tgt and src sequence. So it's better to generate them at the runtime.

No. This is for the case when the data left don't have a full length of args.bptt.

cpuhrsch · 2019-05-30T18:01:39Z

word_language_model/main.py

+
+        if args.model == 'Transformer':
+            if args.bptt > len(train_data) - 1 - i:
+                tgt_mask = model.transformer.generate_square_subsequent_mask(len(train_data) - 1 - i).to(device)


word_language_model/main.py

word_language_model/model.py

word_language_model/main.py

word_language_model/README.md

word_language_model/main.py

word_language_model/model.py

word_language_model/main.py

word_language_model/model.py

word_language_model/generate.py

cpuhrsch · 2019-06-05T21:51:03Z

word_language_model/generate.py

+                word_weights = output[-1].squeeze().div(args.temperature).exp().cpu()
+                word_idx = torch.multinomial(word_weights, 1)[0]
+                word_tensor = torch.Tensor([[word_idx]]).long().to(device)
+                input = torch.cat([input, word_tensor], 0)


Doesn't this mean that input keeps on growing larger and larger? Is that actually necessary?

Not necessary. We can carry a constant length for the input sequence.

word_language_model/generate.py

word_language_model/model.py

Summary: Accidentally rebased the old PR and make it too messy. Find it here (#19274) Create a PR for comments. The model is still WIP but I want to have some feedbacks before moving too far. The transformer model depends on several modules, like MultiheadAttention (landed). Transformer is implemented based on the paper (https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf). Users have the flexibility to build a transformer with self-defined and/or built-in components (i.e encoder, decoder, encoder_layer, decoder_layer). Users could use Transformer class to build a standard transformer model and modify sub-layers as needed. Add a few unit tests for the transformer module, as follow: TestNN.test_Transformer_cell TestNN.test_transformerencoderlayer TestNN.test_transformerdecoderlayer TestNN.test_transformer_args_check TestScript.test_scriptmodule_transformer_cuda There is another demonstration example for applying transformer module on the word language problem. pytorch/examples#555 Pull Request resolved: #20170 Differential Revision: D15417983 Pulled By: zhangguanheng66 fbshipit-source-id: 7ce771a7e27715acd9a23d60bf44917a90d1d572

…ples (pytorch#555) * Use append to accelerate data loading process. * First transformer model working for word language model. * Work for GPU (all the model and data have to be sent to cuda) * Transformer model GPU activated nhead=1 nlayers=1 d_ff=64 test loss 6.55 * Use lr=5.0 test loss 4.8 Encoder/decoder embeddings normalized by sqrt(d_model). test loss 3.84 lr=5.0 Encoder/decoder embeddings normalized by sqrt(d_model). test loss 4.68 lr=20.0 Remove print out. Revise main.py file. Load the best training model through epochs. Update README.md file to include the transformer model. Update the README.md file. Use PositionalEncoding in transformer. test loss 0.30 lr=5.0 * Update main.py to have mask on source sequences. Update generate.py to generate text with transformer.pt model. Add CUDA function to generate.py when running transformer model. Add generate_subsequent_mask() in Transformer Generate transformer model in main.py. Revise generate.py working for both RNN and Transformer models. Remove decoder_data Add some changes because of transformer.py. * No need to provide Trnasform args for generating text. Change d_ff to dim_feedforward. Remove Embeddings and PositionalEncoder out of transformer.py. * Replace tabs with spaces. * Update transformer model in model.py. * Recycle RNN arguments for Transformer model. * Update README.md file. * Remove model.generator in main.py. * Update the warnings in transformer model. * Fix a small bug in model.py. * Remove keyword arguments for consistence. * Create a new function generate_square_subsequent_mask inside the TransformerSeq2Seq * Remove unnecessary attributes. * A minor change. * Move src_mask and tgt_mask as the members of the module. * Move transformer check to model.py * Move masks inside forward function. * User TransformerEncoder for word language model. * Remove Generator module in Transformer. * Merge RNN and Transformer model in model.py * Minor changes. * Minor changes to address reviewer's comments. * Remove reset_parameter function. * Split RNN and Transformer model to keep code readable. * Minor changes.

Guanheng Zhang added 5 commits April 2, 2019 11:33

Use append to accelerate data loading process.

b7e6360

Merge branch 'data_stack_accelerate'

7aaa2eb

First transformer model working for word language model.

a6caf06

Work for GPU (all the model and data have to be sent to cuda)

506e647

Transformer model GPU activated nhead=1 nlayers=1 d_ff=64 test loss 6.55

ca5f3f2

zhangguanheng66 changed the title ~~[WIP] Apply Transformer model word language~~ [WIP] Apply Transformer model for the word language problem in pytorch/examples May 3, 2019

zhangguanheng66 force-pushed the transformer_word_language_model branch from efb5fd7 to 9332536 Compare May 6, 2019 15:20

Guanheng Zhang added 2 commits May 6, 2019 08:22

No need to provide Trnasform args for generating text.

70ebca2

Change d_ff to dim_feedforward. Remove Embeddings and PositionalEncoder out of transformer.py.

zhangguanheng66 force-pushed the transformer_word_language_model branch from 9332536 to 70ebca2 Compare May 6, 2019 15:23

zhangguanheng66 mentioned this pull request May 6, 2019

nn.Transformer pytorch/pytorch#20170

Closed

Guanheng Zhang added 2 commits May 13, 2019 09:53

Replace tabs with spaces.

7a24278

Update transformer model in model.py.

dac9090

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/README.md Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/generate.py Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/main.py Outdated Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/model.py Outdated Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/model.py Outdated Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/model.py Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/model.py Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/main.py Outdated Show resolved Hide resolved

cpuhrsch reviewed May 30, 2019

View reviewed changes

word_language_model/README.md Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/main.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/main.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/main.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/model.py Outdated Show resolved Hide resolved

Guanheng Zhang added 2 commits June 5, 2019 08:51

Move transformer check to model.py

726ea0e

Move masks inside forward function.

6ffcf6e

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/main.py Outdated Show resolved Hide resolved

Guanheng Zhang added 4 commits June 5, 2019 12:23

User TransformerEncoder for word language model.

ab0204e

Remove Generator module in Transformer.

cf776c3

Merge RNN and Transformer model in model.py

912f5d0

Minor changes.

b3742bd

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/main.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/model.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/model.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/model.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/model.py Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/generate.py Outdated Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/generate.py Show resolved Hide resolved

cpuhrsch reviewed Jun 5, 2019

View reviewed changes

word_language_model/model.py Outdated Show resolved Hide resolved

Guanheng Zhang added 4 commits June 5, 2019 15:48

Minor changes to address reviewer's comments.

f7a1d43

Remove reset_parameter function.

755e111

Split RNN and Transformer model to keep code readable.

4080879

Minor changes.

7c96430

zhangguanheng66 changed the title ~~[WIP] Apply Transformer model for the word language problem in pytorch/examples~~ Apply Transformer model for the word language problem in pytorch/examples Jun 12, 2019

zhangguanheng66 mentioned this pull request Jun 18, 2019

nn.TransformerLayer pytorch/pytorch#21876

Open

soumith merged commit 4581968 into pytorch:master Aug 9, 2019

RikoNyberg mentioned this pull request Aug 15, 2019

Removing unused variable #612

Merged

Apply Transformer model for the word language problem in pytorch/examples #555

Apply Transformer model for the word language problem in pytorch/examples #555

Uh oh!

Conversation

zhangguanheng66 commented May 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangguanheng66 commented May 29, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhangguanheng66 commented May 3, 2019 •

edited

Loading