Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/transformer -- moving to the example to the main library #269

Merged
merged 10 commits into from Jan 15, 2020

Conversation

@tscholak
Copy link
Collaborator

tscholak commented Jan 10, 2020

just moving this from the examples to Torch.Typed.NN and making sure everything is up-to-date.

:set prompt "λ> "
:seti -XTypeApplications
:seti -XDataKinds
:seti -XScopedTypeVariables
:seti -XFlexibleContexts
import Torch.HList
import Torch.Typed.NN.Transformer
import Torch.Typed.Tensor
import Torch.Typed.Parameter
import Torch.Typed.NN
import Torch.Typed.Functional
import Torch.Typed.Factories
import qualified Torch.DType as D
import qualified Torch.Device as D
import qualified Torch.NN as A
spec = TransformerLMSpec @2 @3 @10 @0 @16 @32 @128 @'D.Float @'( 'D.CPU, 0) (DropoutSpec 0.2) (TransformerLMLayerSpec (MultiheadAttentionSpec (DropoutSpec 0.2)) (DropoutSpec 0.2) 0.001 (TransformerLMMLPSpec (DropoutSpec 0.2) (DropoutSpec 0.2) (Activation Torch.Typed.Functional.relu) (Activation Torch.Typed.Functional.relu)))
model <- A.sample spec
:t flattenParameters model
flattenParameters model

I also cleanup up Torch.HList and several other modules. There are also some static tests now for the Torch.Typed.Parameter.Parameterized instances.

@tscholak tscholak force-pushed the tscholak:feature/transformer branch from abe97cf to 32b2ffd Jan 10, 2020
@tscholak tscholak changed the title feature/transformer (wip) feature/transformer -- moving to the example to the main library (wip) Jan 10, 2020
tscholak added 8 commits Jan 10, 2020
…itialization, and learned with custom initialization
@tscholak tscholak marked this pull request as ready for review Jan 14, 2020
@tscholak tscholak requested review from junjihashimoto and austinvhuang Jan 14, 2020
@tscholak tscholak changed the title feature/transformer -- moving to the example to the main library (wip) feature/transformer -- moving to the example to the main library Jan 14, 2020
Copy link
Member

austinvhuang left a comment

LGTM! This is great @tscholak !

Would be nice to have a small NLP/vision example to try this out on but that doesn't have to be part of this PR to go ahead and merge.

@tscholak tscholak merged commit f49abcd into hasktorch:master Jan 15, 2020
5 checks passed
5 checks passed
build
Details
build
Details
build
Details
build
Details
build
Details
@tscholak tscholak deleted the tscholak:feature/transformer branch Jan 15, 2020
@tscholak

This comment has been minimized.

Copy link
Collaborator Author

tscholak commented Jan 15, 2020

@austinvhuang I'm working towards completion for the static-transformer example. when done, we will be able to train a gpt-2 model on large datasets like wikipedia on multiple gpus. stay tuned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.