Skip to content

Flexible Python library providing building blocks (layers) for reproducible Transformers research (Tensorflow βœ…, Pytorch πŸ”œ, and Jax πŸ”œ)

License

Notifications You must be signed in to change notification settings

tensorops/TransformerX

Repository files navigation

TransformerX


TransformerX is a Python library for building transformer-based models.

PyPI - Python Version PyPI - Implementation GitHub last commit PyPI - Maintenance PyPI - License PyPI - Format Status Commits Commits

It comes with multiple building blocks and layers you need for creating your model.


Here's why:

  • Your time should be focused on creating more advanced and complex tasks
  • You shouldn't be doing the same tasks over and over

Installation

Install the latest version of TransformerX:

$ pip install transformerx

Getting Started

This example implements a French to English translation model.

Check out these blog posts on Medium:

Note: The data_loader and training modules are still under development and you may want to use your own training and input pipeline. However, the layers package is the core component and will remain the same (you can integrate it with Tensorflow already πŸ”œ Pytorch and JAX).

from transformerx.data_loader import BaseDataset
from transformerx.training import Transformer, Trainer
from transformerx.layers import TransformerEncoder, TransformerDecoder

depth, n_blocks, dropout = 256, 2, 0.2
ffn_num_hiddens, num_heads = 64, 4
key_size, query_size, value_size = 256, 256, 256

data = BaseDataset(batch_size=128)
norm_shape = [2]
encoder = TransformerEncoder(
    len(data.src_vocab),
    depth,
    norm_shape,
    ffn_num_hiddens,
    num_heads,
    n_blocks,
    dropout,
)
decoder = TransformerDecoder(
    len(data.tgt_vocab),
    depth,
    norm_shape,
    ffn_num_hiddens,
    num_heads,
    n_blocks,
    dropout,
)
model = Transformer(encoder, decoder, tgt_pad=data.tgt_vocab["<pad>"], lr=0.001)
trainer = Trainer(max_epochs=2, gradient_clip_val=1)
trainer.fit(model, data)

Features

  • Support CPU/GPU
  • Vectorized operations
  • Standard API

Roadmap

Contribution

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

In case you want to get ideas or just work on a ready-to-solve issue, please check out issues with the label issue list. Here is a list of issue lists

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement" or "bug".

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

If you find this project helpful or valuable, please consider giving it a star ⭐️

Your support helps us to reach a wider audience and improve the project for everyone. Thank you for your support!

Contributors

License

Released under the Apache 2.0 license -> To be changed to MIT license after the first stable release

Contact

Copyright Β© 2021-2023 TensorOps Developers
Soran Ghaderi (soran.gdr.cs@gmail.com)
follow me on Github Twitter Linkedin

Taleb Zarhesh (taleb.zarhesh@gmail.com) follow me on Github Twitter Linkedin

About

Flexible Python library providing building blocks (layers) for reproducible Transformers research (Tensorflow βœ…, Pytorch πŸ”œ, and Jax πŸ”œ)

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages