Implementing wide variety of transformers (improvised version),finetuning as well as trying architectural variants from various research papers and blogs.
- Roberta Pretraining : Basic Implementation , More detail about the model can be found here.
- Implementing Fast Transformers: Failed Attempt, FastTransformer Algorithm can be found here.
- Adaptive Transformers for multimodal Representations: Research paper can be found here.
- Learning Cross-Modality Encoder Representation from transformers: Paper can be found here.
My text generation model trained on Bhagvad_Gita (Holy Hindu Scripture English version) is now available on huggingface and can be found here.
The easiest way to use them is as below:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("epsil/bhagvad_gita")
model = AutoModelForCausalLM.from_pretrained("epsil/bhagvad_gita")
Weak Supervision model build wirh Snorkel and with Finance dataset taken from here. Majority Vote Modelling with some heuristics(rules) and currently work on 3 labelling function. Overall accuracy reached is about 61% and can be enhanced further by dealing with mixed sentence heuristics.
- Finetuning of Sentence Transformer for customized dataset.
- Improvements in Comparison Methods and Advanced Comparison Techniques.
- Performance evaluation of llama-7B Model
- Performance evaluation of Redpajama-3B Model
- Performance evaluation of Bloke Syntara-7B Model