{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":648594989,"defaultBranch":"main","name":"language-modelling-with-pytorch","ownerLogin":"shubham0204","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-06-02T10:39:03.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/41076823?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1685702370.6468601","currentOid":""},"activityList":{"items":[{"before":"515e4b59b8fce7f383ce3067f2877d08fe389460","after":"93307dbc164d4bb60601b1e68f3788686586bdb8","ref":"refs/heads/main","pushedAt":"2024-04-04T15:24:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"update README.md","shortMessageHtmlLink":"update README.md"}},{"before":"1b45d26768cbc19be87936334b3e3c1b8e481d76","after":"515e4b59b8fce7f383ce3067f2877d08fe389460","ref":"refs/heads/main","pushedAt":"2024-04-04T14:52:28.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"move Predictor to predict.py from utils.py, format and lint","shortMessageHtmlLink":"move Predictor to predict.py from utils.py, format and lint"}},{"before":"46e09aadef88b1cdabb170da6d83c900c23c8a1c","after":"1b45d26768cbc19be87936334b3e3c1b8e481d76","ref":"refs/heads/main","pushedAt":"2023-07-20T01:46:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"Updated `README.md`","shortMessageHtmlLink":"Updated <code>README.md</code>"}},{"before":"571531fb6bcc7ec20756121fa54fb6ad63f068cc","after":"46e09aadef88b1cdabb170da6d83c900c23c8a1c","ref":"refs/heads/main","pushedAt":"2023-07-11T03:47:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"- Removed `setup_env.sh` ; Added `requirements.txt` instead\n- `utils.py`: Added model initialization in `Predictor`\n- `app.py`: Added `Predictor`\n- `api.py`: Added `Predictor`\n- Updated `README.md`\n\n(All other changes are due to optimization of imports)","shortMessageHtmlLink":"- Removed <code>setup_env.sh</code> ; Added <code>requirements.txt</code> instead"}},{"before":"58523638449185e48209a9850848369721f4900a","after":"571531fb6bcc7ec20756121fa54fb6ad63f068cc","ref":"refs/heads/main","pushedAt":"2023-06-30T03:10:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`process_data.py`: Removed `PorterStemmer` and added regex to remove hashtags.\n\n`train.py`: `config` is now saved in each model with `torch.save`. Only the best model gets saved overtime.\n\n`predict.py`: The model is now constructed with the `config` loaded from `checkpoint`\n\n`utils.py`: Added `beautify_output` method to format model output.","shortMessageHtmlLink":"<code>process_data.py</code>: Removed <code>PorterStemmer</code> and added regex to remove …"}},{"before":"b4f9adfa3aa549a1e8fef49358d4d9a4023b78c3","after":"58523638449185e48209a9850848369721f4900a","ref":"refs/heads/main","pushedAt":"2023-06-25T01:56:52.088Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`process_data.py`: Removed `PorterStemmer` and added regex to remove hashtags.\n\n`train.py`: Added `print` statements for logging.\n\n`predict.py`: Added `print` statements for logging.\n\n`project_config.toml`: Increased `num_training_iter` to `50000`","shortMessageHtmlLink":"<code>process_data.py</code>: Removed <code>PorterStemmer</code> and added regex to remove …"}},{"before":"2b3be3491a42276c994541c0bcf6b0cacc120ff5","after":"b4f9adfa3aa549a1e8fef49358d4d9a4023b78c3","ref":"refs/heads/main","pushedAt":"2023-06-24T12:11:58.818Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`process_data.py`: Added word-stemming. The vocabulary of words now gets sorted to make a unique order, eliminating\nthe randomness caused by unordered `set`","shortMessageHtmlLink":"<code>process_data.py</code>: Added word-stemming. The vocabulary of words now g…"}},{"before":"02f42e2390578ac6bef1121b8ff58ee0f8da5dd6","after":"2b3be3491a42276c994541c0bcf6b0cacc120ff5","ref":"refs/heads/main","pushedAt":"2023-06-22T11:05:51.061Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`predict.py`: It now loads model's `state_dict` from a saved model\n\n`train.py`: Added `model.to(device)` before initializing optimizer state.\n\nSorted imports in the project.","shortMessageHtmlLink":"<code>predict.py</code>: It now loads model's <code>state_dict</code> from a saved model"}},{"before":"c9342cd3e773323cb481f14838a4f8a36a72a027","after":"02f42e2390578ac6bef1121b8ff58ee0f8da5dd6","ref":"refs/heads/main","pushedAt":"2023-06-20T12:00:32.960Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`train.py`: Added `torch.compile` and `torch.save` now saves `state_dict` of `model` and `optimizer`\n\n`project_config.toml`: Added `compile_model`, `resume_training` and `resume_training_checkpoint_path` settings\n\n`process_data.py`: Renamed `token_number` and `token_linebreak`","shortMessageHtmlLink":"<code>train.py</code>: Added <code>torch.compile</code> and <code>torch.save</code> now saves `state_d…"}},{"before":"5208958e47064e7528c4b5a7aea57059eafbedbf","after":"c9342cd3e773323cb481f14838a4f8a36a72a027","ref":"refs/heads/main","pushedAt":"2023-06-20T02:32:32.325Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"Improved sentence break regex in process_data.py","shortMessageHtmlLink":"Improved sentence break regex in process_data.py"}},{"before":"9c625251f0aa44141d1b42a851c668f210bd6e62","after":"5208958e47064e7528c4b5a7aea57059eafbedbf","ref":"refs/heads/main","pushedAt":"2023-06-20T02:10:53.160Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`layers.py`: Replaced `torch.sqrt` with `math.sqrt`\n\n`process_data.py`: Added filtering for numbers, sentence breaks and word-contractions.\n\n`utils.py`: Added `seq_length` as argument to class `Predictor`\n\n`setup_env.sh`: Added `contractions` package for installation.\n\n`predict.py`: Added `config.data.seq_length` to avoid hard-coding sequence length.","shortMessageHtmlLink":"<code>layers.py</code>: Replaced <code>torch.sqrt</code> with <code>math.sqrt</code>"}},{"before":"70ea3cfc33fa3fb8d18a2e0233806cdbb342f396","after":"9c625251f0aa44141d1b42a851c668f210bd6e62","ref":"refs/heads/main","pushedAt":"2023-06-16T13:28:46.996Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"Added sinusoidal positional embedding","shortMessageHtmlLink":"Added sinusoidal positional embedding"}},{"before":"861ccb3397cf5760400b7cc5f7bf4f988b467b26","after":"70ea3cfc33fa3fb8d18a2e0233806cdbb342f396","ref":"refs/heads/main","pushedAt":"2023-06-15T04:04:08.954Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"Updated training pipeline","shortMessageHtmlLink":"Updated training pipeline"}},{"before":"b6674ddfc8152c6ec50884fcf5b0815c3f625749","after":"861ccb3397cf5760400b7cc5f7bf4f988b467b26","ref":"refs/heads/main","pushedAt":"2023-06-12T10:27:49.555Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"Updated training process","shortMessageHtmlLink":"Updated training process"}},{"before":"75cca1ed7829e943aca0fcabca98cee24f30fb9c","after":"b6674ddfc8152c6ec50884fcf5b0815c3f625749","ref":"refs/heads/main","pushedAt":"2023-06-12T04:33:24.838Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"Added new dataset","shortMessageHtmlLink":"Added new dataset"}},{"before":"502a5e951594027c32ac6525e8227728f60d67d9","after":"75cca1ed7829e943aca0fcabca98cee24f30fb9c","ref":"refs/heads/main","pushedAt":"2023-06-12T04:32:14.459Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"Changed dataset","shortMessageHtmlLink":"Changed dataset"}},{"before":"d7c147d0b06fc39d7921a6a250482e383ca3d5ea","after":"502a5e951594027c32ac6525e8227728f60d67d9","ref":"refs/heads/main","pushedAt":"2023-06-12T01:28:00.750Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`loss.py`: Added `perplexity` metric.\n\n`train.py`: Prints `perplexity` instead of accuracy now. Added `LearningRateScheduler`\n\n`process_data.py`: Added `[SEP]`, `[START]` and `[END]` tokens to sequences.\n\n`setup_env.sh`: It now unzips poems from 10 natural-like topics.\n\n`predict.py`: Added open-text generation with `num_tokens` argument.","shortMessageHtmlLink":"<code>loss.py</code>: Added <code>perplexity</code> metric."}},{"before":"5e9fed456b1cee44978df42a4d78130be0a1c706","after":"d7c147d0b06fc39d7921a6a250482e383ca3d5ea","ref":"refs/heads/main","pushedAt":"2023-06-11T11:38:24.091Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`loss.py`: `logits` and `targets` are now reshaped before computing log-softmax\n\n`process_data.py`: `n_gram_sequences` are now shuffled before saving to file.\n\n`utils.py`: Added `temperature` parameter in softmax computation\n\n`predict.py`: Added `temperature` parameter in softmax computation and `map_location` in `torch.load`","shortMessageHtmlLink":"<code>loss.py</code>: <code>logits</code> and <code>targets</code> are now reshaped before computing l…"}},{"before":"027eba407af9ce73f3fdf0efffba773be3f2a489","after":"5e9fed456b1cee44978df42a4d78130be0a1c706","ref":"refs/heads/main","pushedAt":"2023-06-10T13:54:35.859Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`loss.py`: Added `torch.nn.functional.log_softmax`\n\n`process_data.py`: `n_gram_sequences` are now shuffled before saving to file.\n\n`utils.py`: Added `temperature` parameter to `Predictor`. Also, it now\nsamples words from a multinomial distribution derived from softmax-ed predictions.\n\n`predict.py`: Added `temperature` parameter.","shortMessageHtmlLink":"<code>loss.py</code>: Added <code>torch.nn.functional.log_softmax</code>"}},{"before":"36013967c3ef73c3ad67d01f7ed39fd045a1a84c","after":"027eba407af9ce73f3fdf0efffba773be3f2a489","ref":"refs/heads/main","pushedAt":"2023-06-10T07:18:45.888Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`loss.py`: Added `accuracy` function to calculate categorical accuracy.\n\n`process_data.py`: Modified `make_sequences` method to produce\n\n`train.py`: Moved accuracy to `loss.py`. Loss is now calculated for\nall time-steps across a sequence","shortMessageHtmlLink":"<code>loss.py</code>: Added <code>accuracy</code> function to calculate categorical accuracy."}},{"before":"ddae67954658c5ef8a3768aa8907d1c25562cf39","after":"36013967c3ef73c3ad67d01f7ed39fd045a1a84c","ref":"refs/heads/main","pushedAt":"2023-06-09T03:01:01.973Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`layers.py`: Removed bias from `Linear` used in keys, queries and values\nFixed `mean` in `torch.nn.init.normal_` for biases\n\n`process_data.py`: Added `__name__ == \"__main__\"`\n\n`train.py`: Added `AdamW` with learning rate from `train_config`","shortMessageHtmlLink":"<code>layers.py</code>: Removed bias from <code>Linear</code> used in keys, queries and values"}},{"before":"d80a4a67ea6143af78ee2395290348a4e3e372aa","after":"ddae67954658c5ef8a3768aa8907d1c25562cf39","ref":"refs/heads/main","pushedAt":"2023-06-09T01:37:32.954Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`layers.py`: Added mask for self attention module\n\n`train.py`: W&B login is executed only when `train_config.wandb_logging_enabled`","shortMessageHtmlLink":"<code>layers.py</code>: Added mask for self attention module"}},{"before":"9c24f9418fd6f88336c9b3ba62ef36c97f8bdc6d","after":"d80a4a67ea6143af78ee2395290348a4e3e372aa","ref":"refs/heads/main","pushedAt":"2023-06-08T07:45:32.718Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`layers.py`: Added CUDA `device`\n\n`process_data`: Removed `Fire` dependency. Added print statements\nfor showing no. of samples in train/test datasets\n\n`setup_env.sh`: Unzips `forms/sonnet` and `forms/epic` from the dataset\nto increase training samples\n\n`train.py`: Added `tqdm` progressbar for training and testing loops","shortMessageHtmlLink":"<code>layers.py</code>: Added CUDA <code>device</code>"}},{"before":"61b48be33261711405e7936534dd31c84ef97604","after":"9c24f9418fd6f88336c9b3ba62ef36c97f8bdc6d","ref":"refs/heads/main","pushedAt":"2023-06-06T12:14:00.195Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`layers.py`: Added `device` to `PositionalEmbedding` module\n\n`predict.py`: Added CUDA support for inference\n\n`setup_env.sh`: Downloads dependencies from PyPI, checkout GitHub repo","shortMessageHtmlLink":"<code>layers.py</code>: Added <code>device</code> to <code>PositionalEmbedding</code> module"}},{"before":"47b42af1a4d14be6166aa6a37e8f1f3970fa1c34","after":"61b48be33261711405e7936534dd31c84ef97604","ref":"refs/heads/main","pushedAt":"2023-06-06T02:27:29.387Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`download_dataset.sh`: Bash script to download, unzip and\norganize dataset files from Kaggle\n\n`train.py`: Added `device` for GPU computation\n\n`project_config.toml`: Removed `pytorch.device` attribute","shortMessageHtmlLink":"<code>download_dataset.sh</code>: Bash script to download, unzip and"}},{"before":"90c80ed4cf3e2b75ffdb1fa554750bcae033f6f0","after":"47b42af1a4d14be6166aa6a37e8f1f3970fa1c34","ref":"refs/heads/main","pushedAt":"2023-06-03T03:24:59.531Z","pushType":"push","commitsCount":1,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`config.py`: Reads global configuration from\n`project_config.toml`\n\n`process_data.py`: Removed command-line arguments\nand added config-based parameters\n\n`train.py`: Removed command-line arguments\nand added config-based parameters\n\n`predict.py`: Removed command-line arguments\nand added config-based parameters\n\n`utils.py`: `input_seq` gets converted to a `torch.Tensor`\nand again to `list` for appending predicted tokens","shortMessageHtmlLink":"<code>config.py</code>: Reads global configuration from"}},{"before":null,"after":"90c80ed4cf3e2b75ffdb1fa554750bcae033f6f0","ref":"refs/heads/main","pushedAt":"2023-06-02T10:39:30.646Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"shubham0204","name":"Shubham Panchal","path":"/shubham0204","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/41076823?s=80&v=4"},"commit":{"message":"`train.py`: Completed training pipeline, integrated `wandb`\ntracking\n\n`loss.py`: Created new `sparse_crossentropy_with_logits` loss\nfunction\n\n`process_data.py`: Prints vocab size now","shortMessageHtmlLink":"<code>train.py</code>: Completed training pipeline, integrated <code>wandb</code>"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEJ_R6bgA","startCursor":null,"endCursor":null}},"title":"Activity · shubham0204/language-modelling-with-pytorch"}