This is compilation of the open-source code that was produced as part of my PhD:
| Repository | Publication | Short Description |
|---|---|---|
| helboukkouri/acl_srw_2019 | ACL SRW 2019 [paper] | Better in-domain word representations by leveraging a (small) target task corpus along with pre-trained embeddings from the general domain. |
| helboukkouri/recital_2020 | RECITAL 2020 [paper] | Comparing in-domain BERT (Devlin et al., 2019) models trained from scratch with a medical WordPiece (Wu et al., 2016) vocabulary vs. versions re-trained from general-domain checkpoints with a vocabulary from the general domain. |
| helboukkouri/character-bert | COLING 2020 [paper] | Variant of BERT (Devlin et al., 2019) that consults a word's characters to produce word-level open-vocabulary representations. This model is shown to improve over original BERT in both performance (medical domain) and robustness to misspellings. |
| helboukkouri/character-bert-pretraining | - | Code for pre-training BERT (Devlin et al., 2019) and CharacterBERT (El Boukkouri et al., 2020) models. |
| helboukkouri/mesh-embeddings | - | Pre-trained concept representations for MeSH (a medical knowledge base) built using the node2vec algorithm (Grover and Leskovec, 2016). |
| [Anonymous 🥷🏼] | [Under Submission 🔨👷🏻♂️] | Code for enhancing general-domain representations with knowledge base information. |
Moreover, there is an ongoing pull request for integrating CharacterBERT into the transformers library.