Skip to content

DistilBERT model pre-trained on 131 GB of Japanese web text. The teacher model is BERT-base that built in-house at LINE.

License

Notifications You must be signed in to change notification settings

line/LINE-DistilBERT-Japanese

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 

Repository files navigation

LINE DistilBERT Japanese

This is a DistilBERT model pre-trained on 131 GB of Japanese web text. The teacher model is BERT-base that built in-house at LINE. The model was trained by LINE Corporation.

https://huggingface.co/line-corporation/line-distilbert-base-japanese

For Japanese

README_ja.md is written in Japanese.

How to use

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("line-corporation/line-distilbert-base-japanese", trust_remote_code=True)
model = AutoModel.from_pretrained("line-corporation/line-distilbert-base-japanese")

sentence = "LINE株式会社で[MASK]の研究・開発をしている。"
print(model(**tokenizer(sentence, return_tensors="pt")))

Requirements

fugashi
sentencepiece
unidic-lite

Model architecture

The model architecture is the DitilBERT base model; 6 layers, 768 dimensions of hidden states, 12 attention heads, 66M parameters.

Evaluation

The evaluation by JGLUE is as follows:

model name #Params Marc_ja JNLI JSTS JSQuAD JCommonSenseQA
acc acc Pearson/Spearman EM/F1 acc
LINE-DistilBERT 68M 95.6 88.9 89.2/85.1 87.3/93.3 76.1
Laboro-DistilBERT 68M 94.7 82.0 87.4/82.7 70.2/87.3 73.2
BandaiNamco-DistilBERT 68M 94.6 81.6 86.8/82.1 80.0/88.0 66.5

Tokenization

The texts are first tokenized by MeCab with the Unidic dictionary and then split into subwords by the SentencePiece algorithm. The vocabulary size is 32768.

Licenses

The pretrained models are distributed under the terms of the Apache License, Version 2.0.

To cite this work

We haven't published any paper on this work. Please cite this GitHub repository:

@article{LINE DistilBERT Japanese,
  title = {LINE DistilBERT Japanese},
  author = {"Koga, Kobayashi and Li, Shengzhe and Nakamachi, Akifumi and Sato, Toshinori"},
  year = {2023},
  howpublished = {\url{http://github.com/line/LINE-DistilBERT-Japanese}}
}

About

DistilBERT model pre-trained on 131 GB of Japanese web text. The teacher model is BERT-base that built in-house at LINE.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published