Why use of CNN char embedding?

[cbockman]

"To keep things simple, we use minimal task specific architectures atop BERT-Base and SCIBERT embeddings. Each token is represented as the concatenation of its BERT embedding with a
CNN-based character embedding. If the token has multiple BERT subword units, we use the first one."

Why the use of an additional CNN-based char embeddings? Many (most?) papers using BERT (or similar) solely use the embedding coming out of the LM-based model.

Was there a big additional uptick from layering in the CNN-based char embeddings?

[brendan-ai2]

@ibeltagy , @kyleclo we're getting questions about this issue on AllenNLP's project. Any chance you could follow up? Thanks!

FYI @arunzz

[kyleclo]

Hey @cbockman, your point is well-taken. I'm currently in the process of re-doing the evaluations without these embeddings and there's a minor uptick in the performance due to these. Since BERT-Base was also evaluated using char embeddings, the relative difference between BERT-Base and SciBERT hasn't changed. For example, BC5CDR sees a performance change from 88.94 -> 88.73 (SciBERT-SciVocab) and 85.72 -> 85.08 (BERT-Base) when removing the char embeddings. I'll release the full set of results once they're ready.

[cbockman]

Thanks. What was the rationalization for including? (Empirical > theoretical, of course...) BERT is "supposed" to encapsulate this information (via subwords), anyway. Was this an attempted way to deal with the fact that the BERT layer was frozen (and thus perhaps not able to fully integrate the domain-specific learnings)?

[kyleclo]

@cbockman There wasn't any rationale for including char embeddings. We ran the experiments with a standard NER configuration in AllenNLP that had character-level embeddings & noticed afterwards that it included them. Since experiments are a bit expensive & we felt like it was still a fair comparison between BERT-Base and SciBERT, we didn't rerun everything & reported that we included char embeddings in the arXiv draft. We've since decided that it's worth redoing the experiments to exclude the char embeddings & will update the draft when they're done.

[cbockman]

Thanks! Love the paper (probably obviously).

[arunzz]

Thanks! Will wait for score without char embeddings, Thanks again !