This repository was archived by the owner on Jul 7, 2023. It is now read-only.
This repository was archived by the owner on Jul 7, 2023. It is now read-only.
Weird result on processing tokenization #1497
Description
Description
I used my own dataset on the transformer model, but I got strange results as shown in the picture while processing tokens on t2t-datagen. I guess it is because of my large dataset? My dataset is WikiEd: http://romang.home.amu.edu.pl/wiked/wiked.v1.pdf
Environment information
Ubuntu
tensorflow-gpu 1.13.1
