This repository contains benchmarking tools of various tokenizers.
To perform benchmarking, you have to run the following two steps.
The following commands prepare resources (e.g. model data) and compile source codes.
% git submodule update --init
% ./download_resources.sh
% ./compile_all.sh
To measure the speed of each tokenizer, run the following commands.
If you stop ./run_all.sh in the middle, ./stats.py will calculate statistics from avaiable results.
% ./run_all.sh | tee ./results
% ./stats.py < ./results
This software is developed by LegalForce, Inc., but not an officially supported LegalForce product.
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
For softwares under thirdparty, follow the license terms of each software.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.