Skip to content
Permalink
master
Switch branches/tags
Go to file
 
 
Cannot retrieve contributors at this time

12.1.0

Introduction

Tokei is a fast and accurate code analysis CLI tool and library, allowing you to easily and quickly see how many blank lines, comments, and lines of code are in your codebase. All releases and work on Tokei and tokei.rs (the free companion badge service) are funded by the community through GitHub Sponsors.

You can always download the latest version of tokei through GitHub Releases or Cargo. Tokei is also available through other package managers, though they may not always contain the latest release.

cargo install tokei

What's New?

Added Languages

12.0.0

What's New?

Tokei 12 comes with some of the biggest user facing changes since 1.0, now in the latest version tokei will now analyse and count multiple languages embedded in your source code as well as adding support for Jupyter Notebooks. Now for the first time is able to handle and display different languages contained in a single source file. This currently available for a limited set of languages, with plans to add more support for more in the future. The currently supported languages are;

HTML + Siblings (Vue, Svelte, Etc...)

Tokei will now analyse and report the source code contained in <script>, <style>, and <template> tags in HTML and other similar languages. Tokei will read the value of thetype attribute from the <script> tag and detects the appropriate language based on its mime type or JavaScript if not present. Tokei will do the same for <style> and <template> except reading the lang attribute instead of type and defaulting to CSS and HTML each respectively.

Jupyter Notebooks

Tokei will now read Jupyter Notebook files (.ipynb) and will read the source code and markdown from Jupyter's JSON and output the analysed result.

Markdown

Tokei will now detect any code blocks marked with specified source language and count each as their respective languages or as Markdown if not present or not found. Now you can easily see how many code examples are included in your documentation.

Rust

Tokei will now detect blocks of rustdoc documentation (e.g. //////!) and parse them as markdown.

Verbatim Strings

Tokei is now also capable of handling "verbatim" strings, which are strings that do not accept escape sequences like \. Thanks to @NickHackman for providing the implementation! This is initially supported for C++, C#, F#, and Rust.

New Look

To be able to show these new features, tokei's output has been changed to look like below. For brevity the CLI only displays one level deep in each language, however the library's parser is fully recursive and you can get access to the complete report using the library or by outputting the JSON format.

===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 BASH                    4           49           30           10            9
 JSON                    1         1332         1332            0            0
 Shell                   1           49           38            1           10
 TOML                    2           77           64            4            9
-------------------------------------------------------------------------------
 Markdown                5         1230            0          965          265
 |- JSON                 1           41           41            0            0
 |- Rust                 2           53           42            6            5
 |- Shell                1           22           18            0            4
 (Total)                           1346          101          971          274
-------------------------------------------------------------------------------
 Rust                   19         3349         2782          116          451
 |- Markdown            12          351            5          295           51
 (Total)                           3700         2787          411          502
===============================================================================
 Total                  32         6553         4352         1397          804
===============================================================================

This feature is not just limited to the default output of tokei. You can see it broken down by each file with the --files option.

===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Markdown                5         1230            0          965          265
 |- JSON                 1           41           41            0            0
 |- Rust                 2           53           42            6            5
 |- Shell                1           22           18            0            4
 (Total)                           1346          101          971          274
-------------------------------------------------------------------------------
 ./CODE_OF_CONDUCT.md                46            0           28           18
 ./CHANGELOG.md                     570            0          434          136
-- ./markdown.md --------------------------------------------------------------
 |- Markdown                          4            0            3            1
 |- Rust                              6            4            1            1
 |- (Total)                          10            4            4            2
-- ./README.md ----------------------------------------------------------------
 |- Markdown                        498            0          421           77
 |- Shell                            22           18            0            4
 |- (Total)                         520           18          421           81
-- ./CONTRIBUTING.md ----------------------------------------------------------
 |- Markdown                        112            0           79           33
 |- JSON                             41           41            0            0
 |- Rust                             46           38            4            4
 |- (Total)                         200           79           84           37
===============================================================================
 Total                   5         1346          101          971          274
===============================================================================

Breaking Changes

  • The JSON Output and format of Languages has changed.
  • The JSON feature has been removed and is now included by default.
  • Stats has been split into Report and CodeStats to better represent the separation between analysing a file versus a blob of code.

11.2.0

  • @alexmaco Added shebang and env detection for Crystal.
  • @NickHackman Updated both Vue and HTML to count CSS & JS comments as comments.
  • @XAMPPRocky renamed Perl6's display name to Rakudo.
  • @dbackeus Added erb extension for Ruby HTML.
  • @kobataiwan Tokei will now check for a configuration file in your home directory as well as your current and configuration directory.
  • @dependabot Updated dependencies

Added Languages

  • @alexmaco Dhall
  • @NickHackman Svelte
  • @athas Futhark
  • @morphy2k Gohtml
  • @LucasMW Headache
  • @rosasynstylae Tsx
  • @XAMPPRocky OpenType Feature Files

11.1.0

Added Languages

  • @rubdos Arduino

  • @LuqueDaniel Pan

  • @itkovian Ren'Py

  • Added LanguageType::shebangs, LanguageType::from_file_extension, and LanguageType::from_shebang. (@solanav)

11.0.0

Added languages

  • @bwidawsk GNU Assembly, GDB Script
  • @isker Dust, Apache Velocity
  • @andreblanke FreeMarker

Thanks to some major internal refactoring, Tokei has received significant performance improvements, and is now one of the fastest code counters across any size of codebase. With Tokei 11 showing up to 40–60% faster results than tokei's previous version. To showcase the improvements I've highlighted benchmarks of counting five differently sized codebases. Redis (~220k lines), Rust (~16M lines), and the Unreal Engine (~37.5M lines). In every one of these benchmarks Tokei 11 performed the best by a noticeable margin.

All benchmarks were done on a 15-inch MacBook Pro, with a 2.7GHz Intel Core i7 processor and 16GB 2133 MHz LPDDR3 RAM running macOS Catalina 10.15.3. Your mileage may vary, All benchmarks were done using hyperfine, using default settings for all programs.

Tokei

Note This benchmark is not accurate due to tokei and loc both taking less than 5ms to complete, there is a high degree of error between the times and should mostly be considered equivalent. However it is included because it is notable that scc takes nearly 3x as long to complete on smaller codebases (~5k lines). Graph comparing programs running on the tokei source code

Redis

Graph comparing programs running on the redis source code

Rust

Graph comparing programs running on the rust source code

Unreal

Graph comparing programs running on the unreal source code

10.1.2

  • Added pyw extension to Python.
  • Updated dependencies

10.1.1

  • Fixed .tokeignore always working even when --no-ignore is present.
  • Updated dependencies

Added languages

  • @erikaxel Gherkin (Cucumber)

10.1.0

  • Added cjsx extension to CoffeeScript.
  • Tokei will now recognise files with #!/usr/bin/env ruby as Ruby.
  • Updated dependencies.
  • Tokei now uses crossbeam channels over std::mpsc, which should have a noticeable performance improvement on large repos.
  • Improved documentation for libtokei.

Added languages

  • @lzybkr PowerShell
  • @turbo MoonScript
  • @dtolnay Thrift
  • @Tranzystorek FlatBuffers
  • @NieDzejkob Emojicode
  • @DanteFalzone0 HolyC
  • @sci4me Odin
  • @fkarg Rusty Object Notation (RON)

10.0.0

  • Fixed minor parsing bugs.
  • Width is now limited to 80 unless you use the --files flag.
  • Added the mjs extension to JavaScript.
  • Added the tpp extension to C++.
  • You can now disable Tokei's git ignore detection, similar to ripgrep. See --help for options.
  • You can now add a .tokeignore file to your project to specify file paths for tokei to always ignore. This file uses the same syntax as .gitignore.
  • Improved Pascal representation

Added languages

  • @hobofan solidity
  • @stefanmaric GraphQL
  • @jhpratt PostCSS
  • @evitalis RPM
  • @alexmaco Pony
  • @yjhmelody WASM, LLVM, Pest
  • @XAMPPRocky ASN.1

9.0.0

  • Tokei now has config files. You can now specify some commonly used arguments in a .tokeirc/tokei.toml. Namely columns to set the default column output, types to filter your count to just a single set of languages, and treat_doc_strings_as_comments which is a new option that allows you to specify whether to treat doc strings such as """ in Python as comments or code. The config files can be specified in two places, the current directory tokei is running in and your system configuration directory. The priority of options is as follows CLI > <current_directory> > <configuration_directory>.
  • Tokei is now available on Conda.
  • Tokei's README has been translated to chinese.
  • LanguageType now implements Hash.
  • Tokei now batches it's console output, this should result in a small performance boost.
  • There is now a --columns argument for manually setting tokei's output width.
  • The --sort argument is now case-insensitive.
  • Tokei will now mark languages who's files failed to parse correctly as potentially inaccurate.
  • Due to a bug in trust-ci x86_64-unknown-netbsd versions are will not be available in GitHub releases. (You will still be able to install from source.)
  • Due to toml-rs's lacking enum support the TOML output option has been disabled.

Added languages

  • @t-richards Liquid
  • @diaphore Added the .glsl extension to GLSL.
  • @ahmedelgabri Twig
  • @pmoura Logtalk
  • @alekratz Perl, Not Quite Perl
  • @XAMPPRocky Automake, .NET Resource, HLSL, INI, Unreal Plugin, Unreal Project, Unreal Shader, Unreal Shader Header, Unreal Markdown, Visual Basic, Visual Studio Solution, Visual Studio Project, Xcode Config,
  • @TheMrNomis SWIG
  • @xnorme Added the .vhdl extension to VHDL

8.0.0

  • A language's comments, and quotes are now available through the LanguageType enum.
  • You can filter by language using the -t/--type option. e.g. tokei -t "Rust,C" will print only Rust and C files.
  • Tokei now understands terminal width and will expand to fit it. (Thanks to @Veykril)
  • Added comparison document to compare Tokei to other code counters.
  • Updated dependencies

Added languages

  • @BrandonBoone VB6, VBScript, XSLT
  • @ialpert BrightScript
  • @PJB3005 Dream Maker
  • @schmee edn

7.0.3

Made various optimisations, up to 65% faster in some cases.

Added languages

  • @DenialAdams Added Forsyth-Edwards-Notation (FEN)
  • @DjebbZ Added ClojureC
  • @grimm26 Added HCL/Terraform

7.0.2

  • Updated dependencies.
  • Changed how compilied serialization formats are handled.
  • Fixed minor parser inaccuracies.
  • Tokei should now recognise more python files from their shebang.

Added languages

  • @ignatenko Added Meson
  • @sprang Added Scheme
  • @fengcms Added Vue
  • @mark.knol Added Haxe
  • @rleungx Added ABAP, COBOL, and Groovy
  • @tiehuis Added Zig
  • @murielsilveira Added Mint
  • @notramo Added Elvish Shell and Kakoune
  • @aatxe Added Racket
  • @kamilchm Added ReasonML
  • @cyplp Added XSL

7.0.1

  • Updated dependencies

7.0.0

  • Fixed parsing corner cases
  • Changed storage of comments and quotes from Vec to static slices.
  • Added tracing for debugging single files. Not recommended for use on multiple file
  • Updated log

6.1.0

  • Fixed inaccuracies relating to the end comment being smaller than start comment.

Added languages

  • @mattico Added Xaml
  • @weakish Added Ceylon
  • @theduke Added tsx extension to typescript
  • @vmchale Added Hamlet, Cassius, Lucius, Cabal, Nix, Happy, Alex, and Madlang
  • @notramo Added Crystal

6.0.2

  • Now can recognise file languages based on their filename.

Added Languages:

  • @kazimuth CMake, Dockerfile, Rakefile, Scons

6.0.1

  • Multiple exclude flags now allowed.

Added Languages:

  • @seiks Added Fish Shell
  • @XAMPPRocky Added Module-Definition
  • @tbu- Added Vala

6.0.0

  • Reworked internals
  • Now uses serdederive(_and thusly requires rust v1.15)
  • Now has better file based testing

Added languages:

  • @tuncer Added Ur/Web
  • @svisser Added PureScript
  • @tjodden Add some common extensions for HTML, C++ and Makefile
  • @xd009642 Added VHDL

5.0.0

  • Optimised internals

Added languages:

  • @GungnirInd Added GDScript
  • @tuncer Differentiate between sh and Bash, Added Cogent, F*, F#
  • @pthariensflame Added Agda

4.5.0

  • Added Regex based hueristics so more expensive multi line handling isn't used if there are no multi line comments in the file.
  • Now uses the ignore crate for getting files. Which now also makes determining language from path/file parallelised
  • File counting used to only be parallelised per language, now it is also parallelised per file per language.
  • Updated homepage, and documentation links
  • @rmbreak Tokei will now not add directories with foo.bar like syntax to a language.
  • @Michael-F-Bryan tokei will now exit gracefully when a feature is missing instead of panicing

Added languages:

  • @hauleth Added Elixir support

4.4.0

  • Simplified language definitions, now consolidated into a single JSON file.
  • Fixed regression where lines and files weren't sorted.
  • @llogiq : made clippy fixes
  • @lligo : Added long verbose name

Added languages:

  • @little-dude : Tcl(tcl)
  • @svenstaro : GLSL(vert, tesc, tese, geom, frag, comp)
  • @not-fl3 : Elm(elm)

Changes to existing languages:

  • @xpayn : Added pm extension to Perl.

4.3.0

  • @lligo : Tokei no longer panics on non-character-boundary when printing file names.
  • Fixed regression where no comment style files(json, markdown) weren't counted.
  • Tokei can now handle files in different encodings.(using the encoding library)
  • Tokei now prints errors instead of sliently skipping them.
  • Tokei can now print unused extensions using -v option.

Added languages:

  • Asp(asa, asp)
  • Asp.NET(asax, ascx, asmx, aspx, master, sitemap, webinfo)
  • Hex(hex)
  • Intel Hex(ihex)
  • ReStructuredText(rst)
  • Razor(cshtml)

Changes to existing languages Thanks to @mwilli20 :

  • Another Ada extension(pad)
  • Assembly - Uses ' ' or " " and added another extension(asm)
  • Bash - Uses ' ' or " "
  • Batch - They don't use quotes for strings, added ::
  • Cold Fusion - Uses ' ' or " "
  • D - Uses " " or
  • Dart - Uses " " or ' ' or """ """ or ''' '''
  • Forth - Uses " " but new, doesn't have a preset
  • Fortrans - Use " " or ' '
  • Idris - Uses " " or """ """
  • Julia - Uses " " or """ """
  • Kotlin - Uses " " or """ """
  • Lisp - Comments can be nested
  • Moustache - Uses " " or ' '
  • Nim - Uses " " or """ """
  • Pascal - Uses ' '
  • Perl - Uses " " or ' '
  • Php - Uses " " or ' '
  • Python - Uses " " or ' ' or """ """ or ''' '''
  • Ruby - Uses " " or ' '
  • Sass - Uses " " or ' '
  • Sql - Uses ' '
  • Toml - Uses " " or ' ' or """ """ or ''' '''
  • Typescript - Uses " " or ' ' or
  • Vimscript - Uses " " or ' '
  • Yaml - Uses " " or ' '
  • Zsh - Uses " " or ' '
  • Clojure - Removed #
  • Forth - ( Comment) style comments need a space after the opening paren
  • Haskell - Has nested comments
  • Idris - Has nested comments
  • Jai - Has nested block comments
  • Julia - Has nested block comments
  • Kotlin - Has nested block comments
  • Pascal - Pascal should be multiline from { or (* to } or *)
  • Perl - Perl5 and earlier for multiline comments need =pod to =cut.
  • Swift - Has nested block comments

Tokei's code count

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Rust                   13         2413         1596          601          216
-------------------------------------------------------------------------------
 |ib\language\languages.rs          693          420          197           76
 |anguage\language_type.rs          500          386          102           12
 .\src\main.rs                      314          256           17           41
 |lib\language\language.rs          356          166          166           24
 .\src\lib\utils\fs.rs              129          107            9           13
 |\lib\utils\multi_line.rs          149           89           39           21
 .\src\lib\utils\macros.rs           59           50            3            6
 .\src\lib\stats.rs                  63           45           12            6
 .\src\lib\lib.rs                    76           25           47            4
 .\src\lib\build.rs                  31           23            0            8
 .\src\lib\sort.rs                   28           19            6            3
 .\src\lib\language\mod.rs           11            6            3            2
 .\src\lib\utils\mod.rs               4            4            0            0
-------------------------------------------------------------------------------
 Markdown                4          492          492            0            0
-------------------------------------------------------------------------------
 .\README.md                        252          252            0            0
 .\CHANGELOG.md                     202          202            0            0
 .\CONTRIBUTING.md                   25           25            0            0
 .\CONTRIBUTORS.md                   13           13            0            0
-------------------------------------------------------------------------------
 YAML                    2           70           67            3            0
-------------------------------------------------------------------------------
 .\cli.yml                           53           50            3            0
 .\.travis.yml                       17           17            0            0
-------------------------------------------------------------------------------
 TOML                    1           80           65            0           15
-------------------------------------------------------------------------------
 .\Cargo.toml                        80           65            0           15
-------------------------------------------------------------------------------
 Autoconf                1            9            7            1            1
-------------------------------------------------------------------------------
 .\src\lib\lib.rs.in                  9            7            1            1
-------------------------------------------------------------------------------
 Total                  21         3064         2227          605          232
-------------------------------------------------------------------------------

4.2.0

Tokei is now more precise, and shouldn't ever panic. Tokei now handles comments in quotes and more precise nested comments properly. Fixes #53

Tokei's code count.

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Rust                   13         2303         1487          594          222
-------------------------------------------------------------------------------
 |ib\language\languages.rs          682          401          198           83
 |anguage\language_type.rs          467          359           96           12
 .\src\main.rs                      302          243           17           42
 |lib\language\language.rs          356          166          166           24
 .\src\lib\utils\fs.rs              116           95            9           12
 |\lib\utils\multi_line.rs          156           93           41           22
 .\src\lib\stats.rs                  54           36           12            6
 .\src\lib\build.rs                  31           23            0            8
 .\src\lib\lib.rs                    69           22           43            4
 .\src\lib\utils\macros.rs           27           20            3            4
 .\src\lib\sort.rs                   28           19            6            3
 .\src\lib\language\mod.rs           11            6            3            2
 .\src\lib\utils\mod.rs               4            4            0            0
-------------------------------------------------------------------------------
 YAML                    2           68           65            3            0
-------------------------------------------------------------------------------
 .\cli.yml                           49           46            3            0
 .\.travis.yml                       19           19            0            0
-------------------------------------------------------------------------------
 TOML                    1           71           58            0           13
-------------------------------------------------------------------------------
 .\Cargo.toml                        71           58            0           13
-------------------------------------------------------------------------------
 Autoconf                1            9            7            1            1
-------------------------------------------------------------------------------
 .\src\lib\lib.rs.in                  9            7            1            1
-------------------------------------------------------------------------------
 Total                  17         2451         1617          598          236
-------------------------------------------------------------------------------

4.1.0

Tokei is now ~40% faster.

Added languages

  • Ada
  • Forth

4.0.0

Tokei now has a minimal version without serde for faster compilation.

Updated various dependencies.

Internal dependencies removed.

Regressions

  • CBOR is not supported till it supports serde 0.8

Added languages

  • Handlebars

3.0.0

Tokei is now available as a library.

Tokei now has a lot more tests.

Tokei now supports TOML

Fixed #41

Fixed #44

Fixed #45

2.1.0

Tokei, can now output results in various formats(cbor, json, yaml)

Conversely tokei can now take in results in those formats, and add them to the current run.

Premilarily support for nested comments(currently only supported for rust)

Change in the output format PR #35

Moved .sc from Lisp to Scala.

Internals changed to allow for multiple multi line comment formats.

Added languages:

  • Isabelle

2.0.0

Major rewrite, now parallelized. Can now support sorting files. Added a progress message for when it is counting files. Fixed #29

Added languages:

  • Coq
  • Erlang
  • Kotlin
  • Idris
  • Nim
  • Oz
  • Prolog
  • Qcl
  • Scala
  • Unreal Script
  • Wolfram

1.6.0

Added file counting.

1.5.0

Added Shebang support.

Added languages:

  • Assembly
  • LD Scripts
  • Device Trees
  • Makefiles
  • Plain Text
  • C Shell

1.4.1

Changed the formatting so tokei looks nice for consoles of 80 column width.

1.4.0

Changed from handmade recursive file opening to walkdir