Luau is a fast, small, safe, gradually typed, statically linked, embeddable scripting language derived from Lua. It lies at the heart of Roblox technology - powering all it's user generated content, with Roblox's own internal code having more than 2 millions lines of Luau.
It has sandboxing, type-checking, additional operators & increased performance while maintaining compatibility with Lua.
Lua is faster than Python & Luau is even faster still - more so, as qsv precompiles Luau into bytecode. In addition, luau is embedded into qsv, has debug logging, can do aggregations with its --begin & --end options & has no external dependencies unlike the py command.
It also allows mapping of multiple new computed columns, supports random access with indexed CSV files, and has several helper functions to help ease the development of full-fledged data-wrangling scripts.
As date manipulation is often needed, the LuaDate module is also bundled.
Finally, as qsv's DSL (👑), luau will gain even more features over time compared to the python feature.
Luau 0.604 is currently embedded - qsv's policy is to use the latest stable Luau version at the time of each qsv release.
If you wish to build qsv with the python feature, make sure you have the development libraries for the desired Python version (Python 3.7 and above are supported) installed when doing so (e.g. on Debian/Ubuntu - apt-get install python-dev; on CentOS/RedHat/Amazon Linux - yum install python-devel; on Windows and macOS - use the Python installer for the desired version).
If you plan to distribute your manually built qsv with the python feature, qsv will look for the specific version of Python shared libraries (libpython* on Linux/macOS, python*.dll on Windows) against which it was compiled starting with the current directory & abort with an error if not found, detailing the Python library it was looking for.
Note that this will happen on qsv startup, even if you're NOT running the py command.
When building from source - PyO3 - the underlying crate that enables the python feature, uses a build script to determine the Python version & set the correct linker arguments. By default it uses the python3 executable.
You can override this by setting PYO3_PYTHON (e.g., PYO3_PYTHON=python3.7), before installing/compiling qsv. See the PyO3 User Guide for more information.
Consider using the luau command instead of the py command if the operation you're trying to do can be done with luau - as luau is statically linked, has no external dependencies, much faster than py, can do aggregations, supports random access, has a bevy of qsv helper functions, and allows mapping of multiple new columns.
The py command cannot do aggregations because PyO3's GIL-bound memory limitations will quickly consume a lot of memory (see issue 449 for details).
To prevent this, the py command processes CSVs in batches (default: 30,000 records), with a GIL pool for each batch, so no globals are available across batches.