diff --git a/readme.md b/readme.md index 9c5db888..0075d2cb 100644 --- a/readme.md +++ b/readme.md @@ -1,278 +1,93 @@ # fast_io -fast_io is a new C++20 library for extremely fast input/output and aims to replace iostream and cstdio. It is header-only (module only in the future) for easy inclusion in your project. It requires a capable C++20 compiler supporting concepts. +fast_io is an extremely [fast](readme.md#Benchmarks) C++20 input/output library aiming to replace `` and ``. -## QQ group -1076846339 +It is header only + [MIT licensed](license.txt) for easy inclusion in any project. It requires a capable C++20 compiler supporting concepts. -## Discord URL. - -You can directly ask me questions in Discord. - -https://discord.gg/wYfh8kk - - -## What does "fast" mean in fast_io? - -It does not necessary mean it will be faster than everything else. (Or it would be named as fastest_io. Of course those fmt trolls would claim that.) It means it is significantly faster than stdio.h and iostream for all cases. fast means faster than stdio.h and iostream. - -## Hello World +*Old Repository was archived due to bloat with git through time; old commits were [here](https://bitbucket.org/ejsvifq_mabmip/fast_io_archive-2022-01-23).* +## Examples +### Print to `stdout` ```cpp -#include +#include -int main() -{ - print("Hello World!\n"); +int main() { + print("Hello World!\n"); } ``` +### Load whole file through memory mapping +```cpp +#include -## Old Repository was archived due to bloat with git through time - -Old commits were at: -https://bitbucket.org/ejsvifq_mabmip/fast_io_archive-2022-01-23 +int main() { + fast_io::native_file_loader file_data("text.txt"); + // file_data satisfies std::ranges::contiguous_range +} +``` +### Other examples -## Examples -Deprecated examples are in -https://bitbucket.org/ejsvifq_mabmip/fast_io_deprecated +For other up to date examples please look in the [`examples/`](examples/) folder. -However, they might not work current fast_io since fast_io keeps changing. +Deprecated examples are [here](https://bitbucket.org/ejsvifq_mabmip/fast_io_deprecated) but they might not work anymore since fast_io keeps changing. This i18n repo stores the i18n source files. https://bitbucket.org/ejsvifq_mabmip/fast_io_i18n -## Online docs -https://ewindy.gitee.io/fast_io_rst/index.html - -## Compiler Support -- GCC 12 -- VS 19.29.29917 -- Clang 15 -- Winegcc - -## C++ standard library support - -- GCC libstdc++ -- LLVM libc++ -- MSVC STL - -## Architecture Support -- alpha -- i686 -- x86_64 -- arm -- aarch64 -- riscv -- riscv64 -- powerpc64 -- wasm32-wasi -- mips-abi32 -- mips-abin32 -- mips-abi64 -- avr -- ia64 -- All architectures should work. (Just i do not have time to build all cross-toolchains to test them one by one) - -## Platform Support -- Windows 11 -- Windows 10 -- Windows Server 2019 -- Linux -- BSD platforms (Including FreeBSD, NetBSD, OpenBSD) -- Apple Darwin (MacOS) -- Apple IOS -- Microsoft DOS -- FreeDOS -- Barebone (No any operating system.) -- Cygwin/MSYS2 -- Android -- Wine -- Wine GCC -- ReactOS -- Windows Runtime -- Nintendo DS (devkitPro) -- Webassembly -- Managarm -- Vinix -- limine Bootloader -- AVR -- Banana -- LemonOS -- LLVM raw_fd_ostream -- Windows Kernel Driver -- Linux Kernel Module -- wasm2lua (Compile fast_io to Lua!) -- Any other platform (if you want me to support it, tell me, including your own OS) - -## Kernel Driver Support -- Windows NT kernel Driver -- Linux kernel Module - -## Legacy Windows Version Support (need define _WIN32_WINNT and _WIN32_WINDOWS version by yourself) -- Windows 95 -- Windows 95 Plus! -- Windows Nashville -- Microsoft Memphis -- Windows 98 FE -- Windows 98 SE -- Windows ME -- Microsoft Neptune -- Windows NT -- Windows Odyssey -- Windows 2000 -- Windows Whistler -- Windows XP -- Windows Server 2003 -- Windows Longhorn -- Windows Vista -- Windows Server 2008 -- Windows Vienna -- Windows 7 -- Windows 8 -- Windows 8.1 -- Windows Home Server 2011 -- Windows Server 2012 -- Windows Server 2016 - -## Windows 95 or 95 Plus! Support -You can enable -O3 but you cannot not enable -Ofast. -Also need to install msvcrt.dll - -## Legacy Windows Version Support with DJGPP toolchain -- Windows 1.0 -- Windows 2.0 -- Windows 3.0 -- Windows 3.1 -- Microsoft Chicago -- Windows 95 -- Windows 95 Plus! -- Microsoft Memphis -- Windows 98 FE -- Windows 98 SE -- Windows ME - -## Supported libc FILE* hacks platforms -- glibc -- MSVCRT -- Universal CRT -- Universal CRT with MSVC debugging -- Wine's Emulated MSVCRT -- Wine's Emulated UCRT -- ReactOS Emulated MSVCRT -- ReactOS Emulated UCRT -- BSD libc/Apple Darwin -- MUSL libc -- DJGPP -- Newlib -- Bionic(Android) -- uclibc-ng -- wasi libc -- mlibc (this is a new libc which still contains a lot of bugs currently) -- avrlibc -- picolibc - -## Supported C++ standard library std::streambuf/std::filebuf hacks platforms -- GCC libstdc++ -- LLVM libc++ -- MSVC STL - -## Remote Debugger Support -- WinDbg -- Visual Studio Debugger -- DbgViewer -- Android ADB - -## Special Support -- LLVM Address Sanitizer -- Full freestanding support (which means we do not have any headers besides freestanding headers. Tested with GCC's --disable-hosted-libstdcxx without any libc, including newlib) - -## Design Goal - -A general purpose I/O library to replace stdio.h and iostream +## F.A.Q. +### What does "fast" mean in fast_io? -- As close to system call as possible. -- No default locale. It is optional. -- Correctly deal with EBCDIC exec-charset (where no libc correctly deals it). -- OOP is not allowed. Yes to C with Concepts. No to C with Classes. - -### Safe - -- No easily misused stuff like std::endl -- No internal iomanip states (since it creates security issues) -- Providing RAII for C FILE\*&POSIX fd&win32 HANDLE -- No dual error reporting mechanism. Exception as the ONLY error reporting mechanism. -- No std::error_code and std::system_error -- No useless bounds checking -- No format string nonsense -- Addresss Sanitizer special code for protecting memory safety issues. +It doesn't necessary mean it will be faster than everything else (or it would be named `fastest_io`; of course `{fmt}` trolls would claim that) it means it is significantly faster than `` and `` in all cases. +### What are the supported platforms, compilers, etc. +See [support.md](support.md) for a list of supported "stuff" + platform specific usage notes. +### Other questions +You can directly ask questions on the [Discord server](https://discord.gg/wYfh8kk). +QQ group: `1076846339` -### Easy to use +## Documentation +See either https://ewindy.gitee.io/fast_io_rst/index.html or https://gitee.com/qabeowjbtkwb/fast_io/wikis. -- Unicode/UTF-8 support -- Support POSIX iconv. You can use fast_io for your coding conversion. -- Compatible with C stdio and C++ iostream -- Binary serialization for trivially copyable types and C++ standard library containers -- All fast_io devices can be natively put in C++ containers. std::vector is valid -- Basic/Lua/Python/etc format (print, scan). No support to C and C++ since they are security hazards. -- Static I/O manipulator -- Provide APIs to expose the internal implementation of C FILE* and C++ stream. +## Features +- As close to system call as possible. +- Unicode support (UTF-8, UTF-16, UTF-32) + codecvt for GB18030 and UTF-EBCDIC (no libc correctly deals with it) +- RAII for C `FILE*`, POSIX `fd` and win32 `HANDLE` +- Interop with `` and `` +- No easily misused stuff like `std::endl` +- Static I/O manipulators instead of format string nonsense. +- No locale by default. It is optional. +- No internal `iomanip` states (since it creates security issues) +- Consistent error handling; when available, exceptions as the **only** error reporting mechanism (no `std::error_code`, `std::system_error` or useless bounds checking) +- Freestanding mode +- Addresss Sanitizer special code for protecting memory safety issues. -### Customizability +- Supports POSIX iconv. You can use fast_io for your coding conversion. +- Binary serialization for trivially copyable types and standard containers +- Plays well with C++ containers (e.g. `std::vector` is valid) +- Basic/Lua/Python/etc format (print, scan). No support for C and C++ since they are security hazards. +- Provide APIs to expose the internal implementation of `FILE*` and C++ streams. -- Exception Safe & Exception neutral -- Native Handle Interface +- Native Handle Interface - Extremely easy to support custom devices -- mutex stream without std::mutex since std::mutex is not provided by x86_64-w64-mingw32 with win32 threads. YES TO POSIX PTHREAD and windows Rtl threads. -- Compilation time open mode parse. Supports C style open mode and C++ style open mode. -- No traits_type and EOF +- C and C++ style compile time open mode parsing. +- No `traits_type` and `EOF` - Dynamic Type Support - Multi Process -- Memory map -- debugging IO -- GUI debugging IO -- Freestanding mode +- Memory mapping +- debugging IO (optionally with GUI) - Round-trip floating point algorithm -- Network -- Intrinsic SHA-1, Intrinsic HMAC-SHA1, Intrinsic SHA-256, Intrinsic HMAC-SHA256, SHA-512, HMAC-SHA512 -- zlib compression/decompression support -- Non crypto hash algorithms support. Jenkins Hash +- Hash algorithms support: intrinsic SHA-1, intrinsic HMAC-SHA1, intrinsic SHA-256, intrinsic HMAC-SHA256, SHA-512, HMAC-SHA512 aswell as non crypto hash algorithms, such as Jenkins Hash. +- zlib compression/decompression - Filesystem - OpenSSL BIO, Qt QFile, MFC CFile support - -## Codecvt for File I/O support, including correctly dealing with execution charset of them -- UTF-8 -- UTF-16LE -- UTF-16BE -- UTF-32LE -- UTF-32BE -- GB18030 -- UTF-EBCDIC -- posix iconv is also supported - -## Post C++20 Plan +## Possible Improvements +1. Interfaces for cloud computing algorithms like MapReduce +2. This being in the standard library. +### Post C++20 1. Module support 2. Coroutine support for async IO 3. Improve and refactor code once [Zero-overhead deterministic exceptions](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0709r0.pdf) are added to the standard -## Possible Improvements - 2. Interfaces for cloud computing algorithms like MapReduce - -I'm aiming for this to be in the C++ standard library in the future :) - -## Getting Started -Please see examples in the examples folder. - -compile option: - -`g++ -o example example.cc -Ofast -std=c++20 -s -flto -march=native` - -## Documentation - -See Wiki Page: https://gitee.com/qabeowjbtkwb/fast_io/wikis - ## Benchmarks - 1. I/O 10M integers Goal: Print out ten million integers from 0 to 10M to file. Then reopen that file to scan back. @@ -294,13 +109,13 @@ Notice: I modified libstdc++'s BUFSIZ 1048576 due to BUFSIZE is too small (512 b | fast_io::i/obuf_file | 0.04903s | 0.080996s | | | fast_io::i/obuf_file_mutex | 0.146064s | 0.113155s | thread safe | | c_locale_i/obuf_file ("C") | 0.065988s | 0.086012s | imbued with locale, locale "C" | -| c_locale_i/obuf_file local | 0.153995s | Meaningless | imbued with locale, locale "" | -| fmt::format_int+obuf_file | 0.122999s | Meaningless | | -| fmt::format_int+ofstream | 0.209055s | Meaningless | | -| fmt::format+ofstream | 0.548s | Meaningless | fmt makes things slower | -| fmt::print | 0.663996s | Meaningless | fmt makes things slower | -| std::to_chars+obuf_file | 0.12s | Meaningless | | -| std::to_chars+ofstream | 0.192s | Meaningless | | +| c_locale_i/obuf_file local | 0.153995s | :x: | imbued with locale, locale "" | +| fmt::format_int+obuf_file | 0.122999s | :x: | | +| fmt::format_int+ofstream | 0.209055s | :x: | | +| fmt::format+ofstream | 0.548s | :x: | fmt makes things slower | +| fmt::print | 0.663996s | :x: | fmt makes things slower | +| std::to_chars+obuf_file | 0.12s | :x: | | +| std::to_chars+ofstream | 0.192s | :x: | | | fast_io::c_file_unlocked | 0.098999s | 0.126003s | I hacked MSVCRT's FILE* implementation | | fast_io::c_file | 0.298988s | 0.318001s | Thread Safe. I hacked MSVCRET's FILE* implementation | | fast_io::filebuf_file | 0.048999s | 0.081s | I hacked libstdc++'s streambuf/filebuf implementation| @@ -308,8 +123,8 @@ Notice: I modified libstdc++'s BUFSIZ 1048576 due to BUFSIZE is too small (512 b | fast_io::iobuf_utf8_file_char32| 0.110999s | 0.111011s | UTF-32=>UTF-8 with SSE| | std::wofstream | 2.64s | 3.843735s | wofstream with std::locale codecvt. Extremely slow tbh.| | fast_io::wfilebuf_io_observer | 2.415692s | 2.497704s | wofstream with std::locale codecvt. This proves fstream can never get fixed.| -| Rust language | 0.483s | | RUST IS SLOW. Also Rust does not deal with locale. Think how bad it is. | -| Rust itoa library 0.4.6 | > 0.165s | | I ignore the \n part for it to ensure no bias. | +| Rust language | 0.483s | :x: | RUST IS SLOW. Also Rust does not deal with locale. Think how bad it is. | +| Rust itoa library 0.4.6 | > 0.165s | :x: | I ignored the `\n` part for it to ensure no bias. | Rust language is 10x slower than fast\_io. + binary bloat and itoa library is still extremely slow and usable for me. It is at least 3x slower than fast\_io. @@ -329,8 +144,8 @@ Run the same test on MSVC 19.26.28805. | fstream with rdbuf.sputc trick | 3.3735902s | 3.8145566s | | | fast_io::i/obuf_file | 0.0631433s | 0.1030554s | | | fast_io::i/obuf_file_mutex | 0.2190659s | 0.2485886s | thread safe | -| std::to_chars+obuf_file | 0.1641641s | Meaningless | | -| std::to_chars+ofstream | 0.5461922s | Meaningless | | +| std::to_chars+obuf_file | 0.1641641s | :x: | | +| std::to_chars+ofstream | 0.5461922s | :x: | | | fast_io::c_file_unlocked | 0.1102575s | 0.2399757s | I hacked Universal CRT's FILE* implementation | | fast_io::c_file | 0.2034755s | 0.2621148s | Thread Safe. I hacked UCRT's FILE* implementation | | fast_io::filebuf_file | 0.126661s | 0.2378803s | I hacked MSVC STL's streambuf/filebuf implementation | @@ -349,12 +164,12 @@ Run the same test on GCC 11. glibc + libstdc++ | fast_io::i/obuf_file | 0.050300857s | 0.065372395s | | | fast_io::i/obuf_file_mutex | 0.05290654s | 0.083040518s | thread safe | | c_locale_i/obuf_file ("C") | 0.051939052s | 0.065820056s | imbued with locale, locale "C" | -| c_locale_i/obuf_file local | 0.162406082s | Meaningless | imbued with locale, locale "" | -| std::to_chars+obuf_file | 0.115453587s | Meaningless | | -| fmt::format_int+obuf_file | 0.1183587s | Meaningless | | -| fmt::format_int+ofstream | 0.195914384s | Meaningless | | -| fmt::format+ofstream | 0.633590975s | Meaningless | fmt makes things slower | -| fmt::print | 0.495270371s | Meaningless | fmt makes things slower | +| c_locale_i/obuf_file local | 0.162406082s | :x: | imbued with locale, locale "" | +| std::to_chars+obuf_file | 0.115453587s | :x: | | +| fmt::format_int+obuf_file | 0.1183587s | :x: | | +| fmt::format_int+ofstream | 0.195914384s | :x: | | +| fmt::format+ofstream | 0.633590975s | :x: | fmt makes things slower | +| fmt::print | 0.495270371s | :x: | fmt makes things slower | | boost::iostreams | 0.400906063s | 0.444717051s | Using boost iostreams does not make your code faster | | fast_io::c_file_unlocked | 0.060076723s | 0.073299716s | I hacked glibc's FILE* implementation | | fast_io::c_file | 0.092490191s | 0.104545535s | Thread Safe. I hacked glibc's FILE* implementation | @@ -498,9 +313,9 @@ iconv test: | iconv command | 0.844s | GNU iconv. No BOM which sucks | | utf8_file_to_utf32_file.cc | 0.442s | I use the SSE algorithms provided by the utf-utils project. | -## Credit +## Credits -This project is made possible by referencing from other open-source projects. (I won't take their code directly. They are reimplemented by myself to fit the purpose of this library or it might have integration issues.) +This project is made possible by referencing other open-source projects. (I don't take their code directly, they are reimplemented by myself to fit the purpose of this library or it might have integration issues.) |Project | Url | |---------|-----------------------------------------| diff --git a/support.md b/support.md new file mode 100644 index 00000000..4e5ac1df --- /dev/null +++ b/support.md @@ -0,0 +1,140 @@ +## Compiler Support +- GCC >= 12 +- VS >= 19.29.29917 +- Clang >= 15 +- Winegcc + +## C++ standard library support +- GCC libstdc++ +- LLVM libc++ +- MSVC STL + +## Architecture Support +- alpha +- i686 +- x86_64 +- arm +- aarch64 +- riscv +- riscv64 +- powerpc64 +- wasm32-wasi +- mips-abi32 +- mips-abin32 +- mips-abi64 +- avr +- ia64 +- All other architectures should work, they just aren't tested due to time not being infinite. + +## Platform Support +- Windows 10/11 +- Windows Server 2019 +- Linux +- BSD platforms (Including FreeBSD, NetBSD, OpenBSD) +- Apple Darwin (MacOS) +- Apple IOS +- Microsoft DOS +- FreeDOS +- Barebone (No any operating system.) +- Cygwin/MSYS2 +- Android +- Wine +- Wine GCC +- ReactOS +- Windows Runtime +- Nintendo DS (devkitPro) +- Webassembly +- Managarm +- Vinix +- limine Bootloader +- AVR +- Banana +- LemonOS +- LLVM `raw_fd_ostream` +- Windows Kernel Driver +- Linux Kernel Module +- wasm2lua (Compile fast_io to Lua!) +- Any other platform (if you want me to support it, tell me, including your own OS) + +## Kernel Driver Support +- Windows NT kernel Driver +- Linux kernel Module + +## Legacy Windows Version Support +**You need define `_WIN32_WINNT` and `_WIN32_WINDOWS` version by yourself.** +- Windows 95 +- Windows 95 Plus! +- Windows Nashville +- Microsoft Memphis +- Windows 98 FE +- Windows 98 SE +- Windows ME +- Microsoft Neptune +- Windows NT +- Windows Odyssey +- Windows 2000 +- Windows Whistler +- Windows XP +- Windows Server 2003 +- Windows Longhorn +- Windows Vista +- Windows Server 2008 +- Windows Vienna +- Windows 7 +- Windows 8 +- Windows 8.1 +- Windows Home Server 2011 +- Windows Server 2012 +- Windows Server 2016 + +## Windows 95 or 95 Plus! Support +You can enable `-O3` but not not `-Ofast`. +You also need to install `msvcrt.dll`. + +## Legacy Windows Version Support with DJGPP toolchain +- Windows 1.0 +- Windows 2.0 +- Windows 3.0 +- Windows 3.1 +- Microsoft Chicago +- Windows 95 +- Windows 95 Plus! +- Microsoft Memphis +- Windows 98 FE +- Windows 98 SE +- Windows ME + +## Supported libc FILE* hacks platforms +- glibc +- MSVCRT +- Universal CRT +- Universal CRT with MSVC debugging +- Wine's Emulated MSVCRT +- Wine's Emulated UCRT +- ReactOS Emulated MSVCRT +- ReactOS Emulated UCRT +- BSD libc/Apple Darwin +- MUSL libc +- DJGPP +- Newlib +- Bionic(Android) +- uclibc-ng +- wasi libc +- mlibc (this is a new libc which still contains a lot of bugs currently) +- avrlibc +- picolibc + +## Supported C++ standard library `std::streambuf`/`std::filebuf` hacks platforms +- GCC libstdc++ +- LLVM libc++ +- MSVC STL + +## Remote Debugger Support +- WinDbg +- Visual Studio Debugger +- DbgViewer +- Android ADB + +## Special Support +- LLVM Address Sanitizer +- Full freestanding support (which means we do not have any headers besides freestanding headers. Tested with GCC's --disable-hosted-libstdcxx without any libc, including newlib)