Skip to content

Building

Jump to bottom
YadeWira edited this page Oct 29, 2026 · 11 revisions

Building

Free Pascal 3.2.2 (other 3.2.x should work).

./build.sh                    # out/syc, out/sylzbench
./build.sh -Twin64            # Windows x86-64, needs FPC's win64 units
./build.sh -dSYLZ_PUREPASCAL  # no assembler: the Pascal paths only

build.sh passes its arguments to fpc. Units go to out/units.

platform how release binaries
Linux x86-64 ./build.sh from a Linux host
Linux arm64 cross compiler (below), or natively ./build.sh from a Linux host
Linux x86 (32-bit) a 32-bit FPC (below) from a Linux host
Windows x86-64, x86 cross compilers (below), or natively (below) from a Linux host
macOS arm64 brew install fpc, then ./build.sh from CI

Every build writes the same bytes: CI checks it on every push (see CI).

Linux x86 (32-bit)

scripts/fpc-i386.sh DIR (bash) downloads Debian's i386 packages of FPC 3.2.2 (fp-compiler, fp-units-rtl, fp-units-base), unpacks them into DIR/root without installing anything, and writes DIR/fpc: a wrapper that runs that ppc386 with its unit paths, /usr/lib32, and gcc's 32-bit crtbegin.o directory. It needs curl, dpkg-deb and gcc-multilib (32-bit glibc: the compiler is itself an i386 program).

scripts/fpc-i386.sh /opt/fpc-i386
FPC=/opt/fpc-i386/fpc ./build.sh                # out/syc: ELF 32-bit, i386

build-c.sh asks the compiler for its target (-iTO, -iTP) and compiles the C part with gcc -m32. The package version is pinned in the script (3.2.2+dfsg-46); if Debian drops it from its mirror, update it there.

macOS arm64

brew install fpc
./build.sh

The C part goes through cc (clang) with -mmacosx-version-min=11.0, the version FPC links for. What differs from Linux in the code: the stat field st_mtimensec, utimensat's constants (AT_FDCWD = -2, AT_SYMLINK_NOFOLLOW = 0x20), getentropy (256 bytes a call) for randomness, clock_gettime from libc in sylzbench, and clang's __builtin_shufflevector for gcc's __builtin_shuffle. There is no Mac here: the release binaries are CI's (artifact syc-macos-arm64). They are not signed; after a download, xattr -d com.apple.quarantine syc-macos-arm64.

Windows, natively

With FPC's installer (fpc-3.2.2.win32.and.win64.exe) and MSYS2's mingw-w64 gcc, from Git Bash:

./build.sh -Px86_64 -Twin64   # x86-64: fpc.exe is the i386 driver, -Px86_64 runs ppcrossx64
./build.sh -Twin32            # x86: with C:\msys64\mingw32\bin on PATH

Cross compilers (Windows 32-bit, Linux arm64)

The release builds come from an x86-64 Linux host with FPC 3.2.2 cross compilers built from source, no system install needed. On Debian, fpc-source ships without the generated Makefiles and a few files, so:

cp -r /usr/share/fpcsrc/3.2.2 src && cd src

# compiler: Makefiles, messages
(cd compiler && fpcmake -Tall)
(cd rtl && fpcmake -Tall -r)
mkdir -p compiler/msg && cp /usr/lib/x86_64-linux-gnu/fpc/3.2.2/msg/*.msg compiler/msg/
(cd compiler && ppcx64 utils/msg2inc.pp -FEutils && ./utils/msg2inc msg/errore.msg msg msg)

# cross compilers: PPC_TARGET = the CPU they generate code for
make -C compiler compiler PPC_TARGET=i386    FPC=/usr/bin/ppcx64   # -> compiler/ppc386
make -C compiler compiler PPC_TARGET=aarch64 FPC=/usr/bin/ppcx64   # -> compiler/ppca64

# rtl/<cpu>/makefile.cpu is missing too: it only lists the CPU's .inc files
for c in i386 aarch64; do
  printf 'CPUNAMES=%s\nCPUINCNAMES=$(addsuffix .inc,$(CPUNAMES))\n' \
    "$(cd rtl/$c && ls *.inc | sed 's/\.inc$//' | tr '\n' ' ')" > rtl/$c/makefile.cpu
done

# rtl/linux/aarch64/*prt0.as (start-up code) is missing: take it from the
# release_3_2_2 tag of https://gitlab.com/freepascal.org/fpc/source

make -C rtl/win32 all FPC=$PWD/compiler/ppc386 OS_TARGET=win32 CPU_TARGET=i386
make -C rtl/linux all FPC=$PWD/compiler/ppca64 OS_TARGET=linux CPU_TARGET=aarch64 \
  BINUTILSPREFIX=aarch64-linux-gnu-

Then, from the syc checkout:

FPC=$SRC/compiler/ppc386 ./build.sh -Twin32 -Fu$SRC/rtl/units/i386-win32
FPC=$SRC/compiler/ppca64 ./build.sh -XPaarch64-linux-gnu- -Fu$SRC/rtl/units/aarch64-linux

Win32 links with FPC's internal linker; arm64 needs aarch64-linux-gnu-as and -ld (Debian binutils-aarch64-linux-gnu). The arm64 binaries are static and run on an x86-64 host through qemu-user, which is how the fuzzer is run for arm64.

Every target writes the same bytes. Until syc 0.5.1, 32-bit targets hashed levels 0–3 with 32-bit multiplies only (levels 0–1 on 4 bytes instead of 6), so their output differed; since 0.5.1 they hash as the 64-bit ones do (a few percent slower there, within the noise on real Windows 10). Every build reads every other's files.

C

The C part (src/sylzc.c: the decoder and every level's compressor for 32-bit Windows and arm64, where there is no assembler; on x86_64 the parse of levels 4–8, the candidate pass and DP of levels 9–10) is compiled by build-c.sh with the target's C compiler, which build.sh and plugin/build.sh pick from the compiler's target ($FPC -iTO -iTP) and options: gcc (Linux x86-64 and arm64; -fPIC with -Cg), gcc -m32 (Linux i386), cc (clang, macOS; gcc-only flags left out), x86_64-w64-mingw32-gcc (-Twin64), i686-w64-mingw32-gcc (-Twin32), aarch64-linux-gnu-gcc (-XPaarch64-linux-gnu-); CC overrides. It is freestanding (-ffreestanding -fno-builtin -fno-tree-loop-distribute-patterns -fno-stack-protector -fno-asynchronous-unwind-tables), so it needs no libc or libgcc (on 32-bit Linux the 64-bit trailing-zero count is spelled as two 32-bit ones: some gcc versions call libgcc's __ctzdi2 for it, and FPC does not link libgcc), and lands in the object path fpc gets with -Fo; sylz.pas links it with {$L sylzc.o}. On 32-bit x86 the decoder's C is compiled twice, for SSE2 (target("sse2"); +22-27% decode on Windows 7 32-bit) and for plain i686, and picks one with cpuid on its first call: one binary for any x86. -DSYLZC_NO_SSE2 (via CFLAGS_EXTRA) keeps only the i686 one. The compressors stay i686. -dSYLZ_PUREPASCAL builds without it. Rule: C only where Free Pascal's code falls short, never the main language, always next to a Pascal reference.

Assembler paths

On x86-64, src/sylz.pas uses five assembler routines (the decoder's fast loop, the compressor of levels 0–1, the compressor of levels 2–3, the hash-chain search of levels 4–8, the binary tree of levels 9–10) and src/sylzhuf.pas two (format v3's Huffman bulk loop, with BMI2's shrx when cpuid reports it, and without), for both the SysV (Linux) and Microsoft x64 (Win64) calling conventions. Everywhere else, or with -dSYLZ_PUREPASCAL, the Pascal versions run. At run time SylzUseAsm := False switches sylz to the Pascal paths (assembler and C), and on x86_64 SylzPreferC := True takes the C paths instead of the assembler ones (sylzbench: SYLZ_C=1; the fuzzer rotates all three) (sylzbench: SYLZ_NOASM=1), and HufUseAsm / HufUseBmi2 the Huffman decoder (SYLZ_NOBMI2=1). Both paths give the same results, and the assembler and Pascal compressors the same bytes.

Tests

fpc -O3 -Fusrc -FUout/units -FEout tests/sylzfuzz.pas
out/sylzfuzz 3000 SAMPLE-FILES...

Per case: random, small-alphabet, periodic, run-heavy, long-match or real-file input (slices of the sample files), any level, format v2 or v3 at random; roundtrip; compress again with the other code path and compare bytes; the Huffman decoder rotates through Pascal, assembler and assembler with BMI2; decode four corrupted copies and a truncated one, and decode into a buffer one byte short. All buffers sit flush against a PROT_NONE guard page (16 KiB, a multiple of every target's page, Apple arm64's included), so any out-of-bounds access faults. Unix only (mmap).

Every test at once, as CI runs them:

pip install pynacl cryptography argon2-cffi   # the independent implementations
sh .github/test.sh

It builds cryptotest, sycdedup, sylzedge and sylzfuzz next to out/syc and runs:

  • the crypto vectors (RFC 9106, RFC 8439, XChaCha), and a cross-check of the C and Pascal primitives against hashlib, cryptography, libsodium and Argon2 (OpenSSL 3.2 or later, or else argon2-cffi);
  • the dedup tests (BLAKE2s lanes, CDC cutter);
  • the codec edge cases, with and without assembler;
  • the fuzzer;
  • the archiver end to end (tests/arctest.sh);
  • the format vectors, read back by tools/syc_ref.py.

Then it writes hash.txt: the SHA-256 of what syc compresses from a fixed generated sample (.github/sample.py), as streams and raw blocks, both codecs, levels 0–10. FPC names the compiler, FPCOPTS the options out/syc was built with, WINDOWS=1 skips what needs Unix (fuzzer, arctest, vectors).

CI

.github/workflows/ci.yml, on every push and pull request:

job runner build
linux-x64 ubuntu-latest apt-get install fpc, ./build.sh
linux-arm64 ubuntu-24.04-arm the same, natively
linux-x86 ubuntu-latest gcc-multilib, scripts/fpc-i386.sh
macos-arm64 macos-latest brew install fpc
windows-x64 windows-latest FPC's installer, MSYS2 mingw64, -Px86_64 -Twin64
windows-x86 windows-latest FPC's installer, MSYS2 mingw32, -Twin32
pascal ubuntu-latest -dSYLZ_PUREPASCAL
determinism ubuntu-latest compares the seven hash.txt: one value, or it fails

Each build job runs .github/test.sh. macos-arm64 also uploads syc and sylzbench (artifact syc-macos-arm64), which become the release's macOS binaries.

Tools

  • out/sylzbench [-l LEVELS] [-t SECONDS] [-ec] FILE... — in-memory benchmark. SYLZ_DUMP=prefix writes each compressed block to prefix.LEVEL; SYLZ_LOAD=prefix decodes those blocks instead of compressing (decode-only timing; one file per prefix).
  • bench/sylzstats.pas — literal/match/offset histograms and repeat-offset rates of a dumped block.

Legacy Rust

The v0.1.x sources are archived in legacy/rust/syc-rust-v0.1.29.zip:

cd legacy/rust
unzip syc-rust-v0.1.29.zip
cargo build --release
cargo +1.77.2 build --release --target i686-pc-windows-gnu   # Windows 7 32-bit

syc

Clone this wiki locally