Add GCC and GNU Make to the Nix flake development environment so that ruff can be compiled.

[reivilibre]

I found that compiling/installing ruff needed cc as a linker and make.

Adding these got me away and running.

Base: develop

Original commit schedule, with full messages:

1. Add gcc and GNU make to the Nix flake

[reivilibre]

I found this is also needed for Complement, otherwise Go silently disables CGO and then silently ignores files with import "C" like the bindings to libolm.

[reivilibre]

Andrew seems to have some problem with enabling GCC in the flake breaking his external binaries from the system. Not clear how we fix this...

[sumnerevans]

I needed gcc for pyicu to install correctly. This is the error that I was getting before:

direnv: loading ~/projects/github.com/sumnerevans/beeper/github.com/beeper/synapse/.envrc
direnv: using flake . --impure
direnv: nix-direnv: renewed cache
Installing dependencies from lock file
direnv: ([/nix/store/4c3k16mbpv8idz418pwcai5in552qr4v-direnv-2.32.3/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up.

Package operations: 1 install, 0 updates, 0 removals

  • Installing pyicu (2.11): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel

  (running 'icu-config --version')

  Building PyICU 2.11 for ICU 73.2 (max ICU major version supported: 73)

  (running 'icu-config --cxxflags --cppflags')
  Adding CFLAGS="-std=c++11 -I/nix/store/db79xky20dvsvs5546ffpv5db70yjlxq-icu4c-73.2-dev/include" from /nix/store/59ly20xxpsii9yl9g2xq13by7kymkzw1-devenv-profile/bin/icu-config
  (running 'icu-config --ldflags')
  Adding LFLAGS="-L/nix/store/6i4vwgsiqp8jd8pb8cgf99sx8081rmsj-icu4c-73.2/lib -licui18n -licuuc -licudata" from /nix/store/59ly20xxpsii9yl9g2xq13by7kymkzw1-devenv-profile/bin/icu-config
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-310
  creating build/lib.linux-x86_64-cpython-310/icu
  copying py/icu/__init__.py -> build/lib.linux-x86_64-cpython-310/icu
  running build_ext
  building 'icu._icu_' extension
  creating build/temp.linux-x86_64-cpython-310
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/nhnxylvmaisx7pw8pyyx3a8cq93xanpx-libxcrypt-4.4.36/include -fPIC -I/tmp/tmppbia3ezg/.venv/include -I/nix/store/jhflvwr40xbb0xr6jx4311icp9cym1fp-python3-3.10.12/include/python3.10 -c _icu_.cpp -o build/temp.linux-x86_64-cpython-310/_icu_.o -std=c++11 -std=c++11 -I/nix/store/db79xky20dvsvs5546ffpv5db70yjlxq-icu4c-73.2-dev/include -DPYICU_VER=\"2.11\" -DPYICU_ICU_MAX_VER=\"73\"
  error: command 'gcc' failed: No such file or directory


  at /nix/store/2l4y0jspbrs0ii3wh038lfffnllx3r0w-python3.10-poetry-1.5.1/lib/python3.10/site-packages/poetry/installation/chef.py:147 in _prepare
      143│
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│
      146│             if error is not None:
    → 147│                 raise error from None
      148│
      149│             return path
      150│
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyicu (2.11) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "pyicu (==2.11)"'.

Poetry install failed. Run 'poetry install' manually.
direnv: export +C_INCLUDE_PATH +DEVENV_DOTFILE +DEVENV_PROFILE +DEVENV_ROOT +DEVENV_STATE +GOPATH +GOROOT +IN_NIX_SHELL +LD_LIBRARY_PATH +LIBRARY_PATH +PERL5LIB +PGDATA +PGHOST +PGPORT +PKG_CONFIG_PATH +POETRY_INSTALLER_NO_BINARY +POETRY_VIRTUALENVS_CREATE +POETRY_VIRTUALENVS_IN_PROJECT +POETRY_VIRTUALENVS_PATH +PYTHONPATH +REDISDATA +STARSHIP_SESSION_KEY +STARSHIP_SHELL +VIRTUAL_ENV +name ~PATH ~XDG_CONFIG_DIRS ~XDG_DATA_DIRS

I didn't have any errors installing ruff.

[anoadragon453]

Andrew seems to have some problem with enabling GCC in the flake breaking his external binaries from the system. Not clear how we fix this...

This appears to only occur if your system packages are compiled with a different glibc version what the development environment pulls in by including gcc. This is likely for the vast majority of distros, but unlikely for @reivilibre and I who are running NixOS unstable on our dev machines, which the development environment also uses. Testing in an old Ubuntu 20.04 VM makes the problem clear however.

synapse on  rei/flake_gcc_etc is 📦 v1.90.0rc1 via 🐍 v3.10.12 (matrix-synapse-py3.10) via 🦀 v1.70.0 via ❄️  impure (devenv-shell-env) 
❯ sudo chmod a+r /etc/apt/keyrings/docker.gpg ^C

 -o /etc/apt/keyrings/docker.gpgcker.com/linux/ubuntu/gpg | sudo gpg --dearmor 
curl: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /nix/store/gry70n80hgvpfsgr2zihw69ffg6knnd8-devenv-profile/lib/libsqlite3.so.0)
curl: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /nix/store/gry70n80hgvpfsgr2zihw69ffg6knnd8-devenv-profile/lib/libsqlite3.so.0)
curl: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libm.so.6)
File '/etc/apt/keyrings/docker.gpg' exists. Overwrite? (y/N) y
gpg: no valid OpenPGP data found.

@reivilibre discovered that if we unset LD_LIBRARY_PATH that devenv sets, then we seem to fix the GLIBC errors with no adverse effects. For reference, devenv sets my LD_LIBRARY_PATH env var to the absolute path of .devenv/profile/lib:. That directory contains the following libraries installed by devenv:

Click to show contents of `.devenv/profile/lib`

user@pc:~/synapse$ ls .devenv/profile/lib
adminpack.so                         libssl.so.3
amcheck.so                           libstd-0d91c78a7710ed2e.so
auth_delay.so                        libtest-deccc6ea001455e8.so
auto_explain.so                      libturbojpeg.so
autoinc.so                           libturbojpeg.so.0
bloom.so                             libturbojpeg.so.0.2.0
btree_gin.so                         libwebpdecoder.la
btree_gist.so                        libwebpdecoder.so
citext.so                            libwebpdecoder.so.3
cmake                                libwebpdecoder.so.3.1.7
cube.so                              libwebpdemux.la
cyrillic_and_mic.so                  libwebpdemux.so
dblink.so                            libwebpdemux.so.2
debug                                libwebpdemux.so.2.0.13
dict_int.so                          libwebp.la
dict_snowball.so                     libwebpmux.la
dict_xsyn.so                         libwebpmux.so
earthdistance.so                     libwebpmux.so.3
engines-3                            libwebpmux.so.3.0.12
euc2004_sjis2004.so                  libwebp.so
euc_cn_and_mic.so                    libwebp.so.7
euc_jp_and_sjis.so                   libwebp.so.7.1.7
euc_kr_and_mic.so                    libxml2.la
euc_tw_and_big5.so                   libxml2.so
file_fdw.so                          libxml2.so.2
fuzzystrmatch.so                     libxml2.so.2.11.4
girepository-1.0                     libxmlsec1-gcrypt.la
hstore.so                            libxmlsec1-gcrypt.so
icu                                  libxmlsec1-gcrypt.so.1
insert_username.so                   libxmlsec1-gcrypt.so.1.2.34
_int.so                              libxmlsec1-gnutls.la
isn.so                               libxmlsec1-gnutls.so
latin2_and_win1250.so                libxmlsec1-gnutls.so.1
latin_and_mic.so                     libxmlsec1-gnutls.so.1.2.34
libcrypto.so                         libxmlsec1.la
libcrypto.so.3                       libxmlsec1-nss.la
libecpg_compat.so                    libxmlsec1-nss.so
libecpg_compat.so.3                  libxmlsec1-nss.so.1
libecpg_compat.so.3.14               libxmlsec1-nss.so.1.2.34
libecpg.so                           libxmlsec1-openssl.la
libecpg.so.6                         libxmlsec1-openssl.so
libecpg.so.6.14                      libxmlsec1-openssl.so.1
libexslt.la                          libxmlsec1-openssl.so.1.2.34
libexslt.so                          libxmlsec1.so
libexslt.so.0                        libxmlsec1.so.1
libexslt.so.0.8.21                   libxmlsec1.so.1.2.34
libffi.la                            libxslt.la
libffi.so                            libxslt-plugins
libffi.so.8                          libxslt.so
libffi.so.8.1.2                      libxslt.so.1
libicudata.so                        libxslt.so.1.1.38
libicudata.so.73                     lo.so
libicudata.so.73.2                   ltree.so
libicui18n.so                        moddatetime.so
libicui18n.so.73                     old_snapshot.so
libicui18n.so.73.2                   ossl-modules
libicuio.so                          pageinspect.so
libicuio.so.73                       passwordcheck.so
libicuio.so.73.2                     perl5
libicutest.so                        pg_buffercache.so
libicutest.so.73                     pgcrypto.so
libicutest.so.73.2                   pg_freespacemap.so
libicutu.so                          pgoutput.so
libicutu.so.73                       pg_prewarm.so
libicutu.so.73.2                     pgrowlocks.so
libicuuc.so                          pg_stat_statements.so
libicuuc.so.73                       pgstattuple.so
libicuuc.so.73.2                     pg_surgery.so
libjpeg.so                           pg_trgm.so
libjpeg.so.62                        pg_visibility.so
libjpeg.so.62.3.0                    pgxml.so
libLLVM-16-rust-1.71.0-stable.so     pgxs
libnotify.so                         pkgconfig
libnotify.so.4                       plpgsql.so
libnotify.so.4.0.0                   postgres_fdw.so
libolm.so                            python3.10
libolm.so.3                          refint.so
libolm.so.3.2.15                     rustlib
libpgcommon.a                        seg.so
libpgcommon_shlib.a                  sslinfo.so
libpgfeutils.a                       tablefunc.so
libpgport.a                          tcn.so
libpgport_shlib.a                    test_decoding.so
libpgtypes.so                        tsm_system_rows.so
libpgtypes.so.3                      tsm_system_time.so
libpgtypes.so.3.14                   unaccent.so
libpq.so                             utf8_and_big5.so
libpq.so.5                           utf8_and_cyrillic.so
libpq.so.5.14                        utf8_and_euc2004.so
libpqwalreceiver.so                  utf8_and_euc_cn.so
libpqxx-7.7.so                       utf8_and_euc_jp.so
libpqxx.la                           utf8_and_euc_kr.so
libpqxx.so                           utf8_and_euc_tw.so
libpython3.10.so                     utf8_and_gb18030.so
libpython3.10.so.1.0                 utf8_and_gbk.so
libpython3.so                        utf8_and_iso8859_1.so
librustc_driver-8196f181ccc4f9f6.so  utf8_and_iso8859.so
libsharpyuv.la                       utf8_and_johab.so
libsharpyuv.so                       utf8_and_sjis2004.so
libsharpyuv.so.0                     utf8_and_sjis.so
libsharpyuv.so.0.0.1                 utf8_and_uhc.so
libsqlite3.la                        utf8_and_win.so
libsqlite3.so                        uuid-ossp.so
libsqlite3.so.0                      xmlsec1Conf.sh
libsqlite3.so.0.8.6                  xsltConf.sh
libssl.so

The theory is that if LD_LIBRARY_PATH points to libraries that are built with the pulled in gcc, such as libsqlite3.so.0, then we're effectively injecting separate and unexpected versions of libraries when running system tools. Those tools were compiled with a different glibc version, so we see version 'GLIBC_X.YY' not found errors. Also see this upstream devenv issue: cachix/devenv#555

As a workaround, we can add the following enterShell directive to flake.nix:

# Clear the LD_LIBRARY_PATH environment variable on shell init.
#
# By default, devenv will set LD_LIBRARY_PATH to point to .devenv/profile/lib. This causes
# issues when we include `gcc` as a dependency to build C libraries, as the version of glibc
# that the development environment's cc compiler uses may differ from that of the system.
#
# When LD_LIBRARY_PATH is set, system tools will attempt to use the development environment's
# libraries. Which, when built against an different glibc version lead, to "version 'GLIBC_X.YY' not
# found" errors.
enterShell = ''
    unset LD_LIBRARY_PATH
'';

which stops the injection. Surprisingly, the tools that do need these libraries (sqlite3) still work fine in the development environment. I assume they're finding the built libraries using a different method (pkg-config perhaps).

[reivilibre]

@reivilibre discovered that if we unset LD_LIBRARY_PATH that devenv sets

(it's because I only run NixOS stable and the flake is usually based on some random commit from master that I had to do this, fwiw. It hasn't been causing me any trouble yet as far as I've seen!)

[reivilibre]

aahhhh, seems there's something called an RPATH (runtime library search path) embedded in a binary and that the Nix-built binaries usually point to their libraries in there, but LD_LIBRARY_PATH overrides that and that's why unsetting it fixes it.

NixOS/nix#902 (comment)

[anoadragon453]

lgtm other than a couple tiny nits. Thank you for doing this :)

[anoadragon453]

Suggested change

	# GCC includes a linker; needed for building `ruff`
	# GCC includes a linker; needed for building `ruff`

I've been trying to separate distinct sections with a space :)

[anoadragon453]

Correcting past me.

Suggested change

	# libraries. Which, when built against an different glibc version lead, to "version 'GLIBC_X.YY' not
	# found" errors.
	# libraries. Which, when built against a different glibc version lead, to "version 'GLIBC_X.YY'
	# not found" errors.

[anoadragon453]

@reivilibre did you mean to skip the suggestions?

[reivilibre]

nope, that was a mistake :S