Skip to content

NetBSD GitLab

Jump to bottom Edit New page
Henryk Paluch edited this page Feb 27, 2024 · 5 revisions

Installing GitLab on NetBSD

Experiment: installing GitLab from source on NetBSD.

WARNING! Early research project. Not working yet and there is no guarantee that it will ever be working.!

Roadblock:

NetBSD still (including 10RC4) does not support LC_COLLATE for setlocale(3) which is roadblock - GitLab requires full UTF-8 compatible locale support both in PostgreSQL and also for Ruby language.

See http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/string/strcoll.c?rev=1.12&content-type=text/x-cvsweb-markup&only_with_tag=MAIN function strcoll_l...

In case of PostgreSQL it could be possible to workaround it in 16+ version using ICU library. But I don't know of any meaningful workaround for Ruby lang.

Tested environment

  • NetBSD 10.0 RC4/amd64
  • VM with 1 vCPU, 6GB RAM, 40GB disk (Virtio-BLK), rootfs has 32GB and swap has 8GB

General packages installed

pkgin up
pkgin in curl vim mc wget doas
pkgin in mozilla-rootcerts mozilla-rootcerts-openssl

Building GitLab

We will try version 16.6.6 (or any latest version in 16.6.stable branch).

We will more or less follow official guide for Linux

Packages required for building GitLab.

pkgin in gmake bash

Tested versions:

  • bash-5.2.21nb1
  • gmake-4.4.1
  • Packages required for building GitLab.
pkgin in gmake bash pkg-config

Tested versions:

  • bash-5.2.21nb1
  • gmake-4.4.1
  • pkg-config-0.29.2nb2

Building custom Git

First step is to build custom git for GitLab from Gitaly package, where Gitaly version should be same as desired GitLab version - 16.6.6 in our case.

Install system Git needed to bootstrap build of other Git from Gitaly:

pkgin in git
  • tested: git-2.43.0

Additionally we have to emulate nproc command (to fake number of processors = parallel builds)

  • create directory with mkdir -p /usr/local/bin
  • create script /usr/local/bin/nproc with contents: 
    #!/bin/sh
sysctl -n hw.ncpu
exit 0

Let's try build as non-privileged user - OTHER than git (we will need it later):

mkdir -p ~/projects
cd ~/projects/
curl -fLO https://gitlab.com/gitlab-org/gitaly/-/archive/16-6-stable/gitaly-16-6-stable.tar.gz
tar xf gitaly-16-6-stable.tar.gz 
cd gitaly-16-6-stable

git clone -b 16-6-stable https://gitlab.com/gitlab-org/gitaly.git
cd gitaly

# tested following commit
git describe

  v16.6.7-1-g74f0c2735

git branch -v

  * 16-6-stable 74f0c2735 Merge remote-tracking branch 'dev/16-6-stable' into 16-6-stable

# run "nproc" - to verify that it returns proper value - number of CPUs
nproc

# append V=1 for verbose build commands:
gmake build-git GIT_PREFIX=/usr/local
# this needs root privileges with original PATH (including /usr/pkg/bin)
su
gmake install-git GIT_PREFIX=/usr/local
exit # exit root shell

# test version
/usr/local/bin/git --version

git version 2.42.0

Install Ruby

GitLab requires Ruby 3.0.6 or later 3.x with some specific support.

Trying:

pkgin in ruby

Installed: libyaml-0.2.5 ruby-3.2.2 ruby32-base-3.2.2nb3

Install Go

Go 1.20 is Minimum (I expect).

NOTE: Go is build-only dependency (build binaries are always static and they don't depend on any kind of Go runtime).

pkgin in go
# these links are required by GitLab
ln -s /usr/pkg/go121/bin/go* /usr/local/bin/

# test version
/usr/local/bin/go version

  go version go1.21.5 netbsd/amd64

Installed: go-1.21.5 go121-1.21.5nb1

Installing nodejs and yarn

Nodejs 18.17.x recommended (higher version may work and we will see).

pkgin in 'nodejs>=18<19'

Installed: brotli-1.1.0 icu-74.2 libcares-1.23.0 libuv-1.47.0 nghttp3-1.1.0 ngtcp2-1.1.0 nodejs-18.18.2nb3

But yarn will upgrade it anyway...

pkgin in yarn

# yarn-1.22.21 nodejs-21.4.0

Create git user

Main GitLab installation and also SSH access to GitLab will be provided by git user. Create it with:

/usr/sbin/useradd -m -s /usr/pkg/bin/bash git
# TODO: How to lock password in NetBSD?

Install PostgreSQL

GitLab supports PostgreSQL only, current GitLab requires at least Postgres 13.

There exists wiki page for NetBSD:

Available encodings:

Verify that locale exists:

locale -a | fgrep en_US.UTF-8

en_US.UTF-8

Problem specifying UTF-8 locale:

First try postgresql13 (minimum required by GitLab)

pkgin in postgresql13-server postgresql13-client postgresql13-contrib postgresql13-docs
# installed versions 13.13, script below is in -contrib package:
cp /usr/pkg/share/examples/rc.d/pgsql /etc/rc.d
# enable pgsql startup:
echo pgsql=YES >> /etc/rc.conf

Apply this change:

diff -u /usr/pkg/share/examples/rc.d/pgsql /etc/rc.d/pgsql
--- /usr/pkg/share/examples/rc.d/pgsql  2023-12-29 10:38:28.000000000 +0100
+++ /etc/rc.d/pgsql     2024-02-11 08:55:02.843286319 +0100
@@ -71,7 +71,8 @@
                /usr/sbin/chown ${pgsql_user} ${pgsql_home}
                /usr/bin/chgrp ${pgsql_group} ${pgsql_home}
                /bin/chmod 0700 ${pgsql_home}
-               doit="/usr/bin/su -m ${pgsql_user} -c '${command} init ${command_args}'"
+               doit="/usr/bin/su -m ${pgsql_user} -c '${command} init ${command_args} -o --encoding=UTF8 -o --locale=en_US.UTF-8'"
+               echo "$doit"
                eval $doit
        fi
 }

And boom!

/etc/rc.d/pgsql start

Initializing PostgreSQL databases.
/usr/bin/su -m pgsql -c '/usr/pkg/bin/pg_ctl init -w -s -D /usr/pkg/pgsql/data -m fast -l /usr/pkg/pgsql/errlog -o --encoding=UTF8 -o --locale=en_US.UTF-8'
initdb: error: invalid locale name "en_US.UTF-8"
pg_ctl: database system initialization failed

Trying to find what is in source:

WARNING!

Running make target below will build and INSTALL dependencies to your /usr/pkg target directory! It is generally not recommended to mix binary and source packages in single system!

# run as non-privileged user with 'doas' rights
mkdir -p ~/pkgsrc
cd ~/pkgsrc
curl -fLO https://cdn.netbsd.org/pub/pkgsrc/pkgsrc-2023Q4/pkgsrc.tar.xz
tar xpvf pkgsrc.tar.xz 
doas mv pkgsrc /usr/
# WARNING! initdb is actually in package -client
cd /usr/pkgsrc/databases/postgresql13-client
make print-build-depends-list

  This package requires package(s) \
  "bison-3.8.2nb1 cwrappers-20220403 digest-20220214 \
   gettext-lib-0.22.3 gettext-tools-0.22.3 gmake-4.4.1 libtool-base-2.4.7nb1 \
   libxml2-2.10.4nb6 m4-1.4.19nb1 mktools-20220614 perl-5.38.2 xmlcatmgr-2.2nb1" to build.

# trying to skip build of dependencies:
doas pkgin in bison cwrappers digest gettext-lib gettext-tools gmake libtool-base libxml2 m4 mktools perl xmlcatmg

make print-run-depends-list

  This package requires package(s) "libxml2>=2.10.4nb3 libxml2>=2.6.2" to run.

make patch
# be prepared to enter root's password several times...

fgrep -r "invalid locale name" work/

...
work/postgresql-13.13/src/bin/initdb/initdb.c
...


Apply patch:
```diff
diff -up ../postgresql13-server/work/postgresql-13.13/src/bin/initdb/initdb.c.orig work/postgresql-13.13/src/bin/initdb/initdb.c
--- ../postgresql13-server/work/postgresql-13.13/src/bin/initdb/initdb.c.orig   2024-02-11 09:37:17.979738078 +0100
+++ work/postgresql-13.13/src/bin/initdb/initdb.c       2024-02-11 10:06:29.479501757 +0100
@@ -2157,6 +2157,7 @@ check_locale_name(int category, const ch
 
        /* set the locale with setlocale, to see if it accepts it. */
        res = setlocale(category, locale);
+       pg_log_error("res='%s' setlocale(category=%d, locale='%s')", res, category, locale);
 
        /* save canonical name if requested. */
        if (res && canonname)

Now continue:

make build
make install
# provide root password
# but because package is already installed it will be not reinstalled
# have to do this:
doas pkg_add -U /usr/pkgsrc/packages/All/postgresql13-client-13.13.tgz
/etc/rc.d/pgsql start

Initializing PostgreSQL databases.
/usr/bin/su -m pgsql -c '/usr/pkg/bin/pg_ctl init -w -s -D /usr/pkg/pgsql/data -m fast -l /usr/pkg/pgsql/errlog -o --encoding=UTF8 -o --locale=en_US.UTF-8'
initdb: error: res='en_US.UTF-8' setlocale(category=2, locale='en_US.UTF-8')
initdb: error: res='(null)' setlocale(category=1, locale='en_US.UTF-8')
initdb: error: invalid locale name "en_US.UTF-8"

Here are categories from /usr/include/locale.h

#define LC_COLLATE      1
#define LC_CTYPE        2

So NetBSD does not implement LC_COLLATE (res='(null)' means that setlocale(3) failed).

Expected failure

LC_COLLATE support is not yet fully implemented: /work/src/tests/lib/libc/locale/t_wcscoll.c:120: setlocale(LC_COLLATE, t->locale) != NULL not met

Or /usr/src/lib/libc/string/strcoll.c

int
strcoll_l(const char *s1, const char *s2, locale_t loc)
{
        _DIAGASSERT(s1 != NULL);
        _DIAGASSERT(s2 != NULL);

        /* LC_COLLATE is unimplemented, hence always "C" */
        /* LINTED */ (void)loc;
        return (strcmp(s1, s2));
}

There is man setlocale page and category integer values can be seen at /usr/include/locale.h

Unfortunately same issue with latest stable OpenBSD 7.4 (Feb 2024):

/usr/src/lib/libc/string/strcoll_l.c

/*      $OpenBSD: strcoll_l.c,v 1.1 2017/09/05 03:16:13 schwarze Exp $ */
/*
 * Written in 2017 by Ingo Schwarze <schwarze@openbsd.org>.
 * Released into the public domain.
 */

#include <string.h>

int
strcoll_l(const char *s1, const char *s2,
    locale_t locale __attribute__((__unused__)))
{
        return strcmp(s1, s2);
}

The only BSD from trinity (FreeBSD, NetBSD, OpenBSD) that has proper locale support seems to be FreeBSD (looked on version 14, file /usr/src/lib/libc/string/strcoll.c)

See

The only way how to get around this is to use PostgreSQL >=16 that supports ICU (system independent Collate and Unicode library):

  • to use ICU install rather: 
    pkgin in postgresql16-server postgresql16-client postgresql16-contrib postgresql16-docs postgresql-16
cp /usr/pkg/share/examples/rc.d/pgsql /etc/rc.d/pgsql
  • apply this modified patch: 
     diff -u usr/pkg/share/examples/rc.d/pgsql /etc/rc.d/pgsql
--- usr/pkg/share/examples/rc.d/pgsql   2023-12-29 10:46:38.000000000 +0100
+++ /etc/rc.d/pgsql     2024-02-10 10:18:22.065515519 +0100
@@ -75,7 +75,7 @@
                /usr/sbin/chown ${pgsql_user} ${pgsql_home}
                /usr/bin/chgrp ${pgsql_group} ${pgsql_home}
                /bin/chmod 0700 ${pgsql_home}
-               /usr/bin/su -m ${pgsql_user} -c "${command} init ${command_args}"
+               /usr/bin/su -m ${pgsql_user} -c "${command} init ${command_args} -o --locale-provider=icu -o --encoding=UTF8 -o --icu-locale=en_US.utf8"
        fi
 }
  • and initialize DB: 
    /etc/rc.d/pgsql start
  • check locales - not sure if they are correct: 
    su -m pgsql -c "/usr/pkg/bin/psql -l"
                                               List of databases
   Name    | Owner | Encoding | Locale Provider | Collate | Ctype | ICU Locale | ICU Rules | Access privileges 
-----------+-------+----------+-----------------+---------+-------+------------+-----------+-------------------
 postgres  | pgsql | UTF8     | icu             | C       | C     | en-US      |           | 
 template0 | pgsql | UTF8     | icu             | C       | C     | en-US      |           | =c/pgsql         +
           |       |          |                 |         |       |            |           | pgsql=CTc/pgsql
 template1 | pgsql | UTF8     | icu             | C       | C     | en-US      |           | =c/pgsql         +
           |       |          |                 |         |       |            |           | pgsql=CTc/pgsql
(3 rows)

TODO:

  • redis
  • ...

Copyright © Henryk Paluch. All rights reserved.

Creative Commons License This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

Add a custom sidebar
Clone this wiki locally