Fix musl build (#1164) #1177

leso-kn · 2023-07-10T13:12:54Z

As pointed out by @jefferyto the LFS64 interfaces were marked as deprecated in musl 1.2.4 (release notes, commit), thus go-sqlite3 currently fails to compile on musl.

This PR adjusts CLFAGS to let the C implementation use pread()/pwrite() instead of pread64()/pwrite64().

Note: This change does not affect existing glibc builds as suggested by the pread64 man page:

On Linux, the underlying system calls were renamed in kernel 2.6: pread() became pread64(), and pwrite() became pwrite64(). The system call numbers remained the same. The glibc pread() and pwrite() wrapper functions transparently deal with the change.

leso-kn · 2023-07-10T13:14:03Z

see #1164

omerxx · 2023-07-13T21:53:49Z

Honestly unsure how this isn't urgent to users out there but it saved me.
So for what it's worth I added this to my go.mod:

replace github.com/mattn/go-sqlite3 => github.com/leso-kn/go-sqlite3 v0.0.0-20230710125852-03158dc838ed

ptrcnull · 2023-09-10T13:41:43Z

hey @mattn could we get this merged? it's a really small non-breaking change that would fix all musl builds

mattn · 2023-09-10T14:13:08Z

Can you add test for the musl build ?

rittneje · 2023-09-10T14:27:54Z

I had to do some digging here to check whether this change was safe, because on the surface it isn't. pread takes an off_t while pread64 takes an off64_t. So in a vacuum, this would break on systems where off_t is not 64 bits (overflow). (Note that the comment in the man page is referring to how the glibc wrapper invokes the underlying syscalls and is not relevant to this discussion.)

However, SQLite itself also sets _FILE_OFFSET_BITS to 64 (unless you set SQLITE_DISABLE_LFS), which forces off_t to be an alias for off64_t.

I do think there is a bug in the C library though, since it doesn't appear SQLite will attempt to constrain itself to 32 bits if LFS is disabled.

@jefferyto

Bugfixes - Fixed a bug involving non-English locales of Windows where machine installs using user-mode networking were rejected due to erroneous version detection (openwrt#20209). - Fixed a regression in --env-file handling (openwrt#19565). - Fixed a bug where podman inspect would fail when stat'ing a device failed. API - The network list compat API endpoint is now much faster (openwrt#20035). Openwrt updates: added patch to allow building with musl-1.2.4 Patch source is from gentoo https://github.com/vimproved/gentoo/blob/c4c349f11a4352be1965726eadfe3a8bd8a6fa9c/app-containers/podman/files/podman-4.5.0-fix-build-with-musl-1.2.4.patch Issue was discussed by @jefferyto at mattn/go-sqlite3#1177 Signed-off-by: Oskari Rauta <oskari.rauta@gmail.com>

@jefferyto

Bugfixes - Fixed a bug involving non-English locales of Windows where machine installs using user-mode networking were rejected due to erroneous version detection (#20209). - Fixed a regression in --env-file handling (#19565). - Fixed a bug where podman inspect would fail when stat'ing a device failed. API - The network list compat API endpoint is now much faster (#20035). Openwrt updates: added patch to allow building with musl-1.2.4 Patch source is from gentoo https://github.com/vimproved/gentoo/blob/c4c349f11a4352be1965726eadfe3a8bd8a6fa9c/app-containers/podman/files/podman-4.5.0-fix-build-with-musl-1.2.4.patch Issue was discussed by @jefferyto at mattn/go-sqlite3#1177 Signed-off-by: Oskari Rauta <oskari.rauta@gmail.com>

@jefferyto

Bugfixes - Fixed a bug involving non-English locales of Windows where machine installs using user-mode networking were rejected due to erroneous version detection (openwrt#20209). - Fixed a regression in --env-file handling (openwrt#19565). - Fixed a bug where podman inspect would fail when stat'ing a device failed. API - The network list compat API endpoint is now much faster (openwrt#20035). Openwrt updates: added patch to allow building with musl-1.2.4 Patch source is from gentoo https://github.com/vimproved/gentoo/blob/c4c349f11a4352be1965726eadfe3a8bd8a6fa9c/app-containers/podman/files/podman-4.5.0-fix-build-with-musl-1.2.4.patch Issue was discussed by @jefferyto at mattn/go-sqlite3#1177 Signed-off-by: Oskari Rauta <oskari.rauta@gmail.com> (cherry picked from commit e25d417)

@jefferyto

Bugfixes - Fixed a bug involving non-English locales of Windows where machine installs using user-mode networking were rejected due to erroneous version detection (openwrt#20209). - Fixed a regression in --env-file handling (openwrt#19565). - Fixed a bug where podman inspect would fail when stat'ing a device failed. API - The network list compat API endpoint is now much faster (openwrt#20035). Openwrt updates: added patch to allow building with musl-1.2.4 Patch source is from gentoo https://github.com/vimproved/gentoo/blob/c4c349f11a4352be1965726eadfe3a8bd8a6fa9c/app-containers/podman/files/podman-4.5.0-fix-build-with-musl-1.2.4.patch Issue was discussed by @jefferyto at mattn/go-sqlite3#1177 remarks: removed musl-1.2.4 patch from commit, since that version of musl is not available with openwrt-23.05 Signed-off-by: Oskari Rauta <oskari.rauta@gmail.com> (cherry picked from commit e25d417)

@jefferyto

Bugfixes - Fixed a bug involving non-English locales of Windows where machine installs using user-mode networking were rejected due to erroneous version detection (#20209). - Fixed a regression in --env-file handling (#19565). - Fixed a bug where podman inspect would fail when stat'ing a device failed. API - The network list compat API endpoint is now much faster (#20035). Openwrt updates: added patch to allow building with musl-1.2.4 Patch source is from gentoo https://github.com/vimproved/gentoo/blob/c4c349f11a4352be1965726eadfe3a8bd8a6fa9c/app-containers/podman/files/podman-4.5.0-fix-build-with-musl-1.2.4.patch Issue was discussed by @jefferyto at mattn/go-sqlite3#1177 remarks: removed musl-1.2.4 patch from commit, since that version of musl is not available with openwrt-23.05 Signed-off-by: Oskari Rauta <oskari.rauta@gmail.com> (cherry picked from commit e25d417)

@jefferyto

Bugfixes - Fixed a bug involving non-English locales of Windows where machine installs using user-mode networking were rejected due to erroneous version detection (openwrt#20209). - Fixed a regression in --env-file handling (openwrt#19565). - Fixed a bug where podman inspect would fail when stat'ing a device failed. API - The network list compat API endpoint is now much faster (openwrt#20035). Openwrt updates: added patch to allow building with musl-1.2.4 Patch source is from gentoo https://github.com/vimproved/gentoo/blob/c4c349f11a4352be1965726eadfe3a8bd8a6fa9c/app-containers/podman/files/podman-4.5.0-fix-build-with-musl-1.2.4.patch Issue was discussed by @jefferyto at mattn/go-sqlite3#1177 Signed-off-by: Oskari Rauta <oskari.rauta@gmail.com>

ruzv · 2023-12-08T08:55:37Z

any progress update on this?
musl 1.2.4 was released in May 1, 2023

mattn · 2023-12-08T09:02:23Z

src/os_unix.c have this code. so I guess that this change should work well. @rittneje what do you think?

/* Use pread() and pwrite() if they are available */
#if defined(HAVE_PREAD64) && defined(HAVE_PWRITE64)
# undef USE_PREAD
# undef USE_PWRITE
# define USE_PREAD64 1
# define USE_PWRITE64 1
#elif defined(HAVE_PREAD) && defined(HAVE_PWRITE)
# undef USE_PREAD
# undef USE_PWRITE
# define USE_PREAD64 1
# define USE_PWRITE64 1
#endif

leso-kn · 2023-12-08T15:26:35Z

Thanks for the feedback @mattn, I've changed the patch to your version!

mattn · 2023-12-09T15:45:42Z

Thanks for the feedback @mattn, I've changed the patch to your version!

Sorry, I made you confused. I just said that we can merge previous version of those changes.

mattn · 2023-12-09T15:54:23Z

BTW, now I could reproduce this in https://github.com/mattn/nostr-relay/actions/runs/7151889396/job/19476647857

and confirmed with

#cgo linux,!android CFLAGS: -DHAVE_PREAD=1 -DHAVE_PWRITE=1 -DHAVE_PREAD64=1 -DHAVE_PWRITE64=1

https://github.com/mattn/nostr-relay/pull/2/files#diff-89278323f4af3d0fc58cd9680b5fa55549b914cb147c5f69b9cc4d06d26a780bR24

mattn · 2023-12-09T16:12:29Z

Or we might be possible to remove this line. (not make sure)

mattn/go-sqlite3#1164 mattn/go-sqlite3#1177

james-d-elliott · 2023-12-11T01:18:47Z

For reference if anyone is having this issue in their CI/CD pipelines this affects the latest published golang alpine container image golang:1.21.5-alpine but doesn't affect golang:1.21.4-alpine. I suspect the 1.21.5 image was republished with the latest alpine/musl. It's also fairly likely that it'll affect the latest patch of each supported go version such as 1.20.12 as they likely republish these when the base image changes.

Edit: Looks like a golang:1.21.5-alpine3.18 tag exists. This alleviates the issue with container builds.

Temporary fix for mattn/go-sqlite3#1177 which will be reverted in the near future as that's merged. Signed-off-by: James Elliott <james-d-elliott@users.noreply.github.com>

mattn · 2023-12-13T14:23:05Z

Looks good to me.

I checked this change in mattn/nostr-relay#2

leso-kn · 2023-12-14T18:49:43Z

Rebased and switched back to the #ifdef-variant

mattn · 2023-12-15T01:23:17Z

LGTM, Thank you.

ruzv · 2023-12-15T09:11:43Z

Is there an ETA for the next release that would include this fix?

james-d-elliott · 2023-12-15T09:14:13Z

-7 hours ;) https://github.com/mattn/go-sqlite3/releases/tag/v1.14.19

go-sqlite3 does not build with musl 1.2.4+, which is packaged in some distros (e.g. Alpine 3.19+). This update fixes it. More info: mattn/go-sqlite3#1177

The commit 031e41a is a temporary workaround to fix musl build error, and now to remove `CGO_CFLAGS="-D_LARGEFILE64_SOURCE"` for building docker image, because go-sqlite3 library was upgraded to `1.14.19` [^1] and the musl build was fixed in go-sqlite3 version `1.14.19` [^2]. This reverts commit 031e41a (refer PR #234). --- [^1]: commit hash ad6673b [^2]: mattn/go-sqlite3#1177

Revert "Fix build docker with sqlite3 musl error (#234)" commit. The commit 031e41a is a temporary workaround to fix musl build error, and now to remove `CGO_CFLAGS="-D_LARGEFILE64_SOURCE"` for building docker image, because go-sqlite3 library was upgraded to `1.14.19` [^1] and the musl build was fixed in go-sqlite3 version `1.14.19` [^2]. This reverts commit 031e41a (refer PR #234). --- [^1]: commit hash ad6673b [^2]: mattn/go-sqlite3#1177

…llowing mattn/go-sqlite3#1177

* build(deps): bump golang from 1.21.3-alpine to 1.21.5-alpine Bumps golang from 1.21.3-alpine to 1.21.5-alpine. --- updated-dependencies: - dependency-name: golang dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): update go-sqlite3 to resolve pread/pwrite buile issue following mattn/go-sqlite3#1177 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Idan Frimark <idanf@cisco.com>

see mattn#1177

use same workaround as authelia/authelia#6404 before mattn#1177 fixes the build otherwise

see mattn#1177

use same workaround as authelia/authelia#6404 before mattn#1177 fixes the build otherwise

omerxx mentioned this pull request Jul 13, 2023

Build failure with musl 1.2.4 #1164

Closed

leso-kn force-pushed the pread-pwrite-musl branch from 03158dc to 65d6085 Compare July 15, 2023 10:55

vimproved mentioned this pull request Jul 15, 2023

app-containers/{lxd,podman}: Fix build on musl 1.2.4 gentoo/gentoo#31890

Closed

oskarirauta mentioned this pull request Oct 13, 2023

podman: update to 4.7.1 openwrt/packages#22390

Merged

leso-kn force-pushed the pread-pwrite-musl branch from 65d6085 to 798da2f Compare December 8, 2023 15:20

Fix musl build (mattn#1164)

3cb5556

leso-kn force-pushed the pread-pwrite-musl branch from 798da2f to 55b17a8 Compare December 8, 2023 15:22

sentriz added a commit to sentriz/gonic that referenced this pull request Dec 10, 2023

fix(docker): fix sqlite3 musl build

433829d

mattn/go-sqlite3#1164 mattn/go-sqlite3#1177

james-d-elliott mentioned this pull request Dec 11, 2023

build: explicitly use alpine 3.18 authelia/authelia#6404

Merged

tolzhabayev mentioned this pull request Dec 14, 2023

Chore: Upgrade Go to 1.21.5 grafana/grafana#79329

Merged

leso-kn force-pushed the pread-pwrite-musl branch from 65d6085 to 3cb5556 Compare December 14, 2023 18:48

mattn merged commit 00b02e0 into mattn:master Dec 15, 2023
11 checks passed

OstlerDev mentioned this pull request Dec 20, 2023

Fix Docker Build (issue with go-sqlite3) statping-ng/statping-ng#259

Merged

emersion pushed a commit to emersion/soju that referenced this pull request Dec 21, 2023

Upgrade to go-sqlite3 to fix musl build

c5d0765

go-sqlite3 does not build with musl 1.2.4+, which is packaged in some distros (e.g. Alpine 3.19+). This update fixes it. More info: mattn/go-sqlite3#1177

bmon mentioned this pull request Dec 22, 2023

update sql-migrate rubenv/sql-migrate#259

Merged

joshuaghezzi mentioned this pull request Dec 22, 2023

Fix temporary go-sqlite3 build error mx51/data-dictionary-action#21

Merged

FrimIdan added a commit to openclarity/kubeclarity that referenced this pull request Dec 26, 2023

build(deps): update go-sqlite3 to resolve pread/pwrite buile issue fo…

e7814e2

…llowing mattn/go-sqlite3#1177

FrimIdan added a commit to openclarity/kubeclarity that referenced this pull request Dec 26, 2023

build(deps): update go-sqlite3 to resolve pread/pwrite buile issue fo…

47257e4

…llowing mattn/go-sqlite3#1177

grabpot mentioned this pull request Dec 30, 2023

Docker based setup for self hosting nkanaev/yarr#173

Closed

yvardhineni mentioned this pull request Jan 4, 2024

Bump github.com/mattn/go-sqlite3 from 1.14.17 to 1.14.19 go-gorm/sqlite#175

Closed

phikal mentioned this pull request Jan 4, 2024

Update Dockerfile. Add build ARG to avoid ('pread64' undeclared ) error KWARC/kalah-game#1

Closed

rhafer mentioned this pull request Jan 8, 2024

Removed deprecated envars scheduled for removal with 5.0.0 owncloud/ocis#8149

Merged

tolzhabayev mentioned this pull request Jan 17, 2024

Chore: Bumping go to 1.21.6 grafana/grafana#80709

Merged

3 tasks

jgiannuzzi pushed a commit to jgiannuzzi/go-sqlite3 that referenced this pull request Jan 22, 2024

use locally checked out lib in the docker job

9f71a3a

see mattn#1177

jgiannuzzi pushed a commit to jgiannuzzi/go-sqlite3 that referenced this pull request Jan 22, 2024

temporary fix to make the job pass

b34e636

use same workaround as authelia/authelia#6404 before mattn#1177 fixes the build otherwise

jgiannuzzi pushed a commit to jgiannuzzi/go-sqlite3 that referenced this pull request Jan 22, 2024

use locally checked out lib in the docker job

329cd42

see mattn#1177

jgiannuzzi pushed a commit to jgiannuzzi/go-sqlite3 that referenced this pull request Jan 22, 2024

temporary fix to make the job pass

d6feb2b

use same workaround as authelia/authelia#6404 before mattn#1177 fixes the build otherwise

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix musl build (#1164) #1177

Fix musl build (#1164) #1177

leso-kn commented Jul 10, 2023

leso-kn commented Jul 10, 2023

omerxx commented Jul 13, 2023

ptrcnull commented Sep 10, 2023

mattn commented Sep 10, 2023

rittneje commented Sep 10, 2023

ruzv commented Dec 8, 2023

mattn commented Dec 8, 2023

leso-kn commented Dec 8, 2023

mattn commented Dec 9, 2023

mattn commented Dec 9, 2023

mattn commented Dec 9, 2023

james-d-elliott commented Dec 11, 2023 •

edited

mattn commented Dec 13, 2023

leso-kn commented Dec 14, 2023

mattn commented Dec 15, 2023

ruzv commented Dec 15, 2023

james-d-elliott commented Dec 15, 2023

Fix musl build (#1164) #1177

Fix musl build (#1164) #1177

Conversation

leso-kn commented Jul 10, 2023

leso-kn commented Jul 10, 2023

omerxx commented Jul 13, 2023

ptrcnull commented Sep 10, 2023

mattn commented Sep 10, 2023

rittneje commented Sep 10, 2023

ruzv commented Dec 8, 2023

mattn commented Dec 8, 2023

leso-kn commented Dec 8, 2023

mattn commented Dec 9, 2023

mattn commented Dec 9, 2023

mattn commented Dec 9, 2023

james-d-elliott commented Dec 11, 2023 • edited

mattn commented Dec 13, 2023

leso-kn commented Dec 14, 2023

mattn commented Dec 15, 2023

ruzv commented Dec 15, 2023

james-d-elliott commented Dec 15, 2023

james-d-elliott commented Dec 11, 2023 •

edited