`panic: resource temporarily unavailable` on SunOS Elle 5.11 tribblix-m35 i86pc i386 i86pc

[tsmethurst]

A GoToSocial user is trying some wacky deployment types and is encountering some interesting issues trying to run GtS on the abovementioned operating system. They managed to get some logs out of it which seem pretty interesting!

panic: resource temporarily unavailable
 
goroutine 6 [running]:
github.com/ncruces/go-sqlite3/internal/alloc.Virtual(0x3?, 0x100000000)
        github.com/ncruces/go-sqlite3@v0.19.0/internal/alloc/alloc_unix.go:27 +0xd2
github.com/tetratelabs/wazero/experimental.MemoryAllocatorFunc.Allocate(0x260e660?, 0x40df01?, 0xc0336c3250?)
        github.com/tetratelabs/wazero@v1.8.1/experimental/memory.go:28 +0x1c
github.com/tetratelabs/wazero/internal/wasm.NewMemoryInstance(0xc001cc8110, {0x2f33a60?, 0x2ab5e78?}, {0x2f4dce8, 0xc0334c76c0})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/memory.go:78 +0xa8
github.com/tetratelabs/wazero/internal/wasm.(*ModuleInstance).buildMemory(0xc0337a0600, 0xc0016fa4e0, {0x2f33a60?, 0x2ab5e78?})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/module.go:653 +0x4c
github.com/tetratelabs/wazero/internal/wasm.(*Store).instantiate(0xc000116fc0, {0x2f3f738, 0xc0337dbd40}, 0xc0016fa4e0, {0x0, 0x0}, 0xc033b8e2c0, {0xc001f0c000, 0x3a, 0x3a})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/store.go:370 +0x365
github.com/tetratelabs/wazero/internal/wasm.(*Store).Instantiate(0xc000116fc0, {0x2f3f738, 0xc0337dbd40}, 0x30?, {0x0?, 0x30?}, 0xc00006e908?, {0xc001f0c000, 0x3a, 0x3a})
        github.com/tetratelabs/wazero@v1.8.1/internal/wasm/store.go:327 +0x5a
github.com/tetratelabs/wazero.(*runtime).InstantiateModule(0xc0016f8720, {0x2f3f738, 0xc0337dbd40}, {0x2f48bd0, 0xc001f0a040}, {0x2f56a40, 0xc0051e8d20})
        github.com/tetratelabs/wazero@v1.8.1/runtime.go:318 +0x1e5
github.com/ncruces/go-sqlite3.instantiateSQLite()
        github.com/ncruces/go-sqlite3@v0.19.0/sqlite.go:99 +0x2be
github.com/ncruces/go-sqlite3.newConn({0x2f3f700, 0x60d8460}, {0xc000dc25a0, 0x9b}, 0x46)
        github.com/ncruces/go-sqlite3@v0.19.0/conn.go:76 +0xb4
github.com/ncruces/go-sqlite3.OpenContext(...)
        github.com/ncruces/go-sqlite3@v0.19.0/conn.go:49
github.com/ncruces/go-sqlite3/driver.(*connector).Connect(0xc0016bae10, {0x2f3f700, 0x60d8460})
        github.com/ncruces/go-sqlite3@v0.19.0/driver/driver.go:212 +0xeb
github.com/superseriousbusiness/gotosocial/internal/db/sqlite.(*sqliteConnector).Connect(0x2e5337dbad0?, {0x2f3f700?, 0x60d8460?})
        github.com/superseriousbusiness/gotosocial/internal/db/sqlite/driver.go:64 +0x2a
database/sql.(*DB).conn(0xc0016d8c30, {0x2f3f700, 0x60d8460}, 0x1)
        database/sql/sql.go:1415 +0x71e
database/sql.(*DB).query(0xc0016d8c30, {0x2f3f700, 0x60d8460}, {0xc033b9a000, 0x19}, {0x0, 0x0, 0x0}, 0x0?)
        database/sql/sql.go:1749 +0x57
database/sql.(*DB).QueryContext.func1(0xb0?)
        database/sql/sql.go:1732 +0x4f
database/sql.(*DB).retry(0x411c01?, 0xc0336c3b90)
        database/sql/sql.go:1566 +0x42
database/sql.(*DB).QueryContext(0x0?, {0x2f3f700?, 0x60d8460?}, {0xc033b9a000?, 0x0?}, {0x0?, 0x2f3f700?, 0x60d8460?})
        database/sql/sql.go:1731 +0xc5
github.com/uptrace/bun.(*baseQuery).scan(0xc033ab83c0, {0x2f3f700?, 0x60d8460?}, {0x2f40128?, 0xc033ab83c0?}, {0xc033b9a000, 0x19}, {0x2f368a0, 0xc033753720}, 0x1)
        github.com/uptrace/bun@v1.2.1/query_base.go:562 +0x10c
github.com/uptrace/bun.(*SelectQuery).Scan(0xc033ab83c0, {0x2f3f700, 0x60d8460}, {0xc03379b150?, 0x25aca40?, 0x58?})
        github.com/uptrace/bun@v1.2.1/query_select.go:885 +0x165
github.com/superseriousbusiness/gotosocial/internal/db/bundb.(*applicationDB).GetAllTokens(0xc0053c7370, {0x2f3f700, 0x60d8460})
        github.com/superseriousbusiness/gotosocial/internal/db/bundb/application.go:142 +0x42d
github.com/superseriousbusiness/gotosocial/internal/oauth.(*tokenStore).sweep(0xc0000558a0, {0x2f3f700, 0x60d8460})
        github.com/superseriousbusiness/gotosocial/internal/oauth/tokenstore.go:71 +0x3b
github.com/superseriousbusiness/gotosocial/internal/oauth.newTokenStore.func1({0x2f3f700, 0x60d8460}, 0xc0000558a0)
        github.com/superseriousbusiness/gotosocial/internal/oauth/tokenstore.go:58 +0xfb
created by github.com/superseriousbusiness/gotosocial/internal/oauth.newTokenStore in goroutine 1
        github.com/superseriousbusiness/gotosocial/internal/oauth/tokenstore.go:49 +0xf1

Do you have any idea what might be causing this? Is there maybe some config change they could make?

[NyaaaWhatsUpDoc]

commenting here so i get notified for this thread, and because it's a frankly fascinating setup :p

though i would not be surprised if wazero ended up being the limiting factor in this situation, rather than SQLite

[Here-Be-Saoirse]

we're the user doing terrifying things with Solaris! Yeah, it seemed to work for a while, then eventually something went horribly wrong and everything detonated. Also got "slow database query" warnings in the GTS logs.

[ncruces]

So is this a Solaris 11, or an illumos OS?

The problem with Solaris, is that:

1. it doesn't support OFD locks (to be frank, only Linux does, and on macOS they're a private API)
2. their BSD flock “compatiblity” implementation is broken.

Given that I'm personally uninterested in implementing POSIX locks, the result will always be rather limited. I've tried to come up with some possible simplifications, but POSIX locks simply defy all logic.

That said this doesn't appear to be a locking issue, rather an mmap issue.

How much memory can a process map? The default SQLite configuration tries to mmap 4GB per connection, which is probably excessive. It's uncommitted memory, so just reserving address space, which works out great for Linux 64-bit, (even with overcommit disabled), but maybe it breaks other Unix OSes.

That said, given the locking issues above, if this is Solaris, you can't have more than one connection. I don't know what configuration knobs GoToSocial makes available.

[ncruces]

If I understand this correctly, this is trying to allocate 256MB, and mmap is returning EAGAIN:

github.com/ncruces/go-sqlite3/internal/alloc.Virtual(0x3?, 0x100000000)

So there's already some configuration in place (otherwise we would be allocating the full 4GB).

I'm not sure there's anything I can do here, besides maybe disabling the use of mmap on Solaris (it seems to be of little benefit).

[ncruces]

I was wrong, this is trying to allocate the full 4GB. Still not sure if I can/should do something "auto-magically".

By default, SQLite asks us to allocate the minimum (hardcoded to 320KB) resizable to a maximum (4GB). This is per connection. Something else could be configured by GoToSocial.

This would set the maximum to 64MB (a page is 64KB, 1024 pages are 64MB):

go-sqlite3/sqlite_test.go

Line 14 in 368c900

RuntimeConfig = wazero.NewRuntimeConfig().WithMemoryLimitPages(1024)

This would set both minimum and maximum to 64MB:

go-sqlite3/internal/testcfg/testcfg.go

Lines 19 to 21 in 368c900

	sqlite3.RuntimeConfig = wazero.NewRuntimeConfig().
	WithMemoryCapacityFromMax(true).
	WithMemoryLimitPages(1024)

SQLite can probably live with 64MB in 99% of cases (the default cache size is 2MB, so 8MB would cover a lot). But degenerate cases may need a lot more (e.g. inserting large BLOBs, huge WALs, etc).

The initial call reserves a lot of address space, but tries not to commit it to memory (which is how it works on Linux, but Solaris may think differently):

go-sqlite3/internal/alloc/alloc_unix.go

Lines 23 to 28 in 368c900

	// Reserve max bytes of address space, to ensure we won't need to move it.
	// A protected, private, anonymous mapping should not commit memory.
	b, err := unix.Mmap(-1, 0, int(max), unix.PROT_NONE, unix.MAP_PRIVATE|unix.MAP_ANON)
	if err != nil {
	panic(err)
	}

This ensures we don't need to move memory later, and it allows us to then remap portions of this for shared memory WAL. Not using mmap means we can't use shared memory WAL, as does moving the memory. Figuring out a better strategy would allow (e.g.) Linux 32-bit to support shared memory WAL (fixing many warnings here, @tsmethurst, @NyaaaWhatsUpDoc).

As for Solaris, this is mostly irrelevant, because it doesn't support decent file locks, and so can't use shared memory WAL. On thing you can try @Here-Be-Saoirse is building with the sqlite3_nosys tag and report back.

[Here-Be-Saoirse]

@ncruces This is on Tribblix, an Illumos distro. Apologies, we should have been more clear; we use the term SOlaris for that whole thing

[ncruces]

@ncruces This is on Tribblix, an Illumos distro. Apologies, we should have been more clear; we use the term SOlaris for that whole thing

Oh, that makes a lot more sense.

Unlike Solaris, illumos actually has working BSD locks, and they're even compatible with POSIX locks, just like a BSD. They even claim to have OFD locks, which would be better than the BSDs, but it's a bit deceiving, since they span the entire file (so are simply equivalent to, and likely implemented on top of, flock).

So, I assume the issue is just that illumos (or the VM it's running on, etc) simply doesn't like the idea of reserving 4GB of address space for each connection. Which, tbh, isn't all that surprising. There's a reason I run my parallel tests with the 64MB limit: GitHub Actions doesn't love me having dozens of connections each eating 4GB of continuous address space (even on Linux, which is unusually liberal about this).

This actually validades one of wazero's controversial design decisions: implement sandboxing with bounds checks instead of mapping even more memory and using page faults to sandbox (what browsers do).

So I guess the fix here might be for me to change the default maximum address space to reserve, for example:

• 256MB on 64-bit platforms;
• 32MB on 32-bit platforms.

And then enable shared memory WAL on 32-bit platforms, too. Wdyt @tsmethurst?

That's a rather simple change I can commit to the repo, if it makes it easier for y'all to test, but I'd rather release it only after SQLite 3.47 comes out, as both are “major” changes (this should be in a couple of weeks, there's a release candidate, which passes all my tests).

[Here-Be-Saoirse]

@ncruces That should work. We'd love to test this on actual Solaris, specifically, SPARC64 D Solaris 10u11, but.... golang. Golang is incredibly unfriendly to ports to as-yet unsupported CPU architectures. There doesn't seem to be a frjom-zero bootstrap process, you seem to need golang to build golang, which...... is certainly a choice. Bloody modern developers

[Here-Be-Saoirse]

SPARC64 Solaris 10u11, even. The 'D' there was unintended

[NyaaaWhatsUpDoc]

I'll get a gotosocial pr up in a few :)

[ncruces]

I'll get a gotosocial pr up in a few :)

To configure WithMemoryLimitPages()? That's a good idea; I'll probably do it in go-sqlite3 too (as I wrote above), but that's more immediate for you and I'll love the feedback.

What that won't do is solve WAL on 32-bit platforms for you. Unless you want me to fast track that? I'd rather avoid release notes mentioning breaking compatibility every week. 😬

[ncruces]

SPARC64 Solaris 10u11, even. The 'D' there was unintended

Solaris proper, even if it had a working toolchain, has the file locking issue.

It's really insane, IMO, that just opening and closing a file releases locks acquired through another file descriptor. As if I controlled what files other parts of the process open.

[Here-Be-Saoirse]

@ncruces That's janky as hell. We're guessing that's something Illumos fixed, that <company who's name we shall not speak because of what they did to Sun Microsystems> were too up the ass of their own Linux cloud offerings to bother fixing?

[NyaaaWhatsUpDoc]

I'll get a gotosocial pr up in a few :)

To configure WithMemoryLimitPages()? That's a good idea; I'll probably do it in go-sqlite3 too (as I wrote above), but that's more immediate for you and I'll love the feedback.

What that won't do is solve WAL on 32-bit platforms for you. Unless you want me to fast track that? I'd rather avoid release notes mentioning breaking compatibility every week. 😬

here we are! https://github.com/superseriousbusiness/gotosocial/pull/3441

hopefully we'll be able to sneak in another release-candidate of gotosocial so we can test this final change before release of v0.17.0.

don't worry about fast-tracking 32bit WAL support. i'm not even sure it's a platform that we're going to officially support given we only officially support our WebAssembly embedded ffmpeg, and the performance for that in interpreted mode of wazero is not very servicable :')