SIGBUS: bus error during cgo execution

[mholt]

Hey @mattn -- since you saw my post on X about sqlite3 crashing on macOS 26 Tahoe, I've been doing some more digging and learning and I was hoping to get your thoughts.

My code is running on an M2 Macbook Air, with a sqlite DB on an ExFAT-formatted 4TB Samsung T7 SSD storage device. The DB is opened in WAL mode.

I can reliably reproduce the error, and have done so about a dozen times now. Unfortunately, I can't seem to minimize the reproducer, and the program I'm writing processes tons of personal data, so I can't easily/safely share a reproducer. Still, I can reliably get it to occur after about 10 minutes of running.

Here's the crash:

SIGBUS: bus error
PC=0x1021a0298 m=0 sigcode=1 addr=0x105b673ca
signal arrived during cgo execution

goroutine 10452650 gp=0x1400887c8c0 m=0 mp=0x10507b460 [syscall]:
runtime.cgocall(0x1021740a0, 0x140019b2e08)
        /usr/local/go/src/runtime/cgocall.go:167 +0x44 fp=0x140019b2dd0 sp=0x140019b2d90 pc=0x100e9fa74
github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_internal(0xa1882c640)
        _cgo_gotypes.go:384 +0x30 fp=0x140019b2e00 sp=0x140019b2dd0 pc=0x101319300
github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked.func1(...)
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2233
github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked(0x1400285e060, {0x140012a9b00, 0x8, 0x100eab848?})
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2233 +0x64 fp=0x140019b2f80 sp=0x140019b2e00 pc=0x1013290e4
github.com/mattn/go-sqlite3.(*SQLiteRows).Next.func1()
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2214 +0x2c fp=0x140019b2fd0 sp=0x140019b2f80 pc=0x101328f4c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x140019b2fd0 sp=0x140019b2fd0 pc=0x100eab844
created by github.com/mattn/go-sqlite3.(*SQLiteRows).Next in goroutine 10451559
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2213 +0x17c

The stack trace is the same on each reproduction.

It comes deep inside a syscall within cgo, but if I'm reading this correctly, the calling stack trace says "created by github.com/mattn/go-sqlite3.(*SQLiteRows).Next in goroutine 10451559" which, when I follow that number, takes me to this goroutine stack trace:

goroutine 10451559 gp=0x14001c56540 m=nil [select]:
runtime.gopark(0x140006ab258?, 0x2?, 0x28?, 0xb1?, 0x140006ab22c?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x140006ab0a0 sp=0x140006ab080 pc=0x100ea2fe0
runtime.selectgo(0x140006ab258, 0x140006ab228, 0x140006ab258?, 0x0, 0x5?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x140006ab1e0 sp=0x140006ab0a0 pc=0x100e813dc
github.com/mattn/go-sqlite3.(*SQLiteRows).Next(0x1400285e060, {0x140012a9b00, 0x8, 0x8})
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2216 +0x1c4 fp=0x140006ab2c0 sp=0x140006ab1e0 pc=0x101328d94
database/sql.(*Rows).nextLocked(0x140012e7860)
        /usr/local/go/src/database/sql/sql.go:3066 +0x124 fp=0x140006ab320 sp=0x140006ab2c0 pc=0x1011802e4
database/sql.(*Rows).Next.func1()
        /usr/local/go/src/database/sql/sql.go:3041 +0x2c fp=0x140006ab350 sp=0x140006ab320 pc=0x10118018c
database/sql.withLock({0x103690810, 0x140012e7898}, 0x140006ab3b8)
        /usr/local/go/src/database/sql/sql.go:3572 +0x74 fp=0x140006ab390 sp=0x140006ab350 pc=0x101182874
database/sql.(*Rows).Next(0x140012e7860)
        /usr/local/go/src/database/sql/sql.go:3040 +0x6c fp=0x140006ab3e0 sp=0x140006ab390 pc=0x1011800fc
github.com/timelinize/timelinize/timeline.thumbnailJob.iteratePagesOfTasksFromImportJob({{0x0, 0x0, 0x0}, 0x1, 0x0}, 0x14000d3ec60, 0x1)
        /Users/matt/Dev/timelinize/timeline/thumbnails.go:203 +0x698 fp=0x140006ab9d0 sp=0x140006ab3e0 pc=0x10171e108
github.com/timelinize/timelinize/timeline.thumbnailJob.Run({{0x0, 0x0, 0x0}, 0x1, 0x0}, 0x14000d3ec60, {0x0?, 0x0?, 0x140006abb88?})
        /Users/matt/Dev/timelinize/timeline/thumbnails.go:119 +0x124 fp=0x140006abb10 sp=0x140006ab9d0 pc=0x10171d5d4
github.com/timelinize/timelinize/timeline.(*thumbnailJob).Run(0x103628ae0?, 0x10229f090?, {0x0?, 0x14001170a00?, 0x2?})
        <autogenerated>:1 +0x70 fp=0x140006abb90 sp=0x140006abb10 pc=0x1017307f0
github.com/timelinize/timelinize/timeline.(*Timeline).runJob.func1(_, _, _, {{0x1400f1bd170, 0x24}, 0x2, {0x140012dcb00, 0xa}, 0x0, {0x14003ea0580, ...}, ...}, ...)
        /Users/matt/Dev/timelinize/timeline/jobs.go:463 +0x258 fp=0x140006abea0 sp=0x140006abb90 pc=0x1016ef558
github.com/timelinize/timelinize/timeline.(*Timeline).runJob.gowrap1()
        /Users/matt/Dev/timelinize/timeline/jobs.go:561 +0x5c fp=0x140006abfd0 sp=0x140006abea0 pc=0x1016ef2cc
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x140006abfd0 sp=0x140006abfd0 pc=0x100eab844
created by github.com/timelinize/timelinize/timeline.(*Timeline).runJob in goroutine 336
        /Users/matt/Dev/timelinize/timeline/jobs.go:424 +0x604

The last line of my code it refers to is this one:

https://github.com/timelinize/timelinize/blob/5994da8c752adfed6c8d1f77d8d2bee88afd3d47/timeline/thumbnails.go#L201

(The line number varies slightly because I've since added debug logs, but it is the same line.)

As far as I can tell, my code is correct. It has run millions of times without issues until recently. While there are concurrent goroutines using the DB, each query is inside an appropriate sync.RWMutex lock, and I seem to be closing the rows and transactions. I can do multiple read queries simultaneously, but writes should be exclusive.

I find it hard to believe it's a bug in sqlite3 itself (I have never encountered a bug in sqlite before!?), and I would say it's the kernel or the file system, but the error happens SO reliably that that can't be either, right? If it was the file system doing some sort of sync or flush or power cycle, why would it happen so reliably? If it was the kernel, I'd expect a kernel panic, not a sigbus.

One of my candidate theories is a bug in this wrapper. Still, I doubt it, because the error only started happening after upgrading to Tahoe, and I have to process a specific set of data in my program. So it's still quite likely a bug in my program.

Still, I wanted to ask you... on following the stack trace deeper, I found it referred to this area of code:

go-sqlite3/sqlite3.go

Lines 2211 to 2227 in 8bf7a8a

	resultCh := make(chan error)
	defer close(resultCh)
	go func() {
	resultCh <- rc.nextSyncLocked(dest)
	}()
	select {
	case err := <-resultCh:
	return err
	case <-rc.ctx.Done():
	select {
	case <-resultCh: // no need to interrupt
	default:
	// this is still racy and can be no-op if executed between sqlite3_* calls in nextSyncLocked.
	C.sqlite3_interrupt(rc.s.c.db)
	<-resultCh // ensure goroutine completed
	}
	return rc.ctx.Err()

Specifically the select on L2216. There is a comment in there that says, // this is still racy and can be no-op if executed between sqlite3_* calls in nextSyncLocked.

Is that possibly related?

Do you have any other ideas what might be the cause of this?

---

PS. FWIW, I seem to remember writing code that struggled with this pattern:

go func() {
	resultCh <- rc.nextSyncLocked(dest)
}()

I seem to recall having to instead do:

go func() {
	result := rc.nextSyncLocked(dest)
	resultCh <- result
}()

in order to get the expected results. (I don't remember details, but I think the channel would receive right away, rather than after the nextSyncLocked() function finished.)

I wonder if that could be something else to look into.

Thanks for any help!

[rittneje]

Specifically the select on L2216. There is a comment in there that says, // this is still racy and can be no-op if executed between sqlite3_* calls in nextSyncLocked.

Unless you pass a cancellable context to one of the database/sql methods, it is not possible to hit the select case in question.

---

PS. FWIW, I seem to remember writing code that struggled with this pattern:

Your two code samples are functionally equivalent. If there were a difference, it would represent a bug in Go itself.

---

Does the issue reproduce on any other system? Are your Command Line Developer Tools up-to-date? (xcode-select --install)

[mholt]

Hi @rittneje , thank you so much.

Unless you pass a cancellable context to one of the database/sql methods, it is not possible to hit the select case in question.

Good to know. In my case, I am passing a context that is returned from context.WithCancel, since cancellation is important in my application.

Your two code samples are functionally equivalent. If there were a difference, it would represent a bug in Go itself.

Yeah I would think so. I realize this is likely a separate thing anyway. But let me see if I can dig into that another time and surface more details for you if it turns out to be something. And you are probably right, it might be nothing...

Does the issue reproduce on any other system?

It does not. In fact, here are the systems I have tried:

• MacOS whatever version came before 26 Tahoe, ExFAT storage -- no crash... but since I suspect this bug is non-deterministic, it may have been present but just not manifest. Only performed exact same test maybe once since I didn't know there was an issue yet. Then I upgraded the OS.
• MacOS 26 Tahoe, ExFAT storage -- crash happens pretty consistently (not 100% of the time, but like 90-95%, it's possible I did the test slightly differently by accident the one or two times it didn't crash, or it is possible the bug is non-deterministic)
• MacOS 26 Tahoe, internal APFS storage -- no crash
• Linux, ExFAT storage -- no crash

So the only combination I can reproduce the test reliably is: MacOS on ExFAT.

The test involves processing about 700,000 data points, many/most of which get added to the DB as rows, and then many data files written to the disk next to the database... and right as the import job is finishing (write-heavy), and subsequent thumbnails+embeddings jobs start (read-heavy), is when the crash occurs.

I have noticed that running rm -rf to delete the database and many thousands of data files next to the DB, is MUCH faster on Linux than on MacOS. Same storage device. The Mac even has faster USB. It takes like 20 seconds on Mac, but only about 2-5 seconds on Linux.

I kind of am starting to wonder if the MacOS ExFAT driver is just... bad? Could that possibly have anything to do with this?

Are your Command Line Developer Tools up-to-date? (xcode-select --install)

Yep, just double-checked. They updated about a week ago or so. No updates currently available.

[mholt]

After changing all my locks around DB calls to exclusive locks (using Lock() instead of RLock(), even for SELECT-only transactions), the error goes away. The locking in my code is, after dozens of scrutinizing reviews, theoretically correct, according to my mental model of how sqlite works. But maybe I have some unfounded assumptions when it comes to WAL mode.

I am now wondering if this is a concurrency bug that only the speed characteristics of MacOS+ExFAT expose reliably.

Currently refactoring to follow some of your advice in #1022 (comment) and I will update here.

I still suspect a sloppy ExFAT driver on macOS or "I'm holding it wrong" in my code... lots of searching has shown me that there's not a clear consensus on how to use database/sql for concurrent SQLite code. I was kind of surprised...

[mholt]

Update for you, @rittneje : Following the advice in the linked comment (to use 2 connection pools, one for reads and one for writes, and not using my own mutex), the crash still occurs.

Since I am no longer relying on my own sync mechanisms, and am relying on SQLite's code (and this driver) exclusively -- on top of the ExFAT file system, of course -- I am starting to wonder if there really is a bug in one of those layers?

• ExFAT driver on macOS
• mattn/go-sqlite3
• sqlite itself (let's be real, probably not)

Additionally, with the change following your recommendation, if I try to query the DB while an import is running, the import continues without errors, but the read queries (SELECT-only) give me "database disk image is malformed." Only the read queries give me that error.

I am sure I must still be doing something wrong -- I really want to rule out external bugs or a bug in this library -- but I am not sure what it is.

Here's my new DB opening code, for reference:

	dsn := url.URL{
		Scheme: "file",
		Path:   filepath.Join(repoDir, DBFilename),
	}
	qs := url.Values{
		"mode":          {"rwc"},
		"_journal_mode": {"wal"},
		"_foreign_keys": {"on"},
		"_txlock":       {"immediate"}, // avoid potential "database is locked" errors
	}
	dsn.RawQuery = qs.Encode()

	Log.Info("opening DB write pool", zap.String("dsn", dsn.String()))
	db.WritePool, err = sql.Open("sqlite3", dsn.String())
	if err != nil {
		return nil, fmt.Errorf("opening database write pool: %w", err)
	}
	db.WritePool.SetMaxOpenConns(1) // sqlite doesn't support concurrent writers

	qs.Set("mode", "ro")
	qs.Del("_txlock")
	dsn.RawQuery = qs.Encode()

	Log.Info("opening DB read pool", zap.String("dsn", dsn.String()))
	db.ReadPool, err = sql.Open("sqlite3", dsn.String())
	if err != nil {
		return nil, fmt.Errorf("opening database read pool: %w", err)
	}

[mholt]

I just ran the program again -- the same test -- but I got a different error this time!

fatal error: semasleep on Darwin signal stack
panic during panic

goroutine 0 gp=0x14000372fc0 m=12 mp=0x14000600808 [idle]:
runtime.throw({0x1039ae1d9?, 0x102501604?})
        /usr/local/go/src/runtime/panic.go:1094 +0x34 fp=0x1400103f8b0 sp=0x1400103f880 pc=0x102542ed4
runtime.semasleep(0xffffffffffffffff)
        /usr/local/go/src/runtime/os_darwin.go:49 +0x168 fp=0x1400103f910 sp=0x1400103f8b0 pc=0x102507988
runtime.lock2(0x10674f4d0)
        /usr/local/go/src/runtime/lock_spinbit.go:250 +0x37c fp=0x1400103f970 sp=0x1400103f910 pc=0x1024de8bc
runtime.lockWithRank(...)
        /usr/local/go/src/runtime/lockrank_off.go:24
runtime.lock(...)
        /usr/local/go/src/runtime/lock_spinbit.go:152
runtime.startpanic_m()
        /usr/local/go/src/runtime/panic.go:1376 +0xd0 fp=0x1400103f9a0 sp=0x1400103f970 pc=0x10250a700
runtime.sighandler(0xa, 0x14000600808?, 0x1400103fa40?, 0x14000373180)
        /usr/local/go/src/runtime/signal_unix.go:759 +0x288 fp=0x1400103fa10 sp=0x1400103f9a0 pc=0x102524aa8
runtime.sigtrampgo(0xa, 0x1400103fbb0, 0x1400103fc18)
        /usr/local/go/src/runtime/signal_unix.go:490 +0x108 fp=0x1400103fa90 sp=0x1400103fa10 pc=0x102524328
runtime.sigtrampgo(0xa, 0x1400103fbb0, 0x1400103fc18)
        <autogenerated>:1 +0x1c fp=0x1400103fac0 sp=0x1400103fa90 pc=0x10254dd3c
runtime.sigtramp()
        /usr/local/go/src/runtime/sys_darwin_arm64.s:227 +0x4c fp=0x1400103fb80 sp=0x1400103fac0 pc=0x10254c87c

goroutine 10464907 gp=0x14004e68700 m=12 mp=0x14000600808 [syscall]:
runtime.cgocall(0x103812310, 0x1400095f608)
        /usr/local/go/src/runtime/cgocall.go:167 +0x44 fp=0x1400095f5d0 sp=0x1400095f590 pc=0x10253fa74
github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_internal(0xad9128640)
        _cgo_gotypes.go:384 +0x30 fp=0x1400095f600 sp=0x1400095f5d0 pc=0x1029b9300
github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked.func1(...)
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2233
github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked(0x14002f17920, {0x14000dec980, 0x8, 0x10254b848?})
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2233 +0x64 fp=0x1400095f780 sp=0x1400095f600 pc=0x1029c90e4
github.com/mattn/go-sqlite3.(*SQLiteRows).Next.func1()
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2214 +0x2c fp=0x1400095f7d0 sp=0x1400095f780 pc=0x1029c8f4c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400095f7d0 sp=0x1400095f7d0 pc=0x10254b844
created by github.com/mattn/go-sqlite3.(*SQLiteRows).Next in goroutine 10462025
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2213 +0x17c

goroutine 1 gp=0x140000021c0 m=nil [select (no cases), 12 minutes]:
runtime.gopark(0x1400036c0e0?, 0x0?, 0x3?, 0x0?, 0x1400036de60?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000882d70 sp=0x14000882d50 pc=0x102542fe0
runtime.block()
        /usr/local/go/src/runtime/select.go:104 +0x30 fp=0x14000882da0 sp=0x14000882d70 pc=0x102520d00
github.com/timelinize/timelinize/cmd.getStandardSubcommand.func1()
        /Users/matt/Dev/timelinize/cmd/cmd.go:156 +0x34 fp=0x14000882dc0 sp=0x14000882da0 pc=0x1036ed754
github.com/timelinize/timelinize/cmd.Main({0x104d25ba8, 0x104dc80a0})
        /Users/matt/Dev/timelinize/cmd/cmd.go:63 +0x3dc fp=0x14000882f20 sp=0x14000882dc0 pc=0x1036ec83c
main.main()
        /Users/matt/Dev/timelinize/main.go:57 +0x2c fp=0x14000882f40 sp=0x14000882f20 pc=0x1037b0b7c
runtime.main()
        /usr/local/go/src/runtime/proc.go:285 +0x278 fp=0x14000882fd0 sp=0x14000882f40 pc=0x10250d618
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000882fd0 sp=0x14000882fd0 pc=0x10254b844

goroutine 2 gp=0x14000002c40 m=nil [force gc (idle), 12 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400008ef90 sp=0x1400008ef70 pc=0x102542fe0
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:466
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:373 +0xb4 fp=0x1400008efd0 sp=0x1400008ef90 pc=0x10250d964
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400008efd0 sp=0x1400008efd0 pc=0x10254b844
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:361 +0x24

goroutine 18 gp=0x14000102380 m=nil [GC sweep wait]:
runtime.gopark(0x106716b01?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400008a760 sp=0x1400008a740 pc=0x102542fe0
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:466
runtime.bgsweep(0x14000110000)
        /usr/local/go/src/runtime/mgcsweep.go:323 +0x104 fp=0x1400008a7b0 sp=0x1400008a760 pc=0x1024f5bb4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:212 +0x28 fp=0x1400008a7d0 sp=0x1400008a7b0 pc=0x1024e9808
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400008a7d0 sp=0x1400008a7d0 pc=0x10254b844
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:212 +0x6c

goroutine 19 gp=0x14000102540 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1229678?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400008af60 sp=0x1400008af40 pc=0x102542fe0
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:466
runtime.(*scavengerState).park(0x106719a60)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x5c fp=0x1400008af90 sp=0x1400008af60 pc=0x1024f36cc
runtime.bgscavenge(0x14000110000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0xac fp=0x1400008afb0 sp=0x1400008af90 pc=0x1024f3c6c
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:213 +0x28 fp=0x1400008afd0 sp=0x1400008afb0 pc=0x1024e97a8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400008afd0 sp=0x1400008afd0 pc=0x10254b844
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:213 +0xac

goroutine 3 gp=0x14000003dc0 m=nil [finalizer wait, 7 minutes]:
runtime.gopark(0x0?, 0x104d17e90?, 0x40?, 0x22?, 0x2000000020?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400008e580 sp=0x1400008e560 pc=0x102542fe0
runtime.runFinalizers()
        /usr/local/go/src/runtime/mfinal.go:210 +0x104 fp=0x1400008e7d0 sp=0x1400008e580 pc=0x1024e8784
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400008e7d0 sp=0x1400008e7d0 pc=0x10254b844
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:172 +0x78

goroutine 4 gp=0x14000372380 m=nil [GC worker (idle)]:
runtime.gopark(0x203398044c749?, 0x1?, 0xef?, 0xd5?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000884f10 sp=0x14000884ef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x14000884fb0 sp=0x14000884f10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x14000884fd0 sp=0x14000884fb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000884fd0 sp=0x14000884fd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 20 gp=0x14000102a80 m=nil [GC worker (idle)]:
runtime.gopark(0x203398053af0a?, 0x3?, 0x78?, 0xa0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000885f10 sp=0x14000885ef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x14000885fb0 sp=0x14000885f10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x14000885fd0 sp=0x14000885fb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000885fd0 sp=0x14000885fd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 21 gp=0x14000102c40 m=nil [GC worker (idle)]:
runtime.gopark(0x203398044c772?, 0x3?, 0x80?, 0x37?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x140000a1f10 sp=0x140000a1ef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x140000a1fb0 sp=0x140000a1f10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x140000a1fd0 sp=0x140000a1fb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x140000a1fd0 sp=0x140000a1fd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 34 gp=0x14000504000 m=nil [GC worker (idle)]:
runtime.gopark(0x203398051871e?, 0x3?, 0x20?, 0x56?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000886f10 sp=0x14000886ef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x14000886fb0 sp=0x14000886f10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x14000886fd0 sp=0x14000886fb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000886fd0 sp=0x14000886fd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 5 gp=0x14000372540 m=nil [GC worker (idle)]:
runtime.gopark(0x203398053aeb7?, 0x3?, 0xcf?, 0x16?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400009ff10 sp=0x1400009fef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x1400009ffb0 sp=0x1400009ff10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x1400009ffd0 sp=0x1400009ffb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400009ffd0 sp=0x1400009ffd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 6 gp=0x14000372700 m=nil [GC worker (idle)]:
runtime.gopark(0x106751fe0?, 0x3?, 0xf8?, 0x93?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000883f10 sp=0x14000883ef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x14000883fb0 sp=0x14000883f10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x14000883fd0 sp=0x14000883fb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000883fd0 sp=0x14000883fd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 22 gp=0x14000102e00 m=nil [GC worker (idle)]:
runtime.gopark(0x203398053b004?, 0x1?, 0x10?, 0x28?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000881f10 sp=0x14000881ef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x14000881fb0 sp=0x14000881f10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x14000881fd0 sp=0x14000881fb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000881fd0 sp=0x14000881fd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 35 gp=0x140005041c0 m=nil [GC worker (idle)]:
runtime.gopark(0x203392395e4c4?, 0x3?, 0x2f?, 0x98?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000880f10 sp=0x14000880ef0 pc=0x102542fe0
runtime.gcBgMarkWorker(0x1400005d730)
        /usr/local/go/src/runtime/mgc.go:1463 +0xe0 fp=0x14000880fb0 sp=0x14000880f10 pc=0x1024ebe80
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1373 +0x28 fp=0x14000880fd0 sp=0x14000880fb0 pc=0x1024ebd68
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000880fd0 sp=0x14000880fd0 pc=0x10254b844
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1373 +0x140

goroutine 7 gp=0x14000504540 m=nil [select]:
runtime.gopark(0x140000a0f70?, 0x2?, 0x28?, 0xe?, 0x140000a0f64?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x140000a0de0 sp=0x140000a0dc0 pc=0x102542fe0
runtime.selectgo(0x140000a0f70, 0x140000a0f60, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x140000a0f20 sp=0x140000a0de0 pc=0x1025213dc
github.com/golang/glog.(*fileSink).flushDaemon(0x106719578)
        /Users/matt/go/pkg/mod/github.com/golang/glog@v1.2.4/glog_file.go:380 +0x9c fp=0x140000a0fb0 sp=0x140000a0f20 pc=0x103279eac
github.com/golang/glog.init.1.gowrap1()
        /Users/matt/go/pkg/mod/github.com/golang/glog@v1.2.4/glog_file.go:188 +0x28 fp=0x140000a0fd0 sp=0x140000a0fb0 pc=0x103278fd8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x140000a0fd0 sp=0x140000a0fd0 pc=0x10254b844
created by github.com/golang/glog.init.1 in goroutine 1
        /Users/matt/go/pkg/mod/github.com/golang/glog@v1.2.4/glog_file.go:188 +0x1b0

goroutine 8 gp=0x14000504fc0 m=nil [cleanup wait]:
runtime.gopark(0x140039f7300?, 0x157808200?, 0x78?, 0xbf?, 0x1024e5f54?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400008bf40 sp=0x1400008bf20 pc=0x102542fe0
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:466
runtime.(*cleanupQueue).dequeue(0x10671a1a0)
        /usr/local/go/src/runtime/mcleanup.go:439 +0x110 fp=0x1400008bf80 sp=0x1400008bf40 pc=0x1024e5750
runtime.runCleanups()
        /usr/local/go/src/runtime/mcleanup.go:635 +0x40 fp=0x1400008bfd0 sp=0x1400008bf80 pc=0x1024e5f60
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400008bfd0 sp=0x1400008bfd0 pc=0x10254b844
created by runtime.(*cleanupQueue).createGs in goroutine 1
        /usr/local/go/src/runtime/mcleanup.go:589 +0x108

goroutine 24 gp=0x14000103a40 m=nil [IO wait, 12 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400009eaa0 sp=0x1400009ea80 pc=0x102542fe0
runtime.netpollblock(0x1400009eb38?, 0x25cacdc?, 0x1?)
        /usr/local/go/src/runtime/netpoll.go:575 +0x150 fp=0x1400009eae0 sp=0x1400009eaa0 pc=0x102506ad0
internal/poll.runtime_pollWait(0x109f3c400, 0x72)
        /usr/local/go/src/runtime/netpoll.go:351 +0xa0 fp=0x1400009eb10 sp=0x1400009eae0 pc=0x102542210
internal/poll.(*pollDesc).wait(0x140006fa880?, 0x1025ccd5c?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x1400009eb40 sp=0x1400009eb10 pc=0x1025c67d8
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x140006fa880)
        /usr/local/go/src/internal/poll/fd_unix.go:613 +0x21c fp=0x1400009ebf0 sp=0x1400009eb40 pc=0x1025cadbc
net.(*netFD).accept(0x140006fa880)
        /usr/local/go/src/net/fd_unix.go:161 +0x28 fp=0x1400009ecb0 sp=0x1400009ebf0 pc=0x1026c4558
net.(*TCPListener).accept(0x1400073bf00)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x24 fp=0x1400009ed00 sp=0x1400009ecb0 pc=0x1026db4e4
net.(*TCPListener).Accept(0x1400073bf00)
        /usr/local/go/src/net/tcpsock.go:380 +0x2c fp=0x1400009ed40 sp=0x1400009ed00 pc=0x1026da86c
net/http.(*onceCloseListener).Accept(0x140007550e0?)
        <autogenerated>:1 +0x2c fp=0x1400009ed60 sp=0x1400009ed40 pc=0x10280c4dc
net/http.(*Server).Serve(0x1400073c500, {0x104d37ba0, 0x1400073bf00})
        /usr/local/go/src/net/http/server.go:3463 +0x24c fp=0x1400009ee90 sp=0x1400009ed60 pc=0x1027e72dc
github.com/timelinize/timelinize/tlzapp.(*App).serve.func2()
        /Users/matt/Dev/timelinize/tlzapp/app.go:424 +0x3c fp=0x1400009efd0 sp=0x1400009ee90 pc=0x1036c60ac
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400009efd0 sp=0x1400009efd0 pc=0x10254b844
created by github.com/timelinize/timelinize/tlzapp.(*App).serve in goroutine 1
        /Users/matt/Dev/timelinize/tlzapp/app.go:423 +0x1850

goroutine 192 gp=0x14000602380 m=nil [select]:
runtime.gopark(0x14002be3f48?, 0x3?, 0x68?, 0x5d?, 0x14002be3ea2?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400098fd20 sp=0x1400098fd00 pc=0x102542fe0
runtime.selectgo(0x1400098ff48, 0x14002be3e9c, 0xb?, 0x0, 0x14000091eb0?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x1400098fe60 sp=0x1400098fd20 pc=0x1025213dc
github.com/timelinize/timelinize/timeline.(*Timeline).maintenanceLoop(0x14000158e10)
        /Users/matt/Dev/timelinize/timeline/maintenance.go:67 +0x21c fp=0x1400098ffb0 sp=0x1400098fe60 pc=0x102d9417c
github.com/timelinize/timelinize/timeline.openTimeline.gowrap1()
        /Users/matt/Dev/timelinize/timeline/timeline.go:421 +0x28 fp=0x1400098ffd0 sp=0x1400098ffb0 pc=0x102dc36a8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400098ffd0 sp=0x1400098ffd0 pc=0x10254b844
created by github.com/timelinize/timelinize/timeline.openTimeline in goroutine 42
        /Users/matt/Dev/timelinize/timeline/timeline.go:421 +0x798

goroutine 185 gp=0x14000602540 m=nil [select, 12 minutes]:
runtime.gopark(0x14000091778?, 0x2?, 0x40?, 0x25?, 0x14000091774?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x140000915f0 sp=0x140000915d0 pc=0x102542fe0
runtime.selectgo(0x14000091778, 0x14000091770, 0x140007fccb0?, 0x0, 0x1400070d701?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x14000091730 sp=0x140000915f0 pc=0x1025213dc
database/sql.(*DB).connectionOpener(0x1400099ed00, {0x104d3c7a0, 0x140007574f0})
        /usr/local/go/src/database/sql/sql.go:1261 +0x80 fp=0x140000917a0 sp=0x14000091730 pc=0x102817c60
database/sql.OpenDB.gowrap1()
        /usr/local/go/src/database/sql/sql.go:841 +0x2c fp=0x140000917d0 sp=0x140000917a0 pc=0x102815f1c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x140000917d0 sp=0x140000917d0 pc=0x10254b844
created by database/sql.OpenDB in goroutine 42
        /usr/local/go/src/database/sql/sql.go:841 +0x114

goroutine 32 gp=0x14000103c00 m=nil [IO wait, 12 minutes]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x10256b500?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400090eda0 sp=0x1400090ed80 pc=0x102542fe0
runtime.netpollblock(0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:575 +0x150 fp=0x1400090ede0 sp=0x1400090eda0 pc=0x102506ad0
internal/poll.runtime_pollWait(0x109f3c000, 0x72)
        /usr/local/go/src/runtime/netpoll.go:351 +0xa0 fp=0x1400090ee10 sp=0x1400090ede0 pc=0x102542210
internal/poll.(*pollDesc).wait(0x14000188680?, 0x1400031b400?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x1400090ee40 sp=0x1400090ee10 pc=0x1025c67d8
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000188680, {0x1400031b400, 0x400, 0x400})
        /usr/local/go/src/internal/poll/fd_unix.go:165 +0x1e0 fp=0x1400090eee0 sp=0x1400090ee40 pc=0x1025c79f0
net.(*netFD).Read(0x14000188680, {0x1400031b400?, 0x1400090ef98?, 0x1026d2d14?})
        /usr/local/go/src/net/fd_posix.go:68 +0x28 fp=0x1400090ef30 sp=0x1400090eee0 pc=0x1026c2d58
net.(*conn).Read(0x140000923f8, {0x1400031b400?, 0x1400090efe8?, 0x1036aac7c?})
        /usr/local/go/src/net/net.go:196 +0x34 fp=0x1400090ef80 sp=0x1400090ef30 pc=0x1026d2b94
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*hijackedConn).Read(0x140005eee10, {0x1400031b400?, 0xe9?, 0x1024e16d4?})
        /Users/matt/go/pkg/mod/github.com/caddyserver/caddy/v2@v2.9.1/modules/caddyhttp/responsewriter.go:291 +0x2c fp=0x1400090efb0 sp=0x1400090ef80 pc=0x1036aaaec
bufio.(*Reader).fill(0x14000240d80)
        /usr/local/go/src/bufio/bufio.go:113 +0xe0 fp=0x1400090eff0 sp=0x1400090efb0 pc=0x102626b70
bufio.(*Reader).Peek(0x14000240d80, 0x2)
        /usr/local/go/src/bufio/bufio.go:152 +0x58 fp=0x1400090f010 sp=0x1400090eff0 pc=0x102626cb8
github.com/gorilla/websocket.(*Conn).read(0x140001274a0, 0x0?)
        /Users/matt/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.go:371 +0x24 fp=0x1400090f060 sp=0x1400090f010 pc=0x102b36404
github.com/gorilla/websocket.(*Conn).advanceFrame(0x140001274a0)
        /Users/matt/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.go:809 +0x60 fp=0x1400090f180 sp=0x1400090f060 pc=0x102b38160
github.com/gorilla/websocket.(*Conn).NextReader(0x140001274a0)
        /Users/matt/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.go:1009 +0x88 fp=0x1400090f1d0 sp=0x1400090f180 pc=0x102b38ed8
github.com/gorilla/websocket.(*Conn).ReadMessage(0x140002e58c0?)
        /Users/matt/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.go:1093 +0x1c fp=0x1400090f200 sp=0x1400090f1d0 pc=0x102b3948c
github.com/timelinize/timelinize/tlzapp.server.handleLogs({0x140007e2480, 0x140006fa680, {0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, 0x0, {0x0, ...}, ...}, ...)
        /Users/matt/Dev/timelinize/tlzapp/handlers.go:225 +0x150 fp=0x1400090f310 sp=0x1400090f200 pc=0x1036def70
github.com/timelinize/timelinize/tlzapp.server.handleLogs-fm({0x109f3c620?, 0x14000759000?}, 0x140002eef00?)
        <autogenerated>:1 +0x64 fp=0x1400090f3a0 sp=0x1400090f310 pc=0x1036eb0d4
github.com/timelinize/timelinize/tlzapp.Endpoint.ServeHTTP({{0x10392da98, 0x3}, {0x0, 0x0}, {0x0, 0x0}, 0x140006fd3b0, {0x1039d5c81, 0x2e}}, {0x109f3c620?, ...}, ...)
        /Users/matt/Dev/timelinize/tlzapp/endpoints.go:265 +0x28c fp=0x1400090f4f0 sp=0x1400090f3a0 pc=0x1036d559c
github.com/timelinize/timelinize/tlzapp.(*Endpoint).ServeHTTP(0x14000631530?, {0x109f3c620?, 0x14000759000?}, 0x10399c8c3?)
        <autogenerated>:1 +0x88 fp=0x1400090f5b0 sp=0x1400090f4f0 pc=0x1036e9498
github.com/timelinize/timelinize/tlzapp.(*App).serve.(*App).serve.func1.server.enforceHost.func12({0x109f3c620, 0x14000759000}, 0x140002eef00)
        /Users/matt/Dev/timelinize/tlzapp/server.go:163 +0x78 fp=0x1400090f6a0 sp=0x1400090f5b0 pc=0x1036c9368
github.com/timelinize/timelinize/tlzapp.handlerFunc.ServeHTTP(0x14000631530?, {0x109f3c620?, 0x14000759000?}, 0x140007cc980?)
        /Users/matt/Dev/timelinize/tlzapp/helpers.go:38 +0x38 fp=0x1400090f6d0 sp=0x1400090f6a0 pc=0x1036e1028
github.com/timelinize/timelinize/tlzapp.(*App).serve.(*App).serve.func1.server.enforceOriginAndMethod.func13({0x109f3c620, 0x14000759000}, 0x140002eef00)
        /Users/matt/Dev/timelinize/tlzapp/server.go:204 +0x3a4 fp=0x1400090f820 sp=0x1400090f6d0 pc=0x1036c9224
github.com/timelinize/timelinize/tlzapp.handlerFunc.ServeHTTP(0x14000887888?, {0x109f3c620?, 0x14000759000?}, 0x140008878a8?)
        /Users/matt/Dev/timelinize/tlzapp/helpers.go:38 +0x38 fp=0x1400090f850 sp=0x1400090f820 pc=0x1036e1028
github.com/timelinize/timelinize/tlzapp.(*App).serve.(*App).serve.func1.wrapErrorHandler.func14({0x109f3c620, 0x14000759000}, 0x140002eef00)
        /Users/matt/Dev/timelinize/tlzapp/helpers.go:105 +0x40 fp=0x1400090f890 sp=0x1400090f850 pc=0x1036c8e30
net/http.HandlerFunc.ServeHTTP(0x140007e2540?, {0x109f3c620?, 0x14000759000?}, 0x104c43c20?)
        /usr/local/go/src/net/http/server.go:2322 +0x38 fp=0x1400090f8c0 sp=0x1400090f890 pc=0x1027e3f08
net/http.(*ServeMux).ServeHTTP(0x1066acb90?, {0x109f3c620, 0x14000759000}, 0x140002eef00)
        /usr/local/go/src/net/http/server.go:2861 +0x190 fp=0x1400090f910 sp=0x1400090f8c0 pc=0x1027e59a0
github.com/timelinize/timelinize/tlzapp.server.ServeHTTP({0x140007e2480, 0x140006fa680, {0x104d37ba0, 0x1400073bf00}, 0x0, {0x140005984c8, 0x3, 0x3}, 0x140007e2540, {0x104d268c0, ...}, ...}, ...)
        /Users/matt/Dev/timelinize/tlzapp/server.go:82 +0x24c fp=0x1400090fa40 sp=0x1400090f910 pc=0x1036e235c
github.com/timelinize/timelinize/tlzapp.(*server).ServeHTTP(0x140005f33e0?, {0x104d374f8?, 0x14000158960?}, 0x14000887b50?)
        <autogenerated>:1 +0x90 fp=0x1400090fb40 sp=0x1400090fa40 pc=0x1036ea620
net/http.serverHandler.ServeHTTP({0x14000758f40?}, {0x104d374f8?, 0x14000158960?}, 0x6?)
        /usr/local/go/src/net/http/server.go:3340 +0xb0 fp=0x1400090fb70 sp=0x1400090fb40 pc=0x102800190
net/http.(*conn).serve(0x140003a8750, {0x104d3c768, 0x140006221b0})
        /usr/local/go/src/net/http/server.go:2109 +0x528 fp=0x1400090ffa0 sp=0x1400090fb70 pc=0x1027e22f8
net/http.(*Server).Serve.gowrap3()
        /usr/local/go/src/net/http/server.go:3493 +0x2c fp=0x1400090ffd0 sp=0x1400090ffa0 pc=0x1027e763c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400090ffd0 sp=0x1400090ffd0 pc=0x10254b844
created by net/http.(*Server).Serve in goroutine 24
        /usr/local/go/src/net/http/server.go:3493 +0x384

goroutine 46 gp=0x14000602700 m=nil [select, 12 minutes]:
runtime.gopark(0x1400050c778?, 0x2?, 0x0?, 0x23?, 0x1400050c774?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400050c5f0 sp=0x1400050c5d0 pc=0x102542fe0
runtime.selectgo(0x1400050c778, 0x1400050c770, 0x14000300150?, 0x0, 0x1400050c7a8?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x1400050c730 sp=0x1400050c5f0 pc=0x1025213dc
database/sql.(*DB).connectionOpener(0x14000966000, {0x104d3c7a0, 0x140002ea820})
        /usr/local/go/src/database/sql/sql.go:1261 +0x80 fp=0x1400050c7a0 sp=0x1400050c730 pc=0x102817c60
database/sql.OpenDB.gowrap1()
        /usr/local/go/src/database/sql/sql.go:841 +0x2c fp=0x1400050c7d0 sp=0x1400050c7a0 pc=0x102815f1c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400050c7d0 sp=0x1400050c7d0 pc=0x10254b844
created by database/sql.OpenDB in goroutine 42
        /usr/local/go/src/database/sql/sql.go:841 +0x114

goroutine 47 gp=0x140006028c0 m=nil [select, 12 minutes]:
runtime.gopark(0x1400050cf78?, 0x2?, 0x0?, 0x0?, 0x1400050cf74?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x1400050cdf0 sp=0x1400050cdd0 pc=0x102542fe0
runtime.selectgo(0x1400050cf78, 0x1400050cf70, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x1400050cf30 sp=0x1400050cdf0 pc=0x1025213dc
database/sql.(*DB).connectionOpener(0x140009660d0, {0x104d3c7a0, 0x140002ea870})
        /usr/local/go/src/database/sql/sql.go:1261 +0x80 fp=0x1400050cfa0 sp=0x1400050cf30 pc=0x102817c60
database/sql.OpenDB.gowrap1()
        /usr/local/go/src/database/sql/sql.go:841 +0x2c fp=0x1400050cfd0 sp=0x1400050cfa0 pc=0x102815f1c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x1400050cfd0 sp=0x1400050cfd0 pc=0x10254b844
created by database/sql.OpenDB in goroutine 42
        /usr/local/go/src/database/sql/sql.go:841 +0x114

goroutine 10465929 gp=0x14000602a80 m=nil [select]:
runtime.gopark(0x14000a28f38?, 0x2?, 0x40?, 0x41?, 0x14000a28ee4?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x14000a28d60 sp=0x14000a28d40 pc=0x102542fe0
runtime.selectgo(0x14000a28f38, 0x14000a28ee0, 0x140011e5980?, 0x0, 0x14000af8240?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x14000a28ea0 sp=0x14000a28d60 pc=0x1025213dc
net/http.(*persistConn).writeLoop(0x14000df6360)
        /usr/local/go/src/net/http/transport.go:2597 +0x94 fp=0x14000a28fb0 sp=0x14000a28ea0 pc=0x1027fbf54
net/http.(*Transport).dialConn.gowrap3()
        /usr/local/go/src/net/http/transport.go:1945 +0x28 fp=0x14000a28fd0 sp=0x14000a28fb0 pc=0x1027f9218
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000a28fd0 sp=0x14000a28fd0 pc=0x10254b844
created by net/http.(*Transport).dialConn in goroutine 10465924
        /usr/local/go/src/net/http/transport.go:1945 +0x1164

goroutine 10462025 gp=0x14000602c40 m=nil [select]:
runtime.gopark(0x14000911358?, 0x2?, 0x28?, 0x12?, 0x1400091132c?)
        /usr/local/go/src/runtime/proc.go:460 +0xc0 fp=0x140009111a0 sp=0x14000911180 pc=0x102542fe0
runtime.selectgo(0x14000911358, 0x14000911328, 0x14000911358?, 0x0, 0x5?, 0x1)
        /usr/local/go/src/runtime/select.go:351 +0x6bc fp=0x140009112e0 sp=0x140009111a0 pc=0x1025213dc
github.com/mattn/go-sqlite3.(*SQLiteRows).Next(0x14002f17920, {0x14000dec980, 0x8, 0x8})
        /Users/matt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.32/sqlite3.go:2216 +0x1c4 fp=0x140009113c0 sp=0x140009112e0 pc=0x1029c8d94
database/sql.(*Rows).nextLocked(0x140011f2280)
        /usr/local/go/src/database/sql/sql.go:3066 +0x124 fp=0x14000911420 sp=0x140009113c0 pc=0x1028202e4
database/sql.(*Rows).Next.func1()
        /usr/local/go/src/database/sql/sql.go:3041 +0x2c fp=0x14000911450 sp=0x14000911420 pc=0x10282018c
database/sql.withLock({0x104d30ff0, 0x140011f22b8}, 0x140009114b8)
        /usr/local/go/src/database/sql/sql.go:3572 +0x74 fp=0x14000911490 sp=0x14000911450 pc=0x102822874
database/sql.(*Rows).Next(0x140011f2280)
        /usr/local/go/src/database/sql/sql.go:3040 +0x6c fp=0x140009114e0 sp=0x14000911490 pc=0x1028200fc
github.com/timelinize/timelinize/timeline.thumbnailJob.iteratePagesOfTasksFromImportJob({{0x0, 0x0, 0x0}, 0x1, 0x0}, 0x14002cf99e0, 0x1)
        /Users/matt/Dev/timelinize/timeline/thumbnails.go:201 +0x424 fp=0x140009119f0 sp=0x140009114e0 pc=0x102dbc254
github.com/timelinize/timelinize/timeline.thumbnailJob.Run({{0x0, 0x0, 0x0}, 0x1, 0x0}, 0x14002cf99e0, {0x0?, 0x0?, 0x14000911ba8?})
        /Users/matt/Dev/timelinize/timeline/thumbnails.go:118 +0x124 fp=0x14000911b30 sp=0x140009119f0 pc=0x102dbb994
github.com/timelinize/timelinize/timeline.(*thumbnailJob).Run(0x104cc8ca0?, 0x10393d319?, {0x0?, 0x14000dec700?, 0x2?})
        <autogenerated>:1 +0x70 fp=0x14000911bb0 sp=0x14000911b30 pc=0x102dcea30
github.com/timelinize/timelinize/timeline.(*Timeline).runJob.func1(_, _, _, {{0x14007c020c0, 0x24}, 0x2, {0x14000f823a0, 0xa}, 0x0, {0x14008e7e480, ...}, ...}, ...)
        /Users/matt/Dev/timelinize/timeline/jobs.go:463 +0x254 fp=0x14000911ea0 sp=0x14000911bb0 pc=0x102d8e854
github.com/timelinize/timelinize/timeline.(*Timeline).runJob.gowrap1()
        /Users/matt/Dev/timelinize/timeline/jobs.go:561 +0x5c fp=0x14000911fd0 sp=0x14000911ea0 pc=0x102d8e5cc
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14000911fd0 sp=0x14000911fd0 pc=0x10254b844
created by github.com/timelinize/timelinize/timeline.(*Timeline).runJob in goroutine 318
        /Users/matt/Dev/timelinize/timeline/jobs.go:424 +0x604fatal error: fatal error: semawakeup on Darwin signal stacksemasleep on Darwin signal stack

panic during panic
stack trace unavailable

exit status 4

This looks like something crashed in C or cgo-related code...

fsck_exfat / Disk Utility reports that the disk is OK.

(Another data point: I ran the same test again, and it's back to the same SIGBUS error at the same place.)

[mholt]

I swapped out for modernc.org/sqlite and re-ran the test, and it ALSO crashed at the exact same spot:

unexpected fault address 0x107c415f2
fatal error: fault
[signal SIGBUS: bus error code=0x1 addr=0x107c415f2 pc=0x1028a580c]

goroutine 9660842 gp=0x14003fd5dc0 m=24 mp=0x1400109f008 [running]:
runtime.throw({0x103825224?, 0x1028a12b8?})
        /usr/local/go/src/runtime/panic.go:1094 +0x34 fp=0x14002cfc1d0 sp=0x14002cfc1a0 pc=0x1023aee84
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:922 +0x164 fp=0x14002cfc230 sp=0x14002cfc1d0 pc=0x1023b1064
modernc.org/sqlite/lib._walFindFrame(0x14000d249c0, 0x1363ef028, 0xbf87, 0x136146450)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:49745 +0x1cc fp=0x14002cfc2b0 sp=0x14002cfc240 pc=0x1028a580c
modernc.org/sqlite/lib._sqlite3WalFindFrame(...)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:49790
modernc.org/sqlite/lib._readDbPage(0x14000d249c0, 0x13afcd060)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:42204 +0x8c fp=0x14002cfc330 sp=0x14002cfc2b0 pc=0x102898dbc
modernc.org/sqlite/lib._getPageNormal(0x14000d249c0, 0x13640d028, 0xbf87, 0x136146430, 0x2)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:44608 +0x1c0 fp=0x14002cfc3b0 sp=0x14002cfc330 pc=0x10289d610
modernc.org/sqlite/lib._sqlite3PagerGet(...)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:44700
modernc.org/sqlite/lib._getAndInitPage(0x14000d249c0, 0x1084d8528, 0xbf87, 0x136c163f0, 0x2)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:53849 +0xac fp=0x14002cfc430 sp=0x14002cfc3b0 pc=0x1028abd3c
modernc.org/sqlite/lib._moveToChild(0x14000d249c0, 0x1028b2d14?, 0x2cfc4b8?)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:56756 +0xa0 fp=0x14002cfc470 sp=0x14002cfc430 pc=0x1028b2b80
modernc.org/sqlite/lib._moveToLeftmost(0x14000d249c0, 0x136c16368)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:56927 +0xa0 fp=0x14002cfc4a0 sp=0x14002cfc470 pc=0x1028b30b0
modernc.org/sqlite/lib._sqlite3BtreeNext(0x138b4002c?, 0x136146430?, 0xd249c0?)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:57646 +0x84 fp=0x14002cfc4c0 sp=0x14002cfc4a0 pc=0x1028b45e4
modernc.org/sqlite/lib._btreeNext(0x14000d249c0, 0x136c16368)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:57611 +0x270 fp=0x14002cfc500 sp=0x14002cfc4c0 pc=0x1028b4530
modernc.org/sqlite/lib._sqlite3BtreeNext(0x14000d249c0?, 0x136c159d0?, 0x361461f8?)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:57641 +0xa4 fp=0x14002cfc520 sp=0x14002cfc500 pc=0x1028b4604
modernc.org/sqlite/lib._sqlite3VdbeExec(0x14000d249c0, 0x1363ec628)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:78783 +0x27c4 fp=0x14002cfcd30 sp=0x14002cfc520 pc=0x1028dc094
modernc.org/sqlite/lib._sqlite3Step(0x14000d249c0, 0x1363ec628)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:71069 +0x68 fp=0x14002cfcd70 sp=0x14002cfcd30 pc=0x1028d4dc8
modernc.org/sqlite/lib.Xsqlite3_step(0x14000d249c0, 0x1363ec628)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/lib/sqlite_darwin_arm64.go:71132 +0xc8 fp=0x14002cfcdd0 sp=0x14002cfcd70 pc=0x1028d5158
modernc.org/sqlite.(*conn).step(0x14000e00000, 0x1363ec628)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/sqlite.go:1096 +0x34 fp=0x14002cfce00 sp=0x14002cfcdd0 pc=0x102a115c4
modernc.org/sqlite.(*stmt).query.func2(...)
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/sqlite.go:690
modernc.org/sqlite.(*stmt).query(0x14002ea58b0, {0x104c54ca0, 0x14002d28730}, {0x14002ccc2d0, 0x6, 0x6})
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/sqlite.go:731 +0x1f4 fp=0x14002cfcef0 sp=0x14002cfce00 pc=0x102a0f944
modernc.org/sqlite.(*conn).query(0x13615ffa8?, {0x104c54ca0, 0x14002d28730}, {0x1039101b6?, 0x1?}, {0x14002ccc2d0, 0x6, 0x6})
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/sqlite.go:1843 +0xc0 fp=0x14002cfcf80 sp=0x14002cfcef0 pc=0x102a13d00
modernc.org/sqlite.(*conn).QueryContext(0x14002cfd058?, {0x104c54ca0?, 0x14002d28730?}, {0x1039101b6?, 0x14002ccc2d0?}, {0x14002ccc2d0?, 0x14002cfd058?, 0x10299bcfc?})
        /Users/matt/go/pkg/mod/modernc.org/sqlite@v1.39.0/sqlite_go18.go:63 +0x2c fp=0x14002cfcfd0 sp=0x14002cfcf80 pc=0x102a16cec
database/sql.ctxDriverQuery({0x104c54ca0?, 0x14002d28730?}, {0x1089b5bc8?, 0x14000e00000?}, {0x0?, 0x0?}, {0x1039101b6?, 0x102872f88?}, {0x14002ccc2d0?, 0x10234549c?, ...})
        /usr/local/go/src/database/sql/ctxutil.go:48 +0xac fp=0x14002cfd060 sp=0x14002cfcfd0 pc=0x10267fccc
database/sql.(*DB).queryDC.func1()
        /usr/local/go/src/database/sql/sql.go:1786 +0xe0 fp=0x14002cfd120 sp=0x14002cfd060 pc=0x102687e00
database/sql.withLock({0x104c49598, 0x14000d06900}, 0x14002cfd288)
        /usr/local/go/src/database/sql/sql.go:3572 +0x74 fp=0x14002cfd160 sp=0x14002cfd120 pc=0x10268f4c4
database/sql.(*DB).queryDC(0x14000807930?, {0x104c54ca0, 0x14002d28730}, {0x0, 0x0}, 0x14000d06900, 0x14002ea58a0, {0x1039101b6, 0x2d4}, {0x14002cfd950, ...})
        /usr/local/go/src/database/sql/sql.go:1781 +0x11c fp=0x14002cfd310 sp=0x14002cfd160 pc=0x1026878cc
database/sql.(*DB).query(0x14000807930, {0x104c54ca0, 0x14002d28730}, {0x1039101b6, 0x2d4}, {0x14002cfd950, 0x6, 0x6}, 0x0?)
        /usr/local/go/src/database/sql/sql.go:1764 +0xb4 fp=0x14002cfd390 sp=0x14002cfd310 pc=0x102687754
database/sql.(*DB).QueryContext.func1(0x0?)
        /usr/local/go/src/database/sql/sql.go:1742 +0x40 fp=0x14002cfd400 sp=0x14002cfd390 pc=0x1026875b0
database/sql.(*DB).retry(0x0?, 0x14002cfd480)
        /usr/local/go/src/database/sql/sql.go:1576 +0x4c fp=0x14002cfd450 sp=0x14002cfd400 pc=0x1026864dc
database/sql.(*DB).QueryContext(0x697d3?, {0x104c54ca0?, 0x14002d28730?}, {0x1039101b6?, 0xba?}, {0x14002cfd950?, 0x2?, 0x2?})
        /usr/local/go/src/database/sql/sql.go:1741 +0x80 fp=0x14002cfd4e0 sp=0x14002cfd450 pc=0x102687510
github.com/timelinize/timelinize/timeline.thumbnailJob.iteratePagesOfTasksFromImportJob({{0x0, 0x0, 0x0}, 0x1, 0x0}, 0x140022acb40, 0x1)
        /Users/matt/Dev/timelinize/timeline/thumbnails.go:192 +0x1e4 fp=0x14002cfd9f0 sp=0x14002cfd4e0 pc=0x102dd7404
github.com/timelinize/timelinize/timeline.thumbnailJob.Run({{0x0, 0x0, 0x0}, 0x1, 0x0}, 0x140022acb40, {0x0?, 0x0?, 0x14000af1ba8?})
        /Users/matt/Dev/timelinize/timeline/thumbnails.go:118 +0x124 fp=0x14002cfdb30 sp=0x14002cfd9f0 pc=0x102dd6d84
github.com/timelinize/timelinize/timeline.(*thumbnailJob).Run(0x104bdf340?, 0x10382f50b?, {0x0?, 0x14002ca3c80?, 0x2?})
        <autogenerated>:1 +0x70 fp=0x14002cfdbb0 sp=0x14002cfdb30 pc=0x102de9e20
github.com/timelinize/timelinize/timeline.(*Timeline).runJob.func1(_, _, _, {{0x14001b1ea50, 0x24}, 0x2, {0x14002d15a70, 0xa}, 0x0, {0x14000058b40, ...}, ...}, ...)
        /Users/matt/Dev/timelinize/timeline/jobs.go:463 +0x254 fp=0x14002cfdea0 sp=0x14002cfdbb0 pc=0x102da9c94
github.com/timelinize/timelinize/timeline.(*Timeline).runJob.gowrap1()
        /Users/matt/Dev/timelinize/timeline/jobs.go:561 +0x5c fp=0x14002cfdfd0 sp=0x14002cfdea0 pc=0x102da9a0c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x14002cfdfd0 sp=0x14002cfdfd0 pc=0x1023b7744
created by github.com/timelinize/timelinize/timeline.(*Timeline).runJob in goroutine 293
        /Users/matt/Dev/timelinize/timeline/jobs.go:424 +0x604
...(1,200 more lines)...

Claude Sonnet, which has been following me along this journey (and upgraded to Sonnet 4.5 today), has this to say:

Well, that eliminates the CGO/go-sqlite3 theory entirely. The pure-Go driver is crashing with the same SIGBUS on a misaligned address (0x107c415f2 % 8 = 2).
This is now definitively pointing to macOS's exFAT implementation having a bug with memory-mapped WAL files.

Look at where it's crashing in modernc:

modernc.org/sqlite/lib._walFindFrame(0x14000d249c0, 0x1363ef028, 0xbf87, 0x136146450)

This is reading from the WAL index (the -shm file that's memory-mapped). The fact that both drivers crash when accessing WAL structures suggests the exFAT driver on macOS is returning misaligned data when SQLite mmaps the WAL shared memory file.

This is an Apple Bug

Since both:

• mattn/go-sqlite3 (CGO to C SQLite)
• modernc.org/sqlite (pure Go transpiled SQLite)

...crash the same way on exFAT but not APFS, the common denominator is macOS's exFAT filesystem driver providing misaligned memory mappings for the WAL shared memory file.

I can now confidently rule out the mattn sqlite3 driver. For once I agree with an LLM... I think this is a bug in Apple's ExFAT driver.

[mholt]

Ok. Yeah, I'm just gonna suggest we advise against WAL mode on ExFAT, maybe ExFAT on MacOS specifically. It appears to be broken. The ExFAT driver is returning a misaligned address for ARM64.