disk I/O error during large transactions on Windows using WAL

[Hakkin]

I am getting "disk I/O error" during large transactions with WAL on Windows.

Initially hit this while trying to insert a large amount of hash blobs into a DB in a single transaction, below is a reproducer.

I can't seem to trigger this on Linux, and it only seems to trigger when using WAL journal mode, so I suppose it's something in the Windows-specific WAL code.

Issue occurs on both previous and latest release version.

package main

import (
	"database/sql"
	"encoding/binary"
	"errors"
	"fmt"
	"log"

	"github.com/ncruces/go-sqlite3"
	_ "github.com/ncruces/go-sqlite3/driver"
	_ "github.com/ncruces/go-sqlite3/embed"
)

func sqliteError(err error) error {
	if e := sqlite3.ExtendedErrorCode(0); errors.As(err, &e) {
		return fmt.Errorf("%w (%d)", err, e)
	}
	return err
}

func main() {
	db, err := sql.Open("sqlite3", "file:hashes.db?_pragma=journal_mode(wal)")
	if err != nil {
		log.Fatalf("open: %v", sqliteError(err))
	}
	defer db.Close()

	db.SetMaxOpenConns(1)

	_, err = db.Exec(`
		CREATE TABLE IF NOT EXISTS hashes (
			hash BLOB,
			PRIMARY KEY (hash ASC)
		) WITHOUT ROWID;
	`)
	if err != nil {
		log.Fatalf("create: %v", sqliteError(err))
	}

	stmt, err := db.Prepare(`INSERT INTO hashes(hash) VALUES (?);`)
	if err != nil {
		log.Fatalf("prepare: %v", sqliteError(err))
	}

	var total uint64
	var hash [8]byte
	for batchSize := uint64(1); ; batchSize *= 2 {
		log.Printf("batch = %d", batchSize)
		tx, err := db.Begin()
		if err != nil {
			log.Fatalf("begin: %v", sqliteError(err))
		}
		txStmt := tx.Stmt(stmt)
		for i := uint64(0); i < batchSize; i++ {
			binary.BigEndian.PutUint64(hash[:], total+i)
			_, err = txStmt.Exec(hash[:])
			if err != nil {
				log.Fatalf("exec: %v", sqliteError(err))
			}
		}
		err = tx.Commit()
		if err != nil {
			log.Fatalf("commit: %v", sqliteError(err))
		}
		total += batchSize
	}
}

2025/11/05 19:48:17 batch = 1
2025/11/05 19:48:17 batch = 2
2025/11/05 19:48:17 batch = 4
2025/11/05 19:48:17 batch = 8
2025/11/05 19:48:17 batch = 16
2025/11/05 19:48:17 batch = 32
2025/11/05 19:48:17 batch = 64
2025/11/05 19:48:17 batch = 128
2025/11/05 19:48:17 batch = 256
2025/11/05 19:48:17 batch = 512
2025/11/05 19:48:17 batch = 1024
2025/11/05 19:48:17 batch = 2048
2025/11/05 19:48:17 batch = 4096
2025/11/05 19:48:17 batch = 8192
2025/11/05 19:48:17 batch = 16384
2025/11/05 19:48:17 batch = 32768
2025/11/05 19:48:18 batch = 65536
2025/11/05 19:48:18 batch = 131072
2025/11/05 19:48:19 batch = 262144
2025/11/05 19:48:20 batch = 524288
2025/11/05 19:48:23 batch = 1048576
2025/11/05 19:48:28 batch = 2097152
2025/11/05 19:48:34 exec: sqlite3: disk I/O error (5386)

[Hakkin]

This occurs down to v0.20.2 which is the released that added shared-memory WAL on Windows.

Opening the DB with _pragma=locking_mode(exclusive)&_pragma=journal_mode(wal) stops the error from occurring, so probably something in the Windows shared-memory implementation.

[ncruces]

I don't have access to windows hardware ATM.

Did you run this test with the latest version (v0.30.0)? If not, can you try? If don't expect anything to work, but the error message might be better.

[ncruces]

Also, if you can run this under a debugger, it'd be very helpful to know which of the error exits in this function triggers the behavior:

go-sqlite3/vfs/shm_windows.go

Line 69 in 6d92aa1

func (s *vfsShm) shmMap(ctx context.Context, mod api.Module, id, size int32, extend bool) (_ ptr_t, err error) {

It might seem IOERR_SHMMAP is only returned in one place, but generic errors returned by that function also get that code by default.

[Hakkin]

Yes, I originally encountered this on v0.29.1 but also checked it on the latest release before opening the issue. There doesn't seem to be any difference with the returned error, both versions seems to return sqlite3.ExtendedErrorCode rather than *sqlite3.Error so I don't think there's any other error information besides the extended SQLite result code 5386 (SQLITE_IOERR_SHMMAP)

Setting a breakpoint on all the err returns in that function, it seems like it's util.MapRegion returning the error.

err = error(syscall.Errno) golang.org/x/sys/windows.ERROR_MAPPED_ALIGNMENT (1132) returned here:

go-sqlite3/vfs/shm_windows.go

Lines 99 to 108 in 6d92aa1

	// Maps regions into memory.
	for int(id) >= len(s.shared) {
	r, err := util.MapRegion(ctx, mod, s.File, int64(id)*int64(size), size)
	if err != nil {
	return 0, err
	}
	s.regions = append(s.regions, r)
	s.shared = append(s.shared, r.Data)
	}

[Hakkin]

And the error in util.MapRegion comes from

go-sqlite3/internal/util/mmap_windows.go

Lines 25 to 30 in 6d92aa1

	a, err := windows.MapViewOfFile(h, windows.FILE_MAP_WRITE,
	uint32(offset>>32), uint32(offset), uintptr(size))
	if a == 0 {
	windows.CloseHandle(h)
	return nil, err
	}

where offset = 32768 and size = 32768

[ncruces]

Thanks, that helps. In more ways than one.

The error should've surfaced, I think I know why it didn't. Dumb bug, simple fix.

And I need to figure out the syscall issue, because that's something I'd expect would work.

[ncruces]

@Hakkin are you able to test the most recent commit (5e73c5d) to main?

[Hakkin]

That seems to have fixed it

[ncruces]

Great!

Unsurprisingly, the day I made a new build, wasi-sdk made a pre-release. Let me give it a day or two to see if it goes full release, and I'll tag a release myself.