Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flushing block file to disk failed when using external drive on macOS Ventura #26455

Closed
aureliusltd opened this issue Nov 4, 2022 · 12 comments
Closed

Flushing block file to disk failed when using external drive on macOS Ventura #26455

aureliusltd opened this issue Nov 4, 2022 · 12 comments
Labels
Data corruption macOS Questions and Help UTXO Db and Indexes

Comments

@aureliusltd
Copy link

aureliusltd commented Nov 4, 2022 •
edited

I kept getting an error regarding Flushing Block to disk. I've tried to use bitcoin core V0.23 and it doesn't help; although V0.22 was better at loading the backup file, but I couldn't send any of my bitcoin out. I can't seem to load the wallet or the backup file properly. Not a coder and hoping someone can help. I don't think it's an issue with the hard drive. Any suggestion? I'm on MacOSx Ventura 13.0.

Debug.log file:

2022-11-04T20:34:28Z Using wallet /Volumes/Extreme SSD/bruceltd wallet.dat
2022-11-04T20:34:28Z BerkeleyEnvironment::Open: LogDir=/Volumes/Extreme SSD/database ErrorFile=/Volumes/Extreme SSD/db.log
.
.
.
2022-11-04T20:58:16Z FileCommit: fcntl F_FULLFSYNC failed: 25
2022-11-04T20:58:16Z ERROR: SerializeFileDB: Failed to flush file /Volumes/Extreme SSD/peers.176f

db.log file:
fsync: Inappropriate ioctl for device
PANIC: Inappropriate ioctl for device
txn_checkpoint: failed to flush the buffer cache: DB_RUNRECOVERY: Fatal error, run database recovery
@brl4n
Copy link

brl4n commented Nov 6, 2022 •
edited

also getting similar internal errors for the past few days. MacOS DiskUtility doesn't report any problems. No other apps are having file IO issues. Tried v24.0.0rc3 and same issue. Able to check blocks for a few minutes then crashes. I recently updated to Ventura but don't know enough to decide if they changed something which would cause this. Let me know if you need more info.

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0100000100000012 -> 0x0000000100000012 (possible pointer authentication failure)
Exception Codes: 0x0000000000000001, 0x0100000100000012

Termination Reason: Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process: exc handler [37021]

VM Region Info: 0x100000012 is not in any region. Bytes before following region: 15368174
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
UNUSED SPACE AT START
--->
__TEXT 100ea8000-10282c000 [ 25.5M] r-x/r-x SM=COW ...OS/Bitcoin-Qt

Kernel Triage:
VM - pmap_enter retried due to resource shortage
VM - pmap_enter retried due to resource shortage

@aureliusltd
Copy link
Author

Update on the issue. The current Bitcoin Core software, V0.22 & V0.23, are not working well with the new Mac Software update 13.0 Ventura. I went back to the previous version, Monterey 12.6, and the software works just fine. How can i help to make the new bitcoin core version, V0.24, more compatible with Mac? Thank you!

@hebasto
Copy link
Member

hebasto commented Nov 12, 2022

Update on the issue. The current Bitcoin Core software, V0.22 & V0.23, are not working well with the new Mac Software update 13.0 Ventura.

Looks similar to #26400.

#26400 (comment):

Using external drives is not a configuration which is extensively tested and guaranteed to work in all circumstances. For example, #26397 (comment).

Please try to reproduce the issue while using local drives only. You could use pruning to limit block storage space.

@john-moffett john-moffett mentioned this issue Nov 18, 2022
Closed
@hebasto hebasto changed the title Flushing block file to disk failed. This is likely the result of an I/O error. Flushing block file to disk failed when using external drive on macOS Ventura Nov 18, 2022
@hebasto
Copy link
Member

hebasto commented Nov 18, 2022

The current Bitcoin Core software, V0.22 & V0.23, are not working well with the new Mac Software update 13.0 Ventura. I went back to the previous version, Monterey 12.6, and the software works just fine.

I've updated the issue title.

@hebasto
Copy link
Member

hebasto commented Nov 18, 2022

@aureliusltd @mateusgalasso

What are filesystems on your external drives?

@mateusgalasso
Copy link

@aureliusltd @mateusgalasso

What are filesystems on your external drives?

ExFAT

@john-moffett
Copy link
Contributor

john-moffett commented Nov 18, 2022 •
edited

It looks like other projects (LevelDB, SQLite, etc.) have had similar issues with fcntl using F_FULLFSYNC. Apparently there are certain external drives / filesystems that don't support it, so they return an error. I can only speculate why it seems to be only showing up on macOS Ventura.

Those projects have all used the same strategy: fallback to fsync on error (see previous links or this helpful twitter thread). It seems like the error is specifically ENOTTY in this case, so we could check for that and try fsync.

I've created a branch that does that. Happy to submit as a PR if this approach seems sound.

https://github.com/john-moffett/bitcoin/blob/d99a8bee66656d29017b42fb2c0935de768ccd70/src/util/system.cpp#L1187-L1200

Edit: It may not fix OP's full issue, as BerkeleyDB falls back to fsync only on ENOTSUP. See here:

https://github.com/observerdev/db-4.8.30/blob/3c75267f11336acb1be06bed419e8056cc689d09/os/os_fsync.c#L81-L88

@hebasto
Copy link
Member

hebasto commented Nov 19, 2022

I've created a branch that does that. Happy to submit as a PR if this approach seems sound.

A well motivated PR is better than a comment in an issue :)

@john-moffett john-moffett mentioned this issue Nov 19, 2022
Closed
@hebasto
Copy link
Member

hebasto commented Nov 30, 2022

Apparently, it is an macOS 13 issue. Upgrading to 13.1 should fix it:

Close?

@mateusgalasso
Copy link

Apparently, it is an macOS 13 issue. Upgrading to 13.1 should fix it:

Close?

There is no 13.1
image

@john-moffett
Copy link
Contributor

13.1 is available as a public beta: https://beta.apple.com/sp/betaprogram/welcome

@fanquake fanquake mentioned this issue Dec 6, 2022
Closed
@asylejmani asylejmani mentioned this issue Dec 6, 2022
Closed
@asylejmani
Copy link

I can confirm it works in 13.1! Had v24.0.0 running all night without any issues.

@john-moffett john-moffett mentioned this issue Dec 7, 2022
Merged
fanquake added a commit to bitcoin-core/gui that referenced this issue Jul 20, 2023
@fanquake
Merge bitcoin/bitcoin#26654: util: Show descriptive error messages wh…
ac7c177 
…en FileCommit fails

5408a55 Consolidate Win32-specific error formatting (John Moffett)
c95a443 Show descriptive error messages when FileCommit fails (John Moffett)

Pull request description:

  Only raw [`errno`](https://en.cppreference.com/w/cpp/error/errno) int values are logged if `FileCommit` fails. These values are implementation-specific, so it makes it harder to debug based on user reports. For instance, bitcoin/bitcoin#26455 (comment) and [another](https://bitcointalk.org/index.php?topic=5182526.0#:~:text=FileCommit%3A%20FlushFileBuffers%20failed%3A%205).

  Instead, use `SysErrorString` (or the refactored Windows equivalent `Win32ErrorString`) to display both the raw int value and the descriptive message. All other instances in the code I could find where `errno` or (Windows-only) `GetLastError()`/`WSAGetLastError()` are logged use the full descriptive string. For example:

  https://github.com/bitcoin/bitcoin/blob/1b680948d43b1d39645b9d839a6fa7c6c1786b51/src/util/sock.cpp#L390

  https://github.com/bitcoin/bitcoin/blob/1b680948d43b1d39645b9d839a6fa7c6c1786b51/src/util/sock.cpp#L272

  https://github.com/bitcoin/bitcoin/blob/7e1007a3c6c9a921c2b60919b84a60eaabfe1c5d/src/netbase.cpp#L515-L516

  https://github.com/bitcoin/bitcoin/blob/8ccab65f289e3cce392cbe01d5fc0e7437f51f1e/src/init.cpp#L164

  I refactored the Windows formatting code to put it in `syserror.cpp`, as it's applicable to all Win32 API system errors, not just networking errors. To be clear, the Windows API functions `WSAGetLastError()` and `GetLastError()` are currently [equivalent](https://stackoverflow.com/questions/15586224/is-wsagetlasterror-just-an-alias-for-getlasterror).

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 5408a55 💡

Tree-SHA512: 3921cbac98bd9edaf84d3dd7a43896c7921f144c8ca2cde9bc96d5fb05281f7c55e7cc99db8debf6203b5f916f053025e4fa741f51458fe2c53bb57b0a781027
@hebasto hebasto mentioned this issue Sep 30, 2023
Closed
1 task
@bitcoin bitcoin locked and limited conversation to collaborators Dec 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Data corruption macOS Questions and Help UTXO Db and Indexes
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix macOS failing to flush blockfiles to disk for certain external drives john-moffett/bitcoin
8 participants
@fanquake @brl4n @mateusgalasso @maflcko @asylejmani @hebasto @john-moffett @aureliusltd