Unable to modify files on kbfs under Fedora 25

[kallisti5]

After using for a while, the ability to write to my kbfs is broken:

$ cp /home/kallisti5/msgpack.diff /keybase/public/kallisti5/
cp: cannot create regular file '/keybase/public/kallisti5/msgpack.diff': Input/output error

my log id: 8d754b464904ea349105b31c

2017-03-09T18:27:57.172544-06:00 ▶ [DEBU FUSE serve.go:845] 1d0 -> [ID=0x46] Create error=EIO: Disk limit timeout of 11s reached; requested 313 bytes and 7 files, 44411860935 bytes and 0 files available: context deadline exceeded
github.com/keybase/kbfs/kbfssync.(*Semaphore).Acquire
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/kbfssync/semaphore.go:77
github.com/keybase/kbfs/libkbfs.(*backpressureTracker).beforeBlockPut
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/backpressure_disk_limiter.go:164
github.com/keybase/kbfs/libkbfs.(*backpressureDiskLimiter).beforeBlockPut
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/backpressure_disk_limiter.go:453
github.com/keybase/kbfs/libkbfs.(*tlfJournal).putBlockData
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/tlf_journal.go:1673
github.com/keybase/kbfs/libkbfs.journalBlockServer.Put
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/journal_block_server.go:93
github.com/keybase/kbfs/libkbfs.(*journalBlockServer).Put
	<autogenerated>:1530
github.com/keybase/kbfs/libkbfs.putBlockToServer
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/block_util.go:32
github.com/keybase/kbfs/libkbfs.PutBlockCheckQuota
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/block_util.go:48
github.com/keybase/kbfs/libkbfs.doOneBlockPut
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/block_util.go:61
github.com/keybase/kbfs/libkbfs.doBlockPuts.func1
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/libkbfs/block_util.go:101
github.com/keybase/kbfs/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
	/root/build/gopaths/amd64/src/github.com/keybase/kbfs/vendor/golang.org/x/sync/errgroup/errgroup.go:58
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:2086

[strib]

@kallisti5: thanks for the report. We just added this feature to log writes to local disk first before transferring them over the network, to be faster. However, we're trying to be smart and limit the number of bytes and inodes we use. In this case, it looks like the OS is telling us you have 0 available -- probably because we're doing something wrong.

Can you try df -i on the disk partition hosting your home directory and tell us how many inodes df thinks are available there?

Also, to get around this temporarily, you can do:

echo 1 > /keybase/.kbfs_disable_auto_journals
echo 1 > /keybase/private/kallisti5/.kbfs_disable_journal

to disable this feature for now.

[kallisti5]

$ df -i /home/
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/nvme0n1p4      0     0     0     - /home

$ mount -v | grep home
/dev/nvme0n1p4 on /home type btrfs (rw,relatime,ssd,space_cache,subvolid=258,subvol=/home)

[strib]

Ah, I see. I guess btrfs doesn't report inodes. We'll get a fix for this into master today, thanks!

[kallisti5]

also, the workaround doesn't seem to help. (restarted kbfs as well)

Thanks for all the great work on keybase! great app.

[strib]

Oh. Hrm. Can you post the output of cat /keybase/.kbfs_status and cat /keybase/private/kallisti5/.kbfs_status (assuming you don't see anything you want to keep private in there, I don't think there should be anything), and maybe another keybase log send.

[strib]

(Edited the above with a fixed path for the private folder.)

[kallisti5]

$ cat /keybase/.kbfs_status
cat: /keybase/.kbfs_status: Input/output error

$ cat /keybase/private/kallisti5
cat: /keybase/private/kallisti5: Is a directory

$ ls /keybase/private/kallisti5
<files>

[kallisti5]

log id: 3bfb9da8bf6029531cadb31c

[strib]

ah sorry, I didn't need the ls of your private dir. Feel free to edit that out of the comment. I'll take a look at the log send soon.

[strib]

Huh, I guess this is why you're getting errors looking at the status, will need to look into that. I guess it has to do with the 0 available bytes:

2017-03-10T10:10:02.184302-06:00 ▶ [DEBU FUSE serve.go:845] 0d7 -> [ID=0xa] Lookup error=EIO: json: unsupported value: NaN

As far as why the workaround didn't work, I think there's one more step needed. Try this:

# first cd out of shells that are currently in /keybase
fusermount -u /keybase
rm -rf /home/kallisti5/.local/share/keybase/kbfs_journal/v1/*
run_keybase

That should clear out all evidence of the journals from your disk, and when you restart they should be permanently off. We'll fix the btrfs bug today, so in the next update you should be able to reverse this again by doing:

echo 1 > /keybase/.kbfs_enable_auto_journals

[strib]

@kallisti5: the new linux release should fix the btrfs incompatibility issue, so please re-enable journals as mentioned above, update through your package manager, and re-run run_keybase. If things look ok, please close this out. Thanks!

[kallisti5]

Works fine here now.

Thanks!