Error: EBADF: bad file descriptor, close #581

evbogue · 2018-11-28T14:54:32Z

I'm seeing a lot of these in my terminal. I am using scuttlebot@13.0.3. I'm not scuttlebot@13.2.0, because gossip is broken.

Error: EBADF: bad file descriptor, close

Is anyone else seeing this? Maybe I need to resync my db because something is corrupt?

The text was updated successfully, but these errors were encountered:

evbogue · 2018-11-28T15:20:34Z

cryptix · 2018-11-28T15:37:11Z

have you tried scuttlebot-release with those versions? I fear rolling back to an older version without a package-lock or shrinkwrap will just give you more inconsistencies in the module tree.

evbogue · 2018-11-28T19:29:43Z

I'm not seeing this anymore, so maybe it just had to do with my machine powering off, or a bunk npm install? This is a ghost bug, we can close.

evbogue · 2018-11-29T22:03:46Z

This is happening again, after not happening for awhile, re-opening.

christianbundy · 2018-12-07T18:12:04Z

Just started seeing this as well, you're not alone. 👯‍♂️

christianbundy · 2018-12-07T18:57:13Z

@evbogue Could you try strace?

strace -e trace=desc ./bin.js server &> output.log

I'm seeing two instances of EBADF, but neither are being output to the terminal (only strace):

  253 fstat(0, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 6), ...}) = 0
  254 fstat(1, {st_mode=S_IFREG|0644, st_size=20579, ...}) = 0
  255 fstat(2, {st_mode=S_IFREG|0644, st_size=20636, ...}) = 0
  256 epoll_create1(EPOLL_CLOEXEC)            = 3
  257 pipe2([4, 5], O_CLOEXEC)                = 0
  258 write(5, "*", 1)                        = 1
  259 pipe2([6, 7], O_NONBLOCK|O_CLOEXEC)     = 0
  260 eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 8
  261 ioctl(0, FIOCLEX)                       = 0
  262 ioctl(1, FIOCLEX)                       = 0
  263 ioctl(2, FIOCLEX)                       = 0
  264 ioctl(3, FIOCLEX)                       = 0
  265 ioctl(4, FIOCLEX)                       = 0
  266 ioctl(5, FIOCLEX)                       = 0
  267 ioctl(6, FIOCLEX)                       = 0
  268 ioctl(7, FIOCLEX)                       = 0
  269 ioctl(8, FIOCLEX)                       = 0
  270 ioctl(9, FIOCLEX)                       = -1 EBADF (Bad file descriptor)
  271 ioctl(10, FIOCLEX)                      = -1 EBADF (Bad file descriptor)
  272 ioctl(11, FIOCLEX)                      = -1 EBADF (Bad file descriptor)
  273 ioctl(12, FIOCLEX)                      = -1 EBADF (Bad file descriptor)
  274 ioctl(13, FIOCLEX)                      = -1 EBADF (Bad file descriptor)
  275 ioctl(14, FIOCLEX)                      = -1 EBADF (Bad file descriptor)
  276 ioctl(15, FIOCLEX)                      = -1 EBADF (Bad file descriptor)
  277 ioctl(16, FIOCLEX)                      = -1 EBADF (Bad file descriptor)

75145 epoll_pwait(3, [], 1024, 88, NULL, 8)   = 0
75146 epoll_pwait(3, [], 1024, 139, NULL, 8)  = 0
75147 epoll_pwait(3, [], 1024, 18, NULL, 8)   = 0
75148 epoll_pwait(3, [], 1024, 24, NULL, 8)   = 0
75149 close(25)                               = 0
75150 epoll_pwait(3, [], 1024, 0, NULL, 8)    = 0
75151 epoll_pwait(3, [], 1024, 10, NULL, 8)   = 0
75152 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=94519345283080}}], 1024, 200, NULL, 8) = 1
75153 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
75154 epoll_ctl(3, EPOLL_CTL_DEL, -1, 0x7ffef49db07c) = -1 EBADF (Bad file descriptor)
75155 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=94519345283080}}], 1024, 195, NULL, 8) = 1
75156 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
75157 epoll_ctl(3, EPOLL_CTL_DEL, -1, 0x7ffef49db07c) = -1 EBADF (Bad file descriptor)
75158 epoll_pwait(3, [], 1024, 182, NULL, 8)  = 0
75159 epoll_pwait(3, [], 1024, 200, NULL, 8)  = 0
75160 epoll_pwait(3, [], 1024, 199, NULL, 8)  = 0
75161 epoll_pwait(3, [], 1024, 1, NULL, 8)    = 0
75162 epoll_pwait(3, [], 1024, 107, NULL, 8)  = 0
75163 close(25)                               = 0

I'm going to keep this running and see whether I can capture the error that actually outputs to the terminal, but since it's sporadic it'd be great if you could do the same. When it outputs to the terminal, you should see a line that literally just says "Error: EBADF: bad file descriptor, close" on its own line without any strace prefix.

christianbundy · 2018-12-07T20:39:56Z

Got one:

1400 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1401 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1402 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1403 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1404 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1405 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1406 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1407 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1408 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1409 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1410 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1411 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1412 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1413 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1414 epoll_pwait(3, [{EPOLLIN, {u32=8, u64=9146434982011469832}}], 1024, -1, NULL, 8) = 1
1415 read(8, "\1\0\0\0\0\0\0\0", 1024)       = 8
1416 write(1, "\n> scuttlebot-install@0.0.0 star"..., 123
1417 > scuttlebot-install@0.0.0 start /home/pi/Source/scuttlebot
1418 > sbot server --no-replicate.legacy --host=libra.fraction.io
1419 
1420 ) = 123
1421 pipe2([17, 18], O_CLOEXEC)              = 0
1422 read(4, "*", 1)                         = 1
1423 write(5, "*", 1)                        = 1
1424 close(18)                               = 0
1425 read(17, "", 4)                         = 0
1426 close(17)                               = 0
1427 read(4, "*", 1)                         = 1
1428 write(5, "*", 1)                        = 1
1429 read(4, "*", 1)                         = 1
1430 write(5, "*", 1)                        = 1
1431 epoll_pwait(3, scuttlebot 13.2.1 /home/pi/.ssb logging.level:notice
1432 my key ID: gTsBPQbdSd763h0qoZsBjPlgWbhKKmMFW8kmTv/XrV8=.ed25519
1433 WARNING-DEPRECATION: ssb-links not installed as a plugin. If you are using git-ssb, ssb-npm or patchfoo please consider installing it
1434 Listening on libra.fraction.io:8008 (multiserver net plugin)
1435 Listening on libra.fraction.io:8989 (multiserver ws plugin)
1436 ssb-friends: stream legacy api used
1437 Error: EBADF: bad file descriptor, close
1438 Error: EBADF: bad file descriptor, close
1439 Error: EBADF: bad file descriptor, close
1440 Error: EBADF: bad file descriptor, close
1441 Error: EBADF: bad file descriptor, close
1442 Error: EBADF: bad file descriptor, close
1443 Error: EBADF: bad file descriptor, close
1444 Error: EBADF: bad file descriptor, close
1445 Error: EBADF: bad file descriptor, close
1446 Error: EBADF: bad file descriptor, close
1447 Error: EBADF: bad file descriptor, close
1448 Error: EBADF: bad file descriptor, close
1449 Error: EBADF: bad file descriptor, close
1450 Error: EBADF: bad file descriptor, close
1451 Error: EBADF: bad file descriptor, close
1452 Error: EBADF: bad file descriptor, close
1453 strace: Process 1965 detached
1454  <detached ...>
1455 mmap2(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f8e000
1456 open("/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
1457 fstat64(3, {st_mode=S_IFREG|0644, st_size=42, ...}) = 0
1458 mmap2(NULL, 42, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x76f8d000
1459 close(3)                                = 0
1460 open("/usr/lib/arm-linux-gnueabihf/libarmmem.so", O_RDONLY|O_CLOEXEC) = 3
1461 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\210\5\0\0004\0\0\0"..., 512) = 512
1462 lseek(3, 20868, SEEK_SET)               = 20868
1463 read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1000) = 1000

christianbundy · 2018-12-07T20:41:26Z

@evbogue Any chance you're also running this with something like?

while true; do timeout 1h sbot server; done

I'm wondering whether the issue is coming from timeout restarting the server quickly.

christianbundy · 2018-12-08T03:10:20Z

It's evolving...

$ cd ~/src/patchbay
$ electron index.js
STARTING electron
STARTING SBOT
LOADING config
Listening on undefined:8008 (multiserver net plugin)
listening on socket unix:/home/christianbundy/.ssb/socket
ssb-friends: stream legacy api used
Listening on localhost:8989 (ssb-ws)
  ssb-server connecting longterm +0ms
  ssb-server connecting longterm +2ms
  ssb-server connecting attempt +2ms
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
[5192:1207/190853.264819:FATAL:scoped_file.cc(48)] Check failed: 0 == ret. : Bad file descriptor
#0 0x55b30b3945d7 <unknown>
#1 0x55b30b36696e <unknown>
#2 0x55b30b3671da <unknown>
#3 0x55b30b37c886 <unknown>
#4 0x55b30b350080 <unknown>
#5 0x55b30b34b1f9 <unknown>
#6 0x55b30c623ab7 <unknown>
#7 0x55b30c5576ae <unknown>
#8 0x55b30c550327 <unknown>
#9 0x55b30b3ab7bd <unknown>
#10 0x55b30b3b2923 <unknown>
#11 0x55b30b3b2e40 <unknown>
#12 0x55b30b3b25bc <unknown>
#13 0x55b30b3b0615 <unknown>
#14 0x55b30b3766c3 <unknown>
#15 0x7f76f83f6a9d start_thread
#16 0x7f76f3401b23 __GI___clone

Received signal 6
#0 0x55b30b3945d7 <unknown>
#1 0x55b30b39414f <unknown>
#2 0x7f76f84013c0 <unknown>
#3 0x7f76f333dd7f __GI_raise
#4 0x7f76f3328672 __GI_abort
#5 0x55b30b367442 <unknown>
#6 0x55b30b366f02 <unknown>
#7 0x55b30b3671da <unknown>
#8 0x55b30b37c886 <unknown>
#9 0x55b30b350080 <unknown>
#10 0x55b30b34b1f9 <unknown>
#11 0x55b30c623ab7 <unknown>
#12 0x55b30c5576ae <unknown>
#13 0x55b30c550327 <unknown>
#14 0x55b30b3ab7bd <unknown>
#15 0x55b30b3b2923 <unknown>
#16 0x55b30b3b2e40 <unknown>
#17 0x55b30b3b25bc <unknown>
#18 0x55b30b3b0615 <unknown>
#19 0x55b30b3766c3 <unknown>
#20 0x7f76f83f6a9d start_thread
#21 0x7f76f3401b23 __GI___clone
  r8: 0000000000000000  r9: 00007f76dd85dce0 r10: 0000000000000008 r11: 0000000000000246
 r12: 00007f76f36716c0 r13: 00007f76dd85e440 r14: 00007f76dd85e440 r15: 00007f76dd85e438
  di: 0000000000000002  si: 00007f76dd85dce0  bp: 00007f76dd85e390  bx: 0000000000000006
  dx: 0000000000000000  ax: 0000000000000000  cx: 00007f76f333dd7f  sp: 00007f76dd85dce0
  ip: 00007f76f333dd7f efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]

cryptix · 2018-12-08T11:54:15Z

75154 epoll_ctl(3, EPOLL_CTL_DEL, -1, 0x7ffef49db07c) = -1 EBADF (Bad file descriptor)

i only read https://linux.die.net/man/2/epoll_ctl but I don't think -1 is a valid fdesc to pass there. Without a trace from the js side it's really hard to gues what is trying to do that... maaaybe if you gather the other epoll_ctl calls, you could try to find out if the ones that passed with EPOLL_CTL_ADD are network connections or local files.

evbogue · 2018-12-08T11:58:03Z

@evbogue Any chance you're also running this with something like?
while true; do timeout 1h sbot server; done
I'm wondering whether the issue is coming from timeout restarting the server quickly.

I'm not running my local server in a restart loop. But, my computer has powered off a few times because I forget to check acpi, so my hunch is that this is a corrupt flume somewhere.

christianbundy · 2018-12-09T04:19:18Z

My concern is that somewhere in the stack we're doing something like:

if (err) console.log(err)

I'm not super concerned about the bad file descriptor, but I feel like we should at least be printing a stack trace.

cryptix · 2018-12-13T15:40:46Z

also getting this now after updating scuttle-shells sbot from 13.0 to latest.. I also fear my previous comment about multiple epoll sets might be unhelpful. keks suggested that this is what the v8 event loop uses global to schedule it's IO.. in that case there is not much to dig into.. just even more wondering about what part of our system tries to add -1 to it... sounds definitely like an ignored error.

erde74 · 2018-12-14T19:50:52Z

i see this error too. thought it might be caused because i use NetBSD, but this doesn't seems to be the case.

austinfrey · 2019-01-20T03:14:20Z

I'm getting this too when trying to run my pub. Currently using ssb-server 13.5.2.
is there anything else i can provide from a debugging standpoint to help?

ssb-server 13.5.2 /home/punkmonk/.ssb logging.level:notice
my key ID: iMgGq295k0QoORDrYHbn7vmGvGTForIHAhOq/uiJzh8=.ed25519
WARNING-DEPRECATION: ssb-links not installed as a plugin. If you are using git-ssb, ssb-npm or patchfoo please consider installing it
Listening on 159.203.94.136:8008 (multiserver net plugin)
Listening on 159.203.94.136:8989 (multiserver ws plugin)
ssb-friends: stream legacy api used
Error: EBADF: bad file descriptor, read........... (ooo:42.9%, ebt:80%)
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, read........... (ooo:42.9%, ebt:33.3%)
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, read........... (ooo:42.9%, ebt:66.7%)
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, close
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, read
Error: EBADF: bad file descriptor, close

christianbundy · 2019-01-20T16:45:57Z

It looks like this is actually coming from pull-file: pull-stream/pull-file#6

stale · 2019-06-20T05:10:58Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

christianbundy added the bug label Nov 28, 2018

evbogue closed this as completed Nov 28, 2018

evbogue reopened this Nov 29, 2018

cryptix mentioned this issue Dec 14, 2018

[WIP] scuttle-shell ssbc/patchbay#295

Open

4 tasks

dominictarr added the Backlog label Mar 22, 2019

stale bot added the stale label Jun 20, 2019

evbogue closed this as completed Jun 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: EBADF: bad file descriptor, close #581

Error: EBADF: bad file descriptor, close #581

evbogue commented Nov 28, 2018

evbogue commented Nov 28, 2018

cryptix commented Nov 28, 2018

evbogue commented Nov 28, 2018

evbogue commented Nov 29, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 8, 2018

cryptix commented Dec 8, 2018

evbogue commented Dec 8, 2018

christianbundy commented Dec 9, 2018

cryptix commented Dec 13, 2018

erde74 commented Dec 14, 2018

austinfrey commented Jan 20, 2019

christianbundy commented Jan 20, 2019

stale bot commented Jun 20, 2019

Error: EBADF: bad file descriptor, close #581

Error: EBADF: bad file descriptor, close #581

Comments

evbogue commented Nov 28, 2018

evbogue commented Nov 28, 2018

cryptix commented Nov 28, 2018

evbogue commented Nov 28, 2018

evbogue commented Nov 29, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 7, 2018

christianbundy commented Dec 8, 2018

cryptix commented Dec 8, 2018

evbogue commented Dec 8, 2018

christianbundy commented Dec 9, 2018

cryptix commented Dec 13, 2018

erde74 commented Dec 14, 2018

austinfrey commented Jan 20, 2019

christianbundy commented Jan 20, 2019

stale bot commented Jun 20, 2019