Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to "free" the physical Bluetooth interface so that it can be used by external commands? #17

Closed
jbaldwinroberts opened this issue Nov 22, 2016 · 15 comments

Comments

@jbaldwinroberts
Copy link

I would like to be able to use the physical Bluetooth interface without stopping the go application so I need some way to free up the device once it has been initialised. Is this possible?

While the code is running I get errors as expected:

package main

import (
	"flag"
	"fmt"
	"log"
	"time"

	"github.com/currantlabs/ble/examples/lib/dev"
)

var (
	device = flag.String("device", "default", "implementation of ble")
)

func main() {
	_, err := dev.NewDevice(*device)
	if err != nil {
		log.Fatalf("can't new device : %s", err)
	}
	fmt.Println("opened")

	time.Sleep(100 * time.Second)
}
 ~  gatttool -I -t random -b E0:1A:35:05:47:EB
[E0:1A:35:05:47:EB][LE]> connect
Attempting to connect to E0:1A:35:05:47:EB
Error: connect: No route to host (113)
[E0:1A:35:05:47:EB][LE]> 
[E0:1A:35:05:47:EB][LE]> 
 ~  sudo hciconfig hci0 reset
Can't down device hci0: Device or resource busy (16)
 ✘  ~  sudo hciconfig hci0 down 
Can't down device hci0: Device or resource busy (16)
 ✘  ~  hciconfig dev
hci0:    Type: Primary  Bus: USB
    BD Address: E4:B3:18:4D:40:8F  ACL MTU: 1021:4  SCO MTU: 96:6
    UP RUNNING 
    RX bytes:49338 acl:2 sco:0 events:3888 errors:0
    TX bytes:625918 acl:181 sco:0 commands:4087 errors:0

 ~  gatttool -I -t random -b E0:1A:35:05:47:EB
[E0:1A:35:05:47:EB][LE]> connect
Attempting to connect to E0:1A:35:05:47:EB
Error: connect: No route to host (113)
[E0:1A:35:05:47:EB][LE]> 

Is it possible to free the device created and then use external commands during the sleep period?

@deadprogram
Copy link
Contributor

Hi @josephroberts from a reading of the current code this is not currently possible, at least not in the manner you described.

At least on Linux, the NewDevice() function is opening the HCI and does not release it until Stop() is called, at which point you would need another NewDevice() call to reopen the connection.

@bbartol
Copy link

bbartol commented Jan 28, 2017

Speaking of the "Stop()" method (in ble/linux/device.go), it doesn't appear to work. I try to use it to stop the Gatt server and it is completely ineffectual. Looking at the code I can see why. The method simply calls HCl.Close(), and HCI.Close() does nothing but return nil. I tried modifying the "Stop()" method to instead call HCI.Stop(), but that is ineffectual as well. What I want to do is detect when a peripheral has connected to the Gatt server; get the address of the peripheral; and if it is not what is expected, disconnect the server from the peripheral. I have figured out how to do everything except the disconnect part. I figured I could simply stop the Gatt server to accomplish the disconnect, but I can't find a way to do this. The aforementioned "Stop()" method would seem to be intended to do this but as I wrote above, it's not getting the job done. Any ideas?

Thanks

@roylee17
Copy link
Contributor

@bbartol The Stop() here is to release the HCI device (or HCI socket), although currently it is not implemented yet as you indicated :). #30 is working on this though. I'm pretty sure something like that worked before though I haven't got it work on my current environment yet. You can give it a shot.

However, from your description, it sounds more like a "disconnect" at the connection level.
We used to have something like onConnect(conn), which will be invoked when the server is connected, and you can pull the connection info from that conn. Is this what you were looking for?

We should add that functionality back soon.

@bbartol
Copy link

bbartol commented Jan 28, 2017

@roylee17, Thanks for the shout back. I have figured out how to do everything that you described as onConnect(conn)'s functionality. What's left is to figure out how to disconnect from the server side. Nothing I have tried so far has worked. Calling Disconnect stops my whole program; and as I wrote before, the code to Stop() the server is ineffectual (because it has only been stubbed out so far). I got so far as finding the code where the HCI device is first opened. In that code, an HCIdown and HCIup is executed at the beginning to effect a reset of the device before proceeding to bind the device to the HCI channel. So I thought maybe I could do the same thing to reset the device and that would cause the server to stop. What I found, however, is that neither HCIdown nor HCIup would work once the device is bound to the HCI channel (they both return non-nIl for error). So then I embarked on trying to figure out how to unbind the device. I searched through the code for a couple of hours and found nothing to do this and became concerned that I was embarking down a rabbit hole. So that's where I'm at.

Thanks again.

@roylee17
Copy link
Contributor

roylee17 commented Jan 28, 2017

@bbartol

Calling Disconnect stops my whole program

Is it blocked or crashed? Can you try it with go routine?
Certain cases will re-enter our HCI handling state machine resulting a deadlock. Callback() is one of the cases. I'll add the onConnect/onDisconnect notification/callback back soon, so we can work on it on the same page. This is the preferred approach for scenario.

On the other hand, the device up/down/reset in #30 is another way to "work around" your scenario. As you tried, once the socket got bound, it can't be up/down/reset. With regular TCP/IP sockets, we can use some socket options, or shutdown() instead of close() on the socket. But that doesn't seem to work with the HCI socket.

I was pretty sure that worked for me earlier though I haven't reproduced it in my current environment. Unless I was using multiple HCI USB dongles, and the auto fallback logic (passing -1 as device id) fooled me into believing that I was reusing the same HCI device while it actually picked another for me...

Anyway, I'm building my kernel from source so I can peak and poke the it and track what's happening underneath.

@jbaldwinroberts
Copy link
Author

@roylee17 @bbartol I am spending the week trying to get to the bottom of this. I think the HCI interface is EBUSY the second time around because of an unfinished read on the socket - see this line in the strace log below [pid 6479] 10:22:43.319529 read(3, <unfinished ...>, the read never gets resumed.
Any ideas on how to complete the read either before closing the socket or when opening it?

System specs:

$ uname -a    
Linux x1pro 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux
$ go version
go version go1.7.5 linux/amd64

main.go

package main

import (
	"flag"
	"log"
	"time"

	"github.com/currantlabs/ble/examples/lib/dev"
)

var (
	device = flag.String("device", "default", "implementation of ble")
	name   = flag.String("name", "Gopher", "name of remote peripheral")
	addr   = flag.String("addr", "", "address of remote peripheral (MAC on Linux, UUID on OS X)")
	sub    = flag.Duration("sub", 0, "subscribe to notification and indication for a specified period")
	sd	   = flag.Duration("sd", 5*time.Second, "scanning duration, 0 for indefinitely")
)

func main() {
	d, err := dev.NewDevice(*device)
	if err != nil {
		log.Fatalf("can't new device : %s", err)
	} else {
		log.Printf("Opened")
	}

	time.Sleep(1 * time.Second)

	d.Stop();

	time.Sleep(1 * time.Second)

	d, err = dev.NewDevice(*device)
	if err != nil {
		log.Fatalf("can't new device : %s", err)
	} else {
		log.Printf("Opened")
	}
}

Strace output:

$ sudo strace -f -tt -e trace=ioctl,socket,close,open,read,write ./ble-test-currantlabs
10:22:43.221018 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
10:22:43.221138 close(3)                = 0
10:22:43.221167 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
10:22:43.221193 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
10:22:43.221317 close(3)                = 0
10:22:43.221350 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
10:22:43.221391 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
10:22:43.221546 close(3)                = 0
strace: Process 6478 attached
strace: Process 6479 attached
strace: Process 6480 attached
strace: Process 6481 attached
[pid  6477] 10:22:43.225149 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid  6477] 10:22:43.225372 read(3, "128\n", 4096) = 4
[pid  6477] 10:22:43.225406 read(3, "", 4092) = 0
[pid  6477] 10:22:43.225441 close(3)    = 0
[pid  6477] 10:22:43.225493 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  6477] 10:22:43.225545 close(3)    = 0
[pid  6477] 10:22:43.225592 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  6477] 10:22:43.225653 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid  6477] 10:22:43.225716 close(4)    = 0
[pid  6477] 10:22:43.225744 close(3)    = 0
[pid  6477] 10:22:43.225824 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  6477] 10:22:43.225855 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  6477] 10:22:43.225883 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  6477] 10:22:43.225915 ioctl(3, HCIDEVUP, 0) = 0
[pid  6477] 10:22:43.277745 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  6477] 10:22:43.299644 write(3, "\1\3\f\0", 4) = 4
[pid  6479] 10:22:43.299959 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid  6479] 10:22:43.308560 read(3,  <unfinished ...>
[pid  6477] 10:22:43.308827 write(3, "\1\t\20\0", 4) = 4
[pid  6479] 10:22:43.309392 <... read resumed> "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13
[pid  6479] 10:22:43.309670 read(3,  <unfinished ...>
[pid  6477] 10:22:43.309964 write(3, "\1\5\20\0", 4) = 4
[pid  6479] 10:22:43.310338 <... read resumed> "\4\16\v\1\5\20\0\375\3`\4\0\6\0", 4096) = 14
[pid  6479] 10:22:43.310532 read(3,  <unfinished ...>
[pid  6477] 10:22:43.310753 write(3, "\1\2 \0", 4) = 4
[pid  6479] 10:22:43.311380 <... read resumed> "\4\16\7\1\2 \0\33\0\7", 4096) = 10
[pid  6479] 10:22:43.311686 read(3,  <unfinished ...>
[pid  6477] 10:22:43.312016 write(3, "\1\7 \0", 4) = 4
[pid  6479] 10:22:43.313344 <... read resumed> "\4\16\5\1\7 \0\7", 4096) = 8
[pid  6479] 10:22:43.313570 read(3,  <unfinished ...>
[pid  6480] 10:22:43.314017 write(3, "\1\1 \10\37\0\0\0\0\0\0\0", 12) = 12
[pid  6479] 10:22:43.315353 <... read resumed> "\4\16\4\1\1 \0", 4096) = 7
[pid  6479] 10:22:43.315572 read(3,  <unfinished ...>
[pid  6480] 10:22:43.315747 write(3, "\1\1\f\10\377\377\373\377\7\370\277=", 12) = 12
[pid  6479] 10:22:43.316325 <... read resumed> "\4\16\4\1\1\f\0", 4096) = 7
[pid  6479] 10:22:43.316471 read(3,  <unfinished ...>
[pid  6480] 10:22:43.316671 write(3, "\1m\f\2\1\0", 6) = 6
[pid  6479] 10:22:43.317339 <... read resumed> "\4\16\4\1m\f\0", 4096) = 7
[pid  6479] 10:22:43.317476 read(3,  <unfinished ...>
[pid  6480] 10:22:43.317747 write(3, "\1\6 \17 \0 \0\0\0\0\0\0\0\0\0\0\7\0", 19) = 19
[pid  6479] 10:22:43.318336 <... read resumed> "\4\16\4\1\6 \0", 4096) = 7
[pid  6479] 10:22:43.318591 read(3,  <unfinished ...>
[pid  6480] 10:22:43.318873 write(3, "\1\v \7\1\4\0\4\0\0\0", 11) = 11
[pid  6479] 10:22:43.319337 <... read resumed> "\4\16\4\1\v \0", 4096) = 7
[pid  6479] 10:22:43.319529 read(3,  <unfinished ...>
[pid  6480] 10:22:43.320252 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid  6480] 10:22:43.320411 read(4, "", 4096) = 0
[pid  6480] 10:22:43.320531 close(4)    = 0
[pid  6480] 10:22:43.320747 write(2, "2017/01/30 10:22:43 Opened\n", 272017/01/30 10:22:43 Opened
) = 27
strace: Process 6482 attached
[pid  6480] 10:22:44.322405 write(2, "2017/01/30 10:22:44 device stop\n", 322017/01/30 10:22:44 device stop
) = 32
[pid  6480] 10:22:44.322750 write(2, "2017/01/30 10:22:44 HCI public c"..., 372017/01/30 10:22:44 HCI public close
) = 37
[pid  6480] 10:22:44.323025 write(2, "2017/01/30 10:22:44 HCI private "..., 382017/01/30 10:22:44 HCI private close
) = 38
[pid  6480] 10:22:44.323222 write(2, "2017/01/30 10:22:44 Socket close"..., 332017/01/30 10:22:44 Socket close
) = 33
[pid  6480] 10:22:44.323364 close(3)    = 0
[pid  6480] 10:22:45.324217 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  6480] 10:22:45.324378 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  6480] 10:22:45.324450 ioctl(3, HCIDEVDOWN, 0) = -1 EBUSY (Device or resource busy)
[pid  6480] 10:22:45.324709 write(2, "2017/01/30 10:22:45 can't new de"..., 1282017/01/30 10:22:45 can't new device : can't init hci: no devices available: (hci0: can't down device: device or resource busy)
) = 128
[pid  6480] 10:22:45.324899 +++ exited with 1 +++
[pid  6482] 10:22:45.324923 +++ exited with 1 +++
[pid  6481] 10:22:45.324983 +++ exited with 1 +++
[pid  6478] 10:22:45.324999 +++ exited with 1 +++
[pid  6479] 10:22:45.326381 <... read resumed> <unfinished ...>) = ?
[pid  6479] 10:22:45.326776 +++ exited with 1 +++
10:22:45.326799 +++ exited with 1 +++

@jbaldwinroberts
Copy link
Author

This is the blocking line. Is it necessary to always trigger a write to complete the read? The finished reads always do this...

[pid  6479] 10:22:43.308560 read(3,  <unfinished ...>
[pid  6477] 10:22:43.308827 write(3, "\1\t\20\0", 4) = 4
[pid  6479] 10:22:43.309392 <... read resumed> "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13

The unix.Read function is only used by a couple of libraries, is it possible the unix.Read function has a bug?

I modified the Read function to look like this and Unlocked never prints the last time round.

func (s *Socket) Read(p []byte) (int, error) {
	s.rmu.Lock()
    log.Printf("locked")
	defer s.rmu.Unlock()
	n, err := unix.Read(s.fd, p)
    log.Printf("Unlocked")
	return n, errors.Wrap(err, "can't read hci socket")
}

I did notice something strange - the read resumes when the process is killed. This line in the strace output below [pid 28494] 12:29:24.711194 <... read resumed> <unfinished ...>) = ?

[sudo] password for joe: 
12:29:12.598744 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
12:29:12.599050 close(3)                = 0
12:29:12.599158 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
12:29:12.599247 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
12:29:12.599559 close(3)                = 0
12:29:12.599615 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
12:29:12.599658 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
12:29:12.599797 close(3)                = 0
strace: Process 28493 attached
strace: Process 28494 attached
strace: Process 28495 attached
strace: Process 28496 attached
[pid 28492] 12:29:12.602930 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid 28492] 12:29:12.603137 read(3, "128\n", 4096) = 4
[pid 28492] 12:29:12.603170 read(3, "", 4092) = 0
[pid 28492] 12:29:12.603193 close(3)    = 0
[pid 28492] 12:29:12.603224 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 28492] 12:29:12.603252 close(3)    = 0
[pid 28492] 12:29:12.603279 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 28492] 12:29:12.603328 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid 28492] 12:29:12.603376 close(4)    = 0
[pid 28492] 12:29:12.603398 close(3)    = 0
[pid 28492] 12:29:12.603478 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid 28492] 12:29:12.603503 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid 28492] 12:29:12.603550 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid 28492] 12:29:12.603578 read(4, "", 4096) = 0
[pid 28492] 12:29:12.603599 close(4)    = 0
[pid 28492] 12:29:12.603634 write(2, "2017/01/30 12:29:12 Starting ope"..., 342017/01/30 12:29:12 Starting open
) = 34
[pid 28492] 12:29:12.603660 write(2, "2017/01/30 12:29:12 Polling\n", 282017/01/30 12:29:12 Polling
) = 28
[pid 28492] 12:29:13.605458 write(2, "2017/01/30 12:29:13 Polled\n", 272017/01/30 12:29:13 Polled
) = 27
[pid 28492] 12:29:13.605707 ioctl(3, HCIDEVDOWN, 0) = 0
[pid 28492] 12:29:13.605934 ioctl(3, HCIDEVUP, 0) = 0
[pid 28492] 12:29:13.658808 ioctl(3, HCIDEVDOWN, 0) = 0
[pid 28492] 12:29:13.679811 write(2, "2017/01/30 12:29:13 Finished ope"..., 342017/01/30 12:29:13 Finished open
) = 34
[pid 28492] 12:29:13.680094 write(3, "\1\3\f\0", 4) = 4
[pid 28492] 12:29:13.680231 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28492] 12:29:13.680421 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid 28492] 12:29:13.689222 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28492] 12:29:13.689394 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28492] 12:29:13.689611 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28492] 12:29:13.689722 read(3,  <unfinished ...>
[pid 28494] 12:29:13.689989 write(3, "\1\t\20\0", 4) = 4
[pid 28492] 12:29:13.691121 <... read resumed> "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13
[pid 28492] 12:29:13.691214 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28492] 12:29:13.691386 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28492] 12:29:13.691606 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28492] 12:29:13.691709 read(3,  <unfinished ...>
[pid 28494] 12:29:13.691903 write(3, "\1\5\20\0", 4) = 4
[pid 28492] 12:29:13.693066 <... read resumed> "\4\16\v\1\5\20\0\375\3`\4\0\6\0", 4096) = 14
[pid 28492] 12:29:13.693126 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28492] 12:29:13.693228 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28492] 12:29:13.693399 write(3, "\1\2 \0", 4) = 4
[pid 28494] 12:29:13.693532 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.693635 read(3, "\4\16\7\1\2 \0\33\0\7", 4096) = 10
[pid 28494] 12:29:13.694119 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.694222 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.694413 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.694502 read(3,  <unfinished ...>
[pid 28492] 12:29:13.694618 write(3, "\1\7 \0", 4) = 4
[pid 28494] 12:29:13.695055 <... read resumed> "\4\16\5\1\7 \0\7", 4096) = 8
[pid 28494] 12:29:13.695132 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.695255 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.695448 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.695553 read(3,  <unfinished ...>
[pid 28492] 12:29:13.695677 write(3, "\1\1 \10\37\0\0\0\0\0\0\0", 12) = 12
[pid 28494] 12:29:13.696046 <... read resumed> "\4\16\4\1\1 \0", 4096) = 7
[pid 28494] 12:29:13.696106 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.696209 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.696375 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.696467 read(3,  <unfinished ...>
[pid 28492] 12:29:13.696575 write(3, "\1\1\f\10\377\377\373\377\7\370\277=", 12) = 12
[pid 28494] 12:29:13.697043 <... read resumed> "\4\16\4\1\1\f\0", 4096) = 7
[pid 28494] 12:29:13.697102 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.697206 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.697363 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.697454 read(3,  <unfinished ...>
[pid 28492] 12:29:13.697562 write(3, "\1m\f\2\1\0", 6) = 6
[pid 28494] 12:29:13.698085 <... read resumed> "\4\16\4\1m\f\0", 4096) = 7
[pid 28494] 12:29:13.698146 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.698250 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.698407 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.698499 read(3,  <unfinished ...>
[pid 28492] 12:29:13.698631 write(3, "\1\6 \17 \0 \0\0\0\0\0\0\0\0\0\0\7\0", 19) = 19
[pid 28494] 12:29:13.699042 <... read resumed> "\4\16\4\1\6 \0", 4096) = 7
[pid 28494] 12:29:13.699103 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.699207 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.699366 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.699460 read(3,  <unfinished ...>
[pid 28492] 12:29:13.699563 write(3, "\1\v \7\1\4\0\4\0\0\0", 11) = 11
[pid 28494] 12:29:13.700042 <... read resumed> "\4\16\4\1\v \0", 4096) = 7
[pid 28494] 12:29:13.700102 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.700205 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.700361 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.700456 read(3,  <unfinished ...>
[pid 28492] 12:29:13.700783 write(2, "2017/01/30 12:29:13 Opened\n", 272017/01/30 12:29:13 Opened
) = 27
strace: Process 28500 attached
strace: Process 28501 attached
[pid 28492] 12:29:18.702772 write(2, "2017/01/30 12:29:18 device stop\n", 322017/01/30 12:29:18 device stop
) = 32
[pid 28492] 12:29:18.703115 write(2, "2017/01/30 12:29:18 HCI public c"..., 372017/01/30 12:29:18 HCI public close
) = 37
[pid 28492] 12:29:18.703389 write(2, "2017/01/30 12:29:18 HCI private "..., 382017/01/30 12:29:18 HCI private close
) = 38
[pid 28492] 12:29:18.703623 write(2, "2017/01/30 12:29:18 Socket close"..., 332017/01/30 12:29:18 Socket close
) = 33
[pid 28492] 12:29:18.703778 close(3)    = 0
[pid 28492] 12:29:23.705743 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid 28492] 12:29:23.705995 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid 28492] 12:29:23.706254 write(2, "2017/01/30 12:29:23 Starting ope"..., 342017/01/30 12:29:23 Starting open
) = 34
[pid 28492] 12:29:23.706469 write(2, "2017/01/30 12:29:23 Polling\n", 282017/01/30 12:29:23 Polling
) = 28
[pid 28492] 12:29:24.708944 write(2, "2017/01/30 12:29:24 Polled\n", 272017/01/30 12:29:24 Polled
) = 27
[pid 28492] 12:29:24.709312 ioctl(3, HCIDEVDOWN, 0) = -1 EBUSY (Device or resource busy)
[pid 28492] 12:29:24.709840 write(2, "2017/01/30 12:29:24 can't new de"..., 1282017/01/30 12:29:24 can't new device : can't init hci: no devices available: (hci0: can't down device: device or resource busy)
) = 128
[pid 28501] 12:29:24.710327 +++ exited with 1 +++
[pid 28500] 12:29:24.710381 +++ exited with 1 +++
[pid 28496] 12:29:24.710422 +++ exited with 1 +++
[pid 28495] 12:29:24.710477 +++ exited with 1 +++
[pid 28493] 12:29:24.710512 +++ exited with 1 +++
[pid 28494] 12:29:24.711194 <... read resumed> <unfinished ...>) = ?
[pid 28494] 12:29:24.712223 +++ exited with 1 +++
12:29:24.712284 +++ exited with 1 +++

So far I have tried to force the read to finish with a few methods, none of which work.

I attempted to flush the socket before closing it with each of the commented out lines.

func (s *Socket) Close() error {
    log.Printf("Socket close")
    // unix.Sync()
    // unix.Fsync(s.fd)
    // unix.Fdatasync(s.fd)
	return errors.Wrap(unix.Close(s.fd), "can't close hci socket")
}

I tried to poll for data when the socket is first opened.

func open(fd, id int) (*Socket, error) {
    log.Printf("Starting open")

    log.Printf("Polling")
	// poll for 20ms to see if any data becomes available, then clear it
	pfds := []unix.PollFd{unix.PollFd{Fd: int32(fd), Events: unix.POLLIN}}
	unix.Poll(pfds, 1000)
	if pfds[0].Revents&unix.POLLIN > 0 {
		b := make([]byte, 100)
		unix.Read(fd, b)
	}
    log.Printf("Polled")

	// Reset the device in case previous session didn't cleanup properly.
	if err := ioctl(uintptr(fd), hciDownDevice, uintptr(id)); err != nil {
		return nil, errors.Wrap(err, "can't down device")
	}
	if err := ioctl(uintptr(fd), hciUpDevice, uintptr(id)); err != nil {
		return nil, errors.Wrap(err, "can't up device")
	}

	// HCI User Channel requires exclusive access to the device.
	// The device has to be down at the time of binding.
	if err := ioctl(uintptr(fd), hciDownDevice, uintptr(id)); err != nil {
		return nil, errors.Wrap(err, "can't down device")
	}

	// Bind the RAW socket to HCI User Channel
	sa := unix.SockaddrHCI{Dev: uint16(id), Channel: unix.HCI_CHANNEL_USER}
	if err := unix.Bind(fd, &sa); err != nil {
		return nil, errors.Wrap(err, "can't bind socket to hci user channel")
	}

	// poll for 20ms to see if any data becomes available, then clear it
	pfds = []unix.PollFd{unix.PollFd{Fd: int32(fd), Events: unix.POLLIN}}
	unix.Poll(pfds, 20)
	if pfds[0].Revents&unix.POLLIN > 0 {
		b := make([]byte, 100)
		unix.Read(fd, b)
	}

    log.Printf("Finished open")

	return &Socket{fd: fd}, nil
}

@moogle19
Copy link
Contributor

@josephroberts
Take a look at this.
The read() will block until there is something to read and close() does not unblock the read.

@jbaldwinroberts
Copy link
Author

@moogle19 thanks, so perhaps we are issuing an extra read that is not needed.

@jbaldwinroberts
Copy link
Author

jbaldwinroberts commented Jan 30, 2017

I have narrowed it down to the read and writes - small steps haha - by changing the hci Init() function to comment out and read/write parts.

func (h *HCI) Init() error {
	h.evth[0x3E] = h.handleLEMeta
	h.evth[evt.CommandCompleteCode] = h.handleCommandComplete
	h.evth[evt.CommandStatusCode] = h.handleCommandStatus
	h.evth[evt.DisconnectionCompleteCode] = h.handleDisconnectionComplete
	h.evth[evt.NumberOfCompletedPacketsCode] = h.handleNumberOfCompletedPackets

	h.subh[evt.LEAdvertisingReportSubCode] = h.handleLEAdvertisingReport
	h.subh[evt.LEConnectionCompleteSubCode] = h.handleLEConnectionComplete
	h.subh[evt.LEConnectionUpdateCompleteSubCode] = h.handleLEConnectionUpdateComplete
	h.subh[evt.LELongTermKeyRequestSubCode] = h.handleLELongTermKeyRequest
	// evt.EncryptionChangeCode:                     todo),
	// evt.ReadRemoteVersionInformationCompleteCode: todo),
	// evt.HardwareErrorCode:                        todo),
	// evt.DataBufferOverflowCode:                   todo),
	// evt.EncryptionKeyRefreshCompleteCode:         todo),
	// evt.AuthenticatedPayloadTimeoutExpiredCode:   todo),
	// evt.LEReadRemoteUsedFeaturesCompleteSubCode:   todo),
	// evt.LERemoteConnectionParameterRequestSubCode: todo),

	skt, err := socket.NewSocket(h.id)
	if err != nil {
		return err
	}
	h.skt = skt

	h.chCmdBufs <- make([]byte, 64)

	// go h.sktLoop()
	// h.init()

	// Pre-allocate buffers with additional head room for lower layer headers.
	// HCI header (1 Byte) + ACL Data Header (4 bytes) + L2CAP PDU (or fragment)
	// h.pool = NewPool(1+4+h.bufSize, h.bufCnt-1)

        // h.Send(&h.params.advParams, nil)
        // h.Send(&h.params.scanParams, nil)
	return nil
}

Here is the strace output, showing the interface being Uped and Downed the second time round:

14:03:00.498685 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
14:03:00.498957 close(3)                = 0
14:03:00.499044 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
14:03:00.499107 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
14:03:00.499369 close(3)                = 0
14:03:00.499432 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
14:03:00.499485 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
14:03:00.499704 close(3)                = 0
strace: Process 8274 attached
strace: Process 8275 attached
strace: Process 8276 attached
strace: Process 8277 attached
[pid  8273] 14:03:00.505739 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid  8273] 14:03:00.505999 read(3, "128\n", 4096) = 4
[pid  8273] 14:03:00.506037 read(3, "", 4092) = 0
[pid  8273] 14:03:00.506068 close(3)    = 0
[pid  8273] 14:03:00.506111 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  8273] 14:03:00.506155 close(3)    = 0
[pid  8273] 14:03:00.506190 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  8273] 14:03:00.506264 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid  8273] 14:03:00.506335 close(4)    = 0
[pid  8273] 14:03:00.506388 close(3)    = 0
[pid  8273] 14:03:00.506502 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  8273] 14:03:00.506555 ioctl(3, HCIGETDEVLIST, 0xc42004dc0c) = 0
[pid  8273] 14:03:00.506602 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:00.506650 ioctl(3, HCIDEVUP, 0) = 0
[pid  8273] 14:03:00.559983 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:00.582630 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid  8273] 14:03:00.582725 read(4, "", 4096) = 0
[pid  8273] 14:03:00.582768 close(4)    = 0
[pid  8273] 14:03:00.582832 write(2, "2017/01/30 14:03:00 Opened\n", 272017/01/30 14:03:00 Opened
) = 27
[pid  8273] 14:03:01.583373 close(3)    = 0
[pid  8273] 14:03:02.587812 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  8273] 14:03:02.588231 ioctl(3, HCIGETDEVLIST, 0xc42004dc0c) = 0
[pid  8273] 14:03:02.588587 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:02.588752 ioctl(3, HCIDEVUP, 0) = 0
[pid  8273] 14:03:02.640996 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:02.662194 write(2, "2017/01/30 14:03:02 Opened\n", 272017/01/30 14:03:02 Opened
) = 27
[pid  8276] 14:03:02.662489 +++ exited with 0 +++
[pid  8277] 14:03:02.662541 +++ exited with 0 +++
[pid  8275] 14:03:02.662565 +++ exited with 0 +++
[pid  8274] 14:03:02.663258 +++ exited with 0 +++
14:03:02.663301 +++ exited with 0 +++

@moogle19
Copy link
Contributor

@josephroberts
It is this Read() that is blocking which is called by this loop.
The problem is, that the loop is needed to check if there is data on the socket.

@jbaldwinroberts
Copy link
Author

@moogle19 yep. I am trying to solve this using poll, it gets a little further and then hangs.

Modification to socket.go

func (s *Socket) Read(p []byte) (int, error) {
	s.rmu.Lock()
	defer s.rmu.Unlock()

	pfds := []unix.PollFd{unix.PollFd{Fd: int32(s.fd), Events: unix.POLLIN}}
	unix.Poll(pfds, 0)
	if pfds[0].Revents&unix.POLLIN > 0 {
		n, err := unix.Read(s.fd, p)
		return n, errors.Wrap(err, "can't read hci socket")
	}

	return 0, nil
}

Modification to hci.go

func (h *HCI) sktLoop() {
	b := make([]byte, 4096)
	defer close(h.done)
	for {
		n, err := h.skt.Read(b)
		if err != nil {
			h.err = fmt.Errorf("skt: %s", err)
			return
		}
		if n == 0 {
			continue
		}
		p := make([]byte, n)
		copy(p, b)
		if err := h.handlePkt(p); err != nil {
			h.err = fmt.Errorf("skt: %s", err)
			return
		}
	}
}

Strace output

15:07:56.188948 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
15:07:56.189403 close(3)                = 0
15:07:56.189484 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
15:07:56.189548 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
15:07:56.189754 close(3)                = 0
15:07:56.189815 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
15:07:56.189866 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
15:07:56.190000 close(3)                = 0
strace: Process 7376 attached
strace: Process 7377 attached
strace: Process 7378 attached
strace: Process 7379 attached
[pid  7375] 15:07:56.194198 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid  7375] 15:07:56.194602 read(3, "128\n", 4096) = 4
[pid  7375] 15:07:56.194664 read(3, "", 4092) = 0
[pid  7375] 15:07:56.194737 close(3)    = 0
[pid  7375] 15:07:56.194798 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  7375] 15:07:56.194864 close(3)    = 0
[pid  7375] 15:07:56.194929 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  7375] 15:07:56.195041 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid  7375] 15:07:56.195112 close(4)    = 0
[pid  7375] 15:07:56.195151 close(3)    = 0
[pid  7375] 15:07:56.195292 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  7375] 15:07:56.195339 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  7375] 15:07:56.195384 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7375] 15:07:56.195415 ioctl(3, HCIDEVUP, 0) = 0
[pid  7375] 15:07:56.247351 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7377] 15:07:56.268671 write(3, "\1\3\f\0", 4) = 4
[pid  7375] 15:07:56.281340 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid  7377] 15:07:56.282732 write(3, "\1\t\20\0", 4) = 4
[pid  7375] 15:07:56.291654 read(3, "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13
[pid  7377] 15:07:56.292097 write(3, "\1\5\20\0", 4) = 4
[pid  7375] 15:07:56.293006 read(3, "\4\16\v\1\5\20\0\375\3`\4\0\6\0", 4096) = 14
[pid  7377] 15:07:56.293480 write(3, "\1\2 \0", 4) = 4
[pid  7375] 15:07:56.294515 read(3, "\4\16\7\1\2 \0\33\0\7", 4096) = 10
[pid  7377] 15:07:56.294856 write(3, "\1\7 \0", 4) = 4
[pid  7375] 15:07:56.296026 read(3, "\4\16\5\1\7 \0\7", 4096) = 8
[pid  7377] 15:07:56.296507 write(3, "\1\1 \10\37\0\0\0\0\0\0\0", 12) = 12
[pid  7375] 15:07:56.296967 read(3, "\4\16\4\1\1 \0", 4096) = 7
[pid  7377] 15:07:56.297252 write(3, "\1\1\f\10\377\377\373\377\7\370\277=", 12) = 12
[pid  7375] 15:07:56.297980 read(3, "\4\16\4\1\1\f\0", 4096) = 7
[pid  7377] 15:07:56.298277 write(3, "\1m\f\2\1\0", 6) = 6
[pid  7375] 15:07:56.299000 read(3, "\4\16\4\1m\f\0", 4096) = 7
[pid  7377] 15:07:56.299391 write(3, "\1\6 \17 \0 \0\0\0\0\0\0\0\0\0\0\7\0", 19) = 19
[pid  7375] 15:07:56.300017 read(3, "\4\16\4\1\6 \0", 4096) = 7
[pid  7377] 15:07:56.300639 write(3, "\1\v \7\1\4\0\4\0\0\0", 11) = 11
[pid  7375] 15:07:56.301983 read(3, "\4\16\4\1\v \0", 4096) = 7
[pid  7377] 15:07:56.302713 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid  7377] 15:07:56.302849 read(4, "", 4096) = 0
[pid  7377] 15:07:56.302923 close(4)    = 0
[pid  7377] 15:07:56.302987 write(2, "2017/01/30 15:07:56 Opened\n", 272017/01/30 15:07:56 Opened
) = 27
strace: Process 7380 attached
[pid  7377] 15:07:57.303152 close(3)    = 0
[pid  7377] 15:07:58.303894 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  7377] 15:07:58.303943 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  7377] 15:07:58.303975 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7377] 15:07:58.303999 ioctl(3, HCIDEVUP, 0) = 0
[pid  7377] 15:07:58.354958 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7377] 15:07:58.377053 write(3, "\1\3\f\0", 4) = 4
[pid  7375] 15:07:58.386797 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid  7378] 15:07:58.390170 read(3, ^Cstrace: Process 7375 detached
strace: Process 7376 detached
strace: Process 7377 detached
strace: Process 7378 detached
 <detached ...>
strace: Process 7379 detached
strace: Process 7380 detached

it hangs here: [pid 7378] 15:07:58.390170 read(3, ^Cstrace: Process 7375 detached whilst trying to setup the interface for the second time, so I am gonnna debug that now.

BTW - I am being verbose here for my own notes :P

@roylee17
Copy link
Contributor

Non-blocking I/O Select / Poll/ Epoll breaks the read into two separate system calls, which might introduce latency if context switch happens in between. (latency might be visible or not depends on the platform)

Let's see if we can tackle this within GO land (tricky though...).

@roylee17
Copy link
Contributor

roylee17 commented Feb 2, 2017

Fixed by #32

@roylee17 roylee17 closed this as completed Feb 2, 2017
@bbartol
Copy link

bbartol commented Feb 3, 2017

Thanks everyone for the feedback to my original post. I was able to accomplish the one piece that was vexing me (disconnecting the client from the server side) by calling Device.HCI.conns[ClientMapKey].Close(). The trick was discerning ClientMapKey (the key that references the connection to the client that is currently connected to my Gatt server).

I was able to accomplish everything that I need (at least so far) by adding the attached code to my local copy of the ble library. I organized this code so that none of the existing code (other than the example I discuss below) needed to be modified. If you unzip (keeping the folder structure) into your local copy of the github.com/currantlabs/ble folder (for example, my local copy is at /go/src/github.com/currantlabs/ble), you should be able to rebuild your code with any of the functionality I added and try it out if you want. Included in the attached (in ble/examples/blesh_AGEX) is a scaled down and heavily modified copy of the original blesh example. Changes that I made to said example are commented as "AGEX" (//AGEX). In summary, they are as follows:

main.go:

  • Added 2 parameters to the curr struct - "device_AGEX", and "name":
  var curr struct {
	  device       ble.Device
  	  client       ble.Client
	  clients      map[string]ble.Client
	  uuid         ble.UUID
	  addr         ble.Addr
	  profile      *ble.Profile
	  device_AGEX  ble.Device_AGEX //AGEX
	  name         string //AGEX
  }
  • Modified func main - stripped out everything but support for the "serve" command. Since everything I am trying to do is centered around running a Gatt server, I didn't need the other commands; thus I decided to remove them for simplicity's sake.
  • Modified func setup to call linux.NewDevice() directly rather than through NewDevice->DefaultDevice.
  • Modified func setup to instantiate curr.device_AGEX to get access to the functionality I have added.
  • Modified func cmdServe. I basically turned this function into my sandbox.
  • Added functions "ScanForConnAddr" and "DisconnectAndKillServer". The supporting code for these functions is in the other files (outside of the examples folder).

adv.go:

  • Added this line of code to func advHandler:
    curr.name = a.LocalName()

Other

  • Copied the files count.go, echo,go, and uuids.go in ble/examples/lib over to ble/examples/blesh_AGEX so that I could flatten the code a bit and make my example to be contained all in one folder. I didn't need any of the files in ble/examples/lib/dev nor did I need battery.go in ble/examples/lib.
  • Deleted files lnx.go, util.go, and exp.go in ble/examples/blesh_AGEX - didn't need them

ble.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants