New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Official" Wireshark dissector #3303

Open
FlorianWolters opened this Issue Jul 2, 2017 · 12 comments

Comments

Projects
None yet
7 participants
@FlorianWolters

FlorianWolters commented Jul 2, 2017

Note: I know this is not the best place to post this kind of request, but I do not know where to post this elsewhere. Hopefully other people are also interested in this.


I do wonder what tools the folks at Google use internally to debug and analyze the Protobuf messages transferred over the wire (if using network communication).

The defacto standard to do network analyzes is the network tool Wireshark. The functionality of Wireshark can be extended with a Protocol Dissector (i.e. a plug-in) to support a (custom) protocol.

I've researched a bit and found the following dissectors for Protobuf:

There are plenty of requests for a working and user-friendly dissector on the WWW, e.g.:

I would be very interested in some kind of official Wireshark protocol dissector with the following features:

  1. Supports both proto3 and proto2.
  2. Supports both UDP and TCP (see point 3). The port ranges for both protocols should be configurable with the GUI of Wireshark (i.e. when to activate the dissector).
  3. Supports the configuration of a custom prefix with length-prefix framing. The custom prefix is configured by:
    1. The total length (in bytes) of the custom prefix.
    2. The position of the length-prefix.
    3. The length (in bytes) of the length-prefix.
      That would allow to support TCP as well as custom protocols that define additional fields.
  4. Supports both generic decoding as well as specific decoding (with the help of provided .proto files).
    1. generic: Displays a aessage as [numeric field id:value] pairs. This does not require the .proto file(s).
    2. specific: Displays a message as [field name:value] pairs. This requires the .proto file(s) for the specific protocol. The addition of .proto file(s) should be possible via the GUI of Wireshark and the directory path to the official Protobuf .proto files should be configurable as well (e.g. $PROTOBUF_ROOT/include/google/protobuf).

At the end of the day it would be very nice to have some kind of official protocol dissector contributed to Wireshark.

Are there any efforts towards an official Protobuf dissector for Wireshark? What does the community think? Is this considered useful? Or did I overlook an already existing solution that satisfies the requirements stated above?

@xfxyjwf

This comment has been minimized.

Contributor

xfxyjwf commented Jul 5, 2017

In Google, to send protobuf over the wire, you will be using the rpc library (the internal version of http://grpc.io/) instead of writing your own socket code, and a set of tools/libraries are provided to inspect and diagnose the rpc traffic. For example, with a command line tool you can easily write up a proto message in text format and send it to an arbitrary rpc server and get the response in text format. Hardly anyone needs to inspect the network traffic to understand the proto data being sent and it's also very hard to do because the rpc library will compress and encrypt the proto data. People who work on the rpc library probably need to use wireshark or tcpdump to inspect the network traffic, but again they don't need to inspect the proto data (I know they send payload of 1 byte in their tests). So pretty much nobody in Google will need this protobuf dessector for Wireshark and I doubt it will ever be officially supported...

@ruudsieb

This comment has been minimized.

ruudsieb commented Jul 27, 2017

Actually after reading the OP I tried using the Length Prefixed Protocol Buffer Dissector together with the protobuf_dissector (forked). And it was pretty easy to setup and get it to work in my context which is; Wireshark on Windows, TCP connection, length prefix (little endian, for this I made a small change on the lppb), proto2 message format.

@FlorianWolters

This comment has been minimized.

FlorianWolters commented Jul 27, 2017

@ruudsieb Did you get it to work with the proto3 language?

@ruudsieb

This comment has been minimized.

ruudsieb commented Jul 27, 2017

Sorry, forgot to mention that, no, I'm using proto2 message format.

@jfjffilk

This comment has been minimized.

jfjffilk commented Oct 16, 2017

@FlorianWolters I am also trying to write a wireshark plugin in c to parse google protocol buffers over TCP, but not finding proper source to know how. Have you found any solution for it?

@FlorianWolters

This comment has been minimized.

FlorianWolters commented Oct 16, 2017

@jfjffilk No, as mentioned in the OP there is currently no out-of-the-box solution working for the proto3 language. For general information how-to implement a Wireshark dissector refer to Chapter 9. Packet dissection of the official Wireshark documentation. But I strongly recommend to contact the author(s) of the Wireshark Protobuf Dissector and to analyse/fork/extend that project before beginning from scratch and implementing your own dissector. Please read the OP again for problems related to the existing Protobuf dissectors.

@ndandanov

This comment has been minimized.

ndandanov commented Nov 30, 2017

I would also be interested in an official Wireshark dissector for protobuf and perhaps gRPC.

I agree that it may not be of much use in Google's environment, however I think that many developers and testers would benefit from it.

@ndandanov

This comment has been minimized.

ndandanov commented Nov 30, 2017

Sorry for double-posting, but I tried a few things after posting above.

I found out, that in the official Wireshark repo on GitHub, there is a gRPC dissector - namely the wireshark/epan/dissectors/packet-grpc.c located here: https://github.com/wireshark/wireshark/blob/master/epan/dissectors/packet-grpc.c

It seems that it was officially added in the end of September this year:
https://github.com/wireshark/wireshark/commits/master/epan/dissectors/packet-grpc.c

Note that the original Wireshark repo is located on their website. Please check https://www.wireshark.org/develop.html.

I opted to see how this dissector is working and cloned the Wireshark repo:

git clone https://code.wireshark.org/review/wireshark

My system is a 64-bit Arch Linux, so I built Wireshark with the following commands:

./autogen.sh 
./configure 
make

(using this as a reference: https://www.wireshark.org/docs/wsdg_html_chunked/ChSrcBuildFirstTime.html)

It took around 20-30 minutes to build. The Wireshark version is:
screenshot_2017-11-30_17-09-00

I then ran it as root (in order to be able to capture any traffic) and started capturing the loopback interface.

I cloned, built and started the official gRPC hello world server and client from the grpc-java repo:
https://github.com/grpc/grpc-java/tree/master/examples

The console outputs are, as expected:
screenshot_2017-11-30_16-38-16

The captured packets in Wireshark are:
screenshot_2017-11-30_16-39-16

You can download the .pcapng file from my repo:
https://github.com/ndandanov/wireshark-grpc/blob/master/grpc-hello-world.pcapng

Here you can see that they are decoded as TCP packets.

I clicked with the right mouse button on a packet and selected Decode As...:
screenshot_2017-11-30_16-39-39-1

And configured the packets as HTTP2:
screenshot_2017-11-30_16-40-24

And the packets are now correctly recognized as either HTTP2 or gRPC:
screenshot_2017-11-30_16-40-37

Let's take a look at the packet which carries the actual gRPC hello world request:
screenshot_2017-11-30_16-45-49

I had also opened the .proto file (located here https://github.com/grpc/grpc-java/blob/master/examples/src/main/proto/helloworld.proto) for reference.

The actual value of field 1 from the message HelloRequest, which is string name, is "world" (or 77 6f 72 6c 64 in hex):

screenshot_2017-11-30_16-49-54

And we can verify this:
screenshot_2017-11-30_16-49-39

The gRPC hello world response was of course "Hello world":
screenshot_2017-11-30_16-50-18

The Wireshark interpretation of fields and their values could definitely be extended. I did not come across any option to specify and input .proto file to extract the field names and correct value types. This leads to some doubles represented as just uint64 in Wireshark.

Nevertheless, I think this is a significant improvement in the right direction.

Please share your opinion on this dissector. Has anyone else tried it and published some results/improvements?
Is there any public information regarding the release of this dissector?
Kind regards!

@xfxyjwf

This comment has been minimized.

Contributor

xfxyjwf commented Nov 30, 2017

@ndandanov Great findings! Could you also post this to grpc github repo? I think lots of grpc users will find this useful.

@ndandanov

This comment has been minimized.

ndandanov commented Dec 1, 2017

Hi @xfxyjwf, thank you for the kind words!
I added the above to the following issue: grpc/grpc#13586

I also recorded two videos which show the process in depth. They are somewhat lengthy and unfortunately I could not replicate the same successful packet dissection - probably due to the slightly newer development version of Wireshark.

Nevertheless, they could be of help to somebody:
gRPC Wireshark dissector - Part 1: https://youtu.be/LoGArC58Row
gRPC Wireshark dissector - Part 2: https://youtu.be/Wj9VoJQf-qc

@abhinava

This comment has been minimized.

abhinava commented Dec 21, 2017

@ndandanov This is really great! Do you happen to have any packet captures (PCAP files)?

@abhinava

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment