New explore() function with prompt_toolkit 2.0 integration

[gpotter2]

Current ls() function has different features:

• explore a layer e.g. ls(IP)
• search for a packet e.g. ls("IP")
• list all packets: ls()

The last function, ls() is currently un-readable and very useless, for several reasons:

• There are two many layers in scapy
• There are not ordered
• There are hidden layers (starting with _)

This PR adds a fancy system to discover packets (last case only), when launched in interactive mode, by using the funcionalities of prompt_toolkit (module included in IPython)

Screenshots

Layers

Contribs

Important notes

This PR requires prompt_toolkit>=2.0.0, the new prompt_toolkit version. It is currently unavailable on PyPi, but according to ipython/ipython#10964, both prompt_toolkit>=2.0.0 and ipython supporting it should be released very soon (matter of days).

Both prompt_toolkit 2.0 and IPython 7 have now been released. The PR is now mergeable

[codecov-io]

Codecov Report

Merging #1413 into master will increase coverage by <.01%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master    #1413      +/-   ##
==========================================
+ Coverage   85.31%   85.31%   +<.01%     
==========================================
  Files         179      179              
  Lines       42251    42342      +91     
==========================================
+ Hits        36046    36124      +78     
- Misses       6205     6218      +13

Impacted Files	Coverage Δ
scapy/contrib/dtp.py	100% <ø> (ø)	⬆️
scapy/contrib/ripng.py	100% <ø> (ø)	⬆️
scapy/contrib/cdp.py	90.9% <ø> (ø)	⬆️
scapy/contrib/gtp_v2.py	100% <ø> (ø)	⬆️
scapy/contrib/spbm.py	90.9% <ø> (ø)	⬆️
scapy/asn1packet.py	94.73% <ø> (ø)	⬆️
scapy/layers/skinny.py	100% <ø> (ø)	⬆️
scapy/contrib/cansocket_python_can.py	90% <ø> (ø)	⬆️
scapy/contrib/macsec.py	91.09% <ø> (ø)	⬆️
scapy/contrib/lldp.py	91.86% <ø> (ø)	⬆️
... and 36 more

[p-l-]

I like this (a lot), but I'd rather keep the good old ls(), and rename your function as explore_layers(), maybe?

[gpotter2]

PR status: waiting for ipython/ipython#11177 then it’s release.

I've renamed the function explore() (want to be it as simple as possible), and added a TIP to raw ls() when in interactive mode

[gpotter2]

I messed up my rebase: latest changes are in latest commit :/

[p-l-]

I think it would be better to add this to the docstring (and users will see that using help()). I don't want to be annoyed when I use ls() and know what I'm doing.

[gpotter2]

This will only be printed when the user types ls().
If he uses ls(IP)... or anything similar, it won't be printed.

because ls() result is unreadable, I don't think this really is an issue :/

[gpotter2]

IPython 7 is now live. I’ll rebase this

[gpotter2]

@guedou @p-l- Ready to be reviewed
The codecov decrease is not related

[guedou]

Looks good. Seem my comments.

[guedou]

s/scapy/Scapy/

[guedou]

Several comments are trying the PR:

• while calling ls() the printed line is "TIP: You may use explore() to navigate through all layers using a clear GUI". I don't think that the backquotes are necessary here;

• calling explore without installing the correct modules triggers and ImportError. I think that this error need to be handled correctly:

>>> explore()
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-8aa4a6f0c761> in <module>()
----> 1 explore()
[..]
   1617                           " `pip install ipython`")
   1618     if not int(prompt_toolkit.__version__.split(".")[0]) >= 2:
-> 1619             raise ImportError("prompt_toolkit >= 2.0.0 is required !")
   1620     # Only available with prompt_toolkit > 2.0, not released on PyPi yet
   1621     from prompt_toolkit.shortcuts.dialogs import radiolist_dialog, \

ImportError: prompt_toolkit >= 2.0.0 is required !

[guedou]

More comments after using explore():

• the line "Packet class. Binding mechanism. fuzz() method." should not appear;
• some protocols names end with . others don't;
• explore() could take a parameter for example explore("DNS") and work without the "curses" GUI. That will allow to use this functionality on a 'raw' installation;
• is it possible to search using the GUI?

I am not a big fan of this "GUI" however I really like the idea of exploring layers using a name. It will likely help new comers.

[gpotter2]

@guedou

calling explore without installing the correct modules triggers and ImportError. I think that this error need to be handled correctly

I thought throwing this error (it's scapy who is doing it) was the clearer way of reporting a missing dependency :/

the line "Packet class. Binding mechanism. fuzz() method." should not appear;

I manually added Packet to the list, so that Raw, NoPayload, and Packet also are shown

explore() could take a parameter for example explore("DNS") and work without the "curses" GUI. That will allow to use this functionality on a 'raw' installation;

Would that really be useful ? how to specify if its a module or a layer ?

is it possible to search using the GUI?

Not yet. That's on prompt_toolkit part. I've already submited a few PRs to improve how the list is handled, and am planning to add a few more (support for pagedown and pageup, maybe search...)

[guedou]

@gpotter2 I prefer displaying an error message in interactive mode, but I can leave without it. The non-GUI version could only work for layers, that is a good enough trade-off.

[gpotter2]

@guedou So I added a non-GUI version + tests.
I also pushed prompt-toolkit/python-prompt-toolkit#758 to ask for the searching mechanism on prompt_toolkit

[gpotter2]

Please ignore codacy: the eval is safe as layers only and the import is required by the eval

updated

[guedou]

See my comment. It its good to be merged.