Cache fields initialization

[guedou]

This PR attempts to enhance Scapy performance by caching the initialization of packets fields. Currently, they are initialized at each packet instantiation.

The benchmarks look quite nice and indicate that we can gain between 15% and 35% execution time depending on the use case.

Type	Before - 2.7	Before - 3.6	After - 2.7	After - 3.6
Build	15.77s	12.02s	11.28s (+28%)	9.87s (+17% / + 37%)
Dissect	6.84s	4.62s	4.50s (+34%)	3.82s (+17% / +44%)
Build & dissect	23.58s	17.52s	16.04s (+31%)	14.69s (+16% / +38%)

Here is the script used to bench this PR:

import time

from scapy.all import *
from scapy.modules.six.moves import range

N = 20000
raw_packet = b'E\x00\x00(\x00\x01\x00\x00@\x11|\xc2\x7f\x00\x00\x01\x7f\x00\x00\x01\x005\x005\x00\x14\x00Z\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00'

start = time.time()
for i in range(N):
    p = IP() / UDP() / DNS()
    assert raw(p) == raw_packet
print("Build - %.2fs" % (time.time() - start))

start = time.time()
for i in range(N):
    p = IP(raw_packet)
    assert DNS in p
print("Dissect - %.2fs" % (time.time() - start))

start = time.time()
for i in range(N):
    p = IP() / UDP() / DNS()
    s = raw(p)
    assert s == raw_packet
    p = IP(s)
    assert DNS in p
print("Build & dissect - %.2fs" % (time.time() - start))

fixes #619

[p-l-]

What's the reason of this change?

[guedou]

A RandShort() is consumed while the Packet fields are cached. Therefore, the randomly built packet is slightly different after the patch.

[gpotter2]

Looks like an amazing PR ! Looking forward to it

@guedou Could you have a small look at #619 to see if you could implement the idea with your system ?

PS: you were right about wireshark, #644 failed too...

[guedou]

While debugging the HTTP2 issue @X-Cli found this corner case which is not a smaller version of the issue:

$ cat test_642.py
from scapy.all import *

print '-----'

class SmallPacket(Packet):
  name = 'Small Packet'
  fields_desc = [ ByteField('byte', 0) ]

class TestPacket(Packet):
  name = 'TestPacket'
  fields_desc = [ PacketListField('list', [], SmallPacket) ]

a = TestPacket()
a.list.append(SmallPacket('a'))
a.list.append(SmallPacket('b'))
a.show()

print '-----'
TestPacket().show()

$ python test_642.py
-----
###[ TestPacket ]###
  \list      \
   |###[ Small Packet ]###
   |  byte      = 97
   |###[ Small Packet ]###
   |  byte      = 98

-----
###[ TestPacket ]###
  \list      \
   |###[ Small Packet ]###
   |  byte      = 97
   |###[ Small Packet ]###
   |  byte      = 98

[gpotter2]

@guedou I'm gonna try to get this fixed

[guedou]

Thanks ! I will have a look next week. I also have a fix that I did not push yet.

[guedou]

I have been working on this new patch with @X-Cli but forgot to push it ... It fixes the issue triggered by http2.uts but can slightly change the output of repr().

@gpotter2 & @p-l- I have several questions:

1. is it ok to cache in the Packet class ? We can also cache in subclass like IP or UDP
2. do you agree to alter the repr() behavior ?

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@e64b261). Click here to learn what that means.
The diff coverage is 98.11%.

@@           Coverage Diff            @@
##             master    #642   +/-   ##
========================================
  Coverage          ?   85.3%           
========================================
  Files             ?     179           
  Lines             ?   42300           
  Branches          ?       0           
========================================
  Hits              ?   36082           
  Misses            ?    6218           
  Partials          ?       0

Impacted Files	Coverage Δ
scapy/packet.py	77.11% <98.11%> (ø)

[p-l-]

@guedou: Can you rebase against current master?

Also, can you be more specific about the changes to repr() output: what changes and why?

[guedou]

With the patch, repr() displays the options field as if it was specified by the user. I did not find a way to keep the old behavior. I think that we can introduce this regression as repr() already display more fields than specified (i.e. repr(IP()/TCP()).

Before:

>>> repr(IP())
'<IP  |>'

After:

>>> repr(IP())
'<IP  options=[] |>'

[X-Cli]

Hello,
Just to clarify and shed more light on the problem: @guedou's patch uncovered a very old bug in Scapy.
In fact, in Scapy master revision, any field whose default value has an internal representation as a mutable object (anything except str, int and float, really) is affected by this bug.

When someone fetches the internal representation of such a field and there is no user data for this field, then the returned value is a reference to that mutable object. Any alteration of that object leads to the corruption/alteration of the default value of that field for that Packet.
Before this patch, this was not really a problem, because the default value of each field of each Packet was unique to each field instance of each Packet instance. Almost no one would ever notice this.

@guedou's patch makes all instances of a given Packet type to share the same reference to a unique default value object. So, when one of the Packet instance returns the reference to that shared object, and the referenced object is mutated, the default value of all Packet of the same type are affected by this modification. This is what was happening in the earlier code excerpt that @guedou posted.

To work around this, I submitted a patch to @guedou that deepcopies into self.fields the default value of a field upon a Packet type first instantiation, if that field internal representation of the default value is a mutable object. Unfortunately, this messes with repr() because it displays all fields which contain user values (i.e. values that are in self.fields).

[X-Cli]

Yeah, no. Your patch is similar to one of my early ideas to fix this issue. Unfortunately, it does not work. Returning a new mutable object upon __getattr__ on a field that contains only the default mutable value prevents the default value from being corrupted. However, this also breaks the expectation that self.mylist.append(...) would work uniformely (i.e. when there is already a user value in self.fields vs when there is none).

[gpotter2]

@X-Cli (my bad for the fix)
Question:
What if i suddenly want to use hide_defaults ? (it breaks your part :/)

[gpotter2]

@X-Cli Ok, here's another possibility:

• we keep your change
• we mark hide_defaults as useless / remove it
• we add:

diff --git a/scapy/packet.py b/scapy/packet.py
index debbee7..cf95e6d 100644
--- a/scapy/packet.py
+++ b/scapy/packet.py
@@ -323,9 +323,17 @@ class Packet(BasePacket):
             if isinstance(f, ConditionalField) and not f._evalcond(self):
                 continue
             if f.name in self.fields:
-                val = f.i2repr(self, self.fields[f.name])
+                _fval = self.fields[f.name]
+                _def = self.default_fields[f.name]
+                if _fval.__class__ == _def.__class__ and _fval == _def:
+                    continue
+                val = f.i2repr(self, _fval)

             elif f.name in self.overloaded_fields:
-                val =  f.i2repr(self, self.overloaded_fields[f.name])
+                _over = self.overloaded_fields[f.name]
+                _def = self.default_fields[f.name]
+                if _over.__class__ == _def.__class__ and _over == _def:
+                    continue
+                val =  f.i2repr(self, _over)
             else:
                 continue
             if isinstance(f, Emph) or f in conf.emph:

to keep the same repr behavior ?
(this checks if the value is the same than the default one)...

The triggered question then is the lost of performances... I guess that as __repr__ is only used to show the results, it could be a possible way of doing the things...

[X-Cli]

That sounds good to me. I would probably recommend the use of type(_fval) instead of _fval.__class__, though.
@guedou, @p-l-, any thoughts?

[guedou]

Here is the updated version. I don't know what to do with hide_defaults() =/

[gpotter2]

We can simply remove it as defaults are hidden automatically.

[gpotter2]

So. Where are we here ?

[guedou]

I need to find time to port it to Python3.

[guedou]

@gpotter2 Here is a version that works with Python3. Unfortunately, it does not work with MultiFlagField introduced int #1431

@p-l- can you find a way to fix it ? Given the performances benefit, that is definitely a PR that we need to merge.

[gpotter2]

conflicts

[gpotter2]

I have done several changes to MultipleTypeField in #1485, will check if it fixes anything

[guedou]

@p-l- @gpotter2 ready to be reviewed and discussed!

[gpotter2]

So what does hide_defaults() do now ?

[guedou]

It is broken. You are correct.

[guedou]

I made changes that will make codacity happy.

[gpotter2]

Amazing PR ! All good to me

[p-l-]

Just restarted a failed test, thanks @guedou!