Skip to content

Create benchmark for CBOR format#10

Merged
Im5tu merged 3 commits into
Im5tu:mainfrom
Odonno:feat/cbor
Jan 20, 2025
Merged

Create benchmark for CBOR format#10
Im5tu merged 3 commits into
Im5tu:mainfrom
Odonno:feat/cbor

Conversation

@Odonno
Copy link
Copy Markdown
Contributor

@Odonno Odonno commented Nov 20, 2024

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist

  • I have updated any affected documentation with the changes in this PR
  • I have added tests to cover my changes
  • I have verified all new and existing tests pass
  • I have fully reviewed my own changes before raising the PR

@Odonno
Copy link
Copy Markdown
Contributor Author

Odonno commented Nov 20, 2024

Here are the updates to the output result:

Payload sizes:

Binary:
  Small:
    - MsgPack:        62441
    - ProtoBufNet:    66812
    - AvroConvert:    67772
    - MessagePack:    69309
    - Hyperion:       74317
    - Bebop:          89816
    - CBOR:           128718
    - BSON:           168195
    - GroBuf:         189230
  Medium:
    - MsgPack:        1569257
    - ProtoBufNet:    1685761
    - AvroConvert:    1695880
    - MessagePack:    1737075
    - Hyperion:       1834890
    - Bebop:          2225944
    - CBOR:           3187662
    - BSON:           4155568
    - GroBuf:         4687451
  Large:
    - MsgPack:        6555077
    - ProtoBufNet:    6831765
    - AvroConvert:    6834022
    - MessagePack:    7196839
    - Hyperion:       7369597
    - Bebop:          8938977
    - CBOR:           13014761
    - BSON:           16687507
    - GroBuf:         18823769

Benchmark run output:

| Method                  | data   | Mean          | Error        | StdDev       | Min           | Max           | Gen0      | Gen1      | Gen2     | Allocated   |
|------------------------ |------- |--------------:|-------------:|-------------:|--------------:|--------------:|----------:|----------:|---------:|------------:|
| MemoryPack_Deserialize  | Small  |      69.53 us |     0.859 us |     0.762 us |      68.55 us |      71.24 us |   15.7471 |    5.2490 |        - |    263712 B |
| Bebop_Deserialize       | Small  |      77.08 us |     1.415 us |     1.255 us |      75.34 us |      79.97 us |   16.2354 |    5.4932 |        - |    271976 B |
| GroBuf_Deserialize      | Small  |      96.88 us |     0.979 us |     0.916 us |      95.60 us |      98.78 us |   15.7471 |    5.2490 |        - |    263784 B |
| Hyperion_Deserialize    | Small  |     205.34 us |     0.861 us |     0.719 us |     204.19 us |     206.40 us |   18.7988 |    6.1035 |        - |    315425 B |
| ProtoBufNet_Deserialize | Small  |     214.53 us |     1.588 us |     1.485 us |     212.77 us |     217.33 us |   16.1133 |    5.3711 |        - |    270234 B |
| MessagePack_Deserialize | Small  |     223.62 us |     0.727 us |     0.644 us |     222.43 us |     224.63 us |   15.6250 |    5.1270 |        - |    263713 B |
| MsgPack_Deserialize     | Small  |     247.29 us |     1.449 us |     1.210 us |     245.45 us |     250.06 us |   19.0430 |    6.3477 |        - |    322226 B |
| CBOR_Deserialize        | Small  |     736.48 us |     2.819 us |     2.201 us |     733.27 us |     740.97 us |   15.6250 |    4.8828 |        - |    264019 B |
| BSON_Deserialize        | Small  |     780.21 us |     4.064 us |     3.394 us |     776.88 us |     786.63 us |   41.0156 |   13.6719 |        - |    701859 B |
| AvroConvert_Deserialize | Small  |     979.75 us |     7.327 us |     6.495 us |     970.53 us |     990.92 us |   70.3125 |   23.4375 |        - |   1268960 B |
| MemoryPack_Deserialize  | Medium |   2,435.43 us |    21.264 us |    17.756 us |   2,417.67 us |   2,481.76 us |  386.7188 |  316.4063 |        - |   6532973 B |
| Bebop_Deserialize       | Medium |   2,700.36 us |    20.247 us |    18.939 us |   2,672.77 us |   2,731.93 us |  402.3438 |  339.8438 |        - |   6732432 B |
| GroBuf_Deserialize      | Medium |   3,125.28 us |    41.796 us |    39.096 us |   3,069.61 us |   3,199.06 us |  386.7188 |  316.4063 |        - |   6533045 B |
| Hyperion_Deserialize    | Medium |   6,217.45 us |    44.267 us |    39.241 us |   6,150.78 us |   6,279.93 us |  460.9375 |  390.6250 |        - |   7789515 B |
| ProtoBufNet_Deserialize | Medium |   6,360.58 us |   112.905 us |   100.087 us |   6,133.89 us |   6,446.98 us |  398.4375 |  390.6250 |        - |   6694138 B |
| MessagePack_Deserialize | Medium |   6,378.60 us |    38.940 us |    36.424 us |   6,314.87 us |   6,461.28 us |  382.8125 |  312.5000 |        - |   6532987 B |
| MsgPack_Deserialize     | Medium |   7,198.17 us |    52.487 us |    49.096 us |   7,131.28 us |   7,285.67 us |  476.5625 |  429.6875 |        - |   7992968 B |
| CBOR_Deserialize        | Medium |  20,236.43 us |   102.750 us |    96.113 us |  20,043.32 us |  20,422.88 us |  375.0000 |  250.0000 |        - |   6664316 B |
| BSON_Deserialize        | Medium |  23,744.44 us |    87.362 us |    77.444 us |  23,559.91 us |  23,867.09 us | 1031.2500 |  906.2500 |        - |  17297843 B |
| AvroConvert_Deserialize | Medium |  27,081.65 us |   305.623 us |   270.927 us |  26,459.10 us |  27,513.96 us | 1656.2500 |  906.2500 | 125.0000 |  29296716 B |
| MemoryPack_Deserialize  | Large  |  34,176.96 us |   660.554 us |   926.004 us |  32,212.22 us |  35,759.54 us | 2000.0000 | 1923.0769 | 461.5385 |  26180240 B |
| GroBuf_Deserialize      | Large  |  36,335.19 us |   723.542 us | 1,188.800 us |  33,353.91 us |  38,819.24 us | 1928.5714 | 1857.1429 | 428.5714 |  26180189 B |
| Bebop_Deserialize       | Large  |  41,067.47 us |   815.001 us | 1,646.342 us |  37,424.69 us |  44,223.96 us | 2166.6667 | 2083.3333 | 583.3333 |  26981459 B |
| MessagePack_Deserialize | Large  |  49,041.56 us |   965.639 us | 1,073.305 us |  47,202.33 us |  51,097.17 us | 2000.0000 | 1909.0909 | 454.5455 |  26180291 B |
| ProtoBufNet_Deserialize | Large  |  52,915.37 us | 1,027.193 us | 1,141.722 us |  51,026.05 us |  54,750.99 us | 1900.0000 | 1800.0000 | 400.0000 |  27043526 B |
| Hyperion_Deserialize    | Large  |  63,241.10 us | 1,228.585 us | 1,681.702 us |  60,665.40 us |  66,870.26 us | 2222.2222 | 2111.1111 | 444.4444 |  31181226 B |
| MsgPack_Deserialize     | Large  |  63,699.21 us | 1,168.030 us | 1,092.576 us |  62,135.51 us |  65,547.82 us | 2375.0000 | 2250.0000 | 500.0000 |  32020654 B |
| CBOR_Deserialize        | Large  |  98,928.81 us | 1,200.801 us | 1,064.479 us |  97,911.83 us | 100,791.05 us | 1833.3333 | 1666.6667 | 333.3333 |  28976408 B |
| AvroConvert_Deserialize | Large  | 138,130.42 us | 2,661.123 us | 3,730.523 us | 133,264.67 us | 146,892.27 us | 6250.0000 | 3250.0000 | 250.0000 | 117939758 B |
| BSON_Deserialize        | Large  | 146,646.34 us | 2,704.326 us | 2,656.010 us | 142,567.95 us | 151,809.00 us | 4500.0000 | 4000.0000 | 500.0000 |  69299046 B |
|                         |        |               |              |              |               |               |           |           |          |             |
| MemoryPack_Serialize    | Small  |      42.05 us |     0.284 us |     0.252 us |      41.66 us |      42.55 us |   16.0522 |   16.0522 |  16.0522 |     98835 B |
| GroBuf_Serialize        | Small  |      74.19 us |     0.567 us |     0.531 us |      73.39 us |      75.18 us |   29.5410 |   29.5410 |  29.5410 |    189659 B |
| Bebop_Serialize         | Small  |      81.07 us |     0.324 us |     0.270 us |      80.66 us |      81.57 us |         - |         - |        - |           - |
| MessagePack_Serialize   | Small  |     116.72 us |     0.684 us |     0.571 us |     115.76 us |     117.85 us |    4.0283 |         - |        - |     69336 B |
| ProtoBufNet_Serialize   | Small  |     180.24 us |     1.149 us |     0.959 us |     178.65 us |     181.46 us |   13.4277 |    2.1973 |        - |    228617 B |
| MsgPack_Serialize       | Small  |     196.70 us |     2.612 us |     2.444 us |     193.77 us |     201.87 us |   35.4004 |    7.0801 |        - |    596033 B |
| Hyperion_Serialize      | Small  |     209.84 us |     0.985 us |     0.873 us |     208.89 us |     211.75 us |   38.3301 |   24.6582 |  23.4375 |    384364 B |
| AvroConvert_Serialize   | Small  |     310.17 us |     2.173 us |     1.926 us |     307.22 us |     313.68 us |   71.7773 |   48.8281 |  45.8984 |    703089 B |
| CBOR_Serialize          | Small  |     573.29 us |     0.587 us |     0.490 us |     572.48 us |     573.99 us |   50.7813 |   45.8984 |  45.8984 |    347482 B |
| BSON_Serialize          | Small  |     794.15 us |     5.347 us |     4.740 us |     787.49 us |     805.71 us |  129.8828 |  100.5859 |  98.6328 |   1093995 B |
| MemoryPack_Serialize    | Medium |   1,743.85 us |    33.572 us |    42.458 us |   1,646.98 us |   1,795.76 us |   44.9219 |   44.9219 |  44.9219 |   2501397 B |
| Bebop_Serialize         | Medium |   2,319.44 us |     7.615 us |     6.359 us |   2,305.46 us |   2,330.09 us |         - |         - |        - |     32775 B |
| GroBuf_Serialize        | Medium |   2,591.47 us |    28.304 us |    26.475 us |   2,555.39 us |   2,631.12 us |  160.1563 |  160.1563 | 160.1563 |   4689228 B |
| MessagePack_Serialize   | Medium |   3,294.46 us |    11.836 us |    10.493 us |   3,277.16 us |   3,307.65 us |   70.3125 |   70.3125 |  70.3125 |   1737352 B |
| ProtoBufNet_Serialize   | Medium |   5,192.02 us |    32.494 us |    28.805 us |   5,156.54 us |   5,239.48 us |  265.6250 |  257.8125 | 257.8125 |   7091169 B |
| Hyperion_Serialize      | Medium |   5,371.18 us |    24.494 us |    20.454 us |   5,333.48 us |   5,404.69 us |  289.0625 |  218.7500 | 218.7500 |   7198176 B |
| MemoryPack_Serialize    | Large  |   5,802.84 us |   112.323 us |   124.847 us |   5,540.84 us |   6,021.48 us |   70.3125 |   70.3125 |  70.3125 |  10168937 B |
| MsgPack_Serialize       | Medium |   6,180.95 us |   118.145 us |   110.512 us |   6,038.66 us |   6,369.11 us |  570.3125 |  429.6875 | 429.6875 |  15024468 B |
| GroBuf_Serialize        | Large  |   9,202.98 us |   179.802 us |   159.390 us |   8,912.42 us |   9,534.19 us |  125.0000 |  125.0000 | 125.0000 |  18825078 B |
| Bebop_Serialize         | Large  |   9,742.23 us |    53.710 us |    50.241 us |   9,676.94 us |   9,835.91 us |         - |         - |        - |    524311 B |
| AvroConvert_Serialize   | Medium |  11,324.44 us |   224.199 us |   566.579 us |  10,227.96 us |  12,998.62 us |  625.0000 |  437.5000 | 390.6250 |  14769796 B |
| MessagePack_Serialize   | Large  |  13,867.94 us |    65.614 us |    61.375 us |  13,763.09 us |  13,989.79 us |   31.2500 |   31.2500 |  31.2500 |   7197022 B |
| CBOR_Serialize          | Medium |  15,271.30 us |   103.934 us |    92.134 us |  15,113.38 us |  15,426.75 us |  343.7500 |  265.6250 | 218.7500 |  10462162 B |
| ProtoBufNet_Serialize   | Large  |  20,759.26 us |   206.890 us |   172.762 us |  20,579.79 us |  21,122.24 us |  187.5000 |  187.5000 | 187.5000 |  27893947 B |
| Hyperion_Serialize      | Large  |  23,342.33 us |   428.208 us |   400.546 us |  23,007.41 us |  24,205.40 us |  406.2500 |  125.0000 | 125.0000 |  28827824 B |
| BSON_Serialize          | Medium |  23,620.53 us |   411.202 us |   384.639 us |  22,935.83 us |  24,230.44 us | 1000.0000 |  406.2500 | 406.2500 |  22512040 B |
| MsgPack_Serialize       | Large  |  30,868.42 us |   609.215 us |   598.330 us |  29,813.11 us |  31,784.49 us | 1000.0000 |  531.2500 | 531.2500 |  67712102 B |
| AvroConvert_Serialize   | Large  |  36,923.11 us |   724.022 us | 1,229.444 us |  35,532.65 us |  39,816.21 us | 1133.3333 |  266.6667 | 266.6667 |  59223897 B |
| CBOR_Serialize          | Large  |  64,897.16 us |   355.562 us |   332.593 us |  64,154.34 us |  65,410.18 us |  500.0000 |         - |        - |  39033197 B |
| BSON_Serialize          | Large  |  83,231.18 us |   826.231 us |   732.433 us |  81,281.39 us |  84,442.63 us | 2428.5714 |  142.8571 | 142.8571 |  90285945 B |

@Im5tu Im5tu merged commit 923ed71 into Im5tu:main Jan 20, 2025
@Odonno Odonno deleted the feat/cbor branch January 20, 2025 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants