shared-state: provide compressed cgi-bin endpoints

[spiccinini]

Add compression support to the public synchronization endpoints so that the data is transmitted and received using gzip when gzip is part of the URL query-string.

Why? mainly to help shared-state syncing when there is a bad quality link (low signal, too crowded airspace, etc) reducing the amount of packets that have to be transmitted. Also this leads to reduced bandwidth usage.

This change in its own does not change the current behaviour (uncompressed) but allows to use this feature in the future. There are two main possible mechanisms with different backwards compatibility policy:

1. Provide a backwards compatible mechanism that allows a node to know if the other node supports gzip or not and use the new endpoint in that case. There are many options to do this but all add complexity.
2. Add it in a backward incompatible way but with an upgrade path: provide compression support in the next libremesh release, with compression unused when syncing, and then in a future release switch to "always" compress.

I am more inclined to do the 2nd option as the first option not only adds code complexity but also runtime overhead and/or runtime complexity. What do you think?

[codecov-commenter]

Codecov Report

Merging #911 (f4ee585) into master (95da863) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #911      +/-   ##
==========================================
- Coverage   76.41%   76.38%   -0.03%     
==========================================
  Files          47       47              
  Lines        3934     3934              
==========================================
- Hits         3006     3005       -1     
- Misses        928      929       +1     

Impacted Files	Coverage Δ
...kages/lime-system/files/usr/lib/lua/lime/utils.lua	81.06% <0.00%> (-0.30%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95da863...f4ee585. Read the comment docs.

[ilario]

Second option sounds good.
Shouldn't the original line be erased?
I mean, now you added lines, but you should remove the original code line doing the same in the uncompressed way?

[germanferrero]

I think we could choose 1st option with Accept and Content-Type http headers , and OPTIONS requests, as an standard mechanism

[spiccinini]

I think we could choose 1st option with Accept and Content-Type http headers , and OPTIONS requests, as an standard mechanism

Do you see how without adding additional requests per request or maintaining a state per peer and adding at least one more request in the flow?

[G10h4ck]

Seems a good idea to me, for the future I agree on following the libremesh tradition (so we keep it lightwight) providing upgrade path instead of retro-compatibility.

About the code do I understand well that you forgot to delete a couple of lines from the former code?

[germanferrero]

I think we could choose 1st option with Accept and Content-Type http headers , and OPTIONS requests, as an standard mechanism

Do you see how without adding additional requests per request or maintaining a state per peer and adding at least one more request in the flow?

No, what I had in mind was using an additional OPTION request. Now that I understand the "upgrade path" proposal I also think that it's the best option.

[germanferrero]

Thanks @spiccinini !

[spiccinini]

I've run the following test:

root@LiMe-abc000:/# cat /tmp/shared-state/data/babeld-hosts.json > /tmp/foo
root@LiMe-abc000:/# cat /tmp/shared-state/data/babeld-hosts.json | gzip > /tmp/foo.gzip


root@LiMe-abc000:/# uclient-fetch -q -O- --timeout=3 --post-file='/tmp/foo' 'http://localhost/cgi-bin/shared-state/spam'
root@LiMe-abc000:/# uclient-fetch -q -O- --timeout=3 --post-file='/tmp/foo.gzip' 'http://localhost/cgi-bin/shared-state/bar?gzip=1'

root@LiMe-abc000:/#  md5sum  /tmp/shared-state/data/babeld-hosts.json  /tmp/shared-state/data/bar.json  /tmp/shared-state/data/spam.json
e4b23afbbdcbed68539fa8003d6be29c  /tmp/shared-state/data/babeld-hosts.json
e4b23afbbdcbed68539fa8003d6be29c  /tmp/shared-state/data/bar.json
e4b23afbbdcbed68539fa8003d6be29c  /tmp/shared-state/data/spam.json

[spiccinini]

Hi @ilario , @G10h4ck I have resolved the issues on this, is there anything missing? thanks!

[ilario]

For me whatever works is ok :)
A general thought: The retrocompatibility is not a must, as the compatibility between different LibreMesh releases is not assured and not even tested. (is this right?)

[spiccinini]

For me whatever works is ok :) A general thought: The retrocompatibility is not a must, as the compatibility between different LibreMesh releases is not assured and not even tested. (is this right?)

right but the idea is to minimize issues, even if it is not fully compatible if the shared db does not work while you are upgrading the mesh then it will be harder for the community. So for this at least for me it is important.