Skip to content
This repository has been archived by the owner on Jul 31, 2019. It is now read-only.

Uwsgi Plugin serialization format #42

Closed
tiras-j opened this issue Mar 29, 2017 · 1 comment
Closed

Uwsgi Plugin serialization format #42

tiras-j opened this issue Mar 29, 2017 · 1 comment
Assignees

Comments

@tiras-j
Copy link
Contributor

tiras-j commented Mar 29, 2017

RE: My recent PR we know that marshal is dangerous but fast. However doing some rudimentary testing I think using struct + simple byte concat is faster on pre-Python3.5 (apparently someone decided to overhaul the marshal serialization I suppose). Here's a rudimentary test script I used to run some very basic timings. Using this approach means we have to deal with unicode vs. bytes ourselves, but since clog converts lines into UTF-8 anyways I don't think that will be much of an issue. How about stream names? It doesn't look like the ScribeLogger converts that to UTF-8, so we might need to reconstitute the unicode format by storing the encoding somehow.

Anyways, struct + raw bytes would be safer than marshal and would work in this case since we can make some strong assumptions about the arguments. I'll look into changing it but I think we should get the current PR in first just to fix the marshal.loads problem.

@tiras-j tiras-j self-assigned this Mar 29, 2017
@asottile
Copy link
Contributor

iirc the thrift stuff allows either UTF-8 bytes or text, so encoding the stream name should be safe

tiras-j pushed a commit to tiras-j/yelp_clog that referenced this issue Mar 29, 2017
@tiras-j tiras-j closed this as completed May 8, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants