Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-34093: marshal: Stabilize FLAG_REF usage #8226

Open
wants to merge 3 commits into
base: master
from

Conversation

@methane
Copy link
Member

methane commented Jul 10, 2018

marshal.dumps() tests refcnt(obj)==1 to decide use FLAG_REF or not.

But refcnt of interned string is very unstable.
When compiling same source, refcnt of interned string in the output
may be 1 or >1. It makes FLAG_REF usage unstable.

To help reproducible build, use FLAG_REF for interned string even if
refcnt(obj)==1.

https://bugs.python.org/issue34033

@methane methane force-pushed the methane:stable-marshal branch from 4ebb5cd to e170116 Jul 10, 2018
@methane methane changed the title bpo-34033: marshal: Use FLAG_REF stably bpo-34093: marshal: Use FLAG_REF stably Jul 11, 2018
@serhiy-storchaka serhiy-storchaka self-requested a review Jul 12, 2018
@methane methane force-pushed the methane:stable-marshal branch from 9d7e8f1 to 6c8ea7c Jul 13, 2018
@methane methane removed the DO-NOT-MERGE label Jul 14, 2018
methane added 2 commits Jul 14, 2018
@serhiy-storchaka

This comment has been minimized.

Copy link
Member

serhiy-storchaka commented Jul 14, 2018

I'm in process of reviewing.

@methane methane changed the title bpo-34093: marshal: Use FLAG_REF stably bpo-34093: marshal: Stabilize FLAG_REF usage Jul 14, 2018
@serhiy-storchaka

This comment has been minimized.

Copy link
Member

serhiy-storchaka commented Jul 16, 2018

Do you mind to provide also your initial PR? I want to try to optimize or simplify it.

Copy link
Member

serhiy-storchaka left a comment

Isn't using FLAG_REF for all interned strings slows down unmarshalling and increases the memory consumption for both marshalling and unmarshalling?

@methane

This comment has been minimized.

Copy link
Member Author

methane commented Jul 16, 2018

Isn't using FLAG_REF for all interned strings slows down unmarshalling and increases the memory consumption for both marshalling and unmarshalling?

For unmarshaling speed, it's possible. But interned string has cost of interning; create temoporary string and calling PyDict_SetDefault(). Overhead of FLAG_REF (PyList_Append) is much smaller than it.

For marshaling memory overhead, it will increase hashtable size for each interned string with refcnt==1.
For unmarshaling memory overhead, it will increase list length too.

I don't expect it can be pragmatic problem though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.