-
Notifications
You must be signed in to change notification settings - Fork 370
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Save and load operators in human-readable format #122
Comments
We actually had a large discussion about this back in the early days of FermiLib. See The long story short is that pickle has security issues and also is less efficient. Back in the early days of the project (when we were still developing it with tight ProjectQ integration under the name FermiLib), we did use pickle initially. However, myself and some collaborators started working on a project that involved computation of FermionOperators that are tens of gigabytes and the size of the pickle objects became a real issue for us. However, I think there are also some concerns about marshal that were never really resolved. Looping @Spaceenter into the conversation since he implemented marshal for us. |
Direct text representations are also possible and easy to implement. We could allow for different modes in our save_to_file function that save in different formats - thoughts? Would that address things for you @kevinsung ? |
Yeah, I think adding an option to the |
I think the reason we chose marshal over JSON a few months ago is performance:
That being said, I agree JSON is more readable for very small operators when performance is not a concern. However, if you want human-readable format, is the str method of the operators sufficient? @kevinsung |
One thing is that we have str but can't convert from string. A possible
nice solution would be expanding init to allow longer input strings (as is
only the term operators are possible), and adding "as string" modes for
load/save (which look at a file). I think we could do the first without the
second but it's nice to have it encapsulated.
…On Mon, Dec 11, 2017 at 3:37 PM Wei Sun ***@***.***> wrote:
I think the reason we chose marshal over JSON a few months ago is
performance:
1.
I think we originally implemented using JSON, but @idk3
<https://github.com/idk3> found it uses too much memory, and switching
to marshal reduced the memory usage by an order of magnitude.
2.
Another issue was that JSON requires every key (and value?) must be
string, so we end up having lots of conversions from number to string and
string to number, which is not efficient.
That being said, I agree JSON is more readable for very small operators
when performance is not a concern. However, if you want human-readable
format, is the *str* method of the operators sufficient? @kevinsung
<https://github.com/kevinsung>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#122 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AKRqnTLxWZ2T3W21PqfyvTqCjZbKrgHoks5s_ZJsgaJpZM4Q9iSb>
.
|
@Spaceenter Yeah the |
Yes, that is a great suggestion. I think we should open a new issue about that functionality. |
The string initialization functionality has now been expanded. Accordingly, if anyone wants to work on this, go ahead! |
Hi, I'm Chris, and I'd like to start getting involved with OpenFermion. Has this issue been resolved? It seems as though |
This issue is not resolved yet. Yes, this should use the |
@jarrodmcc always seems to have opinions about how things should be saved. Any thoughts Jarrod? Chris, we should discuss the solution here ideally before you make your PR. |
One key question is why we wanted to human readable format to begin with, and do we really have applications of reading in such a file rather than just outputting one? Knowing this helps determine what we really want to do here. Something like JSON has massive overhead and not what I would call exceptional readability. Other ASCII formats are more readable, but probably require special parsers. My initial inclination would be to keep what we have, and add as Ian suggested a flag to save_to_file to dump to a text format dictated by str. A secondary method could allow one to load in the same way. However I think we should keep the options simple, adding a bunch of different file formats could just increase maintenance complexity later unless we have a good reason for having that format. |
I'm not really sure about the context in which @kevinsung proposed this feature, but my understanding is that a human-readable storage format might be useful in interfacing with other applications or for inspecting simpler operators manually. I can't speak to what kind of overhead might be incurred by using JSON in these situations, but do you think that the following example is readable enough?
Would this be a reasonable way to encode an operator? Is there any other information that should be included or any naming discrepancy that should be corrected? Would it be more human-readable if the list of factors was replaced with the output from the I figure that it would make sense to add a flag to save_operator and load_operator to specify whether a more human-readable format is desired like people have suggested. I don't think there would be much need for many other file formats, unless I'm missing something about why additional formats are useful? |
I actually just wanted to be able to save and load using the |
Currently we have no mechanism for saving and loading FermionOperators and QubitOperators, right? I think the idea was literally just to have a function for saving and loading to/from a text file that is a dump of what comes out when you call This is not the most numerically performant option, but it is certainly human readable, and would be a quick and simple solution that should suffice for most purposes that I can imagine. |
We currently have the functions |
@kevinsung is this issue closed now? |
Ah, yes. Thanks! |
Currently, the
save_operator
function inutils/_operator_utils.py
saves Fermion and QubitOperators using the data serialization modulemarshal
. I think it's desirable to save these objects in a human-readable format, like JSON. What do people think? This would probably best be implemented after resolving issue #43.Also, I'm wondering why
marshal
was chosen rather than the more standardpickle
orcPickle
.The text was updated successfully, but these errors were encountered: