Reading GSD file contents from in-memory buffer

[stukowski]

Description

I would like to propose an extension to the GSD C API, which would allow an application to read data from GSD files stored as blob in memory (and not in the local file system).

The GSD C API function gsd_open() allows opening only physical files, but in certain application scenarios, the GSD data may not be stored in the local file system in the first place. Consider for example a program that downloads a GSD file from a web server, and which would like to read the contained data directly without having to write the received data stream to disk first.

Proposed solution

Additional context

Another scenario where the proposed extension is essential is an application running in a sandbox environment without direct access to the file system and which receives the GSD file data as a blob from some other party. A concrete use case I am working on is a port of the OVITO code running in a web browser. See this demo: https://ovito.org/webdemo/

Here, the original OVITO C++ source code has been compiled to platform-independent WebAssembly code running in a browser tab. When the user picks a local file for import into the application, the browser hands over just the file content as a blob and not the actual path to the file. WebAssembly applications have no access to the file system. Thus, OVITO would need to parse the GSD file content from an in-memory buffer.

Developer

[joaander]

Thanks for the request! GSD internally memory maps most of its data structures, and those that it doesn't are all funneled through a wrapper read/write call. This addition should require relatively minimal changes to the code. I won't get this done for 2.0, but can plan it for 2.1.

An API to open a blob for reading might look like:

int gsd_open_blob(struct gsd_handle* handle, const void *data, size_t data_size);

And then all the other API calls would work with handle like they do for open files.

Would you need write support to memory buffers as well as read? If yes, who should be responsible for allocating memory for new writes to the blob?

[stukowski]

Thanks for the quick response and your willingness to add this capability to the GSD C API. No need to rush it. Work on "OVITO for Web" just started, so I don't have an urgent need for this capability.

The gsd_open_blob() interface you proposed is exactly what I would need.

Having the option to write a GSD file into a memory buffer is also a good idea. In fact, this feature would be required for OVITO's data export function. The program could produce a GSD file in memory and hand the buffer over to the web browser, making its content available to the user as a local file "download".

I can't think of a simple API that would allow the calling application to manage the dynamically growing memory buffer. Nor do I see a scenario where the application really needs the data to go into a pre-allocated buffer. That's why I would propose to let the GSD API manage the memory buffer when writing files. We just need a way for the application to query the address and size of the internal buffer when the data is ready to be transferred somewhere else.

But these are just my thoughts. I would like to leave it to you to decide which party manages the memory buffer in this case. Perhaps there are use cases I haven't thought of, and I would definitely be happy with both options.

[joaander]

That is a good idea. GSD can maintain the in-memory buffer and provide an API to query it. The buffer would be available and kept up to date after each call to gsd_end_frame. gsd_close would free the buffer.

Perhaps even in read mode GSD should copy the provided blob to a GSD handled buffer so that calling application does not need to maintain the input data buffer for the lifetime of the gsd handle.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This issue has been automatically closed because it has not had recent activity.

[joaander]

I do plan to implement this eventually.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has been automatically closed because it has not had recent activity.