Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added classes and unit tests for local_file, orangefs_file and pxfs_file. #1267

Closed
wants to merge 5 commits into from

Conversation

EverYoung124
Copy link
Contributor

Provide asynchronous IO functions for local files, OrangeFS files and PXFS files. All io operations are executed on separate io-thread. local_file and orangefs_file are using server/client model to implement asynchronism while pxfs_file uses the pxfs asynchronous APIs.

@hkaiser
Copy link
Member

hkaiser commented Sep 20, 2014

As it turns out, boost::intrusive_ptr::detach() exists only starting Boost V1.56. That means that the code currently will build only when using this Boost version. Memory management might have to be done explicitly to work around this.

@EverYoung124
Copy link
Contributor Author

It seems detach() is abandoning the ownership of the memory so that it will not get destroyed. Then for boost 1.55.0 and older, is there a way to get out of the intrusive_ptr's control? I tried to use get() and then reset(), not working.

Another way is to modify the intrusive_ptr_release function so that it deletes the memory when count==-1, however, it seems nasty....

@hkaiser
Copy link
Member

hkaiser commented Sep 23, 2014

After thinking some more about this I realized that even the current code is not 100% accurate. The scheme we currently employ is:

void callback(..., void* cb_data)
{
    boost::intrusive_ptr<data> d(cb_data);
    // use d
}

void foo()
{
    boost::intrusive_ptr<data> d(new data);
    some_api_function(..., callback, (void*)d.get());    // extern "C"

    // use d

    // make sure to detach d
    d.detach();
}

This will usually lead leak the data as only one of the reference counts we create will be released (in the destructor of d inside callback).

Solving this problem is not trivial as we have to ensure that the data is released only after both functions are done using it.

@EverYoung124
Copy link
Contributor Author

When calling detach(), the reference count is still 1 and then the intrusive_ptr has abandoned the ownership, thus leaving the data pointer as an orphan and never get deleted. Is this where the problem is?

The foo() function need to return future of the promise inside data, so basically after foo() finishes, we only have access to the future, not the data itself. I am wondering how the data struct can be released.

How did we solve this problem in the client/server model of the hpx component? It should also face the same problem where one action needs to return a future.

I remember that hpx has garbage collection, is it right?

@EverYoung124
Copy link
Contributor Author

Updated the diskperf example using these file classes.

…ile. All io operations are executed on io-thread. local_file and orangefs_file are using server/client model to implement asynchronism while pxfs_file uses the pxfs asynchronous APIs.
@hkaiser
Copy link
Member

hkaiser commented Jan 28, 2015

All if this has been incorporated into the branch pxfs (https://github.com/STEllAR-GROUP/hpx/tree/pxfs). We will continue developing things there.

@hkaiser hkaiser closed this Jan 28, 2015
@hkaiser hkaiser deleted the file_class branch January 28, 2015 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants