Check if HDF5 "file" is a DAOS object

[brtnfld]

PR to skip the POXIS calls used in reading magic number. These POSIX calls can not be used for a DAOS object.

Intel DAOS team feedback is requested as we might be able to remove the HDF5 calls.

This solution assumes Unified Namespace and getfattr is available.

[brtnfld]

We might refrain from using getfattr if we modify H5Fis_accessible to return the VOL access type instead. Investigating.

[christgau]

Wouldn't it be better to directly call getxattr(2) from sys/xattr.h here instead of calling a command which has to be in the path (sure, it is very likely the case) and could potentially do something else than returning file attributes? In the end, getfattr does same call to retrieve the attributes.... at least on a CentOS 7 installation.

[DennisHeimbigner]

I do not think this code is in the proper location. I think it should be part of the later
file reading checks.

[brtnfld]

I do not think this code is in the proper location. I think it should be part of the later
file reading checks.

I think it has to be before phase 7 because that looks like the first place you start to read the magic number. Where do you suggest?

[brtnfld]

I do not think this code is in the proper location. I think it should be part of the later
file reading checks.

Wouldn't it be better to directly call getxattr(2) from sys/xattr.h here instead of calling a command which has to be in the path (sure, it is very likely the case) and could potentially do something else than returning file attributes? In the end, getfattr does same call to retrieve the attributes.... at least on a CentOS 7 installation.

Yes, that would be better.

[DennisHeimbigner]

FYI, I am working on this again.

[DennisHeimbigner]

If there is some way to modify this to avoid using popen, it would be appreciated.

[brtnfld]

Let me see what I can do, the eventual plan is to have H5Fis_accessible return if it is a DAOS container.

[edwardhartnett]

At the EGU I learned of some large data producers wanting to use DAOS for some of their very large data sets. I think it would be great of netCDF handled DAOS better.

I don't think this should hold up the long-awaited 4.9.0 release, but it would be great if this could be figured out for the 4.9.1 release...

[DennisHeimbigner]

To date, we have no solution.

[edwardhartnett]

@WardF can we get this merged? I'd love to test it, and I think it's an important feature. I note that it's scheduled for the 4.9.1 release, so if we can get it in I can test it in the next release candidate...

[brtnfld]

Here is the response I received from Intel:

You can use the extended attribute, check for its existence:
/** extended attribute name that will container the UNS info */
#define DUNS_XATTR_NAME "user.daos"

Ideally you would use that macro, or the duns and call duns_resolve_path(), but you can’t do that I assume in this case since you would need to build netcdf with daos, right?
You can just call lgetxattr on the path with that name and I would think it should be OK. If we change the actual name, one would need to update netCDF but I don’t see that happening.

[DennisHeimbigner]

I see numerous issues with this PR. Most importantly, the DAOS check
appears to be in the wrong place in the code. Should it not be in openmagic?
Oh well, we can fix it later.

[WardF]

@DennisHeimbigner it has not been merged, so now would be a good time to fix it too :)

[DennisHeimbigner]

Well let me take a look.

[DennisHeimbigner]

This is coming back to me. It makes me uneasy from a security point of view to use popen
because it in effect is executing a shell script of sorts as part of the netcdf-c library.
I really wish there was another, more direct, way to test for DAOS.

[DennisHeimbigner]

Also, there is not test case for this. Is there any open DAOS based server against we can test?

[dopplershift]

There's a getxattr() api that can (and probably should) be used instead.

[brtnfld]

I think this only needs to be done now if dfuse is not being used, as a dfuse netcdf file should appear as a POSIX file. This addressed the case of unified name space being used, which seems less common now.

[christgau]

I think this only needs to be done now if dfuse is not being used, as a dfuse netcdf file should appear as a POSIX file.

True.

This addressed the case of unified name space being used, which seems less common now.

But still is a valid use case. Supporting this appears to be lightweight in terms of code. I would highly appreciate it to be available (checking extended file attributes directly, instead via an popen call).

[christgau]

Is there any indicator which DAOS version this branch is developed against?

[brtnfld]

At the time, it would have been DAOS 2.0

[DennisHeimbigner]

Attached is an revised version of dinfermodel.c for you to test. It isolates the DAOS code
and makes it a bit more conformant to the dinfermodel.c coding style.
It also implements the daos check using the API calls listxattr and getxattr
(see dinfermodel.c#isdaoscontainer.c).

Please see it works for your tests.
dinfermodel.c.txt

[brtnfld]

Your patch works and correctly detects the daos file with the daos-vol, which was tested on Google Cloud. I see other testing errors, but I don't think they relate to this issue.