deepspeed.ops.op_builder.async_io.AsyncIOBuilder assert

[stas00]

How can we make it clear to the user that they need to do apt install libaio-dev in this assert?

self = <deepspeed.ops.op_builder.async_io.AsyncIOBuilder object at 0x7f5dbce1f160>
verbose = True
    def jit_load(self, verbose=True):
        if not self.is_compatible():
>           raise RuntimeError(
                f"Unable to JIT load the {self.name} op due to it not being compatible due to hardware/software issue."
            )
E           RuntimeError: Unable to JIT load the async_io op due to it not being compatible due to hardware/software issue.

@tjruwase, can't we just say apt install libaio-dev in the error message?

f"Unable to JIT load the {self.name} op due to it not being compatible due to hardware/software issue. "
"Most likely you need to: 'apt install libaio-dev'"

[tjruwase]

@stas00, if you still have the log can you please check if the assert message is preceded by warning like this: ... requires the libraries: ['libaio-dev'] but are missing. ?

[stas00]

I was given just this part of the trace, so I don't have access to the full log, but as we discussed here: #998 (comment) it must have been the case.

Clearly, the users don't connect the warning and the traceback. Hence the suggestion to include the explicit requirements in the error message.

[tjruwase]

Got it. Okay, this will require a bit more code in the JIT engine. We will look into that.

[stas00]

Perhaps there is a way to check if libaio-dev is installed, other than using apt since it's not the only package manager. But through some other way.

Looking at the contents of the package, other than .so and .a objects, there is /usr/include/libaio.h - I'm not sure if any of these can be checked for directly from python, but probably a tiny .c program could be used to test for it or however gcc's ./configure tool does the discovery.

---

I'm pretty sure I also saw cases of this problem where the WARNING came several lines after the traceback! Different streams...

[dlfrnaos19]

same issue, libaio-dev couldnt make it..
https://github.com/dlfrnaos19/aiffel_test1/blob/master/deepspeed_issue.ipynb

[DevManPGP]

Why does deepspeed say it works in windows when it obviously does not!