[C++] Failed to open local file: Bad file descriptor

[asfimport]

I open an output stream to write a parquet file. Here is the code:

std::shared_ptr<arrow::io::OutputStream> outStream;
arrow::Status err;

err = arrow::io::FileOutputStream::Open(filePath.toStdString(), false, &outStream);
if (err.code() != arrow::StatusCode::OK) {
    std::cout << err.message() << std::endl;
}

Here is the error message I got:

Failed to open local file: <path_to_parquet_file> , error: Bad file descriptor"

I've got this error only when running on Windows version 10.0.10586.1106 (th2_release.170904-1742).

Any idea?

Environment: Windows version 10.0.10586.1106 (th2_release.170904-1742)
Reporter: Tham / @thamht4190

Related issues:

• [C++] Crash at arrow::internal::FileWrite (duplicates)

PRs and other links:

• GitHub Pull Request #4724

_{Note: This issue was originally created as ARROW-5700. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
To start with, which version of Arrow are you using and did you compile it yourself?

[asfimport]

Tham / @thamht4190:
@pitrou  Yes, I build arrow from source myself at this commit: 3eb07b7

 

[asfimport]

Antoine Pitrou / @pitrou:
Does it happen with any output file?

[asfimport]

Antoine Pitrou / @pitrou:
I took a look at the CRT source code, and it's mysterious how EBADF "Invalid file descriptor" could be returned by _wsopen_s.

Since you compiled, you could probably try to debug with the Arrow source code. The function is FileOpenWritable in cpp/src/arrow/util/io-util.cc.

(you could also try to upgrade to a more recent version, as I think the code changed slightly)

[asfimport]

Tham / @thamht4190:

Does it happen with any output file?
I'm not sure. We receive this crash from customer. After this issue makes our application crashed, every time our application is started again, it fails immediately with another message (on same file):

Failed to open local file: <path_to_parquet_file>, error: File exists

So we don't know if it fails with another files.

I will try to install some Windows version (or at least this Windows version) on our office and try to debug as you suggested.

(you could also try to upgrade to a more recent version, as I think the code changed slightly)
I will try with current version first, then upgrade later to see if this issue remains.

[asfimport]

Tham / @thamht4190:

I took a look at the CRT source code, and it's mysterious how EBADF "Invalid file descriptor" could be returned by [_wsopen_s]
Is there any chances that open() is called instead of _wsopen_s(), meaning _WIN32 is not defined?

[asfimport]

Antoine Pitrou / @pitrou:

Is there any chances that open() is called instead of _wsopen_s(), meaning _WIN32 is not defined?

I don't think so, compilation would probably fail.

[asfimport]

Tham / @thamht4190:
@pitrou I can reproduce this issue on this Windows version.

Does it happen with any output file?
Yes, I'm sure now it happens with any output file on this Windows.

Here is the output after _wsopen_s() is called.

_wsopen_s: fd:0000024308936AE8 (*fd): -1 oflag:163721 pmode:384 errno_actual:9

Can you take a look?

[asfimport]

Antoine Pitrou / @pitrou:
Have you checked the value of file_name.ToNative().c_str() ?

[asfimport]

Antoine Pitrou / @pitrou:
Also, what happens if you replace _wsopen_s with the less secure _wsopen ?

[asfimport]

Antoine Pitrou / @pitrou:
Another question still: what does GetLastError() return?

[asfimport]

Antoine Pitrou / @pitrou:
By the way, I've submitted #4724 which should possibly give better error messages, as "Bad file descriptor" doesn't make sense here.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 4724
#4724

[asfimport]

Wes McKinney / @wesm:
The PR didn't actually resolve the problem, just gives better error messages

[asfimport]

Antoine Pitrou / @pitrou:
Right :-)