Csound::Compile() won't work if non-English characters in path.. #632

rorywalsh · 2016-04-21T12:59:05Z

Many of my students have non-English characters in their names. Their home directory, which is the base of all their document paths, therefore contains non English characters. When I use these paths with Csound::Compile() I get:

csound->FileOpen2 failed:: No such file or directory

Below is a simple .cpp file you can test with it. Note that I've only tested this on Windows. I think it's working fine on other platforms. There is no problem when I use the offending path directly with the Csound command line.

#include <stdio.h>
#include "csound.hpp"

int main(int argc, char *argv[])
{
Csound* csound = new Csound();

//doesn't work....
csound->Compile("C:/Users/rory/Desktop/Ádhamh/quickTest.csd");

//works fine....
csound->Compile("C:/Users/rory/Desktop/quickTest.csd");

csound->Start();
csound->Perform();  
delete csound;

return 0;
}

The text was updated successfully, but these errors were encountered:

vlazzarini · 2016-04-21T13:12:19Z

I tried a similar test on OSX

csound->Compile("àèù.csd");

and it passed.

fsateler · 2016-04-21T13:18:03Z

I think the problem is that the string encoding in the source file does not match the encoding used at runtime, thus "special" characters get garbled. @rorywalsh, does the problem persist if you pass the filename via a command line argument to your test program? Also, you can test changing the file encoding to different things (try with utf-8, notepad++ has a Encoding menu that can be used to easily switch)

rorywalsh · 2016-04-21T15:53:24Z

Thanks Fiipe, passing the filename from the command line does work but encoding my files as utf8 doesn't make any difference. Any other ideas?

rorywalsh · 2016-04-22T09:47:44Z

Csound seems to replace the Á with ├ü on Windows? There's hardly a compiler setting I'm overlooking or something?

csound->FileOpen2 failed:: No such file or directory
UnifiedCSD: C:/Users/rory/Desktop/├üdhamh/quickTest.csd

[edit] Just found this link, https://sourceforge.net/p/mingw-w64/wiki2/Unicode%20apps/, perhaps that's the key. I'll try it out and post back when I get the chance.

fsateler · 2016-04-22T11:14:35Z

Rory, did you try other encoding options?

Unfortunately, because c strings have 8bit characters, your sources need to
match the system encoding for this to work. More unfortunately, I have no
idea how to determine the system encoding on windows.
On 22 Apr 2016 06:47, "Rory Walsh" notifications@github.com wrote:

Csound seems to replace the Á with ├ü on Windows? There's hardly a
compiler setting I'm overlooking or something?

csound->FileOpen2 failed:: No such file or directory
UnifiedCSD: C:/Users/rory/Desktop/├üdhamh/quickTest.csd

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#632 (comment)

rorywalsh · 2016-04-22T12:17:42Z

I did Felipe, but none seem to work. I'll keep digging.

rorywalsh · 2016-04-22T14:38:49Z

It seems the fopen() on windows at least will not return a valid FILE pointer if non-ascii chars are in the file path. The solution is to use _wfopen() which I think is Windows only. Using _wfopen() instead of fopen in the following code fixes the issue.

https://github.com/csound/csound/blob/develop/Engine/envvar.c#L1011-L1019

But it's obviously Windows only. Would the devs mind implementing this with some if-def blocks?

vlazzarini · 2016-04-22T15:24:45Z

that could be done for some files, but I wonder if libsndfile would do this too? Can you try and see
if diskin opens a file with wide characters?

jpffitch · 2016-04-22T15:26:44Z

If arguents te sane or near same I would prefer a macro so easy to use all
over

On Fri, 22 Apr 2016, Rory Walsh wrote:

It seems the fopen() on windows at least will not return a valid FILE pointer
if non-ascii chars are in the file path. The solution is to use _wfopen()
which I think is Windows only. Using _wfopen() instead of fopen in the
following code fixes the issue.

https://github.com/csound/csound/blob/develop/Engine/envvar.c#L1011-L1019

But it's obviously Windows only. Would the devs mind implementing this with
some if-def blocks?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on
GitHub[AELNQh9NmclfZKPR5vWvCEboKl3wwcPjks5p6N17gaJpZM4IMpxd.gif]

rorywalsh · 2016-04-22T18:01:21Z

@vlazzarini : diskin2 fails with wide characters. In fact, it seems that any strings with wide chars chars on Windows will not be read properly. Include files that use non-ascii paths also return a not found error.

@jpffitch in those lines I quoted, fullName needs to be a wchar_t_, while params needs to be cast as a wchar_t_. Something like this does it(I say like this because I'm not at the machine I tested it on, but that's more or less what I did):

    if (env == NULL) {     
#if defined(WIN32)
      fullName = (char*) name;
      size_t sz = MultiByteToWideChar(CP_UTF8, 0, name, -1, NULL, 0);
      wchar_t *wfname = alloca(sz);
      MultiByteToWideChar(CP_UTF8, 0, name, -1, wfname, sz);

      sz = MultiByteToWideChar(CP_UTF8, 0, param, -1, NULL, 0);
      wchar_t *wmode = alloca(sz);
      MultiByteToWideChar(CP_UTF8, 0, param, -1, wmode, sz);

      if (type == CSFILE_STD) {
         tmp_f = _wfopen(wfname, wmode);
         if (tmp_f == NULL) {
           perror(Str("csound->FileOpen2 failed:"));
           goto err_return;
         }
      }
#else
      fullName = (char*) name;
      if (type == CSFILE_STD) {
        tmp_f = fopen(fullName, (char*) param);
        if (tmp_f == NULL) {
          perror(Str("csound->FileOpen2 failed:"));
          goto err_return;
        }
      }
#endif

vlazzarini · 2016-04-22T18:12:57Z

not sure if we call fopen or we let libsndfile do it for us. If it is the second case, then you'll need to ask Erik.

rorywalsh · 2016-04-22T18:16:15Z

Considering Csound can't handle non-ascii characters the diskin issue is not as big one for me. (although it may well confuse some users). Not being able to compile a file with the API that has non-ascii characters in its path concerns me more.

vlazzarini · 2016-04-22T19:44:11Z

Ok, I looked and we use libsndfile sf_open etc functions to open soundfile. So that's that. As for standard files, we use fopen, which could be replaced on windows for a more suitable function. It's in envvar.c . You could possibly try it and see if it does anything, then we could look at adding to the sources.

rorywalsh · 2016-04-22T19:51:24Z

The solution is the same as the one I posted a few hours ago, we should use _wfopen().

vlazzarini · 2016-04-22T20:07:27Z

That may be, but it would be good if you could try and let us know. We're linux-osx people here.

Victor Lazzarini
Dean of Arts, Celtic Studies, and Philosophy
Maynooth University
Ireland

On 22 Apr 2016, at 20:51, Rory Walsh notifications@github.com wrote:

The solution is the same as the one I posted a few hours ago, we should use _wfopen().

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

rorywalsh · 2016-04-22T20:12:39Z

I have tried it. I tested it on Windows just before I posted the solution. That's why I refer to it as a "solution" ;) But I'd still rather leave it to you guys to implement it the way you see fit.

vlazzarini · 2016-04-22T20:27:16Z

I see. But we need a Windows developer to implement it all, so we are not doing it blindly.

rorywalsh · 2016-04-24T08:05:12Z

I think this is sorted now. I've updated my original code snippet with working code. I've tested with simple command line API programs and I've also tested it with Cabbage. I left
fullName = (char*) name;
within the if-def block as fullname appears to be used later in that function.

jpffitch · 2016-04-24T14:43:24Z

When you say sorted does it need changes to csound sources?

rorywalsh · 2016-04-24T14:44:31Z

Yes, if you wouldn't mind. The code is posted above.

On 24 April 2016 at 15:43, John ffitch notifications@github.com wrote:

When you say sorted does it need changes to csound sources?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#632 (comment)

jpffitch · 2016-04-24T16:31:44Z

In git; did I get it right?
Please check

rorywalsh · 2016-04-24T18:14:47Z

There seems to be a stray bracket there. I've emailed you the fixed version.

On 24 April 2016 at 17:31, John ffitch notifications@github.com wrote:

In git; did I get it right?
Please check

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#632 (comment)

gogins · 2016-04-24T19:18:02Z

I also found and fixed this.

Regards,
Mike

Michael Gogins
Irreducible Productions
http://michaelgogins.tumblr.com
Michael dot Gogins at gmail dot com

On Sun, Apr 24, 2016 at 2:14 PM, Rory Walsh notifications@github.com
wrote:

There seems to be a stray bracket there. I've attached the fixed version.

On 24 April 2016 at 17:31, John ffitch notifications@github.com wrote:

In git; did I get it right?
Please check

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#632 (comment)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#632 (comment)

jpffitch · 2016-04-24T20:05:24Z

The code is a partial fix at best. There are other calls to fopen and open in envvar.c that are unmodified
so writing to files differs from reading. We need a comprehensible fix. Some sndfile opens are OK as they use the fd from open.

I still cannot see any extra brackets either

rorywalsh · 2016-05-13T08:02:52Z

I just ran into an issue with this latest change when using Microsoft's .NET framework. It seems that it doesn't appreciate the call to _wfopen() and produces a problem with msvcrt.dll. It's not easy to fix as there are no debug tools that can help. But reverting the code does make the issue go away. I'm not sure what's the best course of action here. The fix allows us to open files paths with non-English characters on Windows, but causes a problem for anyone using Csound with .NET. What would people think about a new compile method to deal with non-English paths? Would this be appropriate? Users could just call this instead of the normal csoundCompile() on Windows if they wanted it? Or has anyone any other suggestions?

jpffitch · 2016-10-10T12:59:33Z

is it possible to detect at runtime that .net is being used?

There are a number of calls to fopen in the sources. Which do you think should get the _wfopen treatment?

rorywalsh · 2016-10-10T13:52:49Z

I'm not sure. I wouldn't suggest any of them get the _wfopen() treatment if it means not being able to use Csound with .net. I don't know how best to proceed.

gogins · 2016-10-10T14:15:13Z

Provide a "_w" variant.

Regards,
Mike

On Oct 10, 2016 9:52 AM, "Rory Walsh" notifications@github.com wrote:

I'm not sure. I wouldn't suggest any of them get the _wfopen() treatment
if it means not being able to use Csound with .net. I don't know how best
to proceed.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#632 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGCDJUxsSpu1YxlhN23cQ5m_kgLSjaV6ks5qykMzgaJpZM4IMpxd
.

jpffitch · 2016-10-10T18:13:52Z

I thought the problem was in .net

⁣

Sent from TypeApp

On 10 Oct 2016, 16:35, at 16:35, Michael Gogins notifications@github.com wrote:

Provide a "_w" variant.

Regards,
Mike

On Oct 10, 2016 9:52 AM, "Rory Walsh" notifications@github.com wrote:

I'm not sure. I wouldn't suggest any of them get the _wfopen()
treatment
if it means not being able to use Csound with .net. I don't know how
best
to proceed.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#632 (comment),
or mute
the thread

https://github.com/notifications/unsubscribe-auth/AGCDJUxsSpu1YxlhN23cQ5m_kgLSjaV6ks5qykMzgaJpZM4IMpxd
.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#632 (comment)

rorywalsh · 2016-10-11T09:19:08Z

Here's a quick run down of the issue. Csound won't open files that have
non-English characters on Windows unless we use _wfopen(). This is now
added and working as far as I can tell. But after this change was made I
noticed that _wfopen() causes an issue when using DLLImport from C#. I'm
afraid I don't know enough about C# to be able to suggest a fix. But I can
confirm that removing the call to _wfopen() does fix the issue.

On 10 October 2016 at 19:13, John ffitch notifications@github.com wrote:

I thought the problem was in .net

⁣

Sent from TypeApp

On 10 Oct 2016, 16:35, at 16:35, Michael Gogins notifications@github.com
wrote:

Provide a "_w" variant.

Regards,
Mike

On Oct 10, 2016 9:52 AM, "Rory Walsh" notifications@github.com wrote:

I'm not sure. I wouldn't suggest any of them get the _wfopen()
treatment
if it means not being able to use Csound with .net. I don't know how
best
to proceed.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#632 (comment),
or mute
the thread

<https://github.com/notifications/unsubscribe-
auth/AGCDJUxsSpu1YxlhN23cQ5m_kgLSjaV6ks5qykMzgaJpZM4IMpxd>
.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#632 (comment)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#632 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACkLGfazcJdpCmVecQgPsYseh42H572Pks5qyoBigaJpZM4IMpxd
.

vlazzarini · 2016-10-20T20:08:06Z

I think this is a won't fix case. Can we have an #ifdef C# or something?

jpffitch · 2016-10-24T10:42:04Z

More a cannot fix as Windows does not provide a working fopen. For now we should just keep this problem in mind

rorywalsh · 2016-10-25T18:14:36Z

Can we not use some kind of if/def as Victor suggested?

On 24 October 2016 at 11:42, John ffitch notifications@github.com wrote:

More a cannot fix as Windows does not provide a working fopen. For now we
should just keep this problem in mind

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#632 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACkLGZWEycMP1CscKQOccZX0PvHJYw7Nks5q3It9gaJpZM4IMpxd
.

rorywalsh · 2016-10-25T18:20:59Z

This sounds like a reasonable solution.

On 20 October 2016 at 21:08, vlazzarini notifications@github.com wrote:

I think this is a won't fix case. Can we have an #ifdef C# or something?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#632 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACkLGUisHZxpjhQU2xRCxCiO3MRZQ7rDks5q18ongaJpZM4IMpxd
.

jpffitch added the windows label May 9, 2016

jpffitch added the enhancement label May 17, 2016

kunstmusik added this to the 6.08 milestone Aug 24, 2016

jpffitch added the wontfix label Oct 24, 2016

jpffitch closed this as completed Oct 24, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Csound::Compile() won't work if non-English characters in path.. #632

Csound::Compile() won't work if non-English characters in path.. #632

rorywalsh commented Apr 21, 2016

vlazzarini commented Apr 21, 2016

fsateler commented Apr 21, 2016

rorywalsh commented Apr 21, 2016

rorywalsh commented Apr 22, 2016 •

edited

Loading

fsateler commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

jpffitch commented Apr 22, 2016

rorywalsh commented Apr 22, 2016 •

edited

Loading

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 24, 2016

jpffitch commented Apr 24, 2016

rorywalsh commented Apr 24, 2016

jpffitch commented Apr 24, 2016

rorywalsh commented Apr 24, 2016 •

edited

Loading

gogins commented Apr 24, 2016

jpffitch commented Apr 24, 2016 •

edited

Loading

rorywalsh commented May 13, 2016

jpffitch commented Oct 10, 2016

rorywalsh commented Oct 10, 2016

gogins commented Oct 10, 2016

jpffitch commented Oct 10, 2016

rorywalsh commented Oct 11, 2016

vlazzarini commented Oct 20, 2016

jpffitch commented Oct 24, 2016

rorywalsh commented Oct 25, 2016

rorywalsh commented Oct 25, 2016

Csound::Compile() won't work if non-English characters in path.. #632

Csound::Compile() won't work if non-English characters in path.. #632

Comments

rorywalsh commented Apr 21, 2016

vlazzarini commented Apr 21, 2016

fsateler commented Apr 21, 2016

rorywalsh commented Apr 21, 2016

rorywalsh commented Apr 22, 2016 • edited Loading

fsateler commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

jpffitch commented Apr 22, 2016

rorywalsh commented Apr 22, 2016 • edited Loading

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 22, 2016

vlazzarini commented Apr 22, 2016

rorywalsh commented Apr 24, 2016

jpffitch commented Apr 24, 2016

rorywalsh commented Apr 24, 2016

jpffitch commented Apr 24, 2016

rorywalsh commented Apr 24, 2016 • edited Loading

gogins commented Apr 24, 2016

jpffitch commented Apr 24, 2016 • edited Loading

rorywalsh commented May 13, 2016

jpffitch commented Oct 10, 2016

rorywalsh commented Oct 10, 2016

gogins commented Oct 10, 2016

jpffitch commented Oct 10, 2016

rorywalsh commented Oct 11, 2016

vlazzarini commented Oct 20, 2016

jpffitch commented Oct 24, 2016

rorywalsh commented Oct 25, 2016

rorywalsh commented Oct 25, 2016

rorywalsh commented Apr 22, 2016 •

edited

Loading

rorywalsh commented Apr 22, 2016 •

edited

Loading

rorywalsh commented Apr 24, 2016 •

edited

Loading

jpffitch commented Apr 24, 2016 •

edited

Loading