Cygwin/ssh with opensc-pkcs11.dll: Segmentation fault

[pieterverberne]

Dear all,

Is this a know issue?

$ ssh -I /cygdrive/c/Windows/System32/opensc-pkcs11.dll host.com
Segmentation fault (core dumped)

Also:
$ ssh -I /cygdrive/c/Program\ Files/OpenSC\ Project/PKCS11-Spy/pkcs11-spy.dll host.com

*************** OpenSC PKCS#11 spy *****************
Loaded: "C:\WINDOWS\system32\opensc-pkcs11.dll"

0: C_GetFunctionList
2015-11-13 11:00:22.413
Returned: 0 CKR_OK
Segmentation fault (core dumped)

I'm running Microsoft Windows 7 Professional 64bit, OpenSC 0.15.0 (0.14.0 has the same problem).

Also opensc-pkcs11.dll is not working with Mozilla Firefox. On Linux Debian it is working.

My debugging skills are limited, but I found this on a strace:
--- Process 6148, exception c0000005 at 0000000100441DDE
99 6183880 [main] ssh 6836 build_env: envp 0x180330518, envc 66
43 6183923 [main] ssh 6836 child_info::child_info: subproc_ready 0x240
3180 6183946 [main] ssh 6148 exception::handle: In cygwin_except_handler exception 0xC0000005 at 0x100441DDE sp 0x229F80
46 6183992 [main] ssh 6148 exception::handle: In cygwin_except_handler signal 11 at 0x100441DDE
46 6184038 [main] ssh 6148 _cygtls::inside_kernel: pc 0x100441DDE, h 0x100400000, inside_kernel 0

0xC0000005 is also know as STATUS_ACCESS_VIOLATION.

Gr,
Pieter Verberne

[imasonaz]

Possibly related to missing libraries mentioned in #585 and #586

[frankmorgner]

@pieterverberne where did you get the binaries from?

[pieterverberne]

opensc-0.15.0-win64.msi from
http://sourceforge.net/projects/opensc/files/OpeSC/opensc-0.15.0/opensc-0.15.0-win64.msi/download

I also tried some nightly builds from
http://sourceforge.net/projects/opensc/files/OpenSC/nightly/win64/

When I run depends on opensc-pkcs11.dll (OpenSC 0.15.0 release) I get these results:

I'm not sure what to do with this.

[dengert]

Expand the depends window more, to show all the Warning: messages in bottom frame.
Look for the non windows DLL in the upper left frame.
In right frames look for any other red messages.

[cruzzer]

I'm seeing the same behavior. I've tried x86 and x64 version of 0.14 and 0.15 OpenSC for OpenSSH, but get a segmentation fault. Firefox did although work for me, after I installed x86 version.

What does work:
opensc-tool -l // I can see my smart card
pkcs15-tool --dump // I can see my smart card objects
firefox // I can add pkcs11 module and log-in with my pin, haven't tried with website
(update1:) openvpn works
(update3:) putty_sc works too

OpenSSH wants x64 version and will fail with segmentation fault.
Dependency walker error window is:
Error: At least one required implicit or forwarded dependency was not found.
Warning: At least one delay-load dependency module was not found.

My missing libraries are(I'm on win8.1):
API-MS-WIN-CORE-KERNEL32-PRIVATE-L1-1-1.DLL
API-MS-WIN-CORE-PRIVATEPROFILE-L1-1-1.DLL
API-MS-WIN-SERVICE-PRIVATE-L1-1-1.DLL

My openSSH version is 7.1p1
(update2:) same behavior with openssh 6.9p1

Maybe this is an openssh issue. Does anyone have this working with windows?
(update1&2:) I can confirm that OpenSC works with OpenVPN, putty_sc, which makes openSSH an increasingly likely candidate.

[frankmorgner]

@vletoux do you have any idea?

[vletoux]

there is a lot of trick leading to dependencies not found. This message on itself is not a problem but it has to be correlated to first level dependencies.
On my opinion, there is no problem related to dependencies.

The best would be to collect a crashdump and know on which module (dll) the crash occured.
See https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181%28v=vs.85%29.aspx to generate one.
The call stack (module) can be obtained from windbg or visual studio.
Optionaly, if pdb files are generated by the build (not produced by Appveyor), you can even get the line number on which the crash occured.

[pieterverberne]

I played a bit with Visual Studio and I think I got want you want. Please mail me (pieterverberne at xs4all nl). I will send the dump by mail. (To prevent leaking possible semi interesting data, I will not post it here. I'm just not 100% sure :-) )

[vletoux]

Definitely a ssh problem and not an OpenSC one:

Exception Code: 0xC0000005
Exception Information: The thread tried to read from or write to a virtual address for which it does not have the appropriate access.
And the opensc-pkcs11.dll is loaded in memory.
Thread address is (-1) (0xFFF...FFF)

[frankmorgner]

If pkcs#11 modules from other vendors are working fine, it still could be that OpenSC returns some function pointer or memory which are to thought to be valid by ssh but which are actually bogus.

Am 15. Dezember 2015 09:35:56 MEZ, schrieb vletoux notifications@github.com:

Definitely a ssh problem and not an OpenSC one:

Exception Code: 0xC0000005
Exception Information: The thread tried to read from or write to a
virtual address for which it does not have the appropriate access.
And the opensc-pkcs11.dll is loaded in memory.

---

Reply to this email directly or view it on GitHub:
#607 (comment)

Frank Morgner

[vletoux]

"Unhandled exception at 0x0000000100441DDE (ssh.exe) in ssh.dmp: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF."
=> the memory address is -1, something not random but deliberately set by a program

The best would be to have the pdb associated to ssh.exe (compile it with visual studio) and find where the error is triggered.

[vletoux]

Indeed ssh is calling a derefenced pointer in opensc address space (see image)

I cannot do more with only a dump file.

[frankmorgner]

that's quite a lot already, thanks!

[cruzzer]

As a small update: This behavior seams to have not changed with the newest OpenSSH_7.1p2.

[vletoux]

the best would be to collect the .pdb file of OpenSSH which can be used for crashdump debugging.
Needs to generate ones ...

[cruzzer]

So I've contacted the Cygwin team and this is their response:

Windows and Cygwin DLL don't mix in the manner you expect them to.
You'd need to compile OpenSC for Cygwin.

I guess that sums up what the next step is. Anyone have experience compiling for Cygwin?
(update1:) I've been able to compile opensc for cygwin. When I run "opensc-tool -i" get

OpenSC 0.15.0 [gcc 4.9.3]
Enabled features: zlib openssl pcsc(winscard.dll)

This installed opensc to /usr/bin/* and /usr/lib/ . Sadly when I run "ssh -I /usr/lib/opensc-pkcs11.dll host" I still get a access_violation.

[cruzzer]

Microsoft interestingly decided to start their own openssh port on Github. This was already planned and anounced here. They seam to be making progress.

I grabbed their current build and tried it with opensc. It's a bit buggy, but pkcs11 works with their powershell :) Cygwin/bash does not quite work yet, but that bug has also been reported with powershell, so I'm hopefull this will be resolved for both.

Since I've compiled opensc for cygwin and used it with cygwin/openssh to no avail, my guess is that openssh attempts to load opensc in some way Windows doesn't like. It's just as likely that there are some compile flags for cygwin/opensc that I didn't apply. I'll give the cygwin team an update and see what their thoughts are.

[ghost]

Any updates on this? Just had the same issue with the OpenSSH bundled with Git Bash. I'm ready to do some tests if anyone needs log files or crash dumps.

[cruzzer]

Small update:
I've been revisiting this issue from time to time. The best short term workaround I've found, is mentioned in this article. It is some what old, but using charade with pagent still works.

Long term it would be great if Microsoft re-implements pkcs11 support in their win32 version of OpenSSH, alternatively there is a functioning cygwin opensc package published.

[dwhobrey]

One reason for the segment fault is that the opensc and ssh-agent utils were built with different struct packing alignments, which means when the ssh-agent requests the list of module function pointers, they get returned misaligned. One way to avoid this is to build opensc under cygwin as a Unix app rather than a win app. To do that uncomment the line in opensc's configure.ac, around line 107, that starts "CPPFLAGS... CRYPTO ... FORCE WIN". Run bootstrap and configure, make, make install, with no args, I.e. don't specify host as win or cygwin. If you are using readline, you'll need to specify --disable-strict with ./configure. Remember to first install pkg-config, openssl-devel, zlib, autoconf, automake, libtool, make, gcc-g++, gcc-core etc. That should also install the cygwin ssh utils, which you'll need to use for this to work.

[dengert]

configure.ac line 107 in current master is not a comment. It is in a block that that lets you specify with_cygwin_native

So is the fix already in place?

If not what effect would your change have on other programs that also use opensc-pkcs11.dll?
Should change be to openssh?

[dengert]

Or should the user be using the cygopensc-pkcs11.dll which could also be built if with _cygwin_native is not used.

[dwhobrey]

Digging into it, there are two aspects to consider / resolve:

1. A pure Cygwin approach expects the pkcs11 dynamic lib to have a Unix format.
2. The pkcs11 and ssh utils, such as ssh-agent, need to use the same struct packing when doing IPCs.
   This gets compounded because folks can often have a mixture of ssh utils & OpenSC built either for Windows, or Wnd on Cygwin, and then 32-bit or 64-bit depending on where they got the binaries from and what compiler was used - because it may have used different struct alignment settings. As seen above, when these utils try to talk to each other, problems can arise.

So, to build OpenSC for a pure Cygwin environment, that uses Cygwin's ssh utils out of the box, OpenSC needs to be built according to the Unix flavour build steps. However, there are a couple of points in the current configure.ac build script and pkcs11.h header that complicate this:
A) The pkcs11.h doesn't explicitly set struct alignment, except for a global pragma pack for windows builds. This is a very non-portable situation. In particular, the ck_function_list struct starts with a ck_version struct which is a couple of byte variables. When packing is set to 1, the subsequent function addresses get aligned to something line 2, but for non window builds this is 4, or even 8 on 64-bit. Really struct level alignment attributes should be specified. Without that, we'll see the seg fault issue.
B) The configure.ac script, for the Cygwin build assumes the user wants windows on Cygwin, so it defines -DCRYPTOKI_FORCE_WIN32 on line 107:
CPPFLAGS="${CPPFLAGS} -DCRYPTOKI_FORCE_WIN32"
That's not needed for a pure Cygwin build and is bad because it enables packing to 1 in pkcs11.h.
So commenting out that line makes the build more like the Solaris case above that and makes packing follow the default rules, i.e. aligning in accord with the largest required alignment value for the struct members.
Basically it would be good if the OpenSC maintainers could look into resolving the packing issue, such as adding explicit padding to the version struct so it's always aligned properly, or adding more logic to determining when CRYPTOKI_FORCE_WIN32 is used.

Attached are the instructions for the steps required to get OpenSC working under Cygwin.
InstallingOpenScOnCygwin.txt

[dwhobrey]

Ps: in the install instructions mentioned above, the steps for installing sshpass can be ignored if you don't use it. I have a bunch of scripts that do.

[dengert]

Thanks for the nice explanation.

Note that pkcs11-v2.40/ pkcs11.h says: "The Cryptoki convention on packing is that structures should be 1-byte aligned."

But then goes onto say: "In a UNIX environment, you're on your own for this. You might not need to do (or be able to do!) anything."

And CK_VERSION is in: pkcs11-v2.40/pkcs11t.h

Which as I understand your explanation, treat Cygwin as Unix and leave it up to the compiler?

Since I don't use Cygwin, maybe some other developer who does will look at this.

You are welcome to submit a PR.

[dwhobrey]

I've emailed the Cygwin team asking what they advise - I'll post an update here. Seems like the configure.ac should check whether it's using gcc or MingW, or similar. These aspects are beyond my knowledge :)

[dengert]

Looking at the OpenSSH_7.1p2 as cited above, the pkcs11.h is based on an earlier OpenSC version and has the same #if defined(_WIN32) || defined(CRYPTOKI_FORCE_WIN32) lines.

So OpenSSH must not be using either _WIN32 or CRYPTOKI_FORCE_WIN32 but leaving packing and alignment up to the compiler.

Are there any cygwin packages that use pkcs11 that do alignment differently?

[dengert]

@alonbl can you shed some light on this issue.

From an e-mail Re: [opensc-devel] PKCS#11 header license - replacement headers" 12/8/2006, Alon Bar-Lev to opensc-devel@lists.opensc-projects.org, Alon said:

"2. Added CRYPTOKI_FORCE_WIN32, so that you can force the file
to generate WIN32 fixups. This is important for cygwin environment." and was committed to SVN on 2006-12-9 at that time, and is in OpenSC as commit 44d54b6.

OpenSSH picked up the opensc version of pkcs11.h in the OpenSSH commit d8f60022 on 2010-2-12.

OpenSC configure.ac containing the -D-DCRYPTOKI_FORCE_WIN32 was last modified by commit 0a809dd on 2008-03-06 by @alonbl

[dwhobrey]

I've just done a quick search of the openssh source that Cygwin uses, while it has the v1.3 pkcs11.h from 2013, with the _win32 and cryptoki_force_win32 packing conditionals, none of the configure files reference these variables. So it looks like the binaries that Cygwin ship are built without these options enabled - given they probably auto build.

[alonbl]

As far as I remember, at that time there was no stable cross compiler for windows on windows. the mingw project was on halt and the options were to build native windows binaries were the mingw or cygwin. This flag was added to build native windows binary on cygwin compiler which at that time was better maintained.

Today we have the mingw-w32 project which is maintained and can be used for this purpose.

I believe that the opensc maintainers decided to build windows binaries using microsoft compiler and not use cross compile and automation on linux. So the above is irrelevant anyway.

Unless application is designed to load native PKCS#11 providers on cygwin, it is expected to build cygwin binaries and use the shared library as the PKCS#11 provider and not the windows native dll.

Looking at the openssh code, it seems like it uses dlopen/dlsym to load the library so it must use the cygwin build of opensc. It could have checked for cygwin environment and perform LoadLibrary/GetProcAddress, but then it was limited for only native providers.

It had been long time since I last checked the cygwin wrapping, especially the 64bit, maybe something has changed.

[dwhobrey]

Hi. Hans-Bernard from Cygwin has responded to the advice request in the cygwin at cygwin dot com mailing list - search on "advice cygwin opensc". There he recommends removing the native section entirely as it has not been applicable for a number of years.

[alonbl]

On Thu, Sep 6, 2018 at 8:53 AM Darren ***@***.***> wrote: Hi. Hans-Bernard from Cygwin has responded to the advice request in the cygwin at cygwin dot com mailing list - search on "advice cygwin opensc". There he recommends removing the native section entirely as it has not been applicable for a number of years.

Correct, as I wrote, the mingw-w64 project is providing a great service for the last years.

[frankmorgner]

@dwhobrey thanks for shining light into this issue. Feel free to add InstallingOpenScOnCygwin.txt to the OpenSC wiki and to make a pull request for the obsolete code in configure.ac.

From a cygwin user's perspective it would be great to have a OpenSC package within standard package management. However, I'm not sure if it is really needed with Windows 10 integrating OpenSSH and some PKCS#11 variants of PuTTY available...

[dwhobrey]

Hi Frank, we have a bunch of bash scripts that we normally use on Linux platforms. Being able to run them "out of the box" on Windows via Cygwin, is very convenient. Using PuTTY-CAC for example, requires a very different approach.

[dwhobrey]

Updated Wiki with instructions - added page "Compiling on Cygwin".
Tried to push the change to configure.ac on a branch in order to do a pull request but getting a 403.