Bug in OpenSSL : Timestamping error

[cbgorman]

When attempting to timestamp, we have gotten the error below. Can anybody help us out?

System command failed: 139639965218464:error:2F067065:time stamp routines:TS_CHECK_SIGNING_CERTS:ess signing certificate error:ts_rsp_verify.c:291:, Verification: FAILED”

[NicolasCARPi]

Hello,

Please provide more informations:

• what operating system ?
• what version of elabftw ?
• what version of php/openssl
• do you use the default TSA ?

Can you provide us with a timestamped pdf and a token so we can try to reproduce ?

[Athemis]

Hi Nicolas,
I'm getting a similar error after updating from 1.1.8-p2 to 1.2.0-alpha using the default (DFN) TSA. I can confirm that it was working fine before the update.

System command failed: 1996412112:error:2F067065:time stamp
routines:TS_CHECK_SIGNING_CERTS:ess signing certificate
error:ts_rsp_verify.c:311:, Verification: FAILED

OS: Archlinux x86_64, Kernel 4.5.1
OpenSSL: 1.0.2g
PHP: 7.0.5

Best regards,

Alex

[NicolasCARPi]

Yep, I can confirm too, weird, because it was working fine before, and I don't think I changed anything.

Can you confirm it works in 1.1.8-p2 ? Because if I do a git checkout 1.1.8-p2, it still doesn't work. Or even 1.1.8. And it was working in alpha, because I tested it a few days ago.

Could it be a problem on DFN side ?

[Athemis]

Yep you're right. Downgrading doesn't help, even a fresh install of anything down to 1.1.7 doesn't help.

Could it be a problem on DFN side ?

I'll check if they've changed anything regarding their certificate chain.

[NicolasCARPi]

Yeah thanks, it'll be easier for you (language-wise).

[cbgorman]

Responding to the request for more information:

Installed version: 1.1.8-p2
on a digital ocean droplet running Ubuntu 14.04 x64
PHP 5.5.9-1ubuntu4.11
openSSL 1.0.1f

I didn't customize -- followed the directions at doc/_build/html/install-drop.html

[Athemis]

Just a quick update: The certificate chain is unchanged (aka "the good news") and I can reproduce the faulty behavior with OpenSSL on on the command line level.

The bad news is that we're probably hitting an OpenSSL issue first noticed back in 2013:

• http://openssl.6102.n7.nabble.com/possible-Bug-in-OpenSSL-rfc-3161-TSA-service-tt43128.html#none

The summary is that OpenSSL can't digest TS replies well if attribute certificates are added to them. This scenario (although conformant to the rfc3161) is not supported by OpenSSL. My best guess is that the guys at DFN changed the way their TS replies are formed (software update, etc.). I'll try to get in contact with them to figure out if they are able (and willing) to fix their service for OpenSSL users again. There's absolutely nothing we can do in the meantime (aka "the really bad news").

The consequences are that any timestamps generated before DFN broke for us will still validate because the stored binary responses are ok for OpenSSL, but you will not be able to generate new ones.

I'll report back as soon as I get a response from the DFN.

Cheers,
Alex

[NicolasCARPi]

Yep, thanks for the update Athemis, I saw the page you linked before.

[Athemis]

I got an answer from the DFN. They are aware of the issue. They intentionally included an Attribute Signing Certificate (RFC5035 and RFC5816) to improve security by providing a mechanism to identify the signing certificate inside the timestamping response which means that my initial guess concerning the cause of our problems was correct. They now thoroughly use SHA-2 during the timestamping process which is currently unsupported by openssl. This change happened on April 14th, any timestamps generated before that date will validate without problems. My contact at DFN pointed out that there is a patch under discussion at the openssl github page: openssl/openssl#771

A patched openssl should work again without any problems in combination with the DFN service. Until the patch is mainlined you have to patch openssl yourself. Furthermore they are developing a small java program to verify RFC3161 timestamps and will publish it when it's ready. The release will be announced here: https://blog.pki.dfn.de

Our options are:

1. Wait for the patch beeing mainlined in openssl
2. Patch openssl yourself if using DFN
3. Wait for the DFN to publish their verification tool (which will be usable for all RFC3161 timestamps not only those from the DFN) and provide a backend for their tool as an alternative/in addition to openssl

Cheers,
Alex

EDIT:

To be more precise what the DFN changes mean: They are enforcing at least SHA256 hashes throughout the whole timestamping prodecure including the fingerprints of the signing certificate. In OpenSSL this is hardcoded to SHA1 and that's why it fails.

[NicolasCARPi]

Ok thanks Athemis for this information. I think we'll go with option 1. And maybe I can disable the verification step in timestamping until openssl is patched. And show a message referencing this issue.

[Athemis]

I would opt for keeping the verification functional. There is no point in timestamping if the timestamps are not verified. This issue is not present for all TSAs but only for those who add attribute certificates. We can catch the error message from openssl and display a more meaningful error instead, pointing out that the timestamp cannot be validated because of shortcomings in openssl and suggesting the user to install a patched openssl version or (when mainlined) urging to update to a recent openssl version. Imho that's more transparent then just pretending everything is fine although no verification is done at all.

[NicolasCARPi]

I agree the problem might not arise for another TSA, but I highly doubt people are using something else than the default…

Now I agree that timestamping without actually verifying is useless, so I'll do what you suggest, catch this specific error and display a message :)

[Athemis]

The DFN published their java tool for timestamp verification and also put up a site explaining in detail what changed and why openssl is currently broken in this context.

[NicolasCARPi]

Yep, I was able to verify an old token with their tool. So the question is : do we introduce this tool as a failsafe for validating the token ? This would allow to continue timestamping until the bug in openssl is fixed, what do you think ?

[Athemis]

Taking into account that openssl is not known for merging pull requests very fast (especially the timestamping subsystem doesn't seem to get much love from the devs) this sounds to me like a good idea. We just need to make sure to mention somewhere in the documentation that if the verification fails and one wants to use this fallback mechanism, a functional java installation is required.

[NicolasCARPi]

Yep, I'll do that.

[NicolasCARPi]

So for now, the java validator works on my machine. Didn't work in the docker image for some reason...

I was thinking to include the patched version of openssl with elab. I guess it should work. I'll see about that :)

[Athemis]

Works as well on my machine. Well done! 😄
Distributing a pre-compiled openssl version might prove cumbersome: You'd have to provide statically linked binaries for at least x86/x86_64 and probably ARM (with its numerous variants). Imho it's probably not worth the trouble ;-)

[NicolasCARPi]

As openssl is installed on most machines, and as it is already a requirement for elabftw, I don't need to build a static binary, because libs should be there! Also, I'm pretty sure 99.5% of people are on x86_64 arch. I'll still try, and maybe give up after failing to compile openssl :p

[NicolasCARPi]

I don't manage to validate with the patched version anyway...

Verification: FAILED
140165594154648:error:2F064067:time stamp routines:ts_check_imprints:message imprint mismatch:ts_rsp_verify.c:675:

[NicolasCARPi]

Ok so now the validation of timestamps works in docker also. \o/

I'll soon release 1.2.0 so that users can continue to timestamp their experiments.

[MarioScalabrino]

Hello, I've been struggling for some time with this problem. Could you please tell me how did you solve it? Applying a patch? Using DFN Java verification tool? I talking about:

---

System command failed: 139639965218464:error:2F067065:time stamp routines:TS_CHECK_SIGNING_CERTS:ess signing certificate error:ts_rsp_verify.c:291:, Verification: FAILED”

---

Could you help me I'm not expert enough to apply a patch myself or install the java DFN Java verification without any hint.

[NicolasCARPi]

Hello,

Please try the latest BETA version, it works with Java to validate the timestamps. You can git checkout -b next and git pull origin next.

[MarioScalabrino]

You're talking about patching Openssl right? not using the DFN Java client. I'm not sure. I'm in ubuntu 14.04, I cloned git $ git clone git://git.openssl.org/openssl.git.
Then I tried the two commands but:
certifydocserver@certifydocservervb:~$ git version git version 1.9.1 certifydocserver@certifydocservervb:~$ git pull origin next fatal: Not a git repository (or any of the parent directories): .git certifydocserver@certifydocservervb:~$ git checkout -b next fatal: Not a git repository (or any of the parent directories): .git

My Openssl version:
certifydocserver@certifydocservervb:~$ openssl version OpenSSL 1.0.1f 6 Jan 2014

[NicolasCARPi]

No, I'm talking about using the beta version of elabftw, which uses the Java client to validate timestamps.
https://github.com/elabftw/elabftw/releases/tag/1.2.0-beta
Also, the git commands I gave need to be performed inside the elabftw folder.

[MarioScalabrino]

Would you be so kind to tell me which are the steps I should perform. I really don't understand what should I do.

This is my best guess, please correct me:

1. Download your source code in my Ubuntu 14.04 VM
2. Somehow build it there (I've never done it, I'll study how to do it)
3. Then I suppose I have compiled a service that is in the path for Ubuntu, which I can call it via terminal or PHP.
4. I use my PHPcode to perform the timestamp verification with the TSA certificate with this code (as I do calling Openssl now). Which is the command I should use? Is there a documentation?

$responsefile = self::createTempFile($binary_response_string); $cmd = "openssl ts -verify -digest ".escapeshellarg($hash)." -in ".escapeshellarg($responsefile)." -CAfile ".escapeshellarg($tsa_cert_file); $retarray = array(); exec($cmd." 2>&1", $retarray, $retcode); if(unlink($responsefile)) { If ($debugMN) {echo " File Deleted Tempfile in validate"; } }

[typepub]

Hello,

The only difference I see is the call to « bash » to launch the script. Have you tried to just invoque ./verify.sh with same parameters ? Does-it change anything ?

As Java can’t find the file, I guess everything is OK in the script but maybe a privilege is missing somewhere ?

Le 4 août 2016 à 17:36, Msca notifications@github.com a écrit :

I'm in trouble, I've been fighting for three days now.
In my Ubuntu 14.04 pre-production environment everything works smoothly.

I migrated now the code in the Production environment, which should be exaclty the same. Unfortunately the bash call to verify.sh file doesn't work here, I cannot understand why.

Would you help me please?

Here's the original verify.sh modified to accept programmatically a certificate as the parameter $3.

#!/bin/sh
#Runs the TimeStampVerifier
#modified for use in eLabFTW
request=$1
response=$2
chain=$3
#chain=chain.txt
#root=rootcert.crt
#crl=crl.txt
class=de.dfncert.timestampverifier.TimeStampVerifier
libs=libs/bcpkix-jdk15on-152.jar:libs/bcprov-jdk15on-152.jar
java -cp $libs:. $class "$request" "$response" "$chain"

Terminal:
ubuntu@certifydoc04:/tmp/timestampverifier$ bash ./verify.sh /tmp/tsq.tsq /tmp/tsr.tsr /tmp/ChainZeitstempel.pem
Error: Could not find or load main class de.dfncert.timestampverifier.TimeStampVerifier

I thought it was a Java problem with $JAVA_HOME or $CLASSPATH but I couldn't solve.

That's my verify.sh now

#!/bin/sh
#Runs the TimeStampVerifier
#modified for use in eLabFTW
request=$1
response=$2
chain=$3
#chain=chain.txt
#root=rootcert.crt
#crl=crl.txt
class=de.dfncert.timestampverifier.TimeStampVerifier
libs=libs/bcpkix-jdk15on-152.jar:libs/bcprov-jdk15on-152.jar
#java -cp $libs:. $de.dfncert.timestampverifier.TimeStampVerifier "$request" "$response" "$chain"
#java -cp $libs:. $class "$request" "$response" "$chain" "$root" "$crl"
#java -cp $libs:. $class "$request" "$response" "$chain"

java -cp libs/bcpkix-jdk15on-152.jar:libs/bcprov-jdk15on-152.jar:. de.dfncert.timestampverifier.TimeStampVerifier "$request" "$response" "$chain"

Doesn't work if I use the terminal command.
Terminal:
ubuntu@certifydoc04:/tmp/timestampverifier$ bash ./verify.sh /tmp/tsq.tsq /tmp/tsr.tsr /tmp/ChainZeitstempel.pem
(No such file or directory)ava.io.FileNotFoundException: /tmp/tsq.tsq at
java.io.FileInputStream.open0(Native Method) at
java.io.FileInputStream.open(FileInputStream.java:195) at
java.io.FileInputStream.(FileInputStream.java:138) at
java.io.FileInputStream.(FileInputStream.java:93) at
de.dfncert.timestampverifier.TimeStampVerifier.main(TimeStampVerifier.java:66)

But, it works calling it directly withoput the bash file

Terminal:
ubuntu@certifydoc04:/tmp/timestampverifier$ java -cp libs/bcpkix-jdk15on-152.jar:libs/bcprov-jdk15on-152.jar:. de.dfncert.timestampverifier.TimeStampVerifier /tmp/tsq.tsq /tmp/tsr.tsr /tmp/ChainZeitstempel.pem
Data in response matches data in request.
Signer: C=DE,ST=Berlin,L=Berlin,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,OU=DFN-PKI,CN=PN: Zeitstempel
Time stamp token validated.
Certificate Chain not validated.

How can this be? I cannot go further than this at the moment.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #242 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ATv0HU8-CihcHi_IZd2gzCJbuAB7xVJzks5qcgbogaJpZM4IJ8pL.

[NicolasCARPi]

Yeah, I didn't spot something obvious either...

BTW, to get rid of «it worked in preprod, but not in prod», use Docker, it rocks ;)

[MarioScalabrino]

@typepub, this is what happens if i remove the "bash" or "sh":

ubuntu@certifydoc04:/tmp/timestampverifier$ ./verify.sh /tmp/tsq.tsq /tmp/tsr.tsr /tmp/ChainZeitstempel.pem
bash: ./verify.sh: /bin/sh^M: bad interpreter: No such file or directory

Maybe the bash has a problem?

@nicolas, I'm using a VM image in the Openstack infrastructure, same image to start but then they cannot be synchronized and overtime they diverge, also because they run Wordpress and its MySql database that is hoghly dependent on the main website URL.
There are two instances online with public IP and SSL certificate in two different Openstack clouds (preprod and prod).
Would docker help in that case? My problem is the SSL certificate and the Public IP that have to be different between the preprod and the prod (I already mentioned the dependency on the main website URL on wordpress and its plugins).

[NicolasCARPi]

@MarioScalabrino remove windows line ending (^M) !!!

[MarioScalabrino]

Buff, that was it, solved. Almost 4 days for that silly thing.
Thank you both for helping

[NicolasCARPi]

ahah I knew it ! :D
What text editor do you use ? Maybe configure it to convert line endings to the UNIX one, not the windows one which tend to mess things up ;)
And while you're at it, configure it to show whitespaces at end of lines in red. It always help.

[MarioScalabrino]

Great suggestion, I will take it very seriously now, sooo much time for such a minor thing.

I'm using Sublime text build 3103 and the Netbeans IDE 8.1 patch 1.

Do you know how to do that things you suggest in Sublime text and Netbeans for Ubuntu 14.04?

[NicolasCARPi]

You can use this one : https://github.com/SublimeText/TrailingSpaces
For line ending idk.

[MarioScalabrino]

Great thanks, tomorrow I'll install them.

[gebauer]

Hi,

not all of use have "real" server, so for those who host elabFTW on a standard shared hoster, but still want to somehow time stamp, these service works:
http://tsa.safecreative.org.

It is only freely available for 5 ts per day, but if you have a bigger group, you probably should invest in a real server anyway.
To get it to work you have to download their certificate and put if (via FTP) in your /vendor/ path. Than you should reference it under "SysAdmin > TimeStamp > chain of ...".
The URL is just http://tsa.safecreative.org.

Regards,
Jan

PS: I am not affiliated in any way with above mentioned service.

[NicolasCARPi]

Why wouldn't it work with the default TSA on your shared server?

Thanks for the tip anyway @gebauer :) I added it to the documentation.

BTW, you should use https://tsa.safecreative.org instead of the http address!

[gebauer]

Why wouldn't it work with the default TSA on your shared server?

Because of the openSSL bug?
Java is normally not installed on shared hosts...

[NicolasCARPi]

Just as a follow up, the patch has been merged to master and I could verify a timestamp with openssl from master. Hopefully the next release will contain this patch \o/

[lbonnefond]

Hy,

I got the again the error : "Could not validate the timestamp due to a bug in OpenSSL library. See issue #242. Tried to validate with failsafe method but Java is not installed."

I installed a new instance on a private shared host (Dreamhost). I checked if java was installed, as it seems to be the issue, and the command >java -version gives me this :

java version "1.7.0_151"
OpenJDK Runtime Environment (IcedTea 2.6.11) (7u151-2.6.11-0ubuntu1.14.04.1)
OpenJDK 64-Bit Server VM (build 24.151-b01, mixed mode)

I could go for the safecreative tsa certificate configuration, but I need more than 5 validation per day. (The labnote is for 20 students working at the same time over university wifi network with the same IP...).

Sorry for the noob question.

[NicolasCARPi]

Hello,

What happens if you just run java (because the code is checking for the presence of the word "class" after this command)?

[lbonnefond]

I guessed that the isJavaInstalled() function was doing something like that.

When I run the command java in the shell on my shared host I get this:

Usage: java [-options] class [args...]
           (to execute a class)
   or  java [-options] -jar jarfile [args...]
           (to execute a jar file)
where options include:
    -d32	  use a 32-bit data model if available
    -d64	  use a 64-bit data model if available
    -server	  to select the "server" VM
    -zero	  to select the "zero" VM
    -jamvm	  to select the "jamvm" VM
    -avian	  to select the "avian" VM
    -dcevm	  to select the "dcevm" VM
                  The default VM is server,
                  because you are running on a server-class machine.


    -cp <class search path of directories and zip/jar files>
    -classpath <class search path of directories and zip/jar files>
                  A : separated list of directories, JAR archives,
                  and ZIP archives to search for class files.
    -D<name>=<value>
                  set a system property
    -verbose:[class|gc|jni]
                  enable verbose output
    -version      print product version and exit
    -version:<value>
                  require the specified version to run
    -showversion  print product version and continue
    -jre-restrict-search | -no-jre-restrict-search
                  include/exclude user private JREs in the version search
    -? -help      print this help message
    -X            print help on non-standard options
    -ea[:<packagename>...|:<classname>]
    -enableassertions[:<packagename>...|:<classname>]
                  enable assertions with specified granularity
    -da[:<packagename>...|:<classname>]
    -disableassertions[:<packagename>...|:<classname>]
                  disable assertions with specified granularity
    -esa | -enablesystemassertions
                  enable system assertions
    -dsa | -disablesystemassertions
                  disable system assertions
    -agentlib:<libname>[=<options>]
                  load native agent library <libname>, e.g. -agentlib:hprof
                  see also, -agentlib:jdwp=help and -agentlib:hprof=help
    -agentpath:<pathname>[=<options>]
                  load native agent library by full pathname
    -javaagent:<jarpath>[=<options>]
                  load Java programming language agent, see java.lang.instrument
    -splash:<imagepath>
                  show splash screen with specified image
See http://www.oracle.com/technetwork/java/javase/documentation/index.html for more details.

[NicolasCARPi]

Ok weird. I don't get why java is not correctly detected…

What is which java? (thinking about a path issue)

[lbonnefond]

/usr/bin/java

Yes, I don't understand.

[NicolasCARPi]

Try this:

1/ Comment out these lines. So we disable the check (we know it's installed anyway).

2/ (optional) Edit this line and replace java with /usr/bin/java.

And try to timestamp.

[lbonnefond]

I get an undefined error message after modification 1 and nothing more with modification 2 ...

[NicolasCARPi]

Can you now check in the logs (either on sysadmin panel or the Apache/Nginx logs)?

[lbonnefond]

LOGS on sysadmin panel only show the errors when the IsJavaIntalled() function is active

✪ 2018-01-31 04:06:29 [ Error ] Could not validate the timestamp due to a bug in OpenSSL library. See <a href='https://github.com/elabftw/elabftw/issues/242#issuecomment-212382182'>issue #242</a>. Tried to validate with failsafe method but Java is not installed. (1)
✪ 2018-01-31 02:34:48 [ Error ] Could not validate the timestamp due to a bug in OpenSSL library. See <a href='https://github.com/elabftw/elabftw/issues/242#issuecomment-212382182'>issue #242</a>. Tried to validate with failsafe method but Java is not installed. (1)
✪ 2018-01-31 02:32:00 [ Error ] Could not validate the timestamp due to a bug in OpenSSL library. See <a href='https://github.com/elabftw/elabftw/issues/242#issuecomment-212382182'>issue #242</a>. Tried to validate with failsafe method but Java is not installed. (1)
✪ 2018-01-31 00:28:11 [ Error ] Could not validate the timestamp due to a bug in OpenSSL library. See <a href='https://github.com/elabftw/elabftw/issues/242#issuecomment-212382182'>issue #242</a>. Tried to validate with failsafe method but Java is not installed. (1)
✪ 2018-01-30 01:27:05 [ Error ] Could not validate the timestamp due to a bug in OpenSSL library. See <a href='https://github.com/elabftw/elabftw/issues/242#issuecomment-212382182'>issue #242</a>. Tried to validate with failsafe method but Java is not installed. (1)
✪ 2018-01-30 00:33:05 [ Error ] Could not validate the timestamp due to a bug in OpenSSL library. See <a href='https://github.com/elabftw/elabftw/issues/242#issuecomment-212382182'>issue #242</a>. Tried to validate with failsafe method but Java is not installed. (1)
✪ 2018-01-30 00:12:50 [ Error ] Could not validate the timestamp due to a bug in OpenSSL library. See <a href='https://github.com/elabftw/elabftw/issues/242#issuecomment-212382182'>issue #242</a>. Tried to validate with failsafe method but Java is not installed. (1)

I haven't found the Apache/nginx logs yet...

[NicolasCARPi]

On a shared host you might need to go your control panel to acces them. Or maybe you have a "log" folder somewhere at the root dir ? Or maybe you just can't access them.

Also, as a curiosity, how much do you pay for your shared hosting? 10$/month for 1Go RAM?

Because with DigitalOcean you get twice as much RAM for the same price, and you're root on the machine, and you can install Docker, and you can access the logs, and you can forget about these troubles :p
And if you use this link you even get 10$ for free!

[lbonnefond]

No information either on server error log...

[lbonnefond]

I pay 9.95$/month and have no idea how much RAM is available. But indeed I can't install docker.

[NicolasCARPi]

Yeah, probably 1 Go, and it's shared RAM, so cannot be guaranted afaik. With Docker you get the added benefits of not having to worry about php versions, php available extensions, java version, etc… It just works (once it's setup properly ofc). And at DigitalOcean you have a lot of nice little tools at your disposal for making an entire backup and other stuff.

Out of curiosity, what php version is it running on? (you can see that in the sysadmin panel)

[lbonnefond]

php version of the shell:

PHP 7.0.27 (cli) (built: Jan 30 2018 20:58:12) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.27, Copyright (c) 1999-2017, by Zend Technologies

php version of the site: 7.0 also

Yes I have also a docker install on a VM for the lab note at work, and for this one the timespamping works with the default config but not with the SafeCreative certificate. I bothererd you with this also some time ago (issue #435 ).

Here its a git install on my shared host for students labnote during practicals (haven't asked for hosting by the university IT this year, I wanted to check first if it was working with the students).

Anayway, I am also using my shared host plan for personal blogs, picture gallery, rss agregator and other self-hosted services, so I'm not so eager to change the host...

[lbonnefond]

I've been looking for other free timestamping services that would work with elabftw.

You can get 10 free timestamps per IP with registration on SafeStamper. The certificate is the same as SafeCreative and it works.

I tried also FreeTSA (unknown limitation) and got it to work with https://freetsa.org/tsr and their Freetsa CA Certificate: cacert.pem.

It will solve my problem for the time being.

[NicolasCARPi]

Ok great, I should write documentation for all the different TSA one can use…