Https requests don't work for request time out

[bigwheel]

I read How to Record and Replay an HTTPS Transaction · linkedin/flashback Wiki and tried to proxy https request.
But it didn't work correctly for time out.

Environemnt

• ubuntu 16.10

Reproduce procedure

openssl genrsa 1024 > private-key.pem
openssl rsa -in private-key.pem -pubout -out public-key.pem
openssl req -new -key private-key.pem > my-request.csr # enter for all dialogs, then no password
openssl x509 -req -in my-request.csr -signkey private-key.pem -out public-key.crt -days 3650
openssl pkcs12 -export -in public-key.crt -inkey private-key.pem -out my-identity.p12

# add certificate to os root certificates
sudo cp public-key.crt /usr/share/ca-certificates
sudo echo "public-key.crt" >> /etc/ca-certificats.conf
sudo update-ca-certificates

./startAdminServer.sh -port 1234

# in another console
curl "http://localhost:1234/admin?action=startFlashback" --data '{
  "sceneMode": "record",
  "sceneName": "test1",
  "matchRule": "matchEntireRequest",
  "scenePath": "./scene",
  "proxyHost": "localhost",
  "proxyPort": "5564",
  "caCertPath": "absolute path of my-identity.p12",
  "caCertPwd": "",
  "caAlias": "",
  "caKeyPwd": "",
  "caCertOU": "",
  "caCertO": "",
  "caCertCN": "",
  "caCertL": "",
  "caCertCC": ""
}'

Above commands succeed entirely, however,

Result

curl --proxy localhost:5564 -L 'https://www.google.co.jp/'
curl: (28) Operation timed out after 300570 milliseconds with 0 out of 0 bytes received

And console executes "./startAdminServer.sh -port 1234" doesn't show any logs.

I think my prrocedure is not correct for flashback, but what's wrong ?

[ben-oldcom]

thanks for trying flashback, i will take a look, there might be something wrong with the keys that generated.

[bigwheel]

Thank you for reply.
As this instruction ( https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html ),
I started following key generate command. But result was same time out.

openssl genrsa -aes256 -out private-key.pem 4096

Could you tell me a successful example for https ?

[ben-oldcom]

i did it a while back by following this wiki, let me redo it again tomorrow and have an example cert file for you to try. If i remember correctly, i was using the intermediate cert.

[ben-oldcom]

@bigwheel there is possible something wrong in the Flashback-Admin for passing in the parameters to Flashback (something that being used in ConnectionFlowFactory --> createFullHttpsConnectionFlow), I will need to talk with @sf1152 tomorrow to see what's the best way to fix it. will update you as soon as I get back from him. sorry for the inconvenience.

[bigwheel]

Good news!
I don't need this quickly, then no need temporary solution.
When problem is fixed, I must try it in my project :)

[ben-oldcom]

@bigwheel
I followed exactly what the wiki does: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html, for root and intermediate certificates. (with only the change of "Alice Ltd" --> FLASHBACK which i don't think matters. for all password i keep using the same as in that doc "secretpassword").

then one extra step is to generate the pkcs intermediate ca
openssl pkcs12 -export -out intermediate/intermediate.p12 -inkey intermediate/private/intermediate.key.pem -in intermediate/certs/intermediate.cert.pem -name "flashbackca"
in your flashback folder, start the admin server
./startAdminServer.sh -port 1234

In another console, start flashback
curl "http://localhost:1234/admin?action=startFlashback" --data '{ "sceneMode": "record", "sceneName": "flashbacktest1", "matchRule": "matchEntireRequest", "scenePath": "/tmp", "proxyHost": "localhost", "proxyPort": "5555", "caCertPath": "<absolute path to where you generated your own CA>/intermediate/intermediate.p12", "caCertPwd": "secretpassword", "caAlias": "flashbackca", "caKeyPwd": "secretpassword", "caCertCN": "", "caCertOU": "", "caCertO": "", "caCertL": "", "caCertCC": "" }'
curl google.com
curl --cacert intermediate/certs/intermediate.cert.pem --cert intermediate/intermediate.p12:secretpassword --proxy localhost:5555 -L 'https://www.google.com/'
shutdown flashback
curl "http://localhost:1234/admin?action=shutDownFlashback" --data '{}'
check the recorded
vi /tmp/flashbacktest1

Let me know if this works. I will go ahead and close this issue, feel free to reopen it if still not working.

I really appreciated you are using flashback and actually helped us find couple of issues in the admin/proxy code (everything is working with the above instruction, just enhancement and improvements), i will create new issues and ask help from @sf1152 to fix.

[thestephenstanton]

I went through the steps exactly as you said and suggested (but doing with Scala instead) and I get these errors. Any ideas on what might be going on?

`18/06/22 15:18:52 INFO steps.StopReadingFromClient: Stop reading from client
18/06/22 15:18:52 ERROR channel.ServerChannelHandler: Caught exception on server channel
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1114)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:981)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:934)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
... 11 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1148)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1003)
... 13 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
... 21 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
... 27 more

Unexpected end of file from server
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2055)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
at scalaj.http.DefaultConnectFunc$.apply(Http.scala:471)
at scalaj.http.DefaultConnectFunc$.apply(Http.scala:469)
at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:355)
at scalaj.http.HttpRequest.exec(Http.scala:335)
at scalaj.http.HttpRequest.asString(Http.scala:455)
at com.redventures.databroker.data.HailoClient$.ping(HailoClient.scala:13)
at com.redventures.databroker.data.HailoClientSpec$$anonfun$1.apply(HailoClientSpec.scala:12)
at com.redventures.databroker.data.HailoClientSpec$$anonfun$1.apply(HailoClientSpec.scala:11)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685)
at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1679)
at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692)
at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1692)
at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1685)
at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750)
at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:373)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:410)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1750)
at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1685)
at org.scalatest.Suite$class.run(Suite.scala:1147)
at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1685)
at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795)
at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795)
at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1795)
at com.redventures.flashback.FlashbackFlatSpec.org$scalatest$BeforeAndAfterAll$$super$run(FlashbackFlatSpec.scala:15)
at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
at com.redventures.flashback.FlashbackFlatSpec.run(FlashbackFlatSpec.scala:15)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1340)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1334)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1334)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1011)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1010)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1500)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
at org.scalatest.tools.Runner$.run(Runner.scala:850)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)

`

[sf1152]

I assume you are running in record model. In record model, there are two trust stores. If you client code is running in the same process of Flashback, you need make sure you start flashback first before you point client to your keystore. If not, you might end up using your customized keystore to authenticate with third-party certificate.

[thestephenstanton]

@sf1152 So I start flashback first after newing up a new FlashbackRunner.Builder() and call the method start(), do I need to import this keystore into java somehow? Similar to this post? https://blog.codecentric.de/en/2013/01/how-to-use-self-signed-pem-client-certificates-in-java/