[CI] Upgrade YAML tests fail with javax.crypto.IllegalBlockSizeException

[elasticmachine]

Original comment by @polyfractal:

A triplet of upgrade yaml tests seem to be failing all with the same IllegalBlockSizeException, and it looks they they are only happening on release CI builds at the moment. This triplet LINK REDACTED,time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(_source),index:'build-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:javax.crypto.IllegalBlockSizeException)),sort:!(process.time-start,desc))) but I couldn't see any particular change that may have caused it.

The most recent failure was on 6.x, but it seems to be happening on master too. I'm not sure if it is relevant, but all the failures were on Java 9.0.4+11

LINK REDACTED

Tagging @elastic/es-security to start, although I'm not really sure if this is a security issue or something wonky with the release process/build.

The relevant exception:

Caused by: java.lang.IllegalStateException: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
	at org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:182)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:116)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:72)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:71)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:101)
	at org.elasticsearch.license.LicenseService.registerLicense(LicenseService.java:196)
	at 

Full stack trace from the tests:

{
  "stash" : {
    "body" : {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_state_exception",
            "reason" : "javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher",
            "stack_trace" : "[javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher]; nested: IllegalStateException[javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher]; nested: IllegalBlockSizeException[Input length must be multiple of 16 when decrypting with padded cipher];
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:640)
	at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:585)
	at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)
	at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onFailure(TransportMasterNodeAction.java:160)
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:42)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:657)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
	at org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:182)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:116)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:72)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:71)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:101)
	at org.elasticsearch.license.LicenseService.registerLicense(LicenseService.java:196)
	at org.elasticsearch.license.TransportPutLicenseAction.masterOperation(TransportPutLicenseAction.java:64)
	at org.elasticsearch.license.TransportPutLicenseAction.masterOperation(TransportPutLicenseAction.java:33)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	... 3 more
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:936)
	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:847)
	at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
	at javax.crypto.Cipher.doFinal(Cipher.java:2164)
	at org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:180)
	... 14 more"
          }
        ],
        "type" : "illegal_state_exception",
        "reason" : "javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher",
        "caused_by" : {
          "type" : "illegal_block_size_exception",
          "reason" : "Input length must be multiple of 16 when decrypting with padded cipher",
          "stack_trace" : "javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:936)
	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:847)
	at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
	at javax.crypto.Cipher.doFinal(Cipher.java:2164)
	at org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:180)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:116)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:72)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:71)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:101)
	at org.elasticsearch.license.LicenseService.registerLicense(LicenseService.java:196)
	at org.elasticsearch.license.TransportPutLicenseAction.masterOperation(TransportPutLicenseAction.java:64)
	at org.elasticsearch.license.TransportPutLicenseAction.masterOperation(TransportPutLicenseAction.java:33)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)"
        },
        "stack_trace" : "java.lang.IllegalStateException: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
	at org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:182)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:116)
	at org.elasticsearch.license.CryptUtils.readEncryptedPublicKey(CryptUtils.java:72)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:71)
	at org.elasticsearch.license.LicenseVerifier.verifyLicense(LicenseVerifier.java:101)
	at org.elasticsearch.license.LicenseService.registerLicense(LicenseService.java:196)
	at org.elasticsearch.license.TransportPutLicenseAction.masterOperation(TransportPutLicenseAction.java:64)
	at org.elasticsearch.license.TransportPutLicenseAction.masterOperation(TransportPutLicenseAction.java:33)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:936)
	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:847)
	at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
	at javax.crypto.Cipher.doFinal(Cipher.java:2164)
	at org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:180)
	... 14 more"
      },
      "status" : 500
    }
  }
}]

REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:plugin:integTestRunner \
  -Dtests.seed=1765FEF2C5106B9 \
  -Dtests.class=org.elasticsearch.xpack.test.rest.XPackRestIT \
  -Dtests.method="test {p0=upgrade/10_basic/Upgrade test - should fail as index is already up to date}" \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=ar-LB \
  -Dtests.timezone=AET \
  -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*,xpack/15_basic/*,license/20_put_license/*

REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:plugin:integTestRunner \
  -Dtests.seed=1765FEF2C5106B9 \
  -Dtests.class=org.elasticsearch.xpack.test.rest.XPackRestIT \
  -Dtests.method="test {p0=upgrade/10_basic/Upgrade info - all}" \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=ar-LB \
  -Dtests.timezone=AET \
  -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*,xpack/15_basic/*,license/20_put_license/*

REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:plugin:integTestRunner \
  -Dtests.seed=1765FEF2C5106B9 \
  -Dtests.class=org.elasticsearch.xpack.test.rest.XPackRestIT \
  -Dtests.method="test {p0=upgrade/10_basic/Upgrade test - wait_for_completion:false}" \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=ar-LB \
  -Dtests.timezone=AET \
  -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*,xpack/15_basic/*,license/20_put_license/*

[elasticmachine]

Original comment by @polyfractal:

Starting to get noisy, disabled the tests for now

Master: LINK REDACTED
6.x: LINK REDACTED

[elasticmachine]

Original comment by @tvernum:

@tbrooks8 This looks like it could be license related - any thoughts?

[elasticmachine]

Original comment by @tbrooks8:

Yeah I’ll take a look tomorrow. Thanks for the ping.

[elasticmachine]

Original comment by @tbrooks8:

@rjernst - It looks to be that this is failing while reading the encrypted public key. And it started failing shortly after HASH REDACTED. Could that be related?

[elasticmachine]

Original comment by @rjernst:

It is definitely related. @elasticdog In LINK REDACTED you said no decoding should be necessary. Can you confirm this is still correct? This seems the most likely problem.

[elasticmachine]

Original comment by @elasticdog:

The base64 decoding happens when Ansible pulls the license out of Vault to write it to the disk:

LINK REDACTED

You can verify that the old key and what is in Vault are the same by checking out the commit before HASH REDACTED when the key was removed:

$ cd x-pack-elasticsearch/
$ git checkout HASH REDACTED^
$ vault read -field=pubkey secret/release/license | base64 --decode > decoded.key
$ shasum plugin/keys/prod/public.key decoded.key 
95e0dcb7bb45c4a67c41cd29190319feecd42ef1  plugin/keys/prod/public.key
95e0dcb7bb45c4a67c41cd29190319feecd42ef1  decoded.key

...although, I'm having trouble getting checksums to align with what's actually on the internal-ci machines (I just picked one at random to test):

EMAIL REDACTED shasum /etc/x-pack/license.key
5cc2cb605da4105f2977bb654c4289adf0b653aa  /etc/x-pack/license.key

It does look like they're different sizes, and Ansible might be adding some unexpected content. I've confirmed that the Vault lookup plugin is working as expected, so it may be that Ansible's built-in b64decode filter is not behaving the same as base64 --decode. Investigating now.

[elasticmachine]

Original comment by @elasticdog:

I can see the expected content is there in a hexdump (within some extra junk), so I started digging into the upstream b64decode filter source and Ansible's custom text utility functions. It looks like this may be a known limitation with trying to pass binary files through Jinja templates. I believe in this scenario, the content gets inadvertently UTF-8 encoded. 😢 🐼

I'll split up the Ansible task to use the system binaries for decoding rather than running it through a filter. Sorry for not catching this sooner...you'd think base64 decoding would be a problem devoid of much complexity.

[elasticmachine]

Original comment by @elasticdog:

I've worked around the jinja2 filtering behavior, re-deployed the license file, and confirmed that it matches the expected output:

EMAIL REDACTED shasum /etc/x-pack/license.key
95e0dcb7bb45c4a67c41cd29190319feecd42ef1  /etc/x-pack/license.key

Feel free to resolve this once there's a green release tests job.

[tvernum]

I believe this is sorted out now.

[polyfractal]

@tvernum I noticed the yaml test is still skip: all in master, presumably this should be removed now?

https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/src/test/resources/rest-api-spec/test/upgrade/10_basic.yml#L5

[cbuescher]

@tvernum was directed here by the weekly mail, there still seems to be the above mentioned "skip: all" so I'm just pinging if the skip can be removed.