[Tests] Add http integration test setup #19261

rhoboat · 2018-05-21T15:05:53Z

Notes:
This PR also attempts to improve how we start the Kibana test server. We currently allow all command line options to get passed. This was okay because options that show up later in the list would override the previous options. It doesn't work for the test server when using basePath because --no-base-path was overriding any --server.basePath setting previously set by the user. Now we try to remove overridden options before starting the test server. See comment. 😃

Run

node scripts/functional_tests  \
  --config test/server_integration/http/ssl/config.js \
  --config test/server_integration/http/ssl_redirect/config.js

Todo:

remove --no-base-path option from runs where it doesn't apply
remove base path tests as per discussion about them being redundant
add basic ssl test
add ssl redirect test
run tests during CI

elasticmachine · 2018-05-21T16:11:06Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-23T15:04:13Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-23T15:29:51Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-23T16:30:11Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-06-29T08:20:59Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-06-29T19:42:28Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-06-29T20:13:12Z

retest

elasticmachine · 2018-06-29T21:25:52Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger

I like the direction this is going, but I think we should find a way to validate the certificate that the server is using rather than disabling SSL validation and I'm also curious about a couple seemingly unrelated code removals.

spalger · 2018-07-06T19:00:03Z

tasks/config/run.js

@@ -185,7 +185,6 @@ module.exports = function (grunt) {
      args: [
        'scripts/functional_tests_server',
        '--config', 'test/functional/config.js',
-        '--esFrom', 'source',


Why did this need to go?

I think this was a bad merge...

spalger · 2018-07-06T19:00:16Z

test/api_integration/config.js

@@ -43,7 +43,6 @@ export default async function ({ readConfigFile }) {
    junit: {
      reportName: 'API Integration Tests'
    },
-    env: commonConfig.get('env'),


Why remove this and the one in test/functional/config?

I think this is a bad merge. The esFrom source removal disappeared once I rebased over master again. Let me check.

I have the answer. The env value was removed from test/common/config.js because it used to have the kibana/es server specific stuff. These are left over in the test/functional/config.js and test/api_integration/config.js. It's part of the cleanup that didn't get removed earlier.

Can we remove this env from x-pack test configs (4 places) and schema (src/functional_test_runner/lib/config/schema.js) as well then?

P.S. it seems we have transform_deprecations for deprecated values in test config, wondering who cares about that 🙈

Neither of these is part of this PR, but I did go ahead and remove env. Hopefully that isn't too confusing for future reviewers. :) Can you point to what you're talking about with transform_deprecations?

spalger · 2018-07-06T19:03:42Z

test/integration/http/ssl/index.js

+   * When testing https server, set this to false.
+   * https://github.com/visionmedia/supertest/issues/396
+   */
+  process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';


In general I think we should avoid disabling "reject unauthorized" behavior if possible, and since we are using a specific ssl.certificate when we start Kibana we should be able to use that certificate to indicate that servers using that certificate are "authorized". Probably with something like:

export function KibanaSupertestProvider({ getService }) { const config = getService('config'); const kibanaServerUrl = formatUrl(config.get('servers.kibana')); const agent = supertestAsPromised.agent(); // maybe we should expose a separate config option for this? const kibanaServerCert = config.get('kbnTestServer.serverArgs') .filter(arg => arg.startsWith('--server.ssl.certificate')) .map(arg => arg.split('=').pop()) .map(path => readFileSync(path)) .shift() if (kibanaServerCert) { agent.ca(kibanaServerCert) } return agent(kibanaServerUrl) }

That way we will also be ensuring that the server is using SSL and that it's using the right certificates. That's probably better, don't you think?

Looks good. I agree. I didn't know about this option on supertest. TY!

// maybe we should expose a separate config option for this?

+1 for config option. It's not very convenient to pass one with this provider approach though, I was thinking about something like this:

supertest: (options) => KibanaSupertestProvider({ ...options, ca: require.resolve('../../../dev_certs/server.crt'), }),

@spalger The problem is, the dev cert is self signed, and for some reason there's an error when using it:

│ proc [ftr] └- ✖ fail: "kibana server with ssl handles requests using ssl" │ proc [ftr] │ Error: self signed certificate │ proc [ftr] │ at TLSSocket.<anonymous> (_tls_wrap.js:1105:38) │ proc [ftr] │ at TLSSocket._finishInit (_tls_wrap.js:639:8) │ proc [ftr] │ at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:469:38)

This might be bad advice, but it references the self signed cert issue. Any other ideas? Maybe we can try making a valid cert that isn't self-signed?

Here's more context for what node is doing:
nodejs/node-v0.x-archive#4023

tls, https: validate server certificates by default This commit changes the default value of the rejectUnauthorized option from false to true. What that means is that tls.connect(), https.get() and https.request() will reject invalid server certificates from now on, including self-signed certificates. There is an escape hatch: if you set the NODE_TLS_REJECT_UNAUTHORIZED environment variable to the literal string "0", node.js reverts to its old behavior.

Could it be that when setting process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';, self-signed certs are still being accepted, as this comment implies?
https://github.com/nodejs/node-v0.x-archive/pull/4023/files#diff-cbfd1300d9c62cf7f325159582f45d66R27

// disable strict server certificate validation by the client
process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';

rhoboat · 2018-07-11T18:11:34Z

test/functional/config.js

@@ -106,8 +106,6 @@ export default async function ({ readConfigFile }) {
    },
    servers: commonConfig.get('servers'),

-    env: commonConfig.get('env'),


Cleanup only.

elasticmachine · 2018-07-11T18:38:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

azasypkin · 2018-07-12T08:34:56Z

test/api_integration/config.js

@@ -43,7 +43,6 @@ export default async function ({ readConfigFile }) {
    junit: {
      reportName: 'API Integration Tests'
    },
-    env: commonConfig.get('env'),


Can we remove this env from x-pack test configs (4 places) and schema (src/functional_test_runner/lib/config/schema.js) as well then?

P.S. it seems we have transform_deprecations for deprecated values in test config, wondering who cares about that 🙈

azasypkin · 2018-07-12T09:09:47Z

test/integration/services/supertest.js

+  return supertestAsPromised(kibanaServerUrl);
+}
+
+export function KibanaSupertestWithoutAuthProvider({ getService }) {


note: I don't think we need to separate KibanaSupertestProvider and KibanaSupertestWithoutAuthProvider in OSS kibana since there is no authentication supported, it can probably be just KibanaSupertestProvider since withoutAuth is implicitly expected. What do you think?

Currently these tests are using supertest which is the version with auth. Should they be using without auth? Hm. Doesn't OSS Kibana offer Basic Auth? The default superagent without any auth setting specified includes Basic Auth.

azasypkin · 2018-07-12T09:13:15Z

test/integration/http/basepath/no_rewrite/config.js

+      serverArgs: [
+        ...httpConfig.get('kbnTestServer.serverArgs'),
+        '--server.basePath=/abc/xyz',
+        '--server.rewriteBasePath=true',


note: it's kind of confusing that for no_rewrite tests we use --server.rewriteBasePath=true for rewrite tests we use --server.rewriteBasePath=false, what was the idea behind that?

🙈 I think I swapped the naming.

azasypkin · 2018-07-12T09:33:33Z

test/integration/http/basepath/no_rewrite/index.js

+  describe('Kibana server with basePath and with rewriteBasePath', () => {
+    const basePath = '/abc/xyz';
+
+    it('requires root requests to contain basePath', async () => {


note: would be great if you can add simple test to test that behavior when Kibana notices "old" base path and redirects to new one.

azasypkin · 2018-07-12T10:21:57Z

test/integration/http/ssl/index.js

+   * When testing https server, set this to false.
+   * https://github.com/visionmedia/supertest/issues/396
+   */
+  process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';


// maybe we should expose a separate config option for this?

+1 for config option. It's not very convenient to pass one with this provider approach though, I was thinking about something like this:

supertest: (options) => KibanaSupertestProvider({ ...options, ca: require.resolve('../../../dev_certs/server.crt'), }),

azasypkin · 2018-07-12T10:31:36Z

It's likely not related to this PR, but just curious. I see we run Kibana for tests with the following command:

node scripts/kibana --env.name=development --logging.json=false --server.port=5620 \
 --optimize.watchPort=5630 --optimize.watchPrebuild=true --status.allowAnonymous=true \
 --optimize.enabled=true --elasticsearch.url=http://elastic:changeme@localhost:9200 \
 --elasticsearch.username=elastic --elasticsearch.password=changeme --oss --optimize.enabled=false \
 --elasticsearch.healthCheck.delay=3600000 --server.xsrf.disableProtection=true \
 --server.ssl.enabled=true --server.ssl.key=/projects/elastic/master/kibana/test/dev_certs/server.key \
 --server.ssl.certificate=/projects/elastic/master/kibana/test/dev_certs/server.crt --no-base-path \
 --optimize.bundleDir=/projects/elastic/master/kibana/optimize/bundles

If it's --oss why do we pass --elasticsearch.username, --elasticsearch.password and add elastic:changeme@ to --elasticsearch.url?

elasticmachine · 2018-07-25T16:37:04Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-07-25T22:58:16Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-07-26T17:22:21Z

packages/kbn-test/src/functional_tests/lib/run_kibana_server.js

@@ -59,12 +59,35 @@ function collectCliArgs(config, { installDir, extraKbnOpts }) {
    serverArgs,
    args => (installDir ? args.filter(a => a !== '--oss') : args),
    args => {
-      return installDir ? [...args, ...buildArgs] : [KIBANA_EXEC_PATH, ...args, ...sourceArgs];
+      return installDir ? [...buildArgs, ...args] : [KIBANA_EXEC_PATH, ...sourceArgs, ...args];


args refers to all the options that the user passes in via command line and/or the particular test config.js they're using for starting the Kibana test server.

Previously we were appending sourceArgs (or buildArgs if they passed in --kibana-install-dir) to the end of their passed-in args. But this doesn't work when they're using --server.basePath because sourceArgs was always overriding it with --no-base-path option. In fact for base path settings, the ordering doesn't matter because no-base-path and server.basePath are different keys; just the existence of no-base-path disables the base path proxy server.

Now, I've flipped the ordering so that the user's args always override the default build/source args. And, I've made sure that any base path setting that comes later, whether or not the keys match, will override previous base path settings.

elasticmachine · 2018-08-09T17:03:53Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2018-08-14T01:57:46Z

Would you mind pulling in master? This needs to include the removal of the env from new test configs like #20163

spalger

Works great, diff'd the arguments passed to Kibana in this PR with master and the only things changed are the order of the --no-base-path and --optimize.bundleDir arguments passed in most configs, and the removal of some instances of --server.port, --elasticsearch.url and --optimize.enabled which were being overridden by subsequent flags anyway. I do have a couple notes on the code, but LGTM after you pull master and update the panel_actions config.

spalger · 2018-08-14T01:25:30Z

packages/kbn-test/src/functional_tests/lib/run_kibana_server.js

+ * overridden options
+ */
+function filterCliArgs(args) {
+  const argv = [...args].slice(1);


Out of curiosity, why ignore the first arg? When running from dev it will be the path to scripts/kibana, but that shouldn't be a problem, and otherwise the first argument is a meaningful argument that should be considered, right?

It's true. You're right, except that in the case that it's running from an install-path, that first argument isn't the bin/kibana command. That gets tacked on when we check whether we're on Windows (append the .bat) or not. So.. it's a bit confusing. Ultimately I think you're right, I got to simplify this method by removing that slice. Thanks for noticing!

spalger · 2018-08-14T01:45:48Z

packages/kbn-test/src/functional_tests/lib/run_kibana_server.js

+function filterCliArgs(args) {
+  const argv = [...args].slice(1);
+
+  const filteredArgv = argv.reduce((acc, val, ind) => {


I think this would be a little easier to follow if it was using an object where the keys are unique representations of each arg (unifying the basePath args), and the values were the last version found for each arg. Then this could just return the Object.values() of that object rather than doing so much traversal of the args, and having to keep track of things like index.

elasticmachine · 2018-08-14T03:28:02Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-08-14T18:12:13Z

retest

elasticmachine · 2018-08-14T20:34:13Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-08-14T20:40:01Z

failed on fail: "logstash grok debugger app syntax highlighting applies the correct CSS classes" is that a flaky test?

rhoboat · 2018-08-14T20:40:06Z

retest

elasticmachine · 2018-08-14T22:26:09Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

azasypkin · 2018-08-15T08:36:01Z

You have one r+ already and it should generally be enough. Feel free to request review if you still need additional review though.

rhoboat · 2018-08-15T08:39:34Z

@azasypkin can you just check that the ssl tests are sufficient?

azasypkin · 2018-08-15T08:41:03Z

@azasypkin can you just check that the ssl tests are sufficient?

Sure, looking.....

azasypkin · 2018-08-15T10:11:33Z

Okay, I think it's a good start, thanks for adding these tests! If we need more tests in the future we'll add them in separate PRs.

azasypkin · 2018-08-17T08:43:43Z

6.x/6.5: 8e696b7

* [Tests] Add http integration test setup * Base path tests * SSL tests * Eslint fixes * Remove env from config schema * Rename folders so no_rewrite and rewrite match configs/tests * wip * Use self-signed cert for SSL test * Improve basepath tests * Run base path proxy server in dev mode for now * Remove env from x-pack reporting config * Remove redundant base-path tests * Test SSL with redirectHttpFromPort set * Test SSL with redirectHttpFromPort set * Flesh out comments * Remove some cruft * Add SSL tests to CI run

kimjoar mentioned this pull request May 21, 2018

Integration tests for new platform #12699

Closed

13 tasks

rhoboat requested review from epixa, spalger and azasypkin June 29, 2018 21:41

spalger reviewed Jul 6, 2018

View reviewed changes

rhoboat commented Jul 11, 2018

View reviewed changes

azasypkin reviewed Jul 12, 2018

View reviewed changes

azasypkin mentioned this pull request Jul 12, 2018

Log correct Kibana URL when TLS is enabled and log it only once. #20721

Merged

stacey-gammon mentioned this pull request Jul 12, 2018

Reports fail when PDF generation request includes Transfer-Encoding HTTP header #20724

Closed

archanid added 6 commits July 25, 2018 09:37

[Tests] Add http integration test setup

d8b188c

Base path tests

70d6826

SSL tests

bf493f0

Eslint fixes

d485ffb

Remove env from config schema

b515ad0

Rename folders so no_rewrite and rewrite match configs/tests

8ecbcc0

archanid added 2 commits July 25, 2018 14:49

wip

78cd53c

Use self-signed cert for SSL test

80a52c8

rhoboat commented Jul 26, 2018

View reviewed changes

azasypkin self-requested a review August 10, 2018 19:23

spalger approved these changes Aug 14, 2018

View reviewed changes

archanid added 3 commits August 14, 2018 10:32

Flesh out comments

f937c01

Remove some cruft

90f17d6

Merge branch 'master' into http-integ-tests

9de8a68

Add SSL tests to CI run

68ad563

azasypkin removed their request for review August 15, 2018 08:36

azasypkin merged commit cb8c1de into elastic:master Aug 17, 2018

azasypkin added backport pending v6.5.0 labels Aug 17, 2018

azasypkin mentioned this pull request Aug 17, 2018

[6.x] [Tests] Add http integration test setup (#19261) #22114

Merged

azasypkin added backported and removed backport pending labels Aug 17, 2018

timroes mentioned this pull request Aug 20, 2018

Make panel action tests more generic #22174

Merged

[Tests] Add http integration test setup #19261

[Tests] Add http integration test setup #19261

Conversation

rhoboat commented May 21, 2018 • edited Loading

Run

elasticmachine commented May 21, 2018

💚 Build Succeeded

elasticmachine commented May 23, 2018

💔 Build Failed

elasticmachine commented May 23, 2018

💔 Build Failed

elasticmachine commented May 23, 2018

💔 Build Failed

elasticmachine commented Jun 29, 2018

💔 Build Failed

elasticmachine commented Jun 29, 2018

💔 Build Failed

rhoboat commented Jun 29, 2018

elasticmachine commented Jun 29, 2018

💚 Build Succeeded

spalger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhoboat Jul 25, 2018 • edited Loading

Choose a reason for hiding this comment

rhoboat Jul 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jul 11, 2018

💚 Build Succeeded

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azasypkin commented Jul 12, 2018

elasticmachine commented Jul 25, 2018

💔 Build Failed

elasticmachine commented Jul 25, 2018

💔 Build Failed

Choose a reason for hiding this comment

elasticmachine commented Aug 9, 2018

💚 Build Succeeded

spalger commented Aug 14, 2018

spalger left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Aug 14, 2018

💚 Build Succeeded

rhoboat commented Aug 14, 2018

elasticmachine commented Aug 14, 2018

💔 Build Failed

rhoboat commented Aug 14, 2018

rhoboat commented Aug 14, 2018

elasticmachine commented Aug 14, 2018

💚 Build Succeeded

azasypkin commented Aug 15, 2018

rhoboat commented Aug 15, 2018

azasypkin commented Aug 15, 2018

azasypkin commented Aug 15, 2018

azasypkin commented Aug 17, 2018

rhoboat commented May 21, 2018 •

edited

Loading

rhoboat Jul 25, 2018 •

edited

Loading

rhoboat Jul 25, 2018 •

edited

Loading

spalger left a comment •

edited

Loading