Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[optimizer] extract plugin discovery #14745

Merged
merged 78 commits into from
Dec 6, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
027a1c4
[plugins] extract plugin discover from the kibana server
spalger Nov 1, 2017
b1947e1
integrate plugin discovery module with server
spalger Nov 2, 2017
f1d01c3
[pluginDiscovery] fully extend config before checking enabled status
spalger Nov 2, 2017
b53526c
[pluginDiscovery] limit arbitrary defaults in PluginSpec
spalger Nov 2, 2017
d97d0cc
[ui/navLink] fix tests
spalger Nov 2, 2017
76f8128
[ui/injectedVars] fix tests
spalger Nov 2, 2017
d77ced8
[ui/app] fix tests
spalger Nov 2, 2017
15f0a30
[server/plugins] convert init to callPluginHook tests
spalger Nov 2, 2017
a054329
[build/verifyTranslations] update verify logic
spalger Nov 2, 2017
4d16a5f
[pluginDiscovery] remove rx utils
spalger Nov 2, 2017
ca727dc
fix i18n transaltion key name
spalger Nov 2, 2017
1a01773
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 8, 2017
710254f
[pluginDiscovery] do kibana version checks as a part of discovery
spalger Nov 8, 2017
75fb29e
[pluginDiscovery/createPacksInDirectory$] clarify error handling
spalger Nov 8, 2017
2220f43
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 15, 2017
100a306
[eslint] fix lint errors
spalger Nov 15, 2017
eb9876c
[uiApp/modules] ensure load order matches master
spalger Nov 15, 2017
604344a
[uiBundle] use known uiExport type for providers
spalger Nov 15, 2017
6e7bef9
[uiExports] use the `home` export type
spalger Nov 15, 2017
00c9020
[uiExports] validate that all uiExport types are known
spalger Nov 15, 2017
78951ec
[timelion] remove archaic/broken bwc check
spalger Nov 15, 2017
b37bc7c
revert some stragler changes
spalger Nov 15, 2017
eeb65ae
[pluginSpecs] reformat comments
spalger Nov 15, 2017
00cbaed
[uiBundle] rebel and use more fcb 😬
spalger Nov 15, 2017
646d1d5
correct comment
spalger Nov 15, 2017
edd6a4d
[server/waitForPluginsInit] describe queues var
spalger Nov 15, 2017
b8bbb3a
[server/plugins] prevent multiple calls to next() by using single then()
spalger Nov 15, 2017
5223b70
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 15, 2017
2e134f6
[uiApp] remove archaic deprecation warning
spalger Nov 15, 2017
3dbf0f0
[uiApp] tighten up tests
spalger Nov 15, 2017
eda2690
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 17, 2017
1d8c9de
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 17, 2017
c7a4e5d
[pluginDiscovery/errors] remove $ from symbol var
spalger Nov 17, 2017
ff3b815
[pluginDiscovery/reduceExportSpecs] update docs
spalger Nov 17, 2017
f30dba3
[pluginDiscovery/findPluginSpecs] rightVersion -> isRightVersion
spalger Nov 17, 2017
893cc4a
[pluginDiscovery/findPluginSpecs] fix typos
spalger Nov 17, 2017
7a47910
[uiApps/getById] use Map() rather than memoize
spalger Nov 17, 2017
5c9fdc0
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 17, 2017
0bae3bf
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 18, 2017
424eae5
save
spalger Nov 21, 2017
5d16b95
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 29, 2017
e818704
[savedObjects/mappings] use uiExports.savedObjectMappings
spalger Nov 29, 2017
8bd9c19
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Nov 30, 2017
2cbf032
[server/mapping/indexMapping] update tests, addRootProperties method …
spalger Nov 30, 2017
aff9921
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Dec 1, 2017
cc3fc83
[uiExports] "embeddableHandlers" -> "embeddableFactories"
spalger Dec 1, 2017
dee1198
[pluginDiscovery] fix pluralization of invalidVersionSpec$
spalger Dec 5, 2017
b2dc37a
[pluginDiscover] add README
spalger Dec 5, 2017
9204943
[pluginDiscovery/reduceExportSpecs] don't ignore fasly spec values, j…
spalger Dec 5, 2017
a6f1438
[ui/exportTypes] use better reducer names
spalger Dec 5, 2017
c9eff80
[ui/uiExports] add README
spalger Dec 5, 2017
0606846
fix links
spalger Dec 5, 2017
1898ae4
[pluginDiscovery/readme] expand examples
spalger Dec 5, 2017
89efa7b
[pluginDiscovery/readme] clean up reduceExportSpecs() doc
spalger Dec 5, 2017
6d14b22
[ui/uiExports/readme] cleanup example
spalger Dec 5, 2017
b8a95b7
[pluginDiscovery] remove needless use of lodash
spalger Dec 5, 2017
4cda365
[pluginDiscovery/waitForComplete] use better name
spalger Dec 5, 2017
fc73f4b
[pluginDiscovery/findPluginSpecs] use fixtures rather than core_plugins
spalger Dec 5, 2017
70666a2
[pluginDiscovery/stubSchema] use deafult: false
spalger Dec 5, 2017
b5079a5
[plguinDiscovery/pluginConfig] add tests
spalger Dec 5, 2017
99c1ff1
typo
spalger Dec 5, 2017
88328a4
[uiExports/readme] fix link
spalger Dec 5, 2017
c586cf4
Merge branch 'master' of github.com:elastic/kibana into implement/plu…
spalger Dec 5, 2017
3091c0a
[pluginDiscovery/packAtPath] fail with InvalidPackError if path is no…
spalger Dec 5, 2017
0e2f131
[pluginDiscovery/packAtPath] rely on error.code to detect missing pac…
spalger Dec 5, 2017
36dd268
[pluginDiscovery/packAtPath] only attempt to get pack when observable…
spalger Dec 5, 2017
c35ef96
[pluginDiscovery/packAtPath] add tests
spalger Dec 5, 2017
0d870c5
[pluginDiscovery/pluginPack] move absolute path checks into fs lib
spalger Dec 5, 2017
75dbffe
[pluginDiscovery/packsInDirectory] fix error type check
spalger Dec 5, 2017
b9b75d4
[pluginDiscovery/pluginPack/tests] share some utils
spalger Dec 5, 2017
3dbe93e
[pluginDiscovery/packsInDirectory] add tests
spalger Dec 5, 2017
17912cf
[pluginDiscovery/pluginPack] only cast undefined to array
spalger Dec 5, 2017
139e2b4
[pluginDiscovery/pluginPack] add tests
spalger Dec 5, 2017
66e53df
[pluginDiscovery/pluginSpec/isVersionCompatible] add tests
spalger Dec 5, 2017
03de191
[pluginDiscovery/InvalidPluginError] be less redundant
spalger Dec 5, 2017
1407640
[pluginDiscovery/pluginSpec] verify config service is passed to isEna…
spalger Dec 5, 2017
bad97a8
[pluginDiscovery/pluginSpec] add tests
spalger Dec 6, 2017
a047981
fix "existent" spelling
spalger Dec 6, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/cli/cluster/base_path_proxy.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { map as promiseMap, fromNode } from 'bluebird';
import { Agent as HttpsAgent } from 'https';
import { readFileSync } from 'fs';

import Config from '../../server/config/config';
import { Config } from '../../server/config/config';
import setupConnection from '../../server/http/setup_connection';
import registerHapiPlugins from '../../server/http/register_hapi_plugins';
import setupLogging from '../../server/logging';
Expand Down
10 changes: 9 additions & 1 deletion src/core_plugins/console/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,14 @@ export default function (kibana) {
id: 'console',
require: [ 'elasticsearch' ],

isEnabled(config) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

// console must be disabled when tribe mode is configured
return (
config.get('console.enabled') &&
!config.get('elasticsearch.tribe.url')
);
},

config: function (Joi) {
return Joi.object({
enabled: Joi.boolean().default(true),
Expand Down Expand Up @@ -115,7 +123,7 @@ export default function (kibana) {
}
});

const testApp = kibana.uiExports.apps.hidden.byId['sense-tests'];
const testApp = server.getHiddenUiAppById('sense-tests');
if (testApp) {
server.route({
path: '/app/sense-tests',
Expand Down
10 changes: 9 additions & 1 deletion src/core_plugins/dev_mode/index.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
export default (kibana) => {
if (!kibana.config.get('env.dev')) return;
return new kibana.Plugin({
id: 'dev_mode',

isEnabled(config) {
return (
config.get('env.dev') &&
config.get('dev_mode.enabled')
);
},

uiExports: {
spyModes: [
'plugins/dev_mode/vis_debug_spy_panel'
Expand Down
2 changes: 1 addition & 1 deletion src/core_plugins/kibana/server/lib/manage_uuid.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ export default async function manageUuid(server) {
return result.toString(FILE_ENCODING);
} catch (err) {
if (err.code === 'ENOENT') {
// non-existant uuid file is ok
// non-existent uuid file is ok
return false;
}
server.log(['error', 'read-uuid'], err);
Expand Down
2 changes: 1 addition & 1 deletion src/core_plugins/state_session_storage_redirect/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ export default function (kibana) {
title: 'Redirecting',
id: 'stateSessionStorageRedirect',
main: 'plugins/state_session_storage_redirect',
listed: false,
hidden: true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if hidden is the right word here. To me, if something is hidden, it is inaccessible, but in this case it just means the app does not appear in the sidebar, correct? I can live with it, it just feels off.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually hidden means it doesn’t show in the navigation and isn’t accessible via /app/{id}. listed means that the app isn’t rendered in the nav but is accessible via the app route

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that makes sense. Thanks for clarifying.

}
}
});
Expand Down
52 changes: 31 additions & 21 deletions src/core_plugins/tests_bundle/index.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
import { union } from 'lodash';
import findSourceFiles from './find_source_files';

import { fromRoot } from '../../utils';

import findSourceFiles from './find_source_files';
import { createTestEntryTemplate } from './tests_entry_template';

export default (kibana) => {
return new kibana.Plugin({
config: (Joi) => {
Expand All @@ -13,9 +16,18 @@ export default (kibana) => {
},

uiExports: {
bundle: async (UiBundle, env, apps, plugins) => {
async __bundleProvider__(kbnServer) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this used? I see two other references to __bundleProvider__ in this PR, and both of them are just exports.

Copy link
Contributor Author

@spalger spalger Dec 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other references define the __bundleProvider__ ui export type, which is ultimately defined here. The export there is a reducer, which consumes all __bundleProvider__ exports and writes them to the uiBundleProviders array in the uiExports object. This means that Kibana can access these providers at kbnServer.uiExports.uiBundleProviders

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a README in src/ui/ui_exports would be a good idea

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ for a high-level readme in src/ui/ui_exports

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let modules = [];
const config = kibana.config;

const {
config,
uiApps,
uiBundles,
plugins,
uiExports: {
uiSettingDefaults = {}
}
} = kbnServer;

const testGlobs = [
'src/ui/public/**/*.js',
Expand All @@ -26,20 +38,25 @@ export default (kibana) => {
if (testingPluginIds) {
testGlobs.push('!src/ui/public/**/__tests__/**/*');
testingPluginIds.split(',').forEach((pluginId) => {
const plugin = plugins.byId[pluginId];
if (!plugin) throw new Error('Invalid testingPluginId :: unknown plugin ' + pluginId);
const plugin = plugins
.find(plugin => plugin.id === pluginId);

if (!plugin) {
throw new Error('Invalid testingPluginId :: unknown plugin ' + pluginId);
}

// add the modules from all of this plugins apps
for (const app of plugin.apps) {
modules = union(modules, app.getModules());
for (const app of uiApps) {
if (app.getPluginId() === pluginId) {
modules = union(modules, app.getModules());
}
}

testGlobs.push(`${plugin.publicDir}/**/__tests__/**/*.js`);
});
} else {

// add the modules from all of the apps
for (const app of apps) {
for (const app of uiApps) {
modules = union(modules, app.getModules());
}

Expand All @@ -52,24 +69,17 @@ export default (kibana) => {
for (const f of testFiles) modules.push(f);

if (config.get('tests_bundle.instrument')) {
env.addPostLoader({
uiBundles.addPostLoader({
test: /\.js$/,
exclude: /[\/\\](__tests__|node_modules|bower_components|webpackShims)[\/\\]/,
loader: 'istanbul-instrumenter'
loader: 'istanbul-instrumenter-loader'
});
}

env.defaultUiSettings = plugins.kbnServer.uiExports.consumers
// find the first uiExportsConsumer that has a getUiSettingDefaults method
// See src/ui/ui_settings/ui_exports_consumer.js
.find(consumer => typeof consumer.getUiSettingDefaults === 'function')
.getUiSettingDefaults();

return new UiBundle({
uiBundles.add({
id: 'tests',
modules: modules,
template: require('./tests_entry_template'),
env: env
modules,
template: createTestEntryTemplate(uiSettingDefaults),
});
},

Expand Down
22 changes: 4 additions & 18 deletions src/core_plugins/tests_bundle/tests_entry_template.js
Original file line number Diff line number Diff line change
@@ -1,24 +1,12 @@
import { esTestConfig } from '../../test_utils/es';

export default function ({ env, bundle }) {

const pluginSlug = env.pluginInfo.sort()
.map(p => ' * - ' + p)
.join('\n');

const requires = bundle.modules
.map(m => `require(${JSON.stringify(m)});`)
.join('\n');

return `
export const createTestEntryTemplate = (defaultUiSettings) => (bundle) => `
/**
* Test entry file
*
* This is programatically created and updated, do not modify
*
* context: ${JSON.stringify(env.context)}
* includes code from:
${pluginSlug}
* context: ${bundle.getContext()}
*
*/

Expand Down Expand Up @@ -47,14 +35,12 @@ window.__KBN__ = {
}
},
uiSettings: {
defaults: ${JSON.stringify(env.defaultUiSettings, null, 2).split('\n').join('\n ')},
defaults: ${JSON.stringify(defaultUiSettings, null, 2).split('\n').join('\n ')},
user: {}
}
};

require('ui/test_harness');
${requires}
${bundle.getRequires().join('\n')}
require('ui/test_harness').bootstrap(/* go! */);
`;

}
13 changes: 1 addition & 12 deletions src/core_plugins/timelion/index.js
Original file line number Diff line number Diff line change
@@ -1,15 +1,4 @@
export default function (kibana) {
let mainFile = 'plugins/timelion/app';

const ownDescriptor = Object.getOwnPropertyDescriptor(kibana, 'autoload');
const protoDescriptor = Object.getOwnPropertyDescriptor(kibana.constructor.prototype, 'autoload');
const descriptor = ownDescriptor || protoDescriptor || {};
if (descriptor.get) {
// the autoload list has been replaced with a getter that complains about
// improper access, bypass that getter by seeing if it is defined
mainFile = 'plugins/timelion/app_with_autoload';
}

return new kibana.Plugin({
require: ['kibana', 'elasticsearch'],
uiExports: {
Expand All @@ -18,7 +7,7 @@ export default function (kibana) {
order: -1000,
description: 'Time series expressions for everything',
icon: 'plugins/timelion/icon.svg',
main: mainFile,
main: 'plugins/timelion/app',
injectVars: function (server) {
const config = server.config();
return {
Expand Down
1 change: 1 addition & 0 deletions src/core_plugins/timelion/public/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { SavedObjectRegistryProvider } from 'ui/saved_objects/saved_object_regis
import { notify } from 'ui/notify';
import { timezoneProvider } from 'ui/vis/lib/timezone';

require('ui/autoload/all');
require('plugins/timelion/directives/cells/cells');
require('plugins/timelion/directives/fixed_element');
require('plugins/timelion/directives/fullscreen/fullscreen');
Expand Down
2 changes: 0 additions & 2 deletions src/core_plugins/timelion/public/app_with_autoload.js

This file was deleted.

21 changes: 10 additions & 11 deletions src/optimize/base_optimizer.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,7 @@ const BABEL_EXCLUDE_RE = [

export default class BaseOptimizer {
constructor(opts) {
this.env = opts.env;
this.bundles = opts.bundles;
this.uiBundles = opts.uiBundles;
this.profile = opts.profile || false;

switch (opts.sourceMaps) {
Expand Down Expand Up @@ -60,7 +59,7 @@ export default class BaseOptimizer {
this.compiler.plugin('done', stats => {
if (!this.profile) return;

const path = resolve(this.env.workingDir, 'stats.json');
const path = this.uiBundles.resolvePath('stats.json');
const content = JSON.stringify(stats.toJson());
writeFile(path, content, function (err) {
if (err) throw err;
Expand All @@ -71,7 +70,7 @@ export default class BaseOptimizer {
}

getConfig() {
const cacheDirectory = resolve(this.env.workingDir, '../.cache', this.bundles.hashBundleEntries());
const cacheDirectory = this.uiBundles.getCachePath();

function getStyleLoaders(preProcessors = [], postProcessors = []) {
return ExtractTextPlugin.extract({
Expand Down Expand Up @@ -105,13 +104,13 @@ export default class BaseOptimizer {
const commonConfig = {
node: { fs: 'empty' },
context: fromRoot('.'),
entry: this.bundles.toWebpackEntries(),
entry: this.uiBundles.toWebpackEntries(),

devtool: this.sourceMaps,
profile: this.profile || false,

output: {
path: this.env.workingDir,
path: this.uiBundles.getWorkingDir(),
filename: '[name].bundle.js',
sourceMapFilename: '[file].map',
publicPath: PUBLIC_PATH_PLACEHOLDER,
Expand Down Expand Up @@ -168,7 +167,7 @@ export default class BaseOptimizer {
},
{
test: /\.js$/,
exclude: BABEL_EXCLUDE_RE.concat(this.env.noParse),
exclude: BABEL_EXCLUDE_RE.concat(this.uiBundles.getWebpackNoParseRules()),
use: [
{
loader: 'cache-loader',
Expand All @@ -187,12 +186,12 @@ export default class BaseOptimizer {
},
],
},
...this.env.postLoaders.map(loader => ({
...this.uiBundles.getPostLoaders().map(loader => ({
enforce: 'post',
...loader
})),
],
noParse: this.env.noParse,
noParse: this.uiBundles.getWebpackNoParseRules(),
},

resolve: {
Expand All @@ -205,12 +204,12 @@ export default class BaseOptimizer {
'node_modules',
fromRoot('node_modules'),
],
alias: this.env.aliases,
alias: this.uiBundles.getAliases(),
unsafeCache: this.unsafeCache,
},
};

if (this.env.context.env === 'development') {
if (this.uiBundles.isDevMode()) {
return webpackMerge(commonConfig, {
// In the test env we need to add react-addons (and a few other bits) for the
// enzyme tests to work.
Expand Down
2 changes: 1 addition & 1 deletion src/optimize/bundles_route/__tests__/bundles_route.js
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ describe('optimizer/bundle route', () => {
const server = createServer();

const response = await server.inject({
url: '/bundles/non_existant.js'
url: '/bundles/non_existent.js'
});

expect(response.statusCode).to.be(404);
Expand Down
19 changes: 10 additions & 9 deletions src/optimize/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,21 @@ export default async (kbnServer, server, config) => {
return await kbnServer.mixin(require('./lazy/lazy'));
}

const bundles = kbnServer.bundles;
const { uiBundles } = kbnServer;
server.route(createBundlesRoute({
bundlesPath: bundles.env.workingDir,
bundlesPath: uiBundles.getWorkingDir(),
basePublicPath: config.get('server.basePath')
}));

await bundles.writeEntryFiles();
await uiBundles.writeEntryFiles();

// in prod, only bundle when someing is missing or invalid
const invalidBundles = config.get('optimize.useBundleCache') ? await bundles.getInvalidBundles() : bundles;
const reuseCache = config.get('optimize.useBundleCache')
? await uiBundles.areAllBundleCachesValid()
: false;

// we might not have any work to do
if (!invalidBundles.getIds().length) {
if (reuseCache) {
server.log(
['debug', 'optimize'],
`All bundles are cached and ready to go!`
Expand All @@ -39,21 +41,20 @@ export default async (kbnServer, server, config) => {

// only require the FsOptimizer when we need to
const optimizer = new FsOptimizer({
env: bundles.env,
bundles: bundles,
uiBundles,
profile: config.get('optimize.profile'),
sourceMaps: config.get('optimize.sourceMaps'),
unsafeCache: config.get('optimize.unsafeCache'),
});

server.log(
['info', 'optimize'],
`Optimizing and caching ${bundles.desc()}. This may take a few minutes`
`Optimizing and caching ${uiBundles.getDescription()}. This may take a few minutes`
);

const start = Date.now();
await optimizer.run();
const seconds = ((Date.now() - start) / 1000).toFixed(2);

server.log(['info', 'optimize'], `Optimization of ${bundles.desc()} complete in ${seconds} seconds`);
server.log(['info', 'optimize'], `Optimization of ${uiBundles.getDescription()} complete in ${seconds} seconds`);
};
Loading