Initial threading implementation #23301

rashidkpc · 2018-09-18T22:05:33Z

This is a minimal threading implementation for Canvas. There's still a lot to be done to make this concept great, but this is a start.

What it does:

Creates a server side abstraction on top of the interpreter
Determines where to send the expression by checking the first function to be run
Loads common functions in a separate worker thread on the server.
Routes to a single forked worker (thread), the main thread (server), or the browser (browser), in that order
Defers back to the router when a function isn't found. Fails if the function isn't found in any of the above 3 environments
Times out the worker if it takes too long, and respawns it as needed.
Simplifies the error dialog to remove the stack.

What is does not.:

Round robin a pool of workers
Queue. If one expression in the threaded env fails then anything sent to it in the meantime will fail. The upstream environment handles managing timeouts. I think this would only make sense todo with a pool.
Client side. This doesn't implement web workers, but we could use roughly the same architecture.
Implement a specific, pluggable worker environment on the server. Right now it's just common functions, so plugin authors will always end up in a thread if they put their function in the common directory.

What I don't like:

The socketProvider code. This was reused across the server & browser, but now that it's only used in the browser there's no good reason for the abstraction
The serialize/deserialize stuff feels messy. Do we really need serialization?

x-pack/plugins/canvas/common/interpreter/create_error.js

x-pack/plugins/canvas/public/components/error/error.js

x-pack/plugins/canvas/server/routes/socket.js

x-pack/plugins/canvas/server/lib/route_expression/thread/worker.js

elasticmachine · 2018-10-19T21:44:39Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-10-20T00:09:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

x-pack/plugins/canvas/public/components/app/app.js

x-pack/plugins/canvas/public/lib/browser_registries.js

rashidkpc · 2018-10-22T21:58:32Z

x-pack/plugins/canvas/server/lib/route_expression/thread/index.js

+    worker = getWorker();
+  });
+
+  worker.on('message', msg => {


Do i need to clean these up or does worker.kill do that?

x-pack/plugins/canvas/server/lib/server_registries.js

x-pack/plugins/canvas/server/routes/socket.js

elasticmachine · 2018-10-23T02:08:20Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

w33ble · 2018-10-25T20:43:00Z

Closing in favor of #24598

Replaces #23301 Closes #23080 --- This is a minimal threading implementation for Canvas. There's still a lot to be done to make this concept great, but this is a start. What it does: - Creates a server side abstraction on top of the interpreter - Determines where to send the expression by checking the first function to be run - Loads common functions in a separate worker thread on the server. - Routes to a single forked worker (thread), the main thread (server), or the browser (browser), in that order - Defers back to the router when a function isn't found. Fails if the function isn't found in any of the above 3 environments - Times out the worker if it takes too long, and respawns it as needed. - Simplifies the error dialog to remove the stack. What is does not.: - Round robin a pool of workers - Queue. If one expression in the threaded env fails then anything sent to it in the meantime will fail. The upstream environment handles managing timeouts. I think this would only make sense todo with a pool. - Client side. This doesn't implement web workers, but we could use roughly the same architecture. - Implement a specific, pluggable `worker` environment on the server. Right now it's just common functions, so plugin authors will always end up in a thread if they put their function in the common directory. What I don't like: - The socketProvider code. This was reused across the server & browser, but now that it's only used in the browser there's no good reason for the abstraction - The serialize/deserialize stuff feels messy. Do we really need serialization?

* Translate global navigation bar component (#23993) Translate global navigation bar component * [backport] add back earlier 6.x minor versions We still backport to these branches, primarily for doc changes. * [dev/build] fix invalid assertion * Skip this test until snapshots are updated (#24650) * Feat/expression threading (#24598) Replaces #23301 Closes #23080 --- This is a minimal threading implementation for Canvas. There's still a lot to be done to make this concept great, but this is a start. What it does: - Creates a server side abstraction on top of the interpreter - Determines where to send the expression by checking the first function to be run - Loads common functions in a separate worker thread on the server. - Routes to a single forked worker (thread), the main thread (server), or the browser (browser), in that order - Defers back to the router when a function isn't found. Fails if the function isn't found in any of the above 3 environments - Times out the worker if it takes too long, and respawns it as needed. - Simplifies the error dialog to remove the stack. What is does not.: - Round robin a pool of workers - Queue. If one expression in the threaded env fails then anything sent to it in the meantime will fail. The upstream environment handles managing timeouts. I think this would only make sense todo with a pool. - Client side. This doesn't implement web workers, but we could use roughly the same architecture. - Implement a specific, pluggable `worker` environment on the server. Right now it's just common functions, so plugin authors will always end up in a thread if they put their function in the common directory. What I don't like: - The socketProvider code. This was reused across the server & browser, but now that it's only used in the browser there's no good reason for the abstraction - The serialize/deserialize stuff feels messy. Do we really need serialization? * Polish 6.5 (#24556) * Updates waterfall item design for timeline rows * Adjusts span and tx flyouts and updates tooltips to EUI * Heading size fixes and clean up * Updates tooltip snapshots * Review tweaks and snapshot updates * Revert experiment :) Co-Authored-By: jasonrhodes <jason.matthew.rhodes@gmail.com> * Fixes bug with v1 waterfall state * Fixes bug with timeline bar height * Updates snapshot tests * Updated test so it doesn't mount and rely on EUI makeId() which is non-deterministic per test run * Don't throw errors in optimizer (#24660) * Fixed label position on progress elements (#24623) * [kbn/es] add context to error message (#24664) This just tweaks the kbn-es error message to provide more context than just `Not Found` * [BeatsCM] Beats without tags should return an empty array via the config API (#24665) * [ML] Change file data visualizer JSON format label to NDJSON (#24643) * [ML] Change file datavisualizer JSON format label to NDJSON * [ML] Update edit flyout overrides snapshot * Translations for Coordinate Map (#23952) translate Coordinate Map * Translations for Region Map (#23875) add translations for region_map plugin * [Tools] Add TemplateLiteral parsing to i18n_check tool (#24580) * [Tools] Add TemplateLiteral parsing to i18n_check tool * Add comments * [ML] Remove obsolete sentence from info tooltip. (#24716) * Translate security/users component (#23940) Translate security/users * [Docs] Remove beta notes for ML and Query bar (#24718) * Translations for Table Vis plugin (#23679) add translations for table vis plugin * Feature/translate new nav bar (#24326) translate new_nav_bar * center content in fullscreen mode, hide K7 top nav (#24589) * [APM] Fixes rare cases where KibanaLink is loaded outside of React context (#24705) * Fixes rare cases where KibanaLink will be loaded outside of React context and requires no redux connect dependency * Fixes tests for updated Kibana link component * Removes obsolete snapshot * Secops structure code (#24652) * add basic structure for secops application * finalize skeleton for secops * fix type issue and hapi new version * remove route home, not needed for now * Add configuration + delete noise * prepend elastic license to generated file * Cut down on all tests except for secops tests and one example of infr… (#24693) * Cut down on all tests except for secops tests and one example of infra integration tests * Commented out code for only this branch * Added comments and "please see issue number" * https://github.com/elastic/ingest-dev/issues/60

Rashid Khan added 3 commits September 17, 2018 14:29

Migrate to kibana

c80ee1f

WIP

c7fb927

Everything works

d70211a

rashidkpc added v7.0.0 v6.5.0 Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas labels Sep 18, 2018

rashidkpc requested a review from w33ble September 18, 2018 22:05

rashidkpc changed the title ~~WIP Initial threading implementation~~ [WIP] Initial threading implementation Sep 18, 2018

Rashid Khan added 10 commits September 19, 2018 11:47

Deserialize when returning from browser

b977ac3

Merge branch 'master' of github.com:elastic/kibana into canvas/thread

3a065ce

Fix

1118958

Merge branch 'master' of github.com:elastic/kibana into canvas/thread

1dc24b9

WIP

78178a2

Restart worker as needed

a6e680c

Better error handling

f0d1a7e

Removed debug info in errors. Security, etc

0ced442

Merge branch 'master' of github.com:elastic/kibana into canvas/thread

90531f2

Fix request timeout error passing

c6823e9

rashidkpc changed the title ~~[WIP] Initial threading implementation~~ Initial threading implementation Oct 18, 2018

rashidkpc added the review label Oct 18, 2018

elastic deleted a comment from elasticmachine Oct 19, 2018