Skip to content
This repository has been archived by the owner on May 29, 2021. It is now read-only.

Implement dependency injection into workers, resolves #203 #211

Merged
merged 2 commits into from Nov 5, 2017

Conversation

mvertes
Copy link
Contributor

@mvertes mvertes commented Nov 4, 2017

A new skale context method sc.require is added. It specifies a
set of modules on user side (master) which have to be deployed in
workers for use by callbacks, such as mappers, reducers, etc.

Under the hood, browserify is used on master side to build a
bundle which is serialized and sent to workers (as part of task).
It is then evaluated in worker global context, and dependencies
remain persistent as long as workers live.

This method allows to use in workers any javascript module which
can be browserified, so a large number (almost any pure JS package).

The current commit is for local version, not distributed (code will
be exactly the same). It is experimental for the moment.

An additional statement sc.bundle could be added as well to inject
pre-compiled modules, avoiding the penalty of browserify at each
run.

A new skale context method `sc.require` is added. It specifies a
set of modules on user side (master) which have to be deployed in
workers for use by callbacks, such as mappers, reducers, etc.

Under the hood, [browserify] is used on master side to build a
bundle which is serialized and sent to workers (as part of task).
It is then evaluated in worker global context, and dependencies
remain persistent as long as workers live.

This method allows to use in workers any javascript module which
can be browserified, so a large number (almost any pure JS package).

The current commit is for local version, not distributed (code will
be exactly the same). It is experimental for the moment.

An additional statement `sc.bundle` could be added as well to inject
pre-compiled modules, avoiding the penalty of `browserify` at each
run.

[browserify]: https://github.com/browserify/browserify
@CedricArtigue
Copy link
Contributor

That's really good. I wonder if we can prevent users from having to explicitly call sc.require by just bundling required modules unconditionally. We can find all non-core required modules with require.cache.

It may be much more natural this way as user shouldn't have to think wether the required module will be used on master or worker side.

@mvertes
Copy link
Contributor Author

mvertes commented Nov 5, 2017

Yes, automatic require could be an evolution once feature is stabilized. First, let fix distributed mode, then validate with real-life complex dependencies, such as kafka, cassandra, etc... which may require tuning browserify and/or worker internals.

@mvertes mvertes merged commit 9ab3bfc into skale-me:master Nov 5, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants