feat(transform funcs): stepped transform functions, protected execution #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
after a chat with @dustmop we came to the conclusion that we need more control over what kind of things the user can do and when they can do them in the context of a transformation. The first example we could think of is preventing a transform script that lists a peers datasets & posting them to a random server with the http module. To solve this we're restructuring the execution of transform steps as a set of functions with standard names that the user can declare, and we will call. The stipulation is the output of each of these fuctions can only be data. Each of these opt-in "steps" execute in a predeclared order, with the result of the previous step being the input to the next step. The input to the initial step is any provided dataset data.
dustmop
reviewed
Jun 4, 2018
lib/qri/qri.go
Outdated
|
|
||
| // Load creates a skylark module from a module instance | ||
| func (m *Module) Load() (skylark.StringDict, error) { | ||
| func NewModule(ds *dataset.Dataset, secrets map[string]interface{}) (skylark.StringDict, error) { |
There was a problem hiding this comment.
Since this is no longer returning a Module, how about renaming to NewEnvironment?
after some thought & conversation, we've decided to go a different route with transform functions. Instead of using load methods & protectors, we'll provide each transform func with an "environment object" that comes loaded with the things they can do within the context of that transform function. We think this'll make it easier to reason about what can happen when.
b5
force-pushed
the
dont_call_us_we_call_u
branch
from
b01ab44
to
6bea3cb
Compare
b5
changed the title
dustmop
approved these changes
Jun 4, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
after a chat with @dustmop we came to the conclusion that we need more control over what kind of things the user can do and when they can do them in the context of a transformation. The first example we could think of is preventing a transform script that lists a peers datasets & posting them to a random server with the http module.
To solve this we're restructuring the execution of transform steps, based on the
init, mainpattern from go/c-type programs, or thesetup, drawpattern from various video game engines. In all cases users can optionally declare as a set of functions with standard names & signatures that the user can declare, and qri will call them. The stipulation is the output of each of these functions can only be data. Each of these opt-in "steps" execute in a predeclared order, with the result of the previous step being the input to the next step. The input to the initial step is any provided dataset data.Current steps:
download, transformWhat I'm excited about here is we can later add steps like
mapandreduce, and execute those in parallel environments.