Create an artifact plugin #55

Merged
merged 13 commits into from Mar 31, 2016

Projects

None yet

5 participants

@imbstack
Member

Does what it says on the tin.

@imbstack
Member

@jonasfj, @gregarndt: How does this payload schema look to you? Let's change it to be what we want.

@jonasfj jonasfj commented on an outdated diff Mar 11, 2016
plugins/artifacts/config-schema.yml
+title: config
+description: Artifacts to be published
+type: array
+items:
+ type: object
+ properties:
+ type:
+ title: Upload type
+ description: Artifacts can be either an individual `file` or a `directory` containing potentially multiple files with recursively included subdirectories
+ type: string
+ enum: [file, directory]
+ localPath:
+ title: Local artifact location
+ description: Filesystem path of artifact
+ type: string
+ remotePath:
@jonasfj
jonasfj Mar 11, 2016 Member

I propose path and name...

@jonasfj
jonasfj Mar 11, 2016 Member

remotePath/name should not start with /...

If type is folder it should end with /, and if type is file it should not ever end with /...

Also it should never contain any double slashes...

@gregarndt
Member

Looks good, we just need to get a consensus on the naming of the source and destination stuff.

@selenamarie selenamarie changed the title from [WIP] Create and artifact plugin (Don't merge yet) to [WIP] Create an artifact plugin (Don't merge yet) Mar 11, 2016
@jonasfj
Member
jonasfj commented Mar 13, 2016

I propose path and name... I see the benefit of going with artifactName, the only downside is that everywhere else in artifact end-point documentation it is referred to as <name>..

Honestly, I can't care much... As long as we don't call it remotePath because it is not a path.
Whatever we do, the JSON schema should document it with title: "Artifact Name".


Regarding source I would prefer sourcePath as it's always going to be a path. Just path works fine too... But I care less about that since it doesn't related to queue docs...

@imbstack
Member

I submit a further WIP diff to check and see if this is a good direction to be going in. I feel fairly happy with how this is shaping up. If this is a generally acceptable way of going about this, most of the changes from here on out will be filling out error handling in the plugin itself, and using the client lib stuff (#60) that you all are working on now in the runtime bits.

Currently the tests for this look like

$ go test -v ./plugins/artifacts/
=== RUN   TestArtifactsEmpty
--- PASS: TestArtifactsEmpty (0.00s)
=== RUN   TestArtifactsFile
/public/blah.txt
Hello World
--- PASS: TestArtifactsFile (0.00s)
=== RUN   TestArtifactsDirectory
/public
Hello World
/public
Hello World
/public
Hello World
--- PASS: TestArtifactsDirectory (0.00s)
PASS
ok      github.com/taskcluster/taskcluster-worker/plugins/artifacts     0.017s
@imbstack
Member

A note for @gregarndt: We decided to move ReadSeekCloser to runtime to avoid dependency loops between runtime and engine.

@coveralls

Coverage Status

Coverage increased (+0.1%) to 70.511% when pulling 069cf17 on artifacts into 197638a on master.

@coveralls

Coverage Status

Coverage increased (+0.1%) to 75.214% when pulling c7f400b on artifacts into ea81ad1 on master.

@imbstack
Member

I'd love if I could get a 30% review on this at some point today, just so that we all feel like this is moving in the right direction. As a caveat, nothing is done and the points don't matter. If this might be better done as a guided excercise at this point, I'd be happy to hop on vidyo and talk it through with someone.

@jonasfj, @gregarndt: any takers?

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+
+func (pluginProvider) ConfigSchema() runtime.CompositeSchema {
+ return runtime.NewEmptyCompositeSchema()
+}
+
+type plugin struct {
+ plugins.PluginBase
+}
+
+func (plugin) PayloadSchema() (runtime.CompositeSchema, error) {
+ return configSchema, nil
+}
+
+func (plugin) NewTaskPlugin(options plugins.TaskPluginOptions) (plugins.TaskPlugin, error) {
+ return &taskPlugin{
+ TaskPluginBase: plugins.TaskPluginBase{},
@jonasfj
jonasfj Mar 25, 2016 Member

No need for this line... the zero-value is assigned by default...

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
@@ -0,0 +1,110 @@
+//go:generate go-composite-schema --unexported --required artifacts config-schema.yml generated_configschema.go
@jonasfj
jonasfj Mar 25, 2016 Member

this isn't config, it's payload... let's rename...

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/config-schema.yml
@@ -0,0 +1,28 @@
+$schema: http://json-schema.org/draft-04/schema#
+title: config
@jonasfj
jonasfj Mar 25, 2016 Member

this is payload not config.

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+ }, tp.context)
+ }
+ case "file":
+ fileReader, err := result.ExtractFile(artifact.Path)
+ if err != nil {
+ runtime.CreateErrorArtifact(runtime.ErrorArtifact{
+ Message: fmt.Sprintf("Could not read file '%s'", artifact.Path),
+ Reason: "file-missing-on-worker",
+ Expires: artifact.Expires,
+ }, tp.context)
+ }
+ tp.attemptUpload(fileReader, artifact.Path, artifact.Name, artifact.Expires)
+ }
+ }
+ // TODO: Don't always return true?
+ return true, nil
@jonasfj
jonasfj Mar 25, 2016 Member

If there was errors uploading artifacts (and we used all retries)... we return the http error... or wrap it... this will be a fatal error, no way around... Maybe later we'll make that an internal error.

If there was an illegal filename, feature not supported, MalformedPayloadError, then we create a MalformedPayloadError explaining what when wrong...

In the face of: ErrResourceNotFound
We return false, nil, and we print to log that the task failed because the file or folder was missing.

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+
+type taskPlugin struct {
+ plugins.TaskPluginBase
+ context *runtime.TaskContext
+ payload config
+}
+
+func (tp *taskPlugin) Prepare(context *runtime.TaskContext) error {
+ tp.context = context
+ return nil
+}
+
+func (tp *taskPlugin) Stopped(result engines.ResultSet) (bool, error) {
+ for _, artifact := range tp.payload {
+ switch artifact.Type {
+ case "directory":
@jonasfj
jonasfj Mar 25, 2016 Member

I suspect splitting this into two methods will help..

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+ plugins.TaskPluginBase
+ context *runtime.TaskContext
+ payload config
+}
+
+func (tp *taskPlugin) Prepare(context *runtime.TaskContext) error {
+ tp.context = context
+ return nil
+}
+
+func (tp *taskPlugin) Stopped(result engines.ResultSet) (bool, error) {
+ for _, artifact := range tp.payload {
+ switch artifact.Type {
+ case "directory":
+ err := result.ExtractFolder(artifact.Path, tp.createUploadHandler(artifact.Name, artifact.Expires))
+ if err != nil {
@jonasfj
jonasfj Mar 25, 2016 Member

other errors could happen, but you are required to handle some of them... See:
https://github.com/taskcluster/taskcluster-worker/blob/ea81ad1e3f3de1c876fc3b1ebeda298a60dc7af0/engines/resultset.go#L79-L80

I would say that ErrHandlerInterrupt only happens if you do it...
and ErrNonFatalInternalError probably can't be handled...

If you can't handle errors you forward them... Random errors inside a engine should be fatal.. We do this by returning them...

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+func (tp *taskPlugin) Stopped(result engines.ResultSet) (bool, error) {
+ for _, artifact := range tp.payload {
+ switch artifact.Type {
+ case "directory":
+ err := result.ExtractFolder(artifact.Path, tp.createUploadHandler(artifact.Name, artifact.Expires))
+ if err != nil {
+ runtime.CreateErrorArtifact(runtime.ErrorArtifact{
+ Message: fmt.Sprintf("Could not open directory '%s'", artifact.Path),
+ Reason: "invalid-resource-on-worker",
+ Expires: artifact.Expires,
+ }, tp.context)
+ }
+ case "file":
+ fileReader, err := result.ExtractFile(artifact.Path)
+ if err != nil {
+ runtime.CreateErrorArtifact(runtime.ErrorArtifact{
@jonasfj
jonasfj Mar 25, 2016 Member

Only, in case of ErrResourceNotFound

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+ return true, nil
+}
+
+func (tp taskPlugin) createUploadHandler(name string, expires tcclient.Time) func(string, ioext.ReadSeekCloser) error {
+ return func(path string, stream ioext.ReadSeekCloser) error {
+ return tp.attemptUpload(stream, path, name, expires)
+ }
+}
+
+func (tp taskPlugin) attemptUpload(fileReader ioext.ReadSeekCloser, path string, name string, expires tcclient.Time) error {
+ mimeType := mime.TypeByExtension(filepath.Ext(path))
+ if mimeType == "" {
+ // application/octet-stream is the mime type for "unknown"
+ mimeType = "application/octet-stream"
+ }
+ runtime.UploadS3Artifact(runtime.S3Artifact{
@jonasfj
jonasfj Mar 25, 2016 Member

I assume UploadS3Artifact can return an error

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+type taskPlugin struct {
+ plugins.TaskPluginBase
+ context *runtime.TaskContext
+ payload config
+}
+
+func (tp *taskPlugin) Prepare(context *runtime.TaskContext) error {
+ tp.context = context
+ return nil
+}
+
+func (tp *taskPlugin) Stopped(result engines.ResultSet) (bool, error) {
+ for _, artifact := range tp.payload {
+ switch artifact.Type {
+ case "directory":
+ err := result.ExtractFolder(artifact.Path, tp.createUploadHandler(artifact.Name, artifact.Expires))
@jonasfj
jonasfj Mar 25, 2016 Member

This isn't enough... you need to handle ErrHandlerInterrupt which indicates that an upload failed... but the actual error returned by the handler is eaten, so you have store that in a separate variable... or something like that...

@jonasfj jonasfj commented on an outdated diff Mar 25, 2016
runtime/artifact.go
+ parsp, _, err := context.Queue().CreateArtifact(
+ context.TaskID,
+ strconv.Itoa(context.RunID),
+ artifact.Name,
+ &par,
+ )
+ if err != nil {
+ // TODO: Do something with all of these errors
+ }
+ var resp queue.S3ArtifactResponse
+ err = json.Unmarshal(json.RawMessage(*parsp), &resp)
+ if err != nil {
+ // TODO: Do something with all of these errors
+ }
+
+ putArtifact(resp.PutURL, artifact.Mimetype, artifact.Stream)
@jonasfj
jonasfj Mar 25, 2016 Member

this can return an error

@jonasfj jonasfj commented on the diff Mar 25, 2016
runtime/artifact.go
+ attempts := 0
+ client := &http.Client{}
+ for {
+ attempts++
+ stream.Seek(0, 0)
+ req := &http.Request{
+ Method: "PUT",
+ URL: u,
+ Proto: "HTTP/1.1",
+ ProtoMajor: 1,
+ ProtoMinor: 1,
+ Header: header,
+ Body: stream,
+ ContentLength: contentLength,
+ }
+ resp, err := client.Do(req)
@jonasfj
jonasfj Mar 25, 2016 Member

@imbstack, this is fine short term...

@gregarndt, @imbstack, long term I think we need TaskContext to implement the Context interface from:
https://godoc.org/golang.org/x/net/context

Then we can use: https://godoc.org/golang.org/x/net/context/ctxhttp
Here and ensure that the http request stops uploading if the task is called, or some other plugin returns an error while we are uploading artifacts... then the plugin manager can cancel...(okay for plugin manager to cancel it would have to wrap it somehow)...

So for plugin manager to cancel we would have to pass a Context object with every request. And this would have to be a separate Context object, not just TaskContext...

So maybe:

  1. TaskContext implmenets the Context interface
  2. Every hooks on plugin takes a Context interface as first parameter

Then pluginManager will call plugin.Prepare(Context, TaskContext), and plugin.Stopped(Context, ResultSet), where Context is TaskContext.WithCancel()

@jonasfj
jonasfj Mar 25, 2016 Member

maybe this is something we figure out later... and we just don't support aborting uploads due to cancel... which seems okay,

@gregarndt
gregarndt Mar 25, 2016 Member

I think that if we are in the middle of canceling the most time consuming resource intensive thing has already been done and canceling during this might be overkill.

@jonasfj
jonasfj Mar 25, 2016 Member

it certainly could be overkill... maybe even undesired... Since the artifacts are ready to upload..

@jonasfj jonasfj and 1 other commented on an outdated diff Mar 25, 2016
runtime/ioext/ioext.go
+import (
+ "io"
+)
+
+// ReadSeekCloser implements io.Reader, io.Seeker, and io.Closer. It is trivially implemented by os.File.
+type ReadSeekCloser interface {
+ io.Reader
+ io.Seeker
+ io.Closer
+}
+
+func NopCloser(r io.Reader) ReadSeekNopCloser {
+ return ReadSeekNopCloser{r.(io.ReadSeeker)}
+}
+
+type ReadSeekNopCloser struct {
@jonasfj
jonasfj Mar 25, 2016 Member

This type should not be exported

@imbstack
imbstack Mar 29, 2016 Member

It is used outside of runtime, we end up needing to export it.

@jonasfj jonasfj and 1 other commented on an outdated diff Mar 25, 2016
runtime/ioext/ioext.go
@@ -0,0 +1,24 @@
+package ioext
+
+import (
+ "io"
+)
+
+// ReadSeekCloser implements io.Reader, io.Seeker, and io.Closer. It is trivially implemented by os.File.
+type ReadSeekCloser interface {
+ io.Reader
+ io.Seeker
+ io.Closer
+}
+
+func NopCloser(r io.Reader) ReadSeekNopCloser {
@jonasfj
jonasfj Mar 25, 2016 Member

Exported functions should have a documentation comment

@imbstack
imbstack Mar 25, 2016 Member

Haha, yes. I'll circle back and get these things before we do a real review. Should've defined "30% review" before I asked for one. I just use it to mean something between a design review and checking to see if everything is documented correctly 😄. There's still a great bit of work to do before this is ready to be landed.

@jonasfj
jonasfj Mar 25, 2016 Member

Sure, I was just commenting things I saw :)
bad habit... Should have focused on high-level stuff.. Which seems okay,
well, there is some in error handling.

Regards Jonas Finnemann Jensen.

2016-03-25 11:45 GMT-07:00 Brian Stack notifications@github.com:

In runtime/ioext/ioext.go
#55 (comment)
:

@@ -0,0 +1,24 @@
+package ioext
+
+import (

  • "io"
    +)

+// ReadSeekCloser implements io.Reader, io.Seeker, and io.Closer. It is trivially implemented by os.File.
+type ReadSeekCloser interface {

  • io.Reader
  • io.Seeker
  • io.Closer
    +}

+func NopCloser(r io.Reader) ReadSeekNopCloser {

Haha, yes. I'll circle back and get these things before we do a real
review. Should've defined "30% review" before I asked for one. I just use
it to mean something between a design review and checking to see if
everything is documented correctly [image: 😄]. There's still a
great bit of work to do before this is ready to be landed.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/taskcluster/taskcluster-worker/pull/55/files/641d4358cbd8bfb1d63a428002d6cd29f2372f18#r57476233

@imbstack
imbstack Mar 25, 2016 Member

No worries! I just didn't want you to have to do all of this work this soon.

@jonasfj jonasfj commented on the diff Mar 25, 2016
runtime/ioext/ioext.go
@@ -0,0 +1,24 @@
+package ioext
@jonasfj
jonasfj Mar 25, 2016 Member

Package needs a doc comment... You can do it here, or create a doc.go file with this...

@gregarndt gregarndt commented on the diff Mar 25, 2016
worker/task.go
@@ -173,7 +173,10 @@ func (t *TaskRun) parsePayload() error {
// createTaskPlugins will create a new task plugin to be used during the task lifecycle.
func (t *TaskRun) createTaskPlugins() error {
var err error
- popts := plugins.TaskPluginOptions{TaskInfo: &runtime.TaskInfo{}, Payload: t.pluginPayload}
+ popts := plugins.TaskPluginOptions{TaskInfo: &runtime.TaskInfo{
@gregarndt
gregarndt Mar 25, 2016 Member

Ah, good catch on this, thanks!

@gregarndt gregarndt commented on an outdated diff Mar 25, 2016
plugins/artifacts/artifacts.go
+}
+
+func (plugin) NewTaskPlugin(options plugins.TaskPluginOptions) (plugins.TaskPlugin, error) {
+ return &taskPlugin{
+ TaskPluginBase: plugins.TaskPluginBase{},
+ payload: *(options.Payload.(*config)),
+ }, nil
+}
+
+type taskPlugin struct {
+ plugins.TaskPluginBase
+ context *runtime.TaskContext
+ payload config
+}
+
+func (tp *taskPlugin) Prepare(context *runtime.TaskContext) error {
@gregarndt
gregarndt Mar 25, 2016 Member

I'm not sure if you want to do some validation here about the artifacts. Like making sure artifact expires is <= task.expires otherwise when you go to create the artifact the queue will complain.

@gregarndt gregarndt and 1 other commented on an outdated diff Mar 25, 2016
plugins/artifacts/config-schema.yml
+ enum: [file, directory]
+ path:
+ title: Artifact Path
+ description: Filesystem path of artifact
+ type: string
+ name:
+ title: Artifact Name
+ description: This will be the leading path to directories and the full name for files that are uploaded to s3
+ type: string
+ expires:
+ title: Expiry Tate and Time
+ description: Date when artifact should expire must be in the future
+ type: string
+ format: date-time
+
+ # TODO: Should expires be required?
@gregarndt
gregarndt Mar 25, 2016 Member

right now what we do is default it to task.expires (which is set by the queue if not explicitly declared (default is 1 year from creation I think))

@gregarndt
gregarndt Mar 25, 2016 Member

Note that artifact expiration is required when creating an artifact on the queue, so either we make it standard here as well that's it's required (likely another breaking change in this worker) or we accept that people might not specify it and if that's the care we should default it when creating the artifact.

@jonasfj
jonasfj Mar 25, 2016 Member

I would love of expires to be optional... and default to task.expires. That seems sane...

@imbstack imbstack changed the title from [WIP] Create an artifact plugin (Don't merge yet) to Create an artifact plugin Mar 29, 2016
@imbstack
Member

@gregarndt, @jonasfj: Alright, I think this is pretty much ready (barring inevitable review comments). In particular, I'm pretty sure the error handling I have in here is still overly simplistic, but it's at least moving in the right direction I think.

Example of created artifacts: https://tools.taskcluster.net/task-inspector/#WzLXbfiVTf2MHmaCEJFRmg/0
Example of error artifact: https://tools.taskcluster.net/task-inspector/#ObSgu516SnOhJdOuu3EMsQ/0

@jonasfj jonasfj was assigned by imbstack Mar 29, 2016
@jonasfj jonasfj commented on an outdated diff Mar 29, 2016
runtime/ioext/ioext.go
+
+// ReadSeekCloser implements io.Reader, io.Seeker, and io.Closer. It is trivially implemented by os.File.
+type ReadSeekCloser interface {
+ io.Reader
+ io.Seeker
+ io.Closer
+}
+
+// NopCloser is useful in testing where something that implements ReadSeekCloser is needed
+// If something that implements io.ReadSeeker is passed in, it will give it a noop close function.
+func NopCloser(r io.Reader) ReadSeekNopCloser {
+ return ReadSeekNopCloser{r.(io.ReadSeeker)}
+}
+
+// ReadSeekNopCloser is an implementation of ReadSeekCloser that wraps io.ReadSeekers
+type ReadSeekNopCloser struct {
@jonasfj
jonasfj Mar 29, 2016 Member

You don't need to export this type...

You create it with NopCloser whose return type should ReadSeekCloser

@jonasfj jonasfj commented on an outdated diff Mar 29, 2016
runtime/ioext/ioext.go
+package ioext
+
+import (
+ "io"
+)
+
+// ReadSeekCloser implements io.Reader, io.Seeker, and io.Closer. It is trivially implemented by os.File.
+type ReadSeekCloser interface {
+ io.Reader
+ io.Seeker
+ io.Closer
+}
+
+// NopCloser is useful in testing where something that implements ReadSeekCloser is needed
+// If something that implements io.ReadSeeker is passed in, it will give it a noop close function.
+func NopCloser(r io.Reader) ReadSeekNopCloser {
@jonasfj
jonasfj Mar 29, 2016 Member

return type should be ReadSeekCloser

@jonasfj jonasfj commented on the diff Mar 30, 2016
plugins/artifacts/artifacts.go
+
+func (pluginProvider) NewPlugin(extpoints.PluginOptions) (plugins.Plugin, error) {
+ return plugin{}, nil
+}
+
+type plugin struct {
+ plugins.PluginBase
+}
+
+func (plugin) PayloadSchema() (runtime.CompositeSchema, error) {
+ return payloadSchema, nil
+}
+
+func (plugin) NewTaskPlugin(options plugins.TaskPluginOptions) (plugins.TaskPlugin, error) {
+ if options.Payload == nil {
+ return plugins.TaskPluginBase{}, nil
@jonasfj
jonasfj Mar 30, 2016 Member

nice trick, because payload is optional right?

@imbstack
imbstack Mar 30, 2016 Member

Yeah, without this if the artifacts plugin was enabled, and no artifacts were specified, the whole thing would come to a grinding halt!

@jonasfj
jonasfj Mar 31, 2016 Member

maybe we should modify the plugin manager to ignore a plugin that returns nil, just so it'll be easier to disable things...

On the other hand this is really elegant :)

@jonasfj jonasfj commented on an outdated diff Mar 30, 2016
plugins/artifacts/artifacts.go
+ }
+ return false, err
+ }
+ err = tp.attemptUpload(fileReader, artifact.Path, artifact.Name, artifact.Expires)
+ if err != nil {
+ return false, err
+ }
+ }
+ }
+ return true, nil
+}
+
+func (tp taskPlugin) errorHandled(name string, expires tcclient.Time, err error) bool {
+ if err == engines.ErrFeatureNotSupported || err == engines.ErrResourceNotFound ||
+ err == engines.ErrNonFatalInternalError || err == engines.ErrHandlerInterrupt ||
+ reflect.TypeOf(err).String() == "engines.MalformedPayloadError" {
@jonasfj
jonasfj Mar 30, 2016 Member

Avoid using reflect...

Just make different if statement on the form:

if e, ok := err.(*engines.MalformedPayloadError); ok {
  // Now is e is a MalformedPayloadError type
}

Don't for forget the ok part, otherwise it'll panic on failure.

@jonasfj jonasfj commented on the diff Mar 30, 2016
plugins/artifacts/artifacts.go
+}
+
+func (tp *taskPlugin) Prepare(context *runtime.TaskContext) error {
+ tp.context = context
+ return nil
+}
+
+func (tp *taskPlugin) Stopped(result engines.ResultSet) (bool, error) {
+ var err error
+ for _, artifact := range tp.payload {
+ // If expires is set to this time it's either the default value or has been set to an invalid time anyway
+ if time.Time(artifact.Expires).IsZero() {
+ artifact.Expires = tp.context.TaskInfo.Expires
+ }
+ switch artifact.Type {
+ case "directory":
@jonasfj
jonasfj Mar 30, 2016 Member

hmm, How not to be a pain... But how about we do a method for each case...
Then do all uploads in parallel? Or is that insane?

Take a look at how PluginManager does it with waitgroup and atomicBool

@jonasfj
jonasfj Mar 30, 2016 Member

Since, the only error we're allowed to return is a MalformedPayloadError.
Perhaps all errors we get that aren't fatal should be combined into a single MalformedPayloadError.

Like a bullet point for each thing that was wrong :)

@jonasfj
jonasfj Mar 30, 2016 Member

In which case Atomic bool isn't much use to you, as you'll need a lock when appending the string anyways.

@jonasfj
jonasfj Mar 30, 2016 Member

note, this might be better left for a follow up patch...

@gregarndt
gregarndt Mar 30, 2016 Member

if there was an upload error because of s3/queue/ why would we only be allowed to return a malformed payload error?

@jonasfj
jonasfj Mar 30, 2016 Member

Any other error is fatal as per the interface defintion...

Sorry, you are allowed to return other errors, as long as you keep in mind that such errors are fatal.

@imbstack
imbstack Mar 30, 2016 Member

Perhaps we should update the interface definition to allow more non-fatal errors?

@jonasfj
jonasfj Mar 31, 2016 Member

Perhaps we should update the interface definition to allow more non-fatal errors?
What errors do you propose, and how do you propose we handle them?

Yeah, maybe we should allow: InternalError:
https://github.com/taskcluster/taskcluster-worker/blob/master/engines/errors.go#L117

With the assumption that this must only be used for errors that:
A) Cannot be trigger by the task payload, or behavior of the code running inside the sandbox,
B) We are confident will not affect the next task the worker runs

Most internal errors like out-of-disk-space, out-of-memory, cannot contact docker socket, container suddenly disappeared, file that I know I created don't exist anymore, missing hardware device, file system is corrupted, docker daemon died, do not satisfy condition B.

In cases like above, the sane thing is to report the task exception worker-shutdown, and then polity crash, probably terminate the instance.

In cases like: can't reach the internet, hostname doesn't resolve, credentials are expired or don't work, etc.
The worker may be able to report the task exception internal-error (have the task retried, we probably need to change queue semantics for that), and then try with the next task.

However, if we see a lot of these can't reach the internet, can't resolve DNS, can't upload to S3, credentials expired unexpectedly, etc... Then we should very much die..

@imbstack, We don't want tasks to retry too much, and we definitely don't want a corrupted worker to just churn through the queue, claim tasks and the report them exception... So we have to be extremely careful about what is a non-fatal internal error. Note: errors that potentially leaves something in a dirty state cannot be non-fatal.

If we want to pursue this, rather than just saying everything is fatal. Then we should probably rename InternalError to NonFatalInternalError or CleanInternalError, suggestions?
Then specify in doc string what it can be used for, and what it absolutely cannot be used for. Before we expand what the interfaces can return...

This could be a follow-up, please file a bug for those... :)

@gregarndt
gregarndt Mar 31, 2016 Member

We've discussed before about the use of internal-error and the semantics around retrying. I'm till on the fence with it. I think in the case of failure to upload, most of the time it's a transient issue that I think could be reported as internal-error and the next task that worker tries to complete will not fail for the same reason.

I definitely think in the cases of resource exhaustion or other issues that the worker knows about the host, the worker should shut itself down and resolve tasks as 'worker-shutdown'

@gregarndt
gregarndt Mar 31, 2016 Member

I don't see the need to label it as "NonFatalXXX" if there is not also a "FatalXXX".

Also, with fatal errors, because the shutdown logic is tied into the worker process, we need to make sure those that are treated as fatal are handled appropriately and the shutdown logic (when it exists) will get called rather than just panicking

@jonasfj jonasfj commented on an outdated diff Mar 30, 2016
plugins/artifacts/artifacts.go
+ }
+ case "file":
+ fileReader, err := result.ExtractFile(artifact.Path)
+ if err != nil {
+ if tp.errorHandled(artifact.Name, artifact.Expires, err) {
+ continue
+ }
+ return false, err
+ }
+ err = tp.attemptUpload(fileReader, artifact.Path, artifact.Name, artifact.Expires)
+ if err != nil {
+ return false, err
+ }
+ }
+ }
+ return true, nil
@jonasfj
jonasfj Mar 30, 2016 Member

if you created an error artifact you should return false, MalformedPayloadError

@jonasfj jonasfj commented on an outdated diff Mar 30, 2016
plugins/artifacts/artifacts.go
+ }
+ }
+ return true, nil
+}
+
+func (tp taskPlugin) errorHandled(name string, expires tcclient.Time, err error) bool {
+ if err == engines.ErrFeatureNotSupported || err == engines.ErrResourceNotFound ||
+ err == engines.ErrNonFatalInternalError || err == engines.ErrHandlerInterrupt ||
+ reflect.TypeOf(err).String() == "engines.MalformedPayloadError" {
+ runtime.CreateErrorArtifact(runtime.ErrorArtifact{
+ Name: name,
+ Message: err.Error(),
+ Reason: "invalid-resource-on-worker",
+ Expires: expires,
+ }, tp.context)
+ return true
@jonasfj
jonasfj Mar 30, 2016 Member

We should also log these errors to the task log... tp.context has a method for this...

@jonasfj jonasfj commented on the diff Mar 30, 2016
plugins/artifacts/artifacts.go
+ err == engines.ErrNonFatalInternalError || err == engines.ErrHandlerInterrupt ||
+ reflect.TypeOf(err).String() == "engines.MalformedPayloadError" {
+ runtime.CreateErrorArtifact(runtime.ErrorArtifact{
+ Name: name,
+ Message: err.Error(),
+ Reason: "invalid-resource-on-worker",
+ Expires: expires,
+ }, tp.context)
+ return true
+ }
+ return false
+}
+
+func (tp taskPlugin) createUploadHandler(name, prefix string, expires tcclient.Time) func(string, ioext.ReadSeekCloser) error {
+ return func(path string, stream ioext.ReadSeekCloser) error {
+ return tp.attemptUpload(stream, path, strings.Replace(path, prefix, name, 1), expires)
@jonasfj
jonasfj Mar 30, 2016 Member

Why strings.Replace? what does it do?

@imbstack
imbstack Mar 30, 2016 Member

It's just the way to make the following do the right thing.

"artifacts": [
    {
        "type": "directory",
        "path": "/artifacts",
        "name": "public"
    }
] 

strings.Replace is just turning the path /artifacts/foo.txt given to the UploadHandler into public/foo.txt.

@jonasfj jonasfj commented on an outdated diff Mar 30, 2016
plugins/artifacts/artifacts_test.go
+import (
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/taskcluster/slugid-go/slugid"
+ "github.com/taskcluster/taskcluster-client-go/queue"
+ "github.com/taskcluster/taskcluster-client-go/tcclient"
+ "github.com/taskcluster/taskcluster-worker/plugins/plugintest"
+ "github.com/taskcluster/taskcluster-worker/runtime/client"
+)
+
+type artifactTestCase struct {
+ TestCase *plugintest.Case
@jonasfj
jonasfj Mar 30, 2016 Member

This is a neat trick!!! :)

You can probably just embed plugintest.Case removing some unneeded intentation...

You can still call Test() by using a.plugintest.Case.Test() or a.Case.Test() I think...

@jonasfj jonasfj commented on the diff Mar 30, 2016
plugins/artifacts/artifacts_test.go
+ },
+ "artifacts": [
+ {
+ "type": "directory",
+ "path": "/artifacts",
+ "name": "public"
+ }
+ ]
+ }`,
+ Plugin: "artifacts",
+ TestStruct: t,
+ PluginSuccess: true,
+ EngineSuccess: true,
+ },
+ }.Test()
+}
@jonasfj
jonasfj Mar 30, 2016 Member

In a later PR let's add more test cases... for all the error cases... Might might have to do some tricks, but I think mockEngine can be made to return all sorts of errors.

@jonasfj jonasfj and 2 others commented on an outdated diff Mar 30, 2016
plugins/artifacts/payload-schema.yml
+ properties:
+ type:
+ title: Upload type
+ description: Artifacts can be either an individual `file` or a `directory` containing potentially multiple files with recursively included subdirectories
+ type: string
+ enum: [file, directory]
+ path:
+ title: Artifact Path
+ description: Filesystem path of artifact
+ type: string
+ pattern: "^.*[^/]$"
+ name:
+ title: Artifact Name
+ description: This will be the leading path to directories and the full name for files that are uploaded to s3
+ type: string
+ pattern: "^[^/].*[^/]$"
@jonasfj
jonasfj Mar 30, 2016 Member

@djmitche, @gregarndt If we ever want to enforce artifact naming... Restrict characters to printable ascii or something like that, now is the time...
We can't really make this change on queue.taskcluster.net before we've made sure that all uploaders don't violate this requirement...

However, given that the scope 'queue:get-artifact:<name>' is required to get an artifact... We are already in a situation where any non-public artifact can't be downloaded if name contains characters that isn't ascii printable.

This might be the wrong the place for this discussion, but perhaps we should take the first step in disallowing non-printable ascii chars in artifact names...
I doubt any are in use, but we should probably do a table scan and check before we change the queue. However, locking it down here is a good start.

@gregarndt
gregarndt Mar 31, 2016 Member

I agree that if we're going to start making this a requirement (we should) then we should start with this new worker to set the precedent and make sure at least this worker wouldn't allow it.

@djmitche
djmitche Mar 31, 2016 Contributor

I think that limits similar to scopes make a lot of sense. These are basically filenames, and encodings + filenames are an absolute nightmare, so let's avoid it by sticking to printable ascii

@jonasfj jonasfj commented on an outdated diff Mar 30, 2016
plugins/artifacts/payload-schema.yml
+ title: Artifact Path
+ description: Filesystem path of artifact
+ type: string
+ pattern: "^.*[^/]$"
+ name:
+ title: Artifact Name
+ description: This will be the leading path to directories and the full name for files that are uploaded to s3
+ type: string
+ pattern: "^[^/].*[^/]$"
+ expires:
+ title: Expiry Tate and Time
+ description: Date when artifact should expire must be in the future
+ type: string
+ format: date-time
+
+ # TODO: Should expires be required?
@jonasfj
jonasfj Mar 30, 2016 Member

nope, remove comment

@imbstack imbstack merged commit 37dd9ea into master Mar 31, 2016

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@imbstack imbstack deleted the artifacts branch Mar 31, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment