Get response from all worker pulls

[JennyMet]

Hi,

I want to run multiple requests and get one response (struct or something) that all done.
I mean if I've 10 request and the longest took 10 sec i want after 10 seconds get the all the responses with id or something for each request to know who success and who has failed, is it possible ?

thanks

[oze4]

I am not sure if this is the best approach but using closure/wrapping each "job" + custom "response type" solves this..

package main

import (
	"errors"
	"fmt"
	"time"

	"github.com/gammazero/workerpool"
)

// Response holds job response
type Response struct {
	ID    int
	Error error
	Data  interface{}
}

// wrapJob sends the result of our job to our results channel
func wrapJob(rc chan Response, f func() Response) func() {
	return func() { rc <- f() }
}

var jobs = []func() Response{
	func() Response {
		// do whatever
		// ...
		time.Sleep(time.Second * 3)
		return Response{ID: 1, Data: map[string]string{"hello": "mars"}}
	},
	func() Response {
		// do whatever
		// ...
		time.Sleep(time.Second)
		resp := Response{ID: 2}
		err := errors.New("some error") // simulate error
		if err != nil {
			resp.Error = err
			return resp
		}
		resp.Data = "some result that you may or may not see. depends if there is an error or not."
		return resp
	},
}

func main() {
	wp := workerpool.New(10)
	results := make(chan Response, len(jobs))

	for _, job := range jobs {
		wp.Submit(wrapJob(results, job))
	}

	wp.StopWait()

	close(results)

	for result := range results {
		if result.Error != nil {
			fmt.Printf("Found job with an error : ID '%d' : Error '%s'\n", result.ID, result.Error.Error())
		} else {
			fmt.Printf("Success : ID '%d' : Data '%+v'\n", result.ID, result.Data)
		}
	}
}

Which returns:

// Found job with an error : ID '2' : Error 'some error'
// Success : ID '1' : Data 'map[hello:mars]'

[gammazero]

@JennyMet Yes, that is certainly doable. A simple way to do what you want is to submit a function to workerpool that calls your job function with an id and a channel to put results on.

wp.Submit(func() {
    myFunc(id, resultChan)
})

After the all the jobs are submitted, you can then use select to wait on the result channel or until a timeout. Keep collecting results from the channel until all the results that you expect are collected, or until you timeout waiting.

timeout := time.After(10 * time.Second)
for i := 0; i < jobCount; i++ {
    select {
    case r := <-resultChan:
        handleResult(r)
    case <-timeout:
        return fmt.Errorf("timed out waiting for %d results", jobCount - i)
    } 
}

Here is some code that expands on the idea presented by @oze4 that shows a way to separate the generic mechanics of running and waiting for jobs into a separate package from your code that defines what a job does:
https://gist.github.com/gammazero/bb543631f25a73d1a3f6c8e1a6970eb8

[JennyMet]

@gammazero @oze4 - Thank you!