Collecting Inputs for Resty v2.0 and Plan the release

[jeevatkm]

The goal of this thread to collect feedback's from resty users and plan v2.0.0 release. Action items get recorded here #167.

• Things could be improved
• Things could be added (generalized one's not specific use case)
• Things could be removed

Share your resty experiences for upcoming v2.0.0

[moorereason]

I've used Resty for two small CLI projects that interact with 3rd party REST services. Resty has done everything I wanted it to do and saved me a lot of time. The Resty 1.6 API is adequate for me, but my needs are rather typical so far.

The only hurdles I had to overcome were trying to understand how to use Resty's features. Showing a typical usage example with error handling would have helped me better understand the Resty design at a glance. For example, here's how I typically use Resty when interacting with the Quizlet API:

// GetSets returns all user sets in Quizlet.
func (c *Config) GetSets() ([]Set, error) {
	var sets []Set
	var e QuizletError

	uri := fmt.Sprintf("%s/users/%s/sets", apiBaseURL, c.Username)

	_, err := resty.SetDebug(c.Debug).R().
		SetHeader("Accept", "application/json").
		SetAuthToken(c.AuthToken).
		SetResult(&sets).
		SetError(&e).
		Get(uri)
	if err != nil {
		return nil, err
	}

	if e.Code != 0 {
		return nil, e
	}

	return sets, err
}

// QuizletError represents an error response from the Quizlet API.
type QuizletError struct {
	Code             int      `json:"http_code"`
	QError           string   `json:"error"`
	Title            string   `json:"error_title"`
	Description      string   `json:"error_description"`
	ValidationErrors []string `json:"validation_errors"`
}

// Error implements the error interface.
func (e QuizletError) Error() string {
	return e.Description
}

[jeevatkm]

Thank you for your inputs @moorereason.

Action Item for v2.0 release:

• Improve documentation and examples

[h7kanna]

How about having DefaultClient a DefaultTransport with sane defaults for Timeouts?
For example https://github.com/hashicorp/go-cleanhttp/blob/master/cleanhttp.go#L24

[jeevatkm]

@h7kanna Thank you for your input, I have noted down here #167.

[zjjhzx]

Any plans on supporting "net/http2" ?

[jeevatkm]

@zjjhzx resty already supports http2. Do you face any issues accessing http2 enabled service or website?

[sudo-suhas]

What are your thoughts on functional options?

This can be useful to propagate errors instead of logging(which could be missed) as is the case in func (*Request) SetQueryString

[sudo-suhas]

Another useful feature would be integration with https://github.com/go-playground/form (I prefer this) or https://github.com/google/go-querystring to be able to pass structs for URL encoding.

[jeevatkm]

@sudo-suhas Thanks for your inputs. I will have a look and get back to you.

[h7kanna]

Functional Options Example:
https://github.com/go-kit/kit/blob/master/transport/http/client.go#L53

[jeevatkm]

@sudo-suhas @h7kanna Thanks for your inputs. I have looked into it.

• struct to URL params - both library https://github.com/go-playground/form and https://github.com/google/go-querystring produces the url.Values; it would be good idea to add method in resty to accept url.Values to create query parameters. So that resty user could use their choice of library and supply url.Values into resty.
  • I have created action items for v2.0.0
• Function options - Adding this feature is good however adding functional, error propagation and chained calls together will not be feasible. What do you think of it? Let me know.

[sudo-suhas]

I feel error propagation is the most important and that is the reason I prefer functional options.

[sudo-suhas]

So that resty user could use their choice of library and supply url.Values into resty.

This could be solved by using the following interface:

type Encoder interface {
	Encode(interface{}) (url.Values, error)
}

Another thing I wanted to bring up was use of json.Marshal as opposed to json.NewEncoder. See https://stackoverflow.com/questions/21197239/decoding-json-in-golang-using-json-unmarshal-vs-json-newdecoder-decode. So it might be better to use json.NewEncoder/json.NewDecoder since we are dealing with io.Reader(http.Request.Body) and io.Writer(http.ResponseWriter)

[jeevatkm]

@sudo-suhas

• Could you please describe it further on how this interface gonna be much beneficial with query strings feature?
  • Instead of something likeRequest.SetQueryStringValues(v url.Values).
• Currently resty supports external JSON library registration Suggestion to change json parser in order to improve performance #76 (comment), e621018
  • First let me analysis the impact of migrating to json.{NewEncoder, NewDecoder}, need sometime on this.

[sudo-suhas]

The resty.Client could have a field URLEncoder of type Encoder which would be used for a new method on resty.Request which encodes the value and sets the url.Values on the request. This doesn't really work well with the chained calls API though since it could return an error. Additionally, the user could supply his own Encoder implementation similar to the way JSON library registration works. Let me know if I am not being clear, I can share code examples if required.

[sudo-suhas]

And once again, thanks for all your great work. I really appreciate it.

[jeevatkm]

@sudo-suhas Thank you. We had very good discussion. I will think about the design and then we can have a discussion 😄

[h7kanna]

Hi, Not sure if this is feasible, It may be a big API change.
Can the middleware be based on Roundtripper?

Like
https://github.com/improbable-eng/go-httpwares/blob/master/tripperware.go
Though it is possible now also by passing a custom http client using NewWithClient(hc *http.Client).

Thanks for your work on this handy package.

[jeevatkm]

@h7kanna Thank you, I will have a look on your reference.

[dewitast]

req.SetResult(nil)

Above code will cause panic. I was expecting it to undo previous SetResult or just unset the result type and I think it supposed to be. Is this intended?

[jeevatkm]

@dewitast By default for every request's Result and Error is fresh and applicable to that request only. No reason to supply to nil. Could you describe your use case?

[sudo-suhas]

I recently came across the golang.org/x/sync/singleflight package and I was wondering if something similar could be done with resty to avoid duplicate GET requests. After all, resty knows the HTTP request being made and HTTP is supposed to be stateless. So technically speaking it would be better to skip duplicate simultaneous requests and reuse the response from just one. Realistically speaking this feature should probably be opt-in since there is no guarantee that the HTTP endpoint is stateless.

[jeevatkm]

Thanks @sudo-suhas, nice thoughts I will look into it. Yes, suppress and reuse feature should be provided as opt-in.

[david-l-riley]

Would love to see a built-in mechanism for interacting with NDJSON streams. This can already be done manually, by pulling line-by-line from the raw Response stream and feeding through a JSON decoder, but it would be handy to have it built in.

[jeevatkm]

@david-l-riley Thanks for your input, I will have a look on NDJSON and get back.

[hongnod]

Can you please support Restful server end feature?

[jeevatkm]

@topillar Could you please describe in detail?

Not sure, just taking a guess. Are you looking for RESTful server side framework? if yes then try out my aah framework

[hongnod]

Yes,I noticed aah just after I sent the request.
thanks !

[jeevatkm]

@david-l-riley I have read the NDJSON spec (https://github.com/ndjson/ndjson-spec), it seems each line could be different JSON structure. Not sure how we can generalize this one. Same JSON structure on every line could generalized into resty. Please let me know.

[david-l-riley]

It's true, in NDJSON basically each line is a complete JSON document that could be a different item, suitable for streaming return values (rather than unmarshaling everything at once) If that's something that's difficult to marshal with the built-in JSON library, that's OK; it can be done reasonably simply just by using a line reader and deserializing each line.

[jeevatkm]

@david-l-riley Challenge with handling NDJSON response payload is unmarshal data types for different items.

One of the approach I could think of handling NDJSON response is to have SetResult(type1, type2, type3, so on) and expected response structure should be in the order of SetResult arguments.

For example:

type1 json
type2 json
type3 json
type1 json
type2 json
type3 json
... so on

Does it make sense? Do you have any other approach?

[pborzenkov]

Change: pass error returned by resty.Client.execute() to retry condition callback

Motivation: It is possible for the underlying http.Client to have a round-tripper that does additional HTTP request(s) before issuing an actual user's HTTP request. Example of such round-trippers is golang.org/x/oauth2.Transport which might tries to fetch a token if none is cached already. This round-tripper might return an instance of golang.org/x/oauth2.RetrieveError when it fails to obtain a token. Some of such errors are retry-able (e.g. network failures, response timeouts, etc.), some are not (invalid refresh token, etc.). It'd be good to have this error inside resty's retry condition callback.

[jeevatkm]

@pborzenkov Thanks for your inputs. I will think about the design around it. I have added to action items here #167

[david-l-riley]

So, coming back to the NDJSON: I've been doing some playing around with NDJSON using the current Resty version. It generally works reasonably well if you attach a json.Decoder to the RawBody from the Response, which requires disabling the response parsing, but that loses some useful functionality of the client (most notably for me, the debug logging on the response). I think the most useful way to support this would be to support the return of a json.Decoder attached to the Response's body (with some form of TeeReader allowing for logging). The user would then need to determine what data types to decode from the Decoder, since the NDJSON stream is just a stream of multiple documents, which could be of heterogeneous types.

Other than that, a way of attaching streaming decoders (of all sorts) to the Body without losing the other useful parsing/handling behaviors of Resty's Response would be really useful.

[neganovalexey]

There are some APIs (i. e. https://developers.google.com/drive/api/v3), that return binary content on success and JSON on error. On the one hand, it is not appropriate to read file content into buffer; on the other, error response should be parsed in order to manage retries. Resty client has SetDoNotParseResponse() method, which allows to copy download content from HTTP response directly, but it is very inconvenient to control error response parsing and retries outside the client. I suggest to add something like SetDoParseResponseOnlyOnError() method to Resty. It would be really helpful for such downloads.

[neganovalexey]

I have noticed that Resty unfortunately has poor support for retries of file uploads. It is possible to pass io.Reader to Request.SetBody(), but if it is a file, it will close after the first unsuccessful attempt. My suggestion is following:

1. Add default OnBeforeRequest middleware to check if the request body implements io.Seeker and seek to io.SeekStart if yes;
2. If the response body is io.Reader, prevent calling of Close method by wrapping into structure

// ioutil.NopCloser hides Seek() method
type nopCloser struct {
	io.ReadSeeker
}

func (nopCloser) Close() error { return nil }

[jeevatkm]

I suggest to add something like SetDoParseResponseOnlyOnError() method to Resty.

@neganovalexey I think suggested method should work in conjunction with method SetDoNotParseResponse(). I will think about the design, since these are targeted for v2, I can change the method behavior.

It is possible to pass io.Reader to Request.SetBody(), but if it is a file, it will close after the first unsuccessful attempt.

@neganovalexey I remember resty read and closes the file for multi-part content-type, however I do not remember it closes the io.Reader . Because for io.Reader resty gives control to underling http client, does not close explicitly.

Also FYI RetryConditionFunc func(*Response) (bool, error) has access to underling Resty request object, any changes to request object reflects in the next retry.

[neganovalexey]

@jeevatkm thank you for your response!

@neganovalexey I remember resty read and closes the file for multi-part content-type, however I do not remember it closes the io.Reader . Because for io.Reader resty gives control to underling http client, does not close explicitly.

Golang standard HTTP client closes request body if it implements io.Closer. So if an opened file is passed to Resty's Request.SetBody(), and retries are configured, they will fail.

It is obviously inefficient to reopen file on each attempt.

Moreover, if even io.ReadSeeker does not implement io.Closer (i. e. bytes.Reader), retries will still incorrect, because Resty does not perform Seek() before request.

I understand that it is possible to write own methods in order to fix this, but I think it is not convenient and may lead to bugs.

[xakep666]

I have experience using Resty with servers that returns 200 and error in body. Now I handling it like this:

type MaybeError interface {
	Error() error
}

type SomeResponse struct {
        // some fields here
        ErrorText *string `json:"error"`
}

func (sr *SomeResponse) Error() error {
       if sr.Error != nil {
             return errors.New(*sr.ErrorText)
       }
       return nil
}

func handleMaybeError(c *resty.Client, r *resty.Response) error {
        if rerr, ok := r.Result().(MaybeError); ok {
		return r.Error()
	}
	return nil
}

and using handleMaybeError as OnAfterResponse callback.

I know that returning error response with successive code is very bad practice but it's 3rd party service and I can't change it.

I suggest to add something like MaybeError interface and handleMaybeError middleware to Resty for such cases.

[jeevatkm]

@xakep666 Thank you for sharing your experience in 3rd party service.

I understand your description I think it may not be a candidate for generalized approach within a Resty. I can quickly think of couple of scenario's why it won't-

Scenario 1: Field Name

As you have described field name as error. However I have seen the API return that as a field name message too.

Scenario 2: Response Body Structure

In your example and you're dealing with APIs provide following structure.

{
   "error" : "error message"
}

However, in reality it might have more combinations.

[{
   "error" : "error message 1"
},
{
   "error" : "error message 2"
}]

OR

{
   "message": "some description",
   "errors": [{
	   "error" : "error message 1",
	   "fieldName": "somefield1"
	},
	{
	   "error" : "error message 2",
	   "fieldName": "somefield2"
	}]
}

OR

...

So its very difficult to standardized. If you ask me, You handling above cases very effectively per APIs provider or vendor.

[pborzenkov]

@jeevatkm Unfortunately, returning a non-nil error from OnAfterResponse callback combined with configured retries results in unnecessary retries (e.g. Backoff do retries unconditionally if client.execute() returned a non-nil error). This is something we are facing too.

One way to fix it, as I previously suggested, is to pass this error into retry condition callback and let it figure out whether or not it is actually a retry-able error.

[jeevatkm]

@pborzenkov Thank you for your input, I will have a look and get back to you.

[jeevatkm]

@pborzenkov I have cross checked it. It seems I have already added your suggestion as action items #167 for v2.0.0.

[rhzs]

Hi @jeevatkm, I am newbee on Resty, I have use case to track http call. How to integrate resty with zipkin / opentracing / jaeger?

[jeevatkm]

@incubus8 Resty does not have direct reference to any of the tracing libraries you have mentioned.

Since you could implement tracing/stats easily at your end using any of your choice of library.

Make use of Resty request and response middleware for tracing/stats. Also resty does capture the request time by default.

• https://godoc.org/gopkg.in/resty.v1#Response.Time
• https://godoc.org/gopkg.in/resty.v1#Response.ReceivedAt

[jeevatkm]

I'm closing this input thread, gonna start the v2 development. So we could discuss in the respective issues I will be creating. Thank you all for the great inputs and discussion.