clarification on httpdump example

[davidbirdsong]

I'm using httpdump.go as a reference implementation for a simple forward proxy I'm cooking up that will intercept responses and push them into another system.

I need to send something like the Meta struct down a channel when the response is fully consumed by the downstream client, but more importantly when the body has been consumed by TeeReadCloser.

When LogMeta executes here:
https://github.com/elazarl/goproxy/blob/master/examples/httpdump/httpdump.go#L140
...am I reading it correctly that NewTeeReader hasn't had a chance necessarily to flush the response stream to the FileStream?

Would I need to add a call into TeeReadCloser.Close() ?

[elazarl]

Hi,

Thank you for using goproxy.

I don't think you're reading it correctly. What's happening is, the goproxy hook returns an *http.Response to the proxy, which will read it and write it to the client. We replaced the response body with a reader that would write the response to a file while reading it. Thus when the proxy will call resp.Body.Read(...), it'll write the content to a file in the mean time.

The proxy is also responsible for closing the response after writing it to the client. You only have to worry about providing the response. Let me illustrate that with an example:

C> Hey proxy, give me http://example.com/foo
P> Just a sec client, let me see if I have hooks for that. Actually I do.
   Hey hook, filter the response I got from example.com.
H> Here you go, I cooked a new response with a different body.
P> OK client, I'm reading from the cooked response and writing to you
In the meantime COOKED RESPOSNE is secretely writing down everything in the response.

Let me know if you need any other clarification.

[davidbirdsong]

On Sat, Nov 16, 2013 at 10:03 AM, Elazar Leibovich <notifications@github.com

wrote:

Hi,

Thank you for using goproxy.

I don't think you're reading it correctly. What's happening is, the
goproxy hook returns an *http.Response to the proxy, which will read it
and write it to the client. We replaced the response body with a reader
that would write the response to a file while reading it. Thus when the
proxy will call resp.Body.Read(...), it'll write the content to a file in
the mean time.

Right and that file capture is done with the TeeReederCloser. I'd like to
fire off a background job once the TeeReederCloser is closed. Proxy only
calls resp.Body.Read(...) as the original client consumes the stream and
then I assume golang's builtin http tools call the Close method.

I've modified the TeeReederCloser.Close() to send on a channel I've
created. Is there a more elegant way to be notified when Close is called,
some way that leverages goproxy's framework or golang's built-ins?

The proxy is also responsible for closing the response after writing it to
the client. You only have to worry about providing the response. Let me
illustrate that with an example:

C> Hey proxy, give me http://example.com/foo
P> Just a sec client, let me see if I have hooks for that. Actually I do.
Hey hook, filter the response I got from example.com.
H> Here you go, I cooked a new response with a different body.
P> OK client, I'm reading from the cooked response and writing to you
In the meantime COOKED RESPOSNE is secretely writing down everything in the response.

Let me know if you need any other clarification.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/25#issuecomment-28632022
.

[elazarl]

I'm not sure what exactly you want to achieve, but I don't think you have any other option. Note that TeeReaderCloser.Close might never be called because of a previous Read error, so you must wait until it's actually called.

That said, make sure the channel is buffered, you don't want to keep your goroutine waiting until the background job fires.

Do you have a better design in mind? If there's a way goproxy can help you with that, I'd like to hear that.

[davidbirdsong]

I have a pool of workers that need to pull content from external sources ie. the wild internet. The workers do not have access to the internet and the workers will often pass over content more than once.

I need a proxy that serves as a forward proxy to provide internet access to the workers. The same proxy can serve to intercept the content and forward to a storage layer which can be used for future fetches. I'm effectively building a large read-through cache, but one that's fully deconstructed.

I've been getting by w/ squid and nginx, but I want more programmatic control of the proxy behavior.

I don't think there's any better design, my questions so far are about my specific implementation. I'm new to golang; goproxy has been a great reference for becoming familiar w/ interfaces and golang's concurrency patterns.

Thanks for the note about buffered channels. I keep forgetting that the writer blocks until the reader reads. I guess it's like a socket accept with no listen queue(sort of.)

[davidbirdsong]

BTW, I assume there exists a "go func()" somewhere to provide the concurrency. Is there a goroutine per handler?

[elazarl]

@davidbirdsong I have almost no go func()s in goproxy (except https), the concurrency is handled by the http package where the goroutine resides.

Is it possible for you to show me some of your code? Either privately or in the public?

PS,

If you could write me how well goproxy worked, I'll be delighted.

[elazarl]

Is everything clear? Can I close the issue?

[davidbirdsong]

Hey sorry, totally missed these notifications. Everything's much clearer now. I can share some of the code soon.