From 2f17156c2d8404836f5b32896d1ce4cb05932225 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 14 Dec 2021 23:52:51 -0700 Subject: [PATCH 01/19] Change internal layers stack to be value-based instead of type-based This is very proof-of-concept currently, but wanted to put the code up to get initial feedback/ideas. Here's the gist: * I define 3 abstract subtypes of `Layer` in the Layers module: `InitialLayer`, `RequestLayer`, and `ConnectionLayer`; these correspond to the types of arguments the layer would receive when overloading `request`; `InitialLayer` => `request(method, url, headers, body), `RequestLayer` => `request(url, req, body)`, and `ConnectionLayer` => `request(io, req, body)`. I feel like these are useful abstract types to help distinguish the different _kinds_ of layers based on the arguments they receive in different parts of the overall stack. I feel like it hopefully also solves the "how do I insert my layer in the appropriate level of the stack" because you just pick which abstract layer to subtype * Custom layers are then required to: * be concrete types * must subtype one of the 3 abstract layer types mentioned * must have a constructor of the form: `Layer(next::Layer; kw...) * must have a field to store the `next::Layer` argument * must overload: `request(x::MyLayer, args...; kw...)` where `args` will depend on which abstract layer `MyLayer` subtypes * in the overloaded `request` method, it must, at some point, call `request(layer.next, args...; kw...)` to move to the next layer in the stack * the final requirement is the custom layer must overload `Layers.keywordforlayer(::Val{:kw}) = MyLayer` where `:kw` is a keyword argument "hook" or trigger that will result in `MyLayer` being inserted into the stack What I like about this approach so far: * It feels more straightforward to define my own custom layer: pick which level of the stack I want it in, subtype that abstract layer, register a keyword argument that will include my layer, then define my own `request` overload; we can define all the current HTTP layers in terms of this machinery, custom packages could define their own layers and they all are treated just like any HTTP.jl-baked layer * It also feels like it would be easier to create my own custom "http client"; what I mean by this is that I feel like it's common for API packages to basically have their own `AWS.get`/`AWS.post`/`AWS.put` wrapper methods that eventually call `HTTP.get`/`HTTP.post`/etc., but in between, they're adding custom headers, validating targets, and whatever. A lot of that is _like_ a custom layer, but by having your own `AWS.get`/`AWS.post` wrapper methods, you can _ensure_ certain layers are included; and with this approach that's straightforward because they can indeed make their own custom layer as described above, and then just ensure every `HTTP.request` call includes the new keyword argument to "hook in" their custom layer Potential downsides: * Relying on a global keyword registry; i.e. there's potential for clashes if two different packages try to overload `Layers.keywordforlayer(::Val{:readtimeout}) = ...`. Whichever package is loaded last will "pirate" the keyword arg and overwrite the previous method. We could potentially add some extra safety around this by having another `clientenv=:default` keyword and then devs could overload `Layers.keywordforlayer(::Val{:default}, ::Val{:readtimeout}) = ...` but that feels a tad icky. I also don't expect there to be even dozens of total keywords/layers to process, so with that in mind, it also lowers the risk of clash. * The previous layer functionality allowed specifying the _exact_ layer you wanted your custom layer to be inserted before; do we think that level of specificity/granularity is really necessary? I'm kind of going on a gut feel here that what you really want control over is _what kind of args_ your overloaded `request` method will receive and that's what my 3 new abstract layer subtypes aim to solve. So you can pick 1 of 3 different _general_ layers of where your custom layer gets inserted, but don't have control over anything more specific in terms of what layer comes before/after * One thing I noticed is that the current `stack` function actually has some hard-coded logic that checks keyword arguments and _conditionally_ will include layers based on the _value_ of the keyword arg. We could potentially handle that in the proposed scheme by saying that your custom `Layer(next; kw...)` constructor can return the `nothing` value, in which case, no layer will be inserted. That would give, for example, the `TimeoutLayer` the chance to check if the `readtimeout > 0` before returning a `TimeoutLayer` or `nothing` --- src/ConnectionRequest.jl | 12 ++++--- src/HTTP.jl | 49 +++++++++------------------ src/MessageRequest.jl | 11 +++--- src/StreamRequest.jl | 8 ++--- src/TimeoutRequest.jl | 13 ++++--- src/TopRequest.jl | 2 +- src/exceptions.jl | 7 ---- src/layers.jl | 73 +++++++++++----------------------------- 8 files changed, 64 insertions(+), 111 deletions(-) delete mode 100644 src/exceptions.jl diff --git a/src/ConnectionRequest.jl b/src/ConnectionRequest.jl index 790c72f11..1397d47da 100644 --- a/src/ConnectionRequest.jl +++ b/src/ConnectionRequest.jl @@ -57,12 +57,16 @@ Otherwise leave it open so that it can be reused. `IO` related exceptions from `Base` are wrapped in `HTTP.IOError`. See [`isioerror`](@ref). """ -abstract type ConnectionPoolLayer{Next <: Layer} <: Layer{Next} end +struct ConnectionPoolLayer{Next <: Layer} <: RequestLayer + next::Next + pool::ConnectionPools.Pool +end export ConnectionPoolLayer +ConnectionPoolLayer(next; connectionpool=ConnectionPool.POOL) = ConnectionPoolLayer(next, connectionpool) -function request(::Type{ConnectionPoolLayer{Next}}, url::URI, req, body; +function request(layer::ConnectionPoolLayer, url::URI, req, body; proxy=getproxy(url.scheme, url.host), - socket_type::Type=TCPSocket, kw...) where Next + socket_type::Type=TCPSocket, kw...) if proxy !== nothing target_url = url @@ -99,7 +103,7 @@ function request(::Type{ConnectionPoolLayer{Next}}, url::URI, req, body; req.headers = filter(x->x.first != "Proxy-Authorization", req.headers) end - r = request(Next, io, req, body; kw...) + r = request(layer.next, io, req, body; kw...) if proxy !== nothing && target_url.scheme == "https" close(io) diff --git a/src/HTTP.jl b/src/HTTP.jl index 894bb2282..7c1337496 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -452,7 +452,6 @@ include("ConnectionRequest.jl"); using .ConnectionRequest include("DebugRequest.jl"); using .DebugRequest include("StreamRequest.jl"); using .StreamRequest include("ContentTypeRequest.jl"); using .ContentTypeDetection -include("exceptions.jl") """ The `stack()` function returns the default HTTP Layer-stack type. @@ -568,39 +567,23 @@ relationship with [`HTTP.Response`](@ref), [`HTTP.Parsers`](@ref), ``` *See `docs/src/layers`[`.monopic`](http://monodraw.helftone.com).* """ -function stack(;redirect=true, - aws_authorization=false, - cookies=false, - canonicalize_headers=false, - retry=true, - status_exception=true, - readtimeout=0, - detect_content_type=false, - verbose=0, - kw...) - - NoLayer = Union - stack = TopLayer{ - (redirect ? RedirectLayer : NoLayer){ - BasicAuthLayer{ - (detect_content_type ? ContentTypeDetectionLayer : NoLayer){ - (cookies === true || (cookies isa AbstractDict && !isempty(cookies)) ? - CookieLayer : NoLayer){ - (canonicalize_headers ? CanonicalizeLayer : NoLayer){ - MessageLayer{ - (aws_authorization ? AWS4AuthLayer : NoLayer){ - (retry ? RetryLayer : NoLayer){ - (status_exception ? ExceptionLayer : NoLayer){ - ConnectionPoolLayer{ - (verbose >= 3 || - DEBUG_LEVEL[] >= 3 ? DebugLayer : NoLayer){ - (readtimeout > 0 ? TimeoutLayer : NoLayer){ - StreamLayer{Union{}} - }}}}}}}}}}}}} - - reduce(Layers.EXTRA_LAYERS; init=stack) do stack, (before, custom) - insert(stack, before, custom) +function stack(; kw...) + + layers = stacklayertypes(Layers.ConnectionLayer, StreamLayer(); kw...) + layers = ConnectionPoolLayer(layers; kw...) + layers = stacklayertypes(Layers.RequestLayer, layers; kw...) + layers = MessageLayer(layers; kw...) + return stacklayertypes(Layers.InitialLayer, layers; kw...) +end + +function stacklayertypes(::Type{T}, layers; kw...) where {T} + for (k, _) in pairs(kw) + layer = Layers.keywordforlayer(Val(k)) + if layer !== nothing && layer <: T + layers = layer(layers; kw...) + end end + return layers end include("download.jl") diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index ec02c55c6..5691bb178 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -23,14 +23,17 @@ resource(uri::URI) = string( isempty(uri.path) ? "/" : uri.path, Construct a [`Request`](@ref) object and set mandatory headers. """ -struct MessageLayer{Next <: Layer} <: Layer{Next} end +struct MessageLayer{Next <: Layer} <: RequestLayer + next::Next +end export MessageLayer +MessageLayer(next; kw...) = MessageLayer(next) -function request(::Type{MessageLayer{Next}}, +function request(layer::MessageLayer, method::String, url::URI, headers::Headers, body; http_version=v"1.1", target=resource(url), - parent=nothing, iofunction=nothing, kw...) where Next + parent=nothing, iofunction=nothing, kw...) if isempty(url.port) || (url.scheme == "http" && url.port == "80") || @@ -63,7 +66,7 @@ function request(::Type{MessageLayer{Next}}, req = Request(method, target, headers, bodybytes(body); parent=parent, version=http_version) - return request(Next, url, req, body; iofunction=iofunction, kw...) + return request(layer.next, url, req, body; iofunction=iofunction, kw...) end const USER_AGENT = Ref{Union{String, Nothing}}("HTTP.jl/$VERSION") diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index 6a18961b7..97988c61a 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -19,15 +19,15 @@ immediately so that the transmission can be aborted if the `Response` status indicates that the server does not wish to receive the message body. [RFC7230 6.5](https://tools.ietf.org/html/rfc7230#section-6.5). """ -abstract type StreamLayer{Next <: Layer} <: Layer{Next} end +struct StreamLayer <: ConnectionLayer end export StreamLayer -function request(::Type{StreamLayer{Next}}, io::IO, req::Request, body; +function request(::StreamLayer, io::IO, req::Request, body; reached_redirect_limit=false, response_stream=nothing, iofunction=nothing, verbose::Int=0, - kw...)::Response where Next + kw...)::Response verbose == 1 && printlncompact(req) @@ -84,7 +84,7 @@ function request(::Type{StreamLayer{Next}}, io::IO, req::Request, body; verbose == 1 && printlncompact(response) verbose == 2 && println(response) - return request(Next, response) + return response end function writebody(http::Stream, req::Request, body) diff --git a/src/TimeoutRequest.jl b/src/TimeoutRequest.jl index 0651c8a9e..4f7204739 100644 --- a/src/TimeoutRequest.jl +++ b/src/TimeoutRequest.jl @@ -17,12 +17,17 @@ end Close `IO` if no data has been received for `timeout` seconds. """ -abstract type TimeoutLayer{Next <: Layer} <: Layer{Next} end +struct TimeoutLayer{Next <: Layer} <: ConnectionLayer + next::Next + readtimeout::Int +end export TimeoutLayer +Layers.keywordforlayer(::Val{:readtimeout}) = TimeoutLayer -function request(::Type{TimeoutLayer{Next}}, io::IO, req, body; - readtimeout::Int=0, kw...) where Next +TimeoutLayer(next; readtimeout::Int=0, kw...) = TimeoutLayer(next, readtimeout) +function request(layer::TimeoutLayer, io::IO, req, body; kw...) + readtimeout = layer.readtimeout wait_for_timeout = Ref{Bool}(true) timedout = Ref{Bool}(false) @@ -37,7 +42,7 @@ function request(::Type{TimeoutLayer{Next}}, io::IO, req, body; end try - return request(Next, io, req, body; kw...) + return request(layer.next, io, req, body; kw...) catch e if timedout[] throw(ReadTimeoutError(readtimeout)) diff --git a/src/TopRequest.jl b/src/TopRequest.jl index 54510a74e..e965b542a 100644 --- a/src/TopRequest.jl +++ b/src/TopRequest.jl @@ -10,7 +10,7 @@ export TopLayer This layer is at the top of every stack, and does nothing. It's useful for inserting a custom layer at the top of the stack. """ -abstract type TopLayer{Next <: Layer} <: Layer{Next} end +struct TopLayer{Next <: Layer} <: Layer{Next} end request(::Type{TopLayer{Next}}, args...; kwargs...) where Next = request(Next, args...; kwargs...) diff --git a/src/exceptions.jl b/src/exceptions.jl deleted file mode 100644 index 6ddd09ee5..000000000 --- a/src/exceptions.jl +++ /dev/null @@ -1,7 +0,0 @@ -struct LayerNotFoundException <: Exception - var::String -end - -function Base.showerror(io::IO, e::LayerNotFoundException) - println(io, typeof(e), ": ", e.var) -end diff --git a/src/layers.jl b/src/layers.jl index bb58e53a7..8f9564a32 100644 --- a/src/layers.jl +++ b/src/layers.jl @@ -1,64 +1,29 @@ module Layers -export Layer, next, top_layer, insert, insert_default!, remove_default! +export Layer, keywordforlayer -const EXTRA_LAYERS = Set{Tuple{Union{UnionAll, Type{Union{}}}, UnionAll}}() - -include("exceptions.jl") - -""" -## Request Execution Stack +struct LayerNotFoundException <: Exception + var::String +end -The Request Execution Stack is separated into composable layers. +function Base.showerror(io::IO, e::LayerNotFoundException) + println(io, typeof(e), ": ", e.var) +end -Each layer is defined by a nested type `Layer{Next}` where the `Next` -parameter defines the next layer in the stack. -The `request` method for each layer takes a `Layer{Next}` type as -its first argument and dispatches the request to the next layer -using `request(Next, ...)`. +abstract type Layer end -The example below defines three layers and three stacks each with -a different combination of layers. +abstract type InitialLayer <: Layer end +abstract type RequestLayer <: Layer end +abstract type ConnectionLayer <: Layer end +function keywordforlayer end -```julia -abstract type Layer end -abstract type Layer1{Next <: Layer} <: Layer end -abstract type Layer2{Next <: Layer} <: Layer end -abstract type Layer3 <: Layer end - -request(::Type{Layer1{Next}}, data) where Next = "L1", request(Next, data) -request(::Type{Layer2{Next}}, data) where Next = "L2", request(Next, data) -request(::Type{Layer3}, data) = "L3", data - -const stack1 = Layer1{Layer2{Layer3}} -const stack2 = Layer2{Layer1{Layer3}} -const stack3 = Layer1{Layer3} -``` - -```julia -julia> request(stack1, "foo") -("L1", ("L2", ("L3", "foo"))) - -julia> request(stack2, "bar") -("L2", ("L1", ("L3", "bar"))) - -julia> request(stack3, "boo") -("L1", ("L3", "boo")) -``` - -This stack definition pattern gives the user flexibility in how layers are -combined but still allows Julia to do whole-stack compile time optimisations. - -e.g. the `request(stack1, "foo")` call above is optimised down to a single -function: -```julia -julia> code_typed(request, (Type{stack1}, String))[1].first -CodeInfo(:(begin - return (Core.tuple)("L1", (Core.tuple)("L2", (Core.tuple)("L3", data))) -end)) -``` -""" -abstract type Layer{Next} end +keywordforlayer(kw) = nothing +# custom layers must subtype one of above +# must register a keyword arg for layer +# must have a layer constructor like: Layer(next; kw...) +# must have a field to store `next` layer +# must overload: request(layer::MyLayer, args...; kw...) +# in `request` overload, must call: request(layer.next, args...; kw...) """ next(::Type{S}) where {T, S<:Layer{T}} From 747f7257f274be079307fea689fc2f9892c63623 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 16 Dec 2021 00:09:05 -0700 Subject: [PATCH 02/19] more work for value-based stack --- docs/src/internal_architecture.md | 1 - src/AWS4AuthRequest.jl | 197 ------------------------- src/BasicAuthRequest.jl | 17 ++- src/CanonicalizeRequest.jl | 16 ++- src/ConnectionRequest.jl | 11 +- src/ContentTypeRequest.jl | 15 +- src/CookieRequest.jl | 20 ++- src/DebugRequest.jl | 18 ++- src/ExceptionRequest.jl | 15 +- src/HTTP.jl | 68 ++++----- src/MessageRequest.jl | 10 +- src/RedirectRequest.jl | 17 ++- src/RetryRequest.jl | 19 ++- src/StreamRequest.jl | 6 +- src/TimeoutRequest.jl | 12 +- src/TopRequest.jl | 18 --- src/layers.jl | 58 +------- test/aws4.jl | 232 ------------------------------ test/client.jl | 6 +- test/insert_layers.jl | 55 ------- test/resources/TestRequest.jl | 34 +++-- test/runtests.jl | 2 - 22 files changed, 172 insertions(+), 675 deletions(-) delete mode 100644 src/AWS4AuthRequest.jl delete mode 100644 src/TopRequest.jl delete mode 100644 test/aws4.jl delete mode 100644 test/insert_layers.jl diff --git a/docs/src/internal_architecture.md b/docs/src/internal_architecture.md index 17c754023..f3bfc200c 100644 --- a/docs/src/internal_architecture.md +++ b/docs/src/internal_architecture.md @@ -14,7 +14,6 @@ HTTP.BasicAuthLayer HTTP.CookieLayer HTTP.CanonicalizeLayer HTTP.MessageLayer -HTTP.AWS4AuthLayer HTTP.RetryLayer HTTP.ExceptionLayer HTTP.ConnectionPoolLayer diff --git a/src/AWS4AuthRequest.jl b/src/AWS4AuthRequest.jl deleted file mode 100644 index a5623966e..000000000 --- a/src/AWS4AuthRequest.jl +++ /dev/null @@ -1,197 +0,0 @@ -module AWS4AuthRequest - -using ..Base64 -using ..Dates -using MbedTLS: digest, MD_SHA256, MD_MD5 -import ..Layer, ..request, ..Headers -using URIs -using ..Pairs: getkv, setkv, rmkv -import ..@debug, ..DEBUG_LEVEL - -""" - request(AWS4AuthLayer, ::URI, ::Request, body) -> HTTP.Response - -Add a [AWS Signature Version 4](http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) -`Authorization` header to a `Request`. - - -Credentials are read from environment variables `AWS_ACCESS_KEY_ID`, -`AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN`. -""" -abstract type AWS4AuthLayer{Next <: Layer} <: Layer{Next} end -export AWS4AuthLayer - -function request(::Type{AWS4AuthLayer{Next}}, - url::URI, req, body; kw...) where Next - - if !haskey(kw, :aws_access_key_id) && - !haskey(ENV, "AWS_ACCESS_KEY_ID") - kw = merge(dot_aws_credentials(), kw) - end - - sign_aws4!(req.method, url, req.headers, req.body; kw...) - - return request(Next, url, req, body; kw...) -end - -# Normalize whitespace to the form required in the canonical headers. -# Note that the expected format for multiline headers seems not to be explicitly -# documented, but Amazon provides a test case for it, so we'll match that behavior. -# We replace each `\n` with a `,` and remove all whitespace around the newlines, -# then any remaining contiguous whitespace is replaced with a single space. -function _normalize_ws(s::AbstractString) - if any(isequal('\n'), s) - join(map(_normalize_ws, split(s, '\n')), ',') - else - replace(strip(s), r"\s+" => " ") - end -end - -function sign_aws4!(method::String, - url::URI, - headers::Headers, - body::Vector{UInt8}; - body_sha256::Vector{UInt8}=digest(MD_SHA256, body), - body_md5::Vector{UInt8}=digest(MD_MD5, body), - t::Union{DateTime,Nothing}=nothing, - timestamp::DateTime=now(Dates.UTC), - aws_service::String=String(split(url.host, ".")[1]), - aws_region::String=String(split(url.host, ".")[2]), - aws_access_key_id::String=ENV["AWS_ACCESS_KEY_ID"], - aws_secret_access_key::String=ENV["AWS_SECRET_ACCESS_KEY"], - aws_session_token::String=get(ENV, "AWS_SESSION_TOKEN", ""), - token_in_signature=true, - include_md5=true, - include_sha256=true, - kw...) - if t !== nothing - Base.depwarn("The `t` keyword argument to `sign_aws4!` is deprecated; use " * - "`timestamp` instead.", :sign_aws4!) - timestamp = t - end - - # ISO8601 date/time strings for time of request... - date = Dates.format(timestamp, dateformat"yyyymmdd") - datetime = Dates.format(timestamp, dateformat"yyyymmddTHHMMSS\Z") - - # Authentication scope... - scope = [date, aws_region, aws_service, "aws4_request"] - - # Signing key generated from today's scope string... - signing_key = string("AWS4", aws_secret_access_key) - for element in scope - signing_key = digest(MD_SHA256, element, signing_key) - end - - # Authentication scope string... - scope = join(scope, "/") - - # SHA256 hash of content... - content_hash = bytes2hex(body_sha256) - - # HTTP headers... - rmkv(headers, "Authorization") - setkv(headers, "host", url.host) - setkv(headers, "x-amz-date", datetime) - include_md5 && setkv(headers, "Content-MD5", base64encode(body_md5)) - if (aws_service == "s3" && method == "PUT") || include_sha256 - # This header is required for S3 PUT requests. See the documentation at - # https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html - setkv(headers, "x-amz-content-sha256", content_hash) - end - if aws_session_token != "" - setkv(headers, "x-amz-security-token", aws_session_token) - end - - # Sort and lowercase() Headers to produce canonical form... - unique_header_keys = Vector{String}() - normalized_headers = Dict{String,Vector{String}}() - for (k, v) in sort!([lowercase(k) => v for (k, v) in headers], by=first) - # Some services want the token included as part of the signature - if k == "x-amz-security-token" && !token_in_signature - continue - end - # In Amazon's examples, they exclude Content-Length from signing. This does not - # appear to be addressed in the documentation, so we'll just mimic the example. - if k == "content-length" - continue - end - if !haskey(normalized_headers, k) - normalized_headers[k] = Vector{String}() - push!(unique_header_keys, k) - end - push!(normalized_headers[k], _normalize_ws(v)) - end - canonical_headers = map(unique_header_keys) do k - string(k, ':', join(normalized_headers[k], ',')) - end - signed_headers = join(unique_header_keys, ';') - - # Sort Query String... - query = sort!(collect(queryparams(url.query)), by=first) - - # Paths for requests to S3 should be escaped but not normalized. See - # http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html#canonical-request - # Note that escapepath escapes ~ per RFC 1738, but Amazon includes an example in their - # signature v4 test suite where ~ remains unescaped. We follow the spec here and thus - # deviate from Amazon's example in this case. - path = escapepath(aws_service == "s3" ? url.path : URIs.normpath(url.path)) - - # Create hash of canonical request... - canonical_form = join([method, - path, - escapeuri(query), - join(canonical_headers, "\n"), - "", - signed_headers, - content_hash], "\n") - @debug 3 "AWS4 canonical_form: $canonical_form" - - canonical_hash = bytes2hex(digest(MD_SHA256, canonical_form)) - - # Create and sign "String to Sign"... - string_to_sign = "AWS4-HMAC-SHA256\n$datetime\n$scope\n$canonical_hash" - signature = bytes2hex(digest(MD_SHA256, string_to_sign, signing_key)) - - @debug 3 "AWS4 string_to_sign: $string_to_sign" - @debug 3 "AWS4 signature: $signature" - - # Append Authorization header... - setkv(headers, "Authorization", string( - "AWS4-HMAC-SHA256 ", - "Credential=$aws_access_key_id/$scope, ", - "SignedHeaders=$signed_headers, ", - "Signature=$signature" - )) -end - -using IniFile - -credentials = NamedTuple() - -""" -Load Credentials from [AWS CLI ~/.aws/credentials file] -(http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html). -""" -function dot_aws_credentials()::NamedTuple - - global credentials - if !isempty(credentials) - return credentials - end - - f = get(ENV, "AWS_CONFIG_FILE", joinpath(homedir(), ".aws", "credentials")) - p = get(ENV, "AWS_DEFAULT_PROFILE", get(ENV, "AWS_PROFILE", "default")) - - if !isfile(f) - return NamedTuple() - end - - ini = read(Inifile(), f) - - credentials = ( - aws_access_key_id = String(get(ini, p, "aws_access_key_id")), - aws_secret_access_key = String(get(ini, p, "aws_secret_access_key"))) -end - -end # module AWS4AuthRequest diff --git a/src/BasicAuthRequest.jl b/src/BasicAuthRequest.jl index 7b64b4152..8bb5dc2b1 100644 --- a/src/BasicAuthRequest.jl +++ b/src/BasicAuthRequest.jl @@ -1,22 +1,25 @@ module BasicAuthRequest using ..Base64 - -import ..Layer, ..request +using ..Layers using URIs using ..Pairs: getkv, setkv import ..@debug, ..DEBUG_LEVEL """ - request(BasicAuthLayer, method, ::URI, headers, body) -> HTTP.Response + Layers.request(BasicAuthLayer, method, ::URI, headers, body) -> HTTP.Response Add `Authorization: Basic` header using credentials from url userinfo. """ -abstract type BasicAuthLayer{Next <: Layer} <: Layer{Next} end +struct BasicAuthLayer{Next <: Layer} <: InitialLayer + next::Next +end export BasicAuthLayer +Layers.keywordforlayer(::Val{:basicauth}) = BasicAuthLayer +BasicAuthLayer(next; kw...) = BasicAuthLayer(next) -function request(::Type{BasicAuthLayer{Next}}, - method::String, url::URI, headers, body; kw...) where Next +function Layers.request(layer::BasicAuthLayer, + method::String, url::URI, headers, body; kw...) userinfo = unescapeuri(url.userinfo) @@ -25,7 +28,7 @@ function request(::Type{BasicAuthLayer{Next}}, setkv(headers, "Authorization", "Basic $(base64encode(userinfo))") end - return request(Next, method, url, headers, body; kw...) + return Layers.request(layer.next, method, url, headers, body; kw...) end diff --git a/src/CanonicalizeRequest.jl b/src/CanonicalizeRequest.jl index e3c5b9f9c..01f865373 100644 --- a/src/CanonicalizeRequest.jl +++ b/src/CanonicalizeRequest.jl @@ -1,23 +1,27 @@ module CanonicalizeRequest -import ..Layer, ..request +using ..Layers using ..Messages using ..Strings: tocameldash """ - request(CanonicalizeLayer, method, ::URI, headers, body) -> HTTP.Response + Layers.request(CanonicalizeLayer, method, ::URI, headers, body) -> HTTP.Response Rewrite request and response headers in Canonical-Camel-Dash-Format. """ -abstract type CanonicalizeLayer{Next <: Layer} <: Layer{Next} end +struct CanonicalizeLayer{Next <: Layer} <: InitialLayer + next::Next +end export CanonicalizeLayer +Layers.keywordforlayer(::Val{:canonicalize_headers}) = CanonicalizeLayer +CanonicalizeLayer(next; canonicalize_headers::Bool=true, kw...) = + canonicalize_headers ? CanonicalizeLayer(next) : nothing -function request(::Type{CanonicalizeLayer{Next}}, - method::String, url, headers, body; kw...) where Next +function Layers.request(layer::CanonicalizeLayer, method::String, url, headers, body; kw...) headers = canonicalizeheaders(headers) - res = request(Next, method, url, headers, body; kw...) + res = Layers.request(layer.next, method, url, headers, body; kw...) res.headers = canonicalizeheaders(res.headers) diff --git a/src/ConnectionRequest.jl b/src/ConnectionRequest.jl index 1397d47da..22ec2de32 100644 --- a/src/ConnectionRequest.jl +++ b/src/ConnectionRequest.jl @@ -1,6 +1,6 @@ module ConnectionRequest -import ..Layer, ..request +using ..Layers using URIs, ..Sockets using ..Messages using ..IOExtras @@ -47,7 +47,7 @@ function getproxy(scheme, host) end """ - request(ConnectionPoolLayer, ::URI, ::Request, body) -> HTTP.Response + Layers.request(ConnectionPoolLayer, ::URI, ::Request, body) -> HTTP.Response Retrieve an `IO` connection from the [`ConnectionPool`](@ref). @@ -59,12 +59,11 @@ See [`isioerror`](@ref). """ struct ConnectionPoolLayer{Next <: Layer} <: RequestLayer next::Next - pool::ConnectionPools.Pool end export ConnectionPoolLayer -ConnectionPoolLayer(next; connectionpool=ConnectionPool.POOL) = ConnectionPoolLayer(next, connectionpool) +ConnectionPoolLayer(next; kw...) = ConnectionPoolLayer(next) -function request(layer::ConnectionPoolLayer, url::URI, req, body; +function Layers.request(layer::ConnectionPoolLayer, url::URI, req, body; proxy=getproxy(url.scheme, url.host), socket_type::Type=TCPSocket, kw...) @@ -103,7 +102,7 @@ function request(layer::ConnectionPoolLayer, url::URI, req, body; req.headers = filter(x->x.first != "Proxy-Authorization", req.headers) end - r = request(layer.next, io, req, body; kw...) + r = Layers.request(layer.next, io, req, body; kw...) if proxy !== nothing && target_url.scheme == "https" close(io) diff --git a/src/ContentTypeRequest.jl b/src/ContentTypeRequest.jl index d2f00becc..dfc42b459 100644 --- a/src/ContentTypeRequest.jl +++ b/src/ContentTypeRequest.jl @@ -1,6 +1,6 @@ module ContentTypeDetection -import ..Layer, ..request +using ..Layers using URIs using ..Pairs: getkv, setkv import ..sniff @@ -9,11 +9,16 @@ using ..Messages import ..MessageRequest: bodylength, bodybytes import ..@debug, ..DEBUG_LEVEL -abstract type ContentTypeDetectionLayer{Next <: Layer} <: Layer{Next} end +struct ContentTypeDetectionLayer{Next <: Layer} <: InitialLayer + next::Next +end export ContentTypeDetectionLayer +Layers.keywordforlayer(::Val{:detect_content_type}) = ContentTypeDetectionLayer +ContentTypeDetectionLayer(next; detect_content_type::Bool=true, kw...) = + detect_content_type ? ContentTypeDetectionLayer(netx) : nothing -function request(::Type{ContentTypeDetectionLayer{Next}}, - method::String, url::URI, headers, body; kw...) where Next +function Layers.request(layer::ContentTypeDetectionLayer, + method::String, url::URI, headers, body; kw...) if (getkv(headers, "Content-Type", "") == "" && !isa(body, Form) @@ -24,7 +29,7 @@ function request(::Type{ContentTypeDetectionLayer{Next}}, setkv(headers, "Content-Type", sn) @debug 1 "setting Content-Type header to: $sn" end - return request(Next, method, url, headers, body; kw...) + return Layers.request(layer.next, method, url, headers, body; kw...) end end # module diff --git a/src/CookieRequest.jl b/src/CookieRequest.jl index 7ff0038c5..3d914d156 100644 --- a/src/CookieRequest.jl +++ b/src/CookieRequest.jl @@ -1,7 +1,7 @@ module CookieRequest import ..Dates -import ..Layer, ..request +using ..Layers using URIs using ..Cookies using ..Messages: ascii_lc_isequal @@ -16,20 +16,26 @@ function __init__() end """ - request(CookieLayer, method, ::URI, headers, body) -> HTTP.Response + Layers.request(CookieLayer, method, ::URI, headers, body) -> HTTP.Response Add locally stored Cookies to the request headers. Store new Cookies found in the response headers. """ -abstract type CookieLayer{Next <: Layer} <: Layer{Next} end +struct CookieLayer{Next <: Layer} <: InitialLayer + next::Next + cookiejar::Dict{String, Set{Cookie}} +end export CookieLayer +Layers.keywordforlayer(::Val{:cookies}) = CookieLayer +CookieLayer(next; cookies::Union{Bool, AbstractDict}=true, cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) = + (cookies === true || (cookies isa AbstractDict && !isempty(cookies))) ? CookieLayer(next, cookiejar) : nothing -function request(::Type{CookieLayer{Next}}, +function Layers.request(layer::CookieLayer, method::String, url::URI, headers, body; cookies::Union{Bool, Dict{<:AbstractString, <:AbstractString}}=Dict{String, String}(), - cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), - kw...) where {Next} + kw...) + cookiejar = layer.cookiejar hostcookies = get!(cookiejar, url.host, Set{Cookie}()) cookiestosend = getcookies(hostcookies, url) @@ -42,7 +48,7 @@ function request(::Type{CookieLayer{Next}}, setkv(headers, "Cookie", stringify(getkv(headers, "Cookie", ""), cookiestosend)) end - res = request(Next, method, url, headers, body; kw...) + res = Layers.request(layer.next, method, url, headers, body; kw...) setcookies(hostcookies, url.host, res.headers) diff --git a/src/DebugRequest.jl b/src/DebugRequest.jl index 405688c70..dd456003d 100644 --- a/src/DebugRequest.jl +++ b/src/DebugRequest.jl @@ -1,6 +1,7 @@ module DebugRequest -import ..Layer, ..request +using ..Layers +import ..DEBUG_LEVEL using ..IOExtras const live_mode = true @@ -8,21 +9,26 @@ const live_mode = true include("IODebug.jl") """ - request(DebugLayer, ::IO, ::Request, body) -> HTTP.Response + Layers.request(DebugLayer, ::IO, ::Request, body) -> HTTP.Response Wrap the `IO` stream in an `IODebug` stream and print Message data. """ -abstract type DebugLayer{Next <:Layer} <: Layer{Next} end +struct DebugLayer{Next <:Layer} <: ConnectionLayer + next::Next +end export DebugLayer +Layers.keywordforlayer(::Val{:verbose}) = DebugLayer +DebugLayer(next; verbose=0, kw...) = + (verbose >= 3 || DEBUG_LEVEL[] >= 3) ? DebugLayer(next) : nothing -function request(::Type{DebugLayer{Next}}, io::IO, req, body; kw...) where Next +function Layers.request(layer::DebugLayer, io::IO, req, body; kw...) @static if live_mode - return request(Next, IODebug(io), req, body; kw...) + return Layers.request(layer.next, IODebug(io), req, body; kw...) else iod = IODebug(io) try - return request(Next, iod, req, body; kw...) + return Layers.request(layer.next, iod, req, body; kw...) finally show_log(stdout, iod) end diff --git a/src/ExceptionRequest.jl b/src/ExceptionRequest.jl index 21241f5ab..62aa0125f 100644 --- a/src/ExceptionRequest.jl +++ b/src/ExceptionRequest.jl @@ -2,21 +2,26 @@ module ExceptionRequest export StatusError -import ..Layer, ..request +using ..Layers import ..HTTP using ..Messages: iserror """ - request(ExceptionLayer, ::URI, ::Request, body) -> HTTP.Response + Layers.request(ExceptionLayer, ::URI, ::Request, body) -> HTTP.Response Throw a `StatusError` if the request returns an error response status. """ -abstract type ExceptionLayer{Next <: Layer} <: Layer{Next} end +struct ExceptionLayer{Next <: Layer} <: RequestLayer + next::Next +end export ExceptionLayer +Layers.keywordforlayer(::Val{:status_exception}) = ExceptionLayer +ExceptionLayer(next; status_exception::Bool=true, kw...) = + status_exception ? ExceptionLayer(next) : nothing -function request(::Type{ExceptionLayer{Next}}, a...; kw...) where Next +function Layers.request(layer::ExceptionLayer, a...; kw...) - res = request(Next, a...; kw...) + res = Layers.request(layer.next, a...; kw...) if iserror(res) throw(StatusError(res.status, res.request.method, res.request.target, res)) diff --git a/src/HTTP.jl b/src/HTTP.jl index 7c1337496..d9a7e50f9 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -1,9 +1,6 @@ module HTTP -export startwrite, startread, closewrite, closeread, stack, insert, insert_default!, - remove_default!, AWS4AuthLayer, BasicAuthLayer, CanonicalizeLayer, ConnectionPoolLayer, - ContentTypeDetectionLayer, DebugLayer, ExceptionLayer, MessageLayer, RedirectLayer, - RetryLayer, StreamLayer, TimeoutLayer, TopLayer, +export startwrite, startread, closewrite, closeread, @logfmt_str, common_logfmt, combined_logfmt const DEBUG_LEVEL = Ref(0) @@ -86,7 +83,7 @@ e.g. ```julia HTTP.request("GET", "http://httpbin.org/ip"; retries=4, cookies=true) -HTTP.get("http://s3.us-east-1.amazonaws.com/"; aws_authorization=true) +HTTP.get("http://s3.us-east-1.amazonaws.com/") conf = (readtimeout = 10, retry = false, @@ -153,19 +150,7 @@ SSLContext options Basic Authentication options - Basic authentication is detected automatically from the provided url's `userinfo` (in the form `scheme://user:password@host`) - and adds the `Authorization: Basic` header - - -AWS Authentication options - - - `aws_authorization = false`, enable AWS4 Authentication. - - `aws_service = split(url.host, ".")[1]` - - `aws_region = split(url.host, ".")[2]` - - `aws_access_key_id = ENV["AWS_ACCESS_KEY_ID"]` - - `aws_secret_access_key = ENV["AWS_SECRET_ACCESS_KEY"]` - - `aws_session_token = get(ENV, "AWS_SESSION_TOKEN", "")` - - `body_sha256 = digest(MD_SHA256, body)`, - - `body_md5 = digest(MD_MD5, body)`, + and adds the `Authorization: Basic` header; this can be disabled by passing `basicauth=false` Cookie options @@ -325,9 +310,9 @@ function request(method, url, h=Header[], b=nobody; headers=h, body=b, query=nothing, kw...)::Response return request(HTTP.stack(;kw...), string(method), request_uri(url, query), mkheaders(headers), body; kw...) end -function request(stack::Type{<:Layer}, method, url, h=Header[], b=nobody; +function request(stack::Layer, method, url, h=Header[], b=nobody; headers=h, body=b, query=nothing, kw...)::Response - return request(stack, string(method), request_uri(url, query), mkheaders(headers), body; kw...) + return Layers.request(stack, string(method), request_uri(url, query), mkheaders(headers), body; kw...) end request(::Type{Union{}}, resp::Response) = resp @@ -437,10 +422,8 @@ Shorthand for `HTTP.request("DELETE", ...)`. See [`HTTP.request`](@ref). """ delete(a...; kw...) = request("DELETE", a...; kw...) -include("TopRequest.jl"); using .TopRequest include("RedirectRequest.jl"); using .RedirectRequest include("BasicAuthRequest.jl"); using .BasicAuthRequest -include("AWS4AuthRequest.jl"); using .AWS4AuthRequest include("CookieRequest.jl"); using .CookieRequest include("CanonicalizeRequest.jl"); using .CanonicalizeRequest include("TimeoutRequest.jl"); using .TimeoutRequest @@ -500,8 +483,6 @@ relationship with [`HTTP.Response`](@ref), [`HTTP.Parsers`](@ref), │ ├────────────────────────────────────────────────────────────┤ │ │ │ │ │ request(MessageLayer, method, ::URI, ::Headers, body) │ │ │ │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(AWS4AuthLayer, ::URI, ::Request, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ │ │ request(RetryLayer, ::URI, ::Request, body) │ │ │ │ ├────────────────────────────────────────────────────────────┤ │ │ │ │ │ request(ExceptionLayer, ::URI, ::Request, body) ├ ─ ┘ │ @@ -567,20 +548,43 @@ relationship with [`HTTP.Response`](@ref), [`HTTP.Parsers`](@ref), ``` *See `docs/src/layers`[`.monopic`](http://monodraw.helftone.com).* """ -function stack(; kw...) - - layers = stacklayertypes(Layers.ConnectionLayer, StreamLayer(); kw...) +function stack(; + basicauth=true, + redirect=true, + cookies=false, + canonicalize_headers=false, + retry=true, + status_exception=true, + readtimeout=0, + detect_content_type=false, + verbose=0, + kw...) + + kwargs = Dict{Symbol, Any}(kw) + kwargs[:basicauth] = basicauth + kwargs[:redirect] = redirect + kwargs[:cookies] = cookies + kwargs[:canonicalize_headers] = canonicalize_headers + kwargs[:retry] = retry + kwargs[:status_exception] = status_exception + kwargs[:readtimeout] = readtimeout + kwargs[:detect_content_type] = detect_content_type + kwargs[:verbose] = verbose + layers = stacklayertypes(Layers.ConnectionLayer, StreamLayer(), kwargs) layers = ConnectionPoolLayer(layers; kw...) - layers = stacklayertypes(Layers.RequestLayer, layers; kw...) + layers = stacklayertypes(Layers.RequestLayer, layers, kwargs) layers = MessageLayer(layers; kw...) - return stacklayertypes(Layers.InitialLayer, layers; kw...) + return stacklayertypes(Layers.InitialLayer, layers, kwargs) end -function stacklayertypes(::Type{T}, layers; kw...) where {T} - for (k, _) in pairs(kw) +function stacklayertypes(::Type{T}, layers, kwargs) where {T} + for (k, _) in pairs(kwargs) layer = Layers.keywordforlayer(Val(k)) if layer !== nothing && layer <: T - layers = layer(layers; kw...) + newlayers = layer(layers; kwargs...) + if newlayers !== nothing + layers = newlayers + end end end return layers diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index 5691bb178..527bd7112 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -2,8 +2,8 @@ module MessageRequest export body_is_a_stream, body_was_streamed, setuseragent!, resource - -import ..Layer, ..request +using ..Base64 +using ..Layers using ..IOExtras using URIs using ..Messages @@ -19,7 +19,7 @@ resource(uri::URI) = string( isempty(uri.path) ? "/" : uri.path, !isempty(uri.fragment) ? "#" : "", uri.fragment) """ - request(MessageLayer, method, ::URI, headers, body) -> HTTP.Response + Layers.request(MessageLayer, method, ::URI, headers, body) -> HTTP.Response Construct a [`Request`](@ref) object and set mandatory headers. """ @@ -29,7 +29,7 @@ end export MessageLayer MessageLayer(next; kw...) = MessageLayer(next) -function request(layer::MessageLayer, +function Layers.request(layer::MessageLayer, method::String, url::URI, headers::Headers, body; http_version=v"1.1", target=resource(url), @@ -66,7 +66,7 @@ function request(layer::MessageLayer, req = Request(method, target, headers, bodybytes(body); parent=parent, version=http_version) - return request(layer.next, url, req, body; iofunction=iofunction, kw...) + return Layers.request(layer.next, url, req, body; iofunction=iofunction, kw...) end const USER_AGENT = Ref{Union{String, Nothing}}("HTTP.jl/$VERSION") diff --git a/src/RedirectRequest.jl b/src/RedirectRequest.jl index 9ee5c085c..7e561818a 100644 --- a/src/RedirectRequest.jl +++ b/src/RedirectRequest.jl @@ -1,6 +1,6 @@ module RedirectRequest -import ..Layer, ..request +using ..Layers using URIs using ..Messages using ..Pairs: setkv @@ -8,16 +8,21 @@ import ..Header import ..@debug, ..DEBUG_LEVEL """ - request(RedirectLayer, method, ::URI, headers, body) -> HTTP.Response + Layers.request(RedirectLayer, method, ::URI, headers, body) -> HTTP.Response Redirects the request in the case of 3xx response status. """ -abstract type RedirectLayer{Next <: Layer} <: Layer{Next} end +struct RedirectLayer{Next <: Layer} <: InitialLayer + next::Next +end export RedirectLayer +Layers.keywordforlayer(::Val{:redirect}) = RedirectLayer +RedirectLayer(next; redirect::Bool=true, kw...) = + redirect ? RedirectLayer(next) : nothing -function request(::Type{RedirectLayer{Next}}, +function Layers.request(layer::RedirectLayer, method::String, url::URI, headers, body; - redirect_limit=3, forwardheaders=true, kw...) where Next + redirect_limit=3, forwardheaders=true, kw...) count = 0 while true @@ -25,7 +30,7 @@ function request(::Type{RedirectLayer{Next}}, # the redirect loop to also catch bad redirect URLs. verify_url(url) - res = request(Next, method, url, headers, body; reached_redirect_limit=(count == redirect_limit), kw...) + res = Layers.request(layer.next, method, url, headers, body; reached_redirect_limit=(count == redirect_limit), kw...) if (count == redirect_limit || !isredirect(res) diff --git a/src/RetryRequest.jl b/src/RetryRequest.jl index 66aa7f08e..f3ec30b29 100644 --- a/src/RetryRequest.jl +++ b/src/RetryRequest.jl @@ -1,7 +1,7 @@ module RetryRequest import ..HTTP -import ..Layer, ..request +using ..Layers using ..Sockets using ..IOExtras using ..MessageRequest @@ -9,7 +9,7 @@ using ..Messages import ..@debug, ..DEBUG_LEVEL, ..sprintcompact """ - request(RetryLayer, ::URI, ::Request, body) -> HTTP.Response + Layers.request(RetryLayer, ::URI, ::Request, body) -> HTTP.Response Retry the request if it throws a recoverable exception. @@ -21,14 +21,19 @@ Methods of `isrecoverable(e)` define which exception types lead to a retry. e.g. `HTTP.IOError`, `Sockets.DNSError`, `Base.EOFError` and `HTTP.StatusError` (if status is ``5xx`). """ -abstract type RetryLayer{Next <: Layer} <: Layer{Next} end +struct RetryLayer{Next <: Layer} <: RequestLayer + next::Next +end export RetryLayer +Layers.keywordforlayer(::Val{:retry}) = RetryLayer +RetryLayer(next; retry::Bool=true, kw...) = + retry ? RetryLayer(next) : nothing -function request(::Type{RetryLayer{Next}}, url, req, body; +function Layers.request(layer::RetryLayer, url, req, body; retries::Int=4, retry_non_idempotent::Bool=false, - kw...) where Next + kw...) - retry_request = Base.retry(request, + retry_request = Base.retry(Layers.request, delays=ExponentialBackOff(n = retries), check=(s,ex)->begin retry = isrecoverable(ex, req, retry_non_idempotent) @@ -41,7 +46,7 @@ function request(::Type{RetryLayer{Next}}, url, req, body; return s, retry end) - retry_request(Next, url, req, body; kw...) + retry_request(layer.next, url, req, body; kw...) end isrecoverable(e) = false diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index 97988c61a..affcb933d 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -1,6 +1,6 @@ module StreamRequest -import ..Layer, ..request +using ..Layers using ..IOExtras using ..Messages using ..Streams @@ -9,7 +9,7 @@ using ..MessageRequest import ..@debug, ..DEBUG_LEVEL, ..printlncompact, ..sprintcompact """ - request(StreamLayer, ::IO, ::Request, body) -> HTTP.Response + Layers.request(StreamLayer, ::IO, ::Request, body) -> HTTP.Response Create a [`Stream`](@ref) to send a `Request` and `body` to an `IO` stream and read the response. @@ -22,7 +22,7 @@ indicates that the server does not wish to receive the message body. struct StreamLayer <: ConnectionLayer end export StreamLayer -function request(::StreamLayer, io::IO, req::Request, body; +function Layers.request(::StreamLayer, io::IO, req::Request, body; reached_redirect_limit=false, response_stream=nothing, iofunction=nothing, diff --git a/src/TimeoutRequest.jl b/src/TimeoutRequest.jl index 4f7204739..5028705e9 100644 --- a/src/TimeoutRequest.jl +++ b/src/TimeoutRequest.jl @@ -1,6 +1,6 @@ module TimeoutRequest -import ..Layer, ..request +using ..Layers using ..ConnectionPool import ..@debug, ..DEBUG_LEVEL @@ -13,7 +13,7 @@ function Base.showerror(io::IO, e::ReadTimeoutError) end """ - request(TimeoutLayer, ::IO, ::Request, body) -> HTTP.Response + Layers.request(TimeoutLayer, ::IO, ::Request, body) -> HTTP.Response Close `IO` if no data has been received for `timeout` seconds. """ @@ -23,10 +23,10 @@ struct TimeoutLayer{Next <: Layer} <: ConnectionLayer end export TimeoutLayer Layers.keywordforlayer(::Val{:readtimeout}) = TimeoutLayer +TimeoutLayer(next; readtimeout::Int=0, kw...) = + readtimeout > 0 ? TimeoutLayer(next, readtimeout) : nothing -TimeoutLayer(next; readtimeout::Int=0, kw...) = TimeoutLayer(next, readtimeout) - -function request(layer::TimeoutLayer, io::IO, req, body; kw...) +function Layers.request(layer::TimeoutLayer, io::IO, req, body; kw...) readtimeout = layer.readtimeout wait_for_timeout = Ref{Bool}(true) timedout = Ref{Bool}(false) @@ -42,7 +42,7 @@ function request(layer::TimeoutLayer, io::IO, req, body; kw...) end try - return request(layer.next, io, req, body; kw...) + return Layers.request(layer.next, io, req, body; kw...) catch e if timedout[] throw(ReadTimeoutError(readtimeout)) diff --git a/src/TopRequest.jl b/src/TopRequest.jl deleted file mode 100644 index e965b542a..000000000 --- a/src/TopRequest.jl +++ /dev/null @@ -1,18 +0,0 @@ -module TopRequest - -import ..Layer, ..request - -export TopLayer - -""" - request(TopLayer, args...; kwargs...) - -This layer is at the top of every stack, and does nothing. -It's useful for inserting a custom layer at the top of the stack. -""" -struct TopLayer{Next <: Layer} <: Layer{Next} end - -request(::Type{TopLayer{Next}}, args...; kwargs...) where Next = - request(Next, args...; kwargs...) - -end diff --git a/src/layers.jl b/src/layers.jl index 8f9564a32..ec18571db 100644 --- a/src/layers.jl +++ b/src/layers.jl @@ -1,5 +1,5 @@ module Layers -export Layer, keywordforlayer +export Layer, InitialLayer, RequestLayer, ConnectionLayer struct LayerNotFoundException <: Exception var::String @@ -25,60 +25,6 @@ keywordforlayer(kw) = nothing # must overload: request(layer::MyLayer, args...; kw...) # in `request` overload, must call: request(layer.next, args...; kw...) -""" - next(::Type{S}) where {T, S<:Layer{T}} - -Return the next `Layer` in the stack - -Example: -stack = MessageLayer{ConnectionPoolLayer{StreamLayer{Union{}}} -next(stack) # ConnectionPoolLayer{StreamLayer{Union{}}} -""" -next(::Type{S}) where {T, S<:Layer{T}} = T - -""" - top_layer(::Type{T}) where T <: Layer - -Return the parametric type of the top most `Layer` in the stack - -Example: -stack = MessageLayer{ConnectionPoolLayer{StreamLayer{Union{}}} -top_layer(stack) # MessageLayer -""" -top_layer(::Type{T}) where T <: Layer = T.name.wrapper -top_layer(::Type{Union{}}) = Union{} - -""" - insert(stack::Type{<:Layer}, layer_before::Type{<:Layer}, custom_layer::Type{<:Layer}) - -Insert your `custom_layer` in-front of the `layer_before` - -Example: -stack = MessageLayer{ConnectionPoolLayer{StreamLayer{Union{}}} -result = insert(stack, MessageLayer, TestLayer) # TestLayer{MessageLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}} -""" -function insert(stack::Type{<:Layer}, layer_before::Type{<:Layer}, custom_layer::Type{<:Layer}) - new_stack = Union - head_layer = top_layer(stack) - rest_stack = stack - - while true - if head_layer === layer_before - return new_stack{custom_layer{rest_stack}} - else - head_layer === Union{} && break - new_stack = new_stack{head_layer{T}} where T - rest_stack = next(rest_stack) - head_layer = top_layer(rest_stack) - end - end - throw(LayerNotFoundException("$layer_before not found in $stack")) -end - -insert_default!(before::Type{<:Layer}, custom_layer::Type{<:Layer}) = - push!(EXTRA_LAYERS, (before, custom_layer)) - -remove_default!(before::Type{<:Layer}, custom_layer::Type{<:Layer}) = - delete!(EXTRA_LAYERS, (before, custom_layer)) +function request end end diff --git a/test/aws4.jl b/test/aws4.jl deleted file mode 100644 index 97cb3c67f..000000000 --- a/test/aws4.jl +++ /dev/null @@ -1,232 +0,0 @@ -using Dates -using Test -using HTTP -using HTTP: Headers, URI -using HTTP.AWS4AuthRequest: sign_aws4! - -const useragent = HTTP.MessageRequest.USER_AGENT[] -HTTP.setuseragent!(nothing) - -# Based on https://docs.aws.amazon.com/general/latest/gr/signature-v4-test-suite.html -# and https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html - -function test_sign!(method, headers, params, body=""; opts...) - sign_aws4!(method, - URI("https://example.amazonaws.com/" * params), - headers, - Vector{UInt8}(body); - timestamp=DateTime(2015, 8, 30, 12, 36), - aws_service="service", - aws_region="us-east-1", - # NOTE: These are the example credentials as specified in the AWS docs, - # they are not real - aws_access_key_id="AKIDEXAMPLE", - aws_secret_access_key="wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", - include_md5=false, - include_sha256=false, - opts...) - headers -end - -function test_auth_string(headers, sig, key="AKIDEXAMPLE", date="20150830", service="service") - d = [ - "AWS4-HMAC-SHA256 Credential" => "$key/$date/us-east-1/$service/aws4_request", - "SignedHeaders" => headers, - "Signature" => sig, - ] - join(map(p->join(p, '='), d), ", ") -end - -header_keys(headers) = sort!(map(first, headers)) - -const required_headers = ["Authorization", "host", "x-amz-date"] - -@testset "AWS Signature Version 4" begin - # The signature for requests with no headers where the path ends up as simply / - slash_only_sig = "5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31" - noheaders = [ - ("get-vanilla", "", slash_only_sig), - ("get-vanilla-empty-query-key", "?Param1=value1", "a67d582fa61cc504c4bae71f336f98b97f1ea3c7a6bfe1b6e45aec72011b9aeb"), - ("get-utf8", "ሴ", "8318018e0b0f223aa2bbf98705b62bb787dc9c0e678f255a891fd03141be5d85"), - ("get-relative", "example/..", slash_only_sig), - ("get-relative-relative", "example1/example2/../..", slash_only_sig), - ("get-slash", "/", slash_only_sig), - ("get-slash-dot-slash", "./", slash_only_sig), - ("get-slashes", "example/", "9a624bd73a37c9a373b5312afbebe7a714a789de108f0bdfe846570885f57e84"), - ("get-slash-pointless-dot", "./example", "ef75d96142cf21edca26f06005da7988e4f8dc83a165a80865db7089db637ec5"), - ("get-space", "example space/", "652487583200325589f1fba4c7e578f72c47cb61beeca81406b39ddec1366741"), - ("post-vanilla", "", "5da7c1a2acd57cee7505fc6676e4e544621c30862966e37dddb68e92efbe5d6b"), - ("post-vanilla-empty-query-value", "?Param1=value1", "28038455d6de14eafc1f9222cf5aa6f1a96197d7deb8263271d420d138af7f11"), - ] - @testset "$name" for (name, p, sig) in noheaders - m = startswith(name, "get") ? "GET" : "POST" - headers = test_sign!(m, Headers([]), p) - @test header_keys(headers) == required_headers - d = Dict(headers) - @test d["x-amz-date"] == "20150830T123600Z" - @test d["host"] == "example.amazonaws.com" - @test d["Authorization"] == test_auth_string("host;x-amz-date", sig) - end - - yesheaders = [ - ("get-header-key-duplicate", "", "", - Headers(["My-Header1" => "value2", - "My-Header1" => "value2", - "My-Header1" => "value1"]), - "host;my-header1;x-amz-date", - "c9d5ea9f3f72853aea855b47ea873832890dbdd183b4468f858259531a5138ea"), - ("get-header-value-multiline", "", "", - Headers(["My-Header1" => "value1\n value2\n value3"]), - "host;my-header1;x-amz-date", - "ba17b383a53190154eb5fa66a1b836cc297cc0a3d70a5d00705980573d8ff790"), - ("get-header-value-order", "", "", - Headers(["My-Header1" => "value4", - "My-Header1" => "value1", - "My-Header1" => "value3", - "My-Header1" => "value2"]), - "host;my-header1;x-amz-date", - "08c7e5a9acfcfeb3ab6b2185e75ce8b1deb5e634ec47601a50643f830c755c01"), - ("get-header-value-trim", "", "", - Headers(["My-Header1" => " value1", - "My-Header2" => " \"a b c\""]), - "host;my-header1;my-header2;x-amz-date", - "acc3ed3afb60bb290fc8d2dd0098b9911fcaa05412b367055dee359757a9c736"), - ("post-header-key-sort", "", "", - Headers(["My-Header1" => "value1"]), - "host;my-header1;x-amz-date", - "c5410059b04c1ee005303aed430f6e6645f61f4dc9e1461ec8f8916fdf18852c"), - ("post-header-value-case", "", "", - Headers(["My-Header1" => "VALUE1"]), - "host;my-header1;x-amz-date", - "cdbc9802e29d2942e5e10b5bccfdd67c5f22c7c4e8ae67b53629efa58b974b7d"), - ("post-x-www-form-urlencoded", "", "Param1=value1", - Headers(["Content-Type" => "application/x-www-form-urlencoded", - "Content-Length" => "13"]), - "content-type;host;x-amz-date", - "ff11897932ad3f4e8b18135d722051e5ac45fc38421b1da7b9d196a0fe09473a"), - ("post-x-www-form-urlencoded-parameters", "", "Param1=value1", - Headers(["Content-Type" => "application/x-www-form-urlencoded; charset=utf8", - "Content-Length" => "13"]), - "content-type;host;x-amz-date", - "1a72ec8f64bd914b0e42e42607c7fbce7fb2c7465f63e3092b3b0d39fa77a6fe"), - ] - @testset "$name" for (name, p, body, h, sh, sig) in yesheaders - hh = sort(map(first, h)) - m = startswith(name, "get") ? "GET" : "POST" - test_sign!(m, h, p, body) - @test header_keys(h) == sort(vcat(required_headers, hh)) - d = Dict(h) # collapses duplicates but we don't care here - @test d["x-amz-date"] == "20150830T123600Z" - @test d["host"] == "example.amazonaws.com" - @test d["Authorization"] == test_auth_string(sh, sig) - end - - @testset "AWS Security Token Service" begin - # Not a real security token, provided by AWS as an example - token = string("AQoDYXdzEPT//////////wEXAMPLEtc764bNrC9SAPBSM22wDOk4x4HIZ8j4FZTwd", - "QWLWsKWHGBuFqwAeMicRXmxfpSPfIeoIYRqTflfKD8YUuwthAx7mSEI/qkPpKPi/k", - "McGdQrmGdeehM4IC1NtBmUpp2wUE8phUZampKsburEDy0KPkyQDYwT7WZ0wq5VSXD", - "vp75YU9HFvlRd8Tx6q6fE8YQcHNVXAkiY9q6d+xo0rKwT38xVqr7ZD0u0iPPkUL64", - "lIZbqBAz+scqKmlzm8FDrypNC9Yjc8fPOLn9FX9KSYvKTr4rvx3iSIlTJabIQwj2I", - "CCR/oLxBA==") - @testset "Token included in signature" begin - sh = "host;x-amz-date;x-amz-security-token" - sig = "85d96828115b5dc0cfc3bd16ad9e210dd772bbebba041836c64533a82be05ead" - h = test_sign!("POST", Headers([]), "", aws_session_token=token) - d = Dict(h) - @test d["Authorization"] == test_auth_string(sh, sig) - @test haskey(d, "x-amz-security-token") - end - @testset "Token not included in signature" begin - sh = "host;x-amz-date" - sig = "5da7c1a2acd57cee7505fc6676e4e544621c30862966e37dddb68e92efbe5d6b" - h = test_sign!("POST", Headers([]), "", aws_session_token=token, token_in_signature=false) - d = Dict(h) - @test d["Authorization"] == test_auth_string(sh, sig) - @test haskey(d, "x-amz-security-token") - end - end - - @testset "AWS Simple Storage Service" begin - s3url = "https://examplebucket.s3.amazonaws.com" - opts = (timestamp=DateTime(2013, 5, 24), - aws_service="s3", - aws_region="us-east-1", - # NOTE: These are the example credentials as specified in the AWS docs, - # they are not real - aws_access_key_id="AKIAIOSFODNN7EXAMPLE", - aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - include_md5=false) - - @testset "GET Object" begin - sh = "host;range;x-amz-content-sha256;x-amz-date" - sig = "f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41" - h = Headers(["Range" => "bytes=0-9"]) - sign_aws4!("GET", URI(s3url * "/test.txt"), h, UInt8[]; opts...) - d = Dict(h) - @test d["Authorization"] == test_auth_string(sh, sig, opts.aws_access_key_id, "20130524", "s3") - @test haskey(d, "x-amz-content-sha256") # required for S3 requests - end - - @testset "PUT Object" begin - sh = "date;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class" - sig = "98ad721746da40c64f1a55b78f14c238d841ea1380cd77a1b5971af0ece108bd" - h = Headers(["Date" => "Fri, 24 May 2013 00:00:00 GMT", - "x-amz-storage-class" => "REDUCED_REDUNDANCY"]) - sign_aws4!("PUT", URI(s3url * "/test\$file.text"), h, UInt8[]; - # Override the SHA-256 of the request body, since the actual body is not provided - # for this example in the documentation, only the SHA - body_sha256=hex2bytes("44ce7dd67c959e0d3524ffac1771dfbba87d2b6b4b4e99e42034a8b803f8b072"), - opts...) - d = Dict(h) - @test d["Authorization"] == test_auth_string(sh, sig, opts.aws_access_key_id, "20130524", "s3") - @test haskey(d, "x-amz-content-sha256") - end - - @testset "GET Bucket Lifecycle" begin - sh = "host;x-amz-content-sha256;x-amz-date" - sig = "fea454ca298b7da1c68078a5d1bdbfbbe0d65c699e0f91ac7a200a0136783543" - h = Headers([]) - sign_aws4!("GET", URI(s3url * "/?lifecycle"), h, UInt8[]; opts...) - d = Dict(h) - @test d["Authorization"] == test_auth_string(sh, sig, opts.aws_access_key_id, "20130524", "s3") - @test haskey(d, "x-amz-content-sha256") - end - - @testset "GET Bucket (List Objects)" begin - sh = "host;x-amz-content-sha256;x-amz-date" - sig = "34b48302e7b5fa45bde8084f4b7868a86f0a534bc59db6670ed5711ef69dc6f7" - h = Headers([]) - sign_aws4!("GET", URI(s3url * "/?max-keys=2&prefix=J"), h, UInt8[]; opts...) - d = Dict(h) - @test d["Authorization"] == test_auth_string(sh, sig, opts.aws_access_key_id, "20130524", "s3") - @test haskey(d, "x-amz-content-sha256") - end - end - - @testset "HTTP.request with AWS authentication" begin - resp = HTTP.request("GET", - "https://httpbin.org/headers"; - aws_authorization=true, - timestamp=DateTime(2015, 8, 30, 12, 36), - aws_service="service", - aws_region="us-east-1", - # NOTE: These are the example credentials as specified in the AWS docs, - # they are not real - aws_access_key_id="AKIDEXAMPLE", - aws_secret_access_key="wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", - include_md5=false, - include_sha256=false) - @test resp.status == 200 - headers = JSON.parse(String(resp.body))["headers"] - @test headers["Host"] == "httpbin.org" - @test headers["X-Amz-Date"] == "20150830T123600Z" - auth = "AWS4-HMAC-SHA256 " * - "Credential=AKIDEXAMPLE/20150830/us-east-1/service/aws4_request, " * - "SignedHeaders=accept;host;x-amz-date, " * - "Signature=599128338d879e1e2aee2ce58222da02b47428696b7cf7c5d1b4a9cc75749ff9" - @test headers["Authorization"] == auth - end -end - -HTTP.setuseragent!(useragent) \ No newline at end of file diff --git a/test/client.jl b/test/client.jl index 1781214b5..ec7d8bd23 100644 --- a/test/client.jl +++ b/test/client.jl @@ -8,10 +8,10 @@ using URIs status(r) = r.status @testset "Custom HTTP Stack" begin @testset "Low-level Request" begin - custom_stack = insert(stack(), StreamLayer, TestLayer) - result = request(custom_stack, "GET", "https://httpbin.org/ip") - + wasincluded = Ref(false) + result = HTTP.request("GET", "https://httpbin.org/ip"; httptestlayer=wasincluded) @test status(result) == 200 + @test wasincluded[] end end diff --git a/test/insert_layers.jl b/test/insert_layers.jl deleted file mode 100644 index 27a63153f..000000000 --- a/test/insert_layers.jl +++ /dev/null @@ -1,55 +0,0 @@ -include("../src/exceptions.jl") - -using ..TestRequest - -@testset "HTTP Stack Inserting" begin - @testset "Insert - Beginning" begin - expected = TestLayer{TopLayer{RedirectLayer{BasicAuthLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}}}}}} - result = insert(stack(), TopLayer, TestLayer) - - @test expected == result - end - - @testset "Insert - Middle" begin - expected = TopLayer{RedirectLayer{BasicAuthLayer{MessageLayer{RetryLayer{TestLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}}}}}} - result = insert(stack(), ExceptionLayer, TestLayer) - - @test expected == result - end - - @testset "Insert - End" begin - expected = TopLayer{RedirectLayer{BasicAuthLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{TestLayer{Union{}}}}}}}}}} - result = insert(stack(), Union{}, TestLayer) - - @test expected == result - end - - @testset "Insert - Non-existant layer" begin - @test_throws HTTP.Layers.LayerNotFoundException insert(stack(), AWS4AuthLayer, TestLayer) - end - - @testset "Insert - Multiple Same layer" begin - test_stack = insert(stack(), RetryLayer, ExceptionLayer) - - expected = TopLayer{RedirectLayer{BasicAuthLayer{MessageLayer{TestLayer{ExceptionLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}}}}}}} - result = insert(test_stack, ExceptionLayer, TestLayer) - - @test expected == result - end - - @testset "Inserted final layer runs handler" begin - TestRequest.FLAG[] = false - request(insert(stack(), Union{}, LastLayer), "GET", "https://httpbin.org/anything") - @test TestRequest.FLAG[] - end - - @testset "Insert/remove default layers" begin - top = HTTP.top_layer(stack()) - insert_default!(top, TestLayer) - @test HTTP.top_layer(stack()) <: TestLayer - remove_default!(top, TestLayer) - @test HTTP.top_layer(stack()) <: top - insert_default!(Union{}, TestLayer) - remove_default!(Union{}, TestLayer) - end -end diff --git a/test/resources/TestRequest.jl b/test/resources/TestRequest.jl index 16d8ce1fe..46fae4c18 100644 --- a/test/resources/TestRequest.jl +++ b/test/resources/TestRequest.jl @@ -1,18 +1,32 @@ module TestRequest -import HTTP: Layer, request, Response -abstract type TestLayer{Next <: Layer} <: Layer{Next} end -abstract type LastLayer{Next <: Layer} <: Layer{Next} end -export TestLayer, LastLayer, request +export TestLayer, LastLayer -function request(::Type{TestLayer{Next}}, io::IO, req, body; kw...)::Response where Next - return request(Next, io, req, body; kw...) +using HTTP, HTTP.Layers + +struct TestLayer{Next <: Layer} <: InitialLayer + next::Next + wasincluded::Ref{Bool} +end +Layers.keywordforlayer(::Val{:httptestlayer}) = TestLayer +TestLayer(next; httptestlayer=Ref(false), kw...) = TestLayer(next, httptestlayer) + +function Layers.request(layer::TestLayer, meth, url, headers, body; kw...) + layer.wasincluded[] = true + return Layers.request(layer.next, meth, url, headers, body; kw...) +end + +struct LastLayer{Next <: Layer} <: ConnectionLayer + next::Next + wasincluded::Ref{Bool} end +Layers.keywordforlayer(::Val{:httplastlayer}) = LastLayer +LastLayer(next; httplastlayer=Ref(false), kw...) = LastLayer(next, httplastlayer) -const FLAG = Ref(false) -function request(::Type{LastLayer{Next}}, resp)::Response where Next - FLAG[] = true - return request(Next, resp) +function Layers.request(layer::LastLayer, io::IO, req, body; kw...) + resp = Layers.request(layer.next, io, req, body; kw...) + layer.wasincluded[] = true + return resp end end diff --git a/test/runtests.jl b/test/runtests.jl index 3e831cbcc..202239042 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -20,8 +20,6 @@ include(joinpath(dir, "resources/TestRequest.jl")) "handlers.jl", "server.jl", "async.jl", - "aws4.jl", - "insert_layers.jl", "mwe.jl", ] file = joinpath(dir, f) From ee3aec8fc6944744627ab692ff0b094cbc8449aa Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 16 Dec 2021 00:24:00 -0700 Subject: [PATCH 03/19] fix tests --- src/HTTP.jl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/HTTP.jl b/src/HTTP.jl index d9a7e50f9..3346b5d48 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -307,8 +307,8 @@ end ``` """ function request(method, url, h=Header[], b=nobody; - headers=h, body=b, query=nothing, kw...)::Response - return request(HTTP.stack(;kw...), string(method), request_uri(url, query), mkheaders(headers), body; kw...) + headers=h, body=b, kw...)::Response + return request(HTTP.stack(;kw...), method, url, headers, body; kw...) end function request(stack::Layer, method, url, h=Header[], b=nobody; headers=h, body=b, query=nothing, kw...)::Response From ad7c1a75150f26976caa2af213d91d33fe3c07cb Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 16 Dec 2021 09:52:09 -0700 Subject: [PATCH 04/19] Fix 32-bit by ensuring consistent order of layers --- src/HTTP.jl | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/src/HTTP.jl b/src/HTTP.jl index 3346b5d48..7ef4a7872 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -550,7 +550,6 @@ relationship with [`HTTP.Response`](@ref), [`HTTP.Parsers`](@ref), """ function stack(; basicauth=true, - redirect=true, cookies=false, canonicalize_headers=false, retry=true, @@ -558,18 +557,17 @@ function stack(; readtimeout=0, detect_content_type=false, verbose=0, + redirect=true, kw...) - kwargs = Dict{Symbol, Any}(kw) - kwargs[:basicauth] = basicauth - kwargs[:redirect] = redirect - kwargs[:cookies] = cookies - kwargs[:canonicalize_headers] = canonicalize_headers - kwargs[:retry] = retry - kwargs[:status_exception] = status_exception - kwargs[:readtimeout] = readtimeout - kwargs[:detect_content_type] = detect_content_type - kwargs[:verbose] = verbose + kwargs = merge( + (; + basicauth, cookies, canonicalize_headers, + retry, status_exception, readtimeout, + detect_content_type, verbose, redirect + ), + kw + ) layers = stacklayertypes(Layers.ConnectionLayer, StreamLayer(), kwargs) layers = ConnectionPoolLayer(layers; kw...) layers = stacklayertypes(Layers.RequestLayer, layers, kwargs) From 65188edc92c45fe5c081cd761b530ee5685de718 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 13 Jan 2022 13:04:19 -0700 Subject: [PATCH 05/19] more work --- src/BasicAuthRequest.jl | 6 ++---- src/CanonicalizeRequest.jl | 9 +++++---- src/ConnectionRequest.jl | 15 +++++++++------ src/ContentTypeRequest.jl | 10 +++++----- src/CookieRequest.jl | 16 +++++++++------- src/DebugRequest.jl | 10 +++++----- src/ExceptionRequest.jl | 13 +++++++------ src/HTTP.jl | 8 +++++--- src/MessageRequest.jl | 18 +++++++++++------- src/RedirectRequest.jl | 19 +++++++++++++------ src/RetryRequest.jl | 12 ++++++------ src/StreamRequest.jl | 30 ++++++++++++++++++++---------- src/TimeoutRequest.jl | 8 ++++---- src/layers.jl | 7 ++++++- 14 files changed, 107 insertions(+), 74 deletions(-) diff --git a/src/BasicAuthRequest.jl b/src/BasicAuthRequest.jl index 8bb5dc2b1..1c2a77e8f 100644 --- a/src/BasicAuthRequest.jl +++ b/src/BasicAuthRequest.jl @@ -18,9 +18,7 @@ export BasicAuthLayer Layers.keywordforlayer(::Val{:basicauth}) = BasicAuthLayer BasicAuthLayer(next; kw...) = BasicAuthLayer(next) -function Layers.request(layer::BasicAuthLayer, - method::String, url::URI, headers, body; kw...) - +function Layers.request(layer::BasicAuthLayer, method::String, url::URI, headers, body) userinfo = unescapeuri(url.userinfo) if !isempty(userinfo) && getkv(headers, "Authorization", "") == "" @@ -28,7 +26,7 @@ function Layers.request(layer::BasicAuthLayer, setkv(headers, "Authorization", "Basic $(base64encode(userinfo))") end - return Layers.request(layer.next, method, url, headers, body; kw...) + return Layers.request(layer.next, method, url, headers, body) end diff --git a/src/CanonicalizeRequest.jl b/src/CanonicalizeRequest.jl index 01f865373..dc29b37d9 100644 --- a/src/CanonicalizeRequest.jl +++ b/src/CanonicalizeRequest.jl @@ -14,14 +14,15 @@ struct CanonicalizeLayer{Next <: Layer} <: InitialLayer end export CanonicalizeLayer Layers.keywordforlayer(::Val{:canonicalize_headers}) = CanonicalizeLayer -CanonicalizeLayer(next; canonicalize_headers::Bool=true, kw...) = - canonicalize_headers ? CanonicalizeLayer(next) : nothing +Layers.shouldinclude(::Type{CanonicalizeLayer}; canonicalize_headers::Bool=true, kw...) = + canonicalize_headers +CanonicalizeLayer(next; kw...) = CanonicalizeLayer(next) -function Layers.request(layer::CanonicalizeLayer, method::String, url, headers, body; kw...) +function Layers.request(layer::CanonicalizeLayer, method::String, url, headers, body) headers = canonicalizeheaders(headers) - res = Layers.request(layer.next, method, url, headers, body; kw...) + res = Layers.request(layer.next, method, url, headers, body) res.headers = canonicalizeheaders(res.headers) diff --git a/src/ConnectionRequest.jl b/src/ConnectionRequest.jl index 22ec2de32..174d244be 100644 --- a/src/ConnectionRequest.jl +++ b/src/ConnectionRequest.jl @@ -59,14 +59,17 @@ See [`isioerror`](@ref). """ struct ConnectionPoolLayer{Next <: Layer} <: RequestLayer next::Next + proxy::String + socket_type::Any end export ConnectionPoolLayer -ConnectionPoolLayer(next; kw...) = ConnectionPoolLayer(next) - -function Layers.request(layer::ConnectionPoolLayer, url::URI, req, body; - proxy=getproxy(url.scheme, url.host), - socket_type::Type=TCPSocket, kw...) +ConnectionPoolLayer(next; + proxy=getproxy(url.scheme, url.host), + socket_type::Type=TCPSocket, + kw...) = ConnectionPoolLayer(next, proxy, socket_type) +function Layers.request(layer::ConnectionPoolLayer, url::URI, req, body) + proxy, socket_type = layer.proxy, layer.socket_type if proxy !== nothing target_url = url url = URI(proxy) @@ -102,7 +105,7 @@ function Layers.request(layer::ConnectionPoolLayer, url::URI, req, body; req.headers = filter(x->x.first != "Proxy-Authorization", req.headers) end - r = Layers.request(layer.next, io, req, body; kw...) + r = Layers.request(layer.next, io, req, body) if proxy !== nothing && target_url.scheme == "https" close(io) diff --git a/src/ContentTypeRequest.jl b/src/ContentTypeRequest.jl index dfc42b459..77fee9993 100644 --- a/src/ContentTypeRequest.jl +++ b/src/ContentTypeRequest.jl @@ -14,11 +14,11 @@ struct ContentTypeDetectionLayer{Next <: Layer} <: InitialLayer end export ContentTypeDetectionLayer Layers.keywordforlayer(::Val{:detect_content_type}) = ContentTypeDetectionLayer -ContentTypeDetectionLayer(next; detect_content_type::Bool=true, kw...) = - detect_content_type ? ContentTypeDetectionLayer(netx) : nothing +Layers.shouldinclude(::Type{ContentTypeDetectionLayer}; + detect_content_type::Bool=true) = detect_content_type +ContentTypeDetectionLayer(next; kw...) = ContentTypeDetectionLayer(netx) -function Layers.request(layer::ContentTypeDetectionLayer, - method::String, url::URI, headers, body; kw...) +function Layers.request(layer::ContentTypeDetectionLayer, method::String, url::URI, headers, body) if (getkv(headers, "Content-Type", "") == "" && !isa(body, Form) @@ -29,7 +29,7 @@ function Layers.request(layer::ContentTypeDetectionLayer, setkv(headers, "Content-Type", sn) @debug 1 "setting Content-Type header to: $sn" end - return Layers.request(layer.next, method, url, headers, body; kw...) + return Layers.request(layer.next, method, url, headers, body) end end # module diff --git a/src/CookieRequest.jl b/src/CookieRequest.jl index 3d914d156..8ad876be7 100644 --- a/src/CookieRequest.jl +++ b/src/CookieRequest.jl @@ -23,17 +23,19 @@ Store new Cookies found in the response headers. """ struct CookieLayer{Next <: Layer} <: InitialLayer next::Next + cookies cookiejar::Dict{String, Set{Cookie}} end export CookieLayer Layers.keywordforlayer(::Val{:cookies}) = CookieLayer -CookieLayer(next; cookies::Union{Bool, AbstractDict}=true, cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) = - (cookies === true || (cookies isa AbstractDict && !isempty(cookies))) ? CookieLayer(next, cookiejar) : nothing +Layers.shouldinclude(::Type{CookieLayer}; cookies=true, kw...) = + cookies === true || (cookies isa AbstractDict && !isempty(cookies)) +CookieLayer(next; + cookies=true, + cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) = + CookieLayer(next, cookies, cookiejar) -function Layers.request(layer::CookieLayer, - method::String, url::URI, headers, body; - cookies::Union{Bool, Dict{<:AbstractString, <:AbstractString}}=Dict{String, String}(), - kw...) +function Layers.request(layer::CookieLayer, method::String, url::URI, headers, body) cookiejar = layer.cookiejar hostcookies = get!(cookiejar, url.host, Set{Cookie}()) @@ -48,7 +50,7 @@ function Layers.request(layer::CookieLayer, setkv(headers, "Cookie", stringify(getkv(headers, "Cookie", ""), cookiestosend)) end - res = Layers.request(layer.next, method, url, headers, body; kw...) + res = Layers.request(layer.next, method, url, headers, body) setcookies(hostcookies, url.host, res.headers) diff --git a/src/DebugRequest.jl b/src/DebugRequest.jl index dd456003d..59217f282 100644 --- a/src/DebugRequest.jl +++ b/src/DebugRequest.jl @@ -18,17 +18,17 @@ struct DebugLayer{Next <:Layer} <: ConnectionLayer end export DebugLayer Layers.keywordforlayer(::Val{:verbose}) = DebugLayer -DebugLayer(next; verbose=0, kw...) = - (verbose >= 3 || DEBUG_LEVEL[] >= 3) ? DebugLayer(next) : nothing +Layers.shouldinclude(::Type{DebugLayer}; verbose=0, kw...) = (verbose >= 3 || DEBUG_LEVEL[] >= 3) +DebugLayer(next; kw...) = DebugLayer(next) -function Layers.request(layer::DebugLayer, io::IO, req, body; kw...) +function Layers.request(layer::DebugLayer, io::IO, req, body) @static if live_mode - return Layers.request(layer.next, IODebug(io), req, body; kw...) + return Layers.request(layer.next, IODebug(io), req, body) else iod = IODebug(io) try - return Layers.request(layer.next, iod, req, body; kw...) + return Layers.request(layer.next, iod, req, body) finally show_log(stdout, iod) end diff --git a/src/ExceptionRequest.jl b/src/ExceptionRequest.jl index 62aa0125f..7e3918790 100644 --- a/src/ExceptionRequest.jl +++ b/src/ExceptionRequest.jl @@ -7,21 +7,22 @@ import ..HTTP using ..Messages: iserror """ - Layers.request(ExceptionLayer, ::URI, ::Request, body) -> HTTP.Response + Layers.request(ExceptionLayer, ::Response) -> HTTP.Response Throw a `StatusError` if the request returns an error response status. """ -struct ExceptionLayer{Next <: Layer} <: RequestLayer +struct ExceptionLayer{Next <: Layer} <: ResponseLayer next::Next end export ExceptionLayer Layers.keywordforlayer(::Val{:status_exception}) = ExceptionLayer -ExceptionLayer(next; status_exception::Bool=true, kw...) = - status_exception ? ExceptionLayer(next) : nothing +Layers.shouldinclude(::Type{<:ExceptionLayer}; status_exception::Bool=true) = + status_exception +ExceptionLayer(next; kw...) = ExceptionLayer(next) -function Layers.request(layer::ExceptionLayer, a...; kw...) +function Layers.request(layer::ExceptionLayer, resp) - res = Layers.request(layer.next, a...; kw...) + res = Layers.request(layer.next, resp) if iserror(res) throw(StatusError(res.status, res.request.method, res.request.target, res)) diff --git a/src/HTTP.jl b/src/HTTP.jl index 7ef4a7872..117964e78 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -311,8 +311,8 @@ function request(method, url, h=Header[], b=nobody; return request(HTTP.stack(;kw...), method, url, headers, body; kw...) end function request(stack::Layer, method, url, h=Header[], b=nobody; - headers=h, body=b, query=nothing, kw...)::Response - return Layers.request(stack, string(method), request_uri(url, query), mkheaders(headers), body; kw...) + headers=h, body=b, query=nothing)::Response + return Layers.request(stack, string(method), request_uri(url, query), mkheaders(headers), body) end request(::Type{Union{}}, resp::Response) = resp @@ -568,7 +568,9 @@ function stack(; ), kw ) - layers = stacklayertypes(Layers.ConnectionLayer, StreamLayer(), kwargs) + layers = stacklayertypes(Layers.ResponseLayer, Union{}, kwargs) + layers = StreamLayer(layers; kw...) + layers = stacklayertypes(Layers.ConnectionLayer, layers, kwargs) layers = ConnectionPoolLayer(layers; kw...) layers = stacklayertypes(Layers.RequestLayer, layers, kwargs) layers = MessageLayer(layers; kw...) diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index 527bd7112..e630bf918 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -25,15 +25,19 @@ Construct a [`Request`](@ref) object and set mandatory headers. """ struct MessageLayer{Next <: Layer} <: RequestLayer next::Next + http_version::VersionNumber + target::String + parent::Union{Nothing, Request} + iofunction end export MessageLayer -MessageLayer(next; kw...) = MessageLayer(next) +MessageLayer(next; + http_version=v"1.1", + target=resource(url), + parent=nothing, iofunction=nothing, + kw...) = MessageLayer(next, http_version, target, parent, iofunction) -function Layers.request(layer::MessageLayer, - method::String, url::URI, headers::Headers, body; - http_version=v"1.1", - target=resource(url), - parent=nothing, iofunction=nothing, kw...) +function Layers.request(layer::MessageLayer, method::String, url::URI, headers::Headers, body) if isempty(url.port) || (url.scheme == "http" && url.port == "80") || @@ -66,7 +70,7 @@ function Layers.request(layer::MessageLayer, req = Request(method, target, headers, bodybytes(body); parent=parent, version=http_version) - return Layers.request(layer.next, url, req, body; iofunction=iofunction, kw...) + return Layers.request(layer.next, url, req, body) end const USER_AGENT = Ref{Union{String, Nothing}}("HTTP.jl/$VERSION") diff --git a/src/RedirectRequest.jl b/src/RedirectRequest.jl index 7e561818a..bb14bed86 100644 --- a/src/RedirectRequest.jl +++ b/src/RedirectRequest.jl @@ -14,22 +14,29 @@ Redirects the request in the case of 3xx response status. """ struct RedirectLayer{Next <: Layer} <: InitialLayer next::Next + redirect_limit::Int + forwardheaders::Bool end + export RedirectLayer + Layers.keywordforlayer(::Val{:redirect}) = RedirectLayer -RedirectLayer(next; redirect::Bool=true, kw...) = - redirect ? RedirectLayer(next) : nothing -function Layers.request(layer::RedirectLayer, - method::String, url::URI, headers, body; - redirect_limit=3, forwardheaders=true, kw...) +Layers.shouldinclude(::Type{RedirectLayer}; redirect::Bool=true, kw...) = redirect + +RedirectLayer(next; redirect_limit=3, forwardheaders=true, kw...) = + RedirectLayer(next, redirect_limit, forwardheaders) + +function Layers.request(layer::RedirectLayer, method::String, url::URI, headers, body) + redirect_limit = layer.redirect_limit + forwardheaders = layer.forwardheaders count = 0 while true # Verify the url before making the request. Verification is done in # the redirect loop to also catch bad redirect URLs. verify_url(url) - + # FIXME: can't pass keywords to other layers? res = Layers.request(layer.next, method, url, headers, body; reached_redirect_limit=(count == redirect_limit), kw...) if (count == redirect_limit diff --git a/src/RetryRequest.jl b/src/RetryRequest.jl index f3ec30b29..c29e599f5 100644 --- a/src/RetryRequest.jl +++ b/src/RetryRequest.jl @@ -23,15 +23,15 @@ e.g. `HTTP.IOError`, `Sockets.DNSError`, `Base.EOFError` and `HTTP.StatusError` """ struct RetryLayer{Next <: Layer} <: RequestLayer next::Next + retries::Int + retry_non_idempotent::Bool end export RetryLayer Layers.keywordforlayer(::Val{:retry}) = RetryLayer -RetryLayer(next; retry::Bool=true, kw...) = - retry ? RetryLayer(next) : nothing +RetryLayer(next; retry::Bool=true, retries::int=4, retry_non_idempotent=false, kw...) = + retry ? RetryLayer(next, retries, retry_non_idempotent) : nothing -function Layers.request(layer::RetryLayer, url, req, body; - retries::Int=4, retry_non_idempotent::Bool=false, - kw...) +function Layers.request(layer::RetryLayer, url, req, body) retry_request = Base.retry(Layers.request, delays=ExponentialBackOff(n = retries), @@ -46,7 +46,7 @@ function Layers.request(layer::RetryLayer, url, req, body; return s, retry end) - retry_request(layer.next, url, req, body; kw...) + retry_request(layer.next, url, req, body) end isrecoverable(e) = false diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index affcb933d..c47150d32 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -19,16 +19,26 @@ immediately so that the transmission can be aborted if the `Response` status indicates that the server does not wish to receive the message body. [RFC7230 6.5](https://tools.ietf.org/html/rfc7230#section-6.5). """ -struct StreamLayer <: ConnectionLayer end +struct StreamLayer{Next <: Layer} <: ConnectionLayer + next::Next + reached_redirect_limit::Bool + response_stream + iofunction + verbose +end export StreamLayer - -function Layers.request(::StreamLayer, io::IO, req::Request, body; - reached_redirect_limit=false, - response_stream=nothing, - iofunction=nothing, - verbose::Int=0, - kw...)::Response - +StreamLayer(next; + reached_redirect_limit=false, + response_stream=nothing, + iofunction=nothing, + verbose::Int=0, + kw...) = StreamLayer(next, reached_redirect_limit, response_stream, iofunction, verbose) + +function Layers.request(layer::StreamLayer, io::IO, req::Request, body)::Response + reached_redirect_limit = layer.reached_redirect_limit + response_stream = layer.response_stream + iofunction = layer.iofunction + verbose = layer.verbose verbose == 1 && printlncompact(req) response = req.response @@ -84,7 +94,7 @@ function Layers.request(::StreamLayer, io::IO, req::Request, body; verbose == 1 && printlncompact(response) verbose == 2 && println(response) - return response + return Layers.request(layer.next, response) end function writebody(http::Stream, req::Request, body) diff --git a/src/TimeoutRequest.jl b/src/TimeoutRequest.jl index 5028705e9..b3cd3980e 100644 --- a/src/TimeoutRequest.jl +++ b/src/TimeoutRequest.jl @@ -23,10 +23,10 @@ struct TimeoutLayer{Next <: Layer} <: ConnectionLayer end export TimeoutLayer Layers.keywordforlayer(::Val{:readtimeout}) = TimeoutLayer -TimeoutLayer(next; readtimeout::Int=0, kw...) = - readtimeout > 0 ? TimeoutLayer(next, readtimeout) : nothing +Layers.shouldinclude(::Type{<:TimeoutLayer}; readtimeout::Int=0, kw...) = readtimeout > 0 +TimeoutLayer(next; readtimeout::Int=0, kw...) = TimeoutLayer(next, readtimeout) -function Layers.request(layer::TimeoutLayer, io::IO, req, body; kw...) +function Layers.request(layer::TimeoutLayer, io::IO, req, body) readtimeout = layer.readtimeout wait_for_timeout = Ref{Bool}(true) timedout = Ref{Bool}(false) @@ -42,7 +42,7 @@ function Layers.request(layer::TimeoutLayer, io::IO, req, body; kw...) end try - return Layers.request(layer.next, io, req, body; kw...) + return Layers.request(layer.next, io, req, body) catch e if timedout[] throw(ReadTimeoutError(readtimeout)) diff --git a/src/layers.jl b/src/layers.jl index ec18571db..ed821fd11 100644 --- a/src/layers.jl +++ b/src/layers.jl @@ -1,5 +1,5 @@ module Layers -export Layer, InitialLayer, RequestLayer, ConnectionLayer +export Layer, InitialLayer, RequestLayer, ConnectionLayer, ResponseLayer struct LayerNotFoundException <: Exception var::String @@ -14,10 +14,15 @@ abstract type Layer end abstract type InitialLayer <: Layer end abstract type RequestLayer <: Layer end abstract type ConnectionLayer <: Layer end +abstract type ResponseLayer <: Layer end function keywordforlayer end keywordforlayer(kw) = nothing + +function shouldinclude end + +shouldinclude(T; kw...) = true # custom layers must subtype one of above # must register a keyword arg for layer # must have a layer constructor like: Layer(next; kw...) From 3e7677badbd178b6a355698ee15a6c4d603e1069 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 8 Mar 2022 21:34:25 -0700 Subject: [PATCH 06/19] move stuff around --- src/HTTP.jl | 92 ++++++++++++++++++++++++--------------------------- src/layers.jl | 4 +-- 2 files changed, 44 insertions(+), 52 deletions(-) diff --git a/src/HTTP.jl b/src/HTTP.jl index 117964e78..e170467d2 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -307,15 +307,51 @@ end ``` """ function request(method, url, h=Header[], b=nobody; - headers=h, body=b, kw...)::Response - return request(HTTP.stack(;kw...), method, url, headers, body; kw...) -end -function request(stack::Layer, method, url, h=Header[], b=nobody; - headers=h, body=b, query=nothing)::Response - return Layers.request(stack, string(method), request_uri(url, query), mkheaders(headers), body) + headers=h, body=b, query=nothing, kw...)::Response + return Layers.request(HTTP.stack(; kw...), string(method), request_uri(url, query), mkheaders(headers), body) end -request(::Type{Union{}}, resp::Response) = resp +""" + HTTP.get(url [, headers]; ) -> HTTP.Response + +Shorthand for `HTTP.request("GET", ...)`. See [`HTTP.request`](@ref). +""" +get(a...; kw...) = request("GET", a...; kw...) + +""" + HTTP.put(url, headers, body; ) -> HTTP.Response + +Shorthand for `HTTP.request("PUT", ...)`. See [`HTTP.request`](@ref). +""" +put(a...; kw...) = request("PUT", a...; kw...) + +""" + HTTP.post(url, headers, body; ) -> HTTP.Response + +Shorthand for `HTTP.request("POST", ...)`. See [`HTTP.request`](@ref). +""" +post(a...; kw...) = request("POST", a...; kw...) + +""" + HTTP.patch(url, headers, body; ) -> HTTP.Response + +Shorthand for `HTTP.request("PATCH", ...)`. See [`HTTP.request`](@ref). +""" +patch(a...; kw...) = request("PATCH", a...; kw...) + +""" + HTTP.head(url; ) -> HTTP.Response + +Shorthand for `HTTP.request("HEAD", ...)`. See [`HTTP.request`](@ref). +""" +head(u; kw...) = request("HEAD", u; kw...) + +""" + HTTP.delete(url [, headers]; ) -> HTTP.Response + +Shorthand for `HTTP.request("DELETE", ...)`. See [`HTTP.request`](@ref). +""" +delete(a...; kw...) = request("DELETE", a...; kw...) request_uri(url, query) = URI(URI(url); query=query) request_uri(url, ::Nothing) = URI(url) @@ -380,48 +416,6 @@ function openraw(method::Union{String,Symbol}, url, headers=Header[]; kw...)::Tu take!(socketready) end -""" - HTTP.get(url [, headers]; ) -> HTTP.Response - -Shorthand for `HTTP.request("GET", ...)`. See [`HTTP.request`](@ref). -""" -get(a...; kw...) = request("GET", a...; kw...) - -""" - HTTP.put(url, headers, body; ) -> HTTP.Response - -Shorthand for `HTTP.request("PUT", ...)`. See [`HTTP.request`](@ref). -""" -put(u, h=[], b=""; kw...) = request("PUT", u, h, b; kw...) - -""" - HTTP.post(url, headers, body; ) -> HTTP.Response - -Shorthand for `HTTP.request("POST", ...)`. See [`HTTP.request`](@ref). -""" -post(u, h=[], b=""; kw...) = request("POST", u, h, b; kw...) - -""" - HTTP.patch(url, headers, body; ) -> HTTP.Response - -Shorthand for `HTTP.request("PATCH", ...)`. See [`HTTP.request`](@ref). -""" -patch(u, h=[], b=""; kw...) = request("PATCH", u, h, b; kw...) - -""" - HTTP.head(url; ) -> HTTP.Response - -Shorthand for `HTTP.request("HEAD", ...)`. See [`HTTP.request`](@ref). -""" -head(u; kw...) = request("HEAD", u; kw...) - -""" - HTTP.delete(url [, headers]; ) -> HTTP.Response - -Shorthand for `HTTP.request("DELETE", ...)`. See [`HTTP.request`](@ref). -""" -delete(a...; kw...) = request("DELETE", a...; kw...) - include("RedirectRequest.jl"); using .RedirectRequest include("BasicAuthRequest.jl"); using .BasicAuthRequest include("CookieRequest.jl"); using .CookieRequest diff --git a/src/layers.jl b/src/layers.jl index ed821fd11..8603b8289 100644 --- a/src/layers.jl +++ b/src/layers.jl @@ -16,9 +16,7 @@ abstract type RequestLayer <: Layer end abstract type ConnectionLayer <: Layer end abstract type ResponseLayer <: Layer end -function keywordforlayer end - -keywordforlayer(kw) = nothing +const LAYERS = Dict function shouldinclude end From 206b847cb566e824a7c7dab4cc6f5df2bbee84e4 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Wed, 9 Mar 2022 23:11:40 -0700 Subject: [PATCH 07/19] more iterating/ideas around revamping the client request stack/stacking mechanisms --- src/BasicAuthRequest.jl | 23 +++-- src/CanonicalizeRequest.jl | 21 +++-- src/ConnectionRequest.jl | 24 ++++-- src/ContentTypeRequest.jl | 12 ++- src/CookieRequest.jl | 40 ++++----- src/DebugRequest.jl | 16 ++-- src/ExceptionRequest.jl | 18 ++-- src/HTTP.jl | 67 ++++++++++----- src/MessageRequest.jl | 25 ++---- src/Messages.jl | 18 ++-- src/RedirectRequest.jl | 34 ++++---- src/RetryRequest.jl | 23 ++--- src/StreamRequest.jl | 23 +++-- src/TimeoutRequest.jl | 10 ++- src/layers.jl | 153 +++++++++++++++++++++++++++++++--- test/client.jl | 4 +- test/resources/TestRequest.jl | 29 +++---- 17 files changed, 344 insertions(+), 196 deletions(-) diff --git a/src/BasicAuthRequest.jl b/src/BasicAuthRequest.jl index 1c2a77e8f..2eb0de099 100644 --- a/src/BasicAuthRequest.jl +++ b/src/BasicAuthRequest.jl @@ -13,21 +13,20 @@ Add `Authorization: Basic` header using credentials from url userinfo. """ struct BasicAuthLayer{Next <: Layer} <: InitialLayer next::Next + basicauth::Bool end export BasicAuthLayer -Layers.keywordforlayer(::Val{:basicauth}) = BasicAuthLayer -BasicAuthLayer(next; kw...) = BasicAuthLayer(next) - -function Layers.request(layer::BasicAuthLayer, method::String, url::URI, headers, body) - userinfo = unescapeuri(url.userinfo) - - if !isempty(userinfo) && getkv(headers, "Authorization", "") == "" - @debug 1 "Adding Authorization: Basic header." - setkv(headers, "Authorization", "Basic $(base64encode(userinfo))") +BasicAuthLayer(next; basicauth::Bool=true, kw...) = BasicAuthLayer(next, basicauth) + +function Layers.request(layer::BasicAuthLayer, ctx, method::String, url::URI, headers, body) + if layer.basicauth + userinfo = unescapeuri(url.userinfo) + if !isempty(userinfo) && getkv(headers, "Authorization", "") == "" + @debug 1 "Adding Authorization: Basic header." + setkv(headers, "Authorization", "Basic $(base64encode(userinfo))") + end end - - return Layers.request(layer.next, method, url, headers, body) + return Layers.request(layer.next, ctx, method, url, headers, body) end - end # module BasicAuthRequest diff --git a/src/CanonicalizeRequest.jl b/src/CanonicalizeRequest.jl index dc29b37d9..ec0d408d9 100644 --- a/src/CanonicalizeRequest.jl +++ b/src/CanonicalizeRequest.jl @@ -11,21 +11,20 @@ Rewrite request and response headers in Canonical-Camel-Dash-Format. """ struct CanonicalizeLayer{Next <: Layer} <: InitialLayer next::Next + canonicalize_headers::Bool end export CanonicalizeLayer -Layers.keywordforlayer(::Val{:canonicalize_headers}) = CanonicalizeLayer -Layers.shouldinclude(::Type{CanonicalizeLayer}; canonicalize_headers::Bool=true, kw...) = - canonicalize_headers -CanonicalizeLayer(next; kw...) = CanonicalizeLayer(next) +CanonicalizeLayer(next; canonicalize_headers::Bool=false, kw...) = CanonicalizeLayer(next, canonicalize_headers) -function Layers.request(layer::CanonicalizeLayer, method::String, url, headers, body) - - headers = canonicalizeheaders(headers) - - res = Layers.request(layer.next, method, url, headers, body) - - res.headers = canonicalizeheaders(res.headers) +function Layers.request(layer::CanonicalizeLayer, ctx, method::String, url, headers, body) + if layer.canonicalize_headers + headers = canonicalizeheaders(headers) + end + res = Layers.request(layer.next, ctx, method, url, headers, body) + if layer.canonicalize_headers + res.headers = canonicalizeheaders(res.headers) + end return res end diff --git a/src/ConnectionRequest.jl b/src/ConnectionRequest.jl index 174d244be..2939e125e 100644 --- a/src/ConnectionRequest.jl +++ b/src/ConnectionRequest.jl @@ -5,11 +5,13 @@ using URIs, ..Sockets using ..Messages using ..IOExtras using ..ConnectionPool -using MbedTLS: SSLContext +using MbedTLS: SSLContext, SSLConfig using ..Pairs: getkv, setkv using Base64: base64encode import ..@debug, ..DEBUG_LEVEL +const nosslconfig = SSLConfig() + # hasdotsuffix reports whether s ends in "."+suffix. hasdotsuffix(s, suffix) = endswith(s, "." * suffix) @@ -61,17 +63,19 @@ struct ConnectionPoolLayer{Next <: Layer} <: RequestLayer next::Next proxy::String socket_type::Any + kw end export ConnectionPoolLayer ConnectionPoolLayer(next; - proxy=getproxy(url.scheme, url.host), + proxy="", socket_type::Type=TCPSocket, - kw...) = ConnectionPoolLayer(next, proxy, socket_type) + kw...) = ConnectionPoolLayer(next, proxy, socket_type, kw) -function Layers.request(layer::ConnectionPoolLayer, url::URI, req, body) - proxy, socket_type = layer.proxy, layer.socket_type +function Layers.request(layer::ConnectionPoolLayer, ctx, req, body) + proxy = layer.proxy != "" ? layer.proxy : getproxy(req.url.scheme, req.url.host) + socket_type = layer.socket_type if proxy !== nothing - target_url = url + target_url = req.url url = URI(proxy) if target_url.scheme == "http" req.target = string(target_url) @@ -82,12 +86,14 @@ function Layers.request(layer::ConnectionPoolLayer, url::URI, req, body) @debug 1 "Adding Proxy-Authorization: Basic header." setkv(req.headers, "Proxy-Authorization", "Basic $(base64encode(userinfo))") end + else + url = req.url end IOType = sockettype(url, socket_type) local io try - io = newconnection(IOType, url.host, url.port; kw...) + io = newconnection(IOType, url.host, url.port; layer.kw...) catch e rethrow(isioerror(e) ? IOError(e, "during request($url)") : e) end @@ -101,11 +107,11 @@ function Layers.request(layer::ConnectionPoolLayer, url::URI, req, body) close(io) return r end - io = ConnectionPool.sslupgrade(io, target_url.host; kw...) + io = ConnectionPool.sslupgrade(io, target_url.host; layer.kw...) req.headers = filter(x->x.first != "Proxy-Authorization", req.headers) end - r = Layers.request(layer.next, io, req, body) + r = Layers.request(layer.next, ctx, io, req, body) if proxy !== nothing && target_url.scheme == "https" close(io) diff --git a/src/ContentTypeRequest.jl b/src/ContentTypeRequest.jl index 77fee9993..186d5ef73 100644 --- a/src/ContentTypeRequest.jl +++ b/src/ContentTypeRequest.jl @@ -11,16 +11,14 @@ import ..@debug, ..DEBUG_LEVEL struct ContentTypeDetectionLayer{Next <: Layer} <: InitialLayer next::Next + detect_content_type::Bool end export ContentTypeDetectionLayer -Layers.keywordforlayer(::Val{:detect_content_type}) = ContentTypeDetectionLayer -Layers.shouldinclude(::Type{ContentTypeDetectionLayer}; - detect_content_type::Bool=true) = detect_content_type -ContentTypeDetectionLayer(next; kw...) = ContentTypeDetectionLayer(netx) +ContentTypeDetectionLayer(next; detect_content_type::Bool=false, kw...) = ContentTypeDetectionLayer(next, detect_content_type) -function Layers.request(layer::ContentTypeDetectionLayer, method::String, url::URI, headers, body) +function Layers.request(layer::ContentTypeDetectionLayer, ctx, method::String, url::URI, headers, body) - if (getkv(headers, "Content-Type", "") == "" + if layer.detect_content_type && (getkv(headers, "Content-Type", "") == "" && !isa(body, Form) && bodylength(body) != unknown_length && bodylength(body) > 0) @@ -29,7 +27,7 @@ function Layers.request(layer::ContentTypeDetectionLayer, method::String, url::U setkv(headers, "Content-Type", sn) @debug 1 "setting Content-Type header to: $sn" end - return Layers.request(layer.next, method, url, headers, body) + return Layers.request(layer.next, ctx, method, url, headers, body) end end # module diff --git a/src/CookieRequest.jl b/src/CookieRequest.jl index 8ad876be7..5409591d2 100644 --- a/src/CookieRequest.jl +++ b/src/CookieRequest.jl @@ -27,34 +27,34 @@ struct CookieLayer{Next <: Layer} <: InitialLayer cookiejar::Dict{String, Set{Cookie}} end export CookieLayer -Layers.keywordforlayer(::Val{:cookies}) = CookieLayer -Layers.shouldinclude(::Type{CookieLayer}; cookies=true, kw...) = - cookies === true || (cookies isa AbstractDict && !isempty(cookies)) CookieLayer(next; cookies=true, cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) = CookieLayer(next, cookies, cookiejar) -function Layers.request(layer::CookieLayer, method::String, url::URI, headers, body) +function Layers.request(layer::CookieLayer, ctx, method::String, url::URI, headers, body) + cookies = layer.cookies + if cookies === true || (cookies isa AbstractDict && !isempty(cookies)) + cookiejar = layer.cookiejar + hostcookies = get!(cookiejar, url.host, Set{Cookie}()) - cookiejar = layer.cookiejar - hostcookies = get!(cookiejar, url.host, Set{Cookie}()) - - cookiestosend = getcookies(hostcookies, url) - if !(cookies isa Bool) - for (name, value) in cookies - push!(cookiestosend, Cookie(name, value)) + cookiestosend = getcookies(hostcookies, url) + if !(cookies isa Bool) + for (name, value) in cookies + push!(cookiestosend, Cookie(name, value)) + end + end + if !isempty(cookiestosend) + setkv(headers, "Cookie", stringify(getkv(headers, "Cookie", ""), cookiestosend)) end - end - if !isempty(cookiestosend) - setkv(headers, "Cookie", stringify(getkv(headers, "Cookie", ""), cookiestosend)) - end - - res = Layers.request(layer.next, method, url, headers, body) - - setcookies(hostcookies, url.host, res.headers) - return res + res = Layers.request(layer.next, ctx, method, url, headers, body) + setcookies(hostcookies, url.host, res.headers) + return res + else + # skip + return Layers.request(layer.next, ctx, method, url, headers, body) + end end function getcookies(cookies, url) diff --git a/src/DebugRequest.jl b/src/DebugRequest.jl index 59217f282..3f8d2ea6f 100644 --- a/src/DebugRequest.jl +++ b/src/DebugRequest.jl @@ -15,20 +15,22 @@ Wrap the `IO` stream in an `IODebug` stream and print Message data. """ struct DebugLayer{Next <:Layer} <: ConnectionLayer next::Next + verbose::Int end export DebugLayer -Layers.keywordforlayer(::Val{:verbose}) = DebugLayer -Layers.shouldinclude(::Type{DebugLayer}; verbose=0, kw...) = (verbose >= 3 || DEBUG_LEVEL[] >= 3) -DebugLayer(next; kw...) = DebugLayer(next) - -function Layers.request(layer::DebugLayer, io::IO, req, body) +DebugLayer(next; verbose=0, kw...) = DebugLayer(next, verbose) +function Layers.request(layer::DebugLayer, ctx, io::IO, req, body) + # if not debugging, just call to next layer + if !(layer.verbose >= 3 || DEBUG_LEVEL[] >= 3) + return Layers.request(layer.next, ctx, io, req, body) + end @static if live_mode - return Layers.request(layer.next, IODebug(io), req, body) + return Layers.request(layer.next, ctx, IODebug(io), req, body) else iod = IODebug(io) try - return Layers.request(layer.next, iod, req, body) + return Layers.request(layer.next, ctx, iod, req, body) finally show_log(stdout, iod) end diff --git a/src/ExceptionRequest.jl b/src/ExceptionRequest.jl index 7e3918790..b36edaa5d 100644 --- a/src/ExceptionRequest.jl +++ b/src/ExceptionRequest.jl @@ -13,22 +13,18 @@ Throw a `StatusError` if the request returns an error response status. """ struct ExceptionLayer{Next <: Layer} <: ResponseLayer next::Next + status_exception::Bool end export ExceptionLayer -Layers.keywordforlayer(::Val{:status_exception}) = ExceptionLayer -Layers.shouldinclude(::Type{<:ExceptionLayer}; status_exception::Bool=true) = - status_exception -ExceptionLayer(next; kw...) = ExceptionLayer(next) +ExceptionLayer(next; status_exception::Bool=true) = ExceptionLayer(next, status_exception) -function Layers.request(layer::ExceptionLayer, resp) - - res = Layers.request(layer.next, resp) - - if iserror(res) +function Layers.request(layer::ExceptionLayer, ctx, resp) + res = Layers.request(layer.next, ctx, resp) + if layer.status_exception && iserror(res) throw(StatusError(res.status, res.request.method, res.request.target, res)) + else + return res end - - return res end """ diff --git a/src/HTTP.jl b/src/HTTP.jl index e170467d2..5e5b76995 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -308,7 +308,27 @@ end """ function request(method, url, h=Header[], b=nobody; headers=h, body=b, query=nothing, kw...)::Response - return Layers.request(HTTP.stack(; kw...), string(method), request_uri(url, query), mkheaders(headers), body) + return request(HTTP.stack(; kw...), method, url, headers, body, query) +end + +const Context = Dict{Symbol, Any} + +function request(stack::Layers.Layer, method, url, h=Header[], b=nobody, q=nothing; + headers=h, body=b, query=q)::Response + return Layers.request(stack, Context(), string(method), request_uri(url, query), mkheaders(headers), body) +end + +macro client(layertypes...) + esc(quote + get(a...; kw...) = request("GET", a...; kw...) + put(a...; kw...) = request("PUT", a...; kw...) + post(a...; kw...) = request("POST", a...; kw...) + patch(a...; kw...) = request("PATCH", a...; kw...) + head(u; kw...) = request("HEAD", u; kw...) + delete(a...; kw...) = request("DELETE", a...; kw...) + request(method, url, h=HTTP.Header[], b=HTTP.nobody; headers=h, body=b, query=nothing, kw...)::HTTP.Response = + HTTP.request(HTTP.stack($(layertypes...); kw...), method, url, headers, body, query) + end) end """ @@ -542,7 +562,8 @@ relationship with [`HTTP.Response`](@ref), [`HTTP.Parsers`](@ref), ``` *See `docs/src/layers`[`.monopic`](http://monodraw.helftone.com).* """ -function stack(; +function stack(layertypes::Type{<:Layers.Layer}...; + # default keyword arg values basicauth=true, cookies=false, canonicalize_headers=false, @@ -554,31 +575,31 @@ function stack(; redirect=true, kw...) - kwargs = merge( - (; - basicauth, cookies, canonicalize_headers, - retry, status_exception, readtimeout, - detect_content_type, verbose, redirect - ), - kw - ) - layers = stacklayertypes(Layers.ResponseLayer, Union{}, kwargs) - layers = StreamLayer(layers; kw...) - layers = stacklayertypes(Layers.ConnectionLayer, layers, kwargs) + # ResponseLayers + layers = ExceptionLayer(BottomLayer(); status_exception=status_exception) + layers = stacklayertypes(Layers.ResponseLayer, layers, layertypes; kw...) + # transition ConnectionLayer => ResponseLayer + layers = StreamLayer(layers; verbose=verbose, kw...) + # ConnectionLayers + layers = DebugLayer(TimeoutLayer(layers; readtimeout=readtimeout); verbose=verbose) + layers = stacklayertypes(Layers.ConnectionLayer, layers, layertypes; kw...) + # transition RequestLayer => ConnectionLayer layers = ConnectionPoolLayer(layers; kw...) - layers = stacklayertypes(Layers.RequestLayer, layers, kwargs) + # RequestLayers + layers = RetryLayer(layers; retry=retry, kw...) + layers = stacklayertypes(Layers.RequestLayer, layers, layertypes; kw...) + # transition InitialLayer => RequestLayer layers = MessageLayer(layers; kw...) - return stacklayertypes(Layers.InitialLayer, layers, kwargs) + layers = BasicAuthLayer(CanonicalizeLayer(ContentTypeDetectionLayer(RedirectLayer(CookieLayer(layers; + cookies=cookies, kw...); redirect=redirect, kw...); detect_content_type=detect_content_type, kw...); + canonicalize_headers=canonicalize_headers, kw...); basicauth=basicauth, kw...) + return stacklayertypes(Layers.InitialLayer, layers, layertypes; kw...) end -function stacklayertypes(::Type{T}, layers, kwargs) where {T} - for (k, _) in pairs(kwargs) - layer = Layers.keywordforlayer(Val(k)) - if layer !== nothing && layer <: T - newlayers = layer(layers; kwargs...) - if newlayers !== nothing - layers = newlayers - end +function stacklayertypes(::Type{T}, layers::Layers.Layer, layertypes; kw...) where {T} + for LayerType in layertypes + if LayerType <: T + layers = LayerType(layers; kw...) end end return layers diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index e630bf918..2b12079a2 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -11,13 +11,6 @@ import ..Messages: bodylength import ..Headers import ..Form, ..content_type -""" -"request-target" per https://tools.ietf.org/html/rfc7230#section-5.3 -""" -resource(uri::URI) = string( isempty(uri.path) ? "/" : uri.path, - !isempty(uri.query) ? "?" : "", uri.query, - !isempty(uri.fragment) ? "#" : "", uri.fragment) - """ Layers.request(MessageLayer, method, ::URI, headers, body) -> HTTP.Response @@ -26,18 +19,15 @@ Construct a [`Request`](@ref) object and set mandatory headers. struct MessageLayer{Next <: Layer} <: RequestLayer next::Next http_version::VersionNumber - target::String - parent::Union{Nothing, Request} iofunction end export MessageLayer MessageLayer(next; http_version=v"1.1", - target=resource(url), - parent=nothing, iofunction=nothing, - kw...) = MessageLayer(next, http_version, target, parent, iofunction) + iofunction=nothing, + kw...) = MessageLayer(next, http_version, iofunction) -function Layers.request(layer::MessageLayer, method::String, url::URI, headers::Headers, body) +function Layers.request(layer::MessageLayer, ctx, method::String, url::URI, headers::Headers, body) if isempty(url.port) || (url.scheme == "http" && url.port == "80") || @@ -58,7 +48,7 @@ function Layers.request(layer::MessageLayer, method::String, url::URI, headers:: l = bodylength(body) if l != unknown_length setheader(headers, "Content-Length" => string(l)) - elseif method == "GET" && iofunction isa Function + elseif method == "GET" && layer.iofunction isa Function setheader(headers, "Content-Length" => "0") end end @@ -66,11 +56,10 @@ function Layers.request(layer::MessageLayer, method::String, url::URI, headers:: # "Content-Type" => "multipart/form-data; boundary=..." setheader(headers, content_type(body)) end + parent = get(ctx, :parentrequest, nothing) + req = Request(method, resource(url), headers, bodybytes(body); url=url, version=layer.http_version, parent=parent) - req = Request(method, target, headers, bodybytes(body); - parent=parent, version=http_version) - - return Layers.request(layer.next, url, req, body) + return Layers.request(layer.next, ctx, req, body) end const USER_AGENT = Ref{Union{String, Nothing}}("HTTP.jl/$VERSION") diff --git a/src/Messages.jl b/src/Messages.jl index c03acb1b6..1f8028566 100644 --- a/src/Messages.jl +++ b/src/Messages.jl @@ -55,7 +55,7 @@ Streaming of request and response bodies is handled by the module Messages export Message, Request, Response, - reset!, status, method, headers, uri, body, + reset!, status, method, headers, uri, body, resource, iserror, isredirect, ischunked, issafe, isidempotent, header, hasheader, headercontains, setheader, defaultheader!, appendheader, mkheaders, readheaders, headerscomplete, @@ -66,6 +66,7 @@ export Message, Request, Response, import ..HTTP +using ..URIs using ..Pairs using ..IOExtras using ..Parsers @@ -202,8 +203,6 @@ Represents a HTTP Request Message. - `response`, the `Response` to this `Request` -- `txcount`, number of times this `Request` has been sent (see RetryRequest.jl). - - `parent`, the `Response` (if any) that led to this request (e.g. in the case of a redirect). [RFC7230 6.4](https://tools.ietf.org/html/rfc7231#section-6.4) @@ -218,7 +217,7 @@ mutable struct Request <: Message headers::Headers body::Vector{UInt8} response::Response - txcount::Int + url::URI parent end @@ -231,19 +230,26 @@ Constructor for `HTTP.Request`. For daily use, see [`HTTP.request`](@ref). """ function Request(method::String, target, headers=[], body=UInt8[]; - version=v"1.1", parent=nothing) + version=v"1.1", url::URI=URI(), parent=nothing) r = Request(method, target == "" ? "/" : target, version, mkheaders(headers), bytes(body), Response(0), - 0, + url, parent) r.response.request = r return r end +""" +"request-target" per https://tools.ietf.org/html/rfc7230#section-5.3 +""" +resource(uri::URI) = string( isempty(uri.path) ? "/" : uri.path, + !isempty(uri.query) ? "?" : "", uri.query, + !isempty(uri.fragment) ? "#" : "", uri.fragment) + mkheaders(h::Headers) = h mkheaders(h)::Headers = Header[string(k) => string(v) for (k,v) in h] diff --git a/src/RedirectRequest.jl b/src/RedirectRequest.jl index bb14bed86..5b9ff8d6f 100644 --- a/src/RedirectRequest.jl +++ b/src/RedirectRequest.jl @@ -14,21 +14,23 @@ Redirects the request in the case of 3xx response status. """ struct RedirectLayer{Next <: Layer} <: InitialLayer next::Next + redirect::Bool redirect_limit::Int forwardheaders::Bool end export RedirectLayer -Layers.keywordforlayer(::Val{:redirect}) = RedirectLayer +RedirectLayer(next; redirect::Bool=true, redirect_limit=3, forwardheaders=true, kw...) = + RedirectLayer(next, redirect, redirect_limit, forwardheaders) -Layers.shouldinclude(::Type{RedirectLayer}; redirect::Bool=true, kw...) = redirect - -RedirectLayer(next; redirect_limit=3, forwardheaders=true, kw...) = - RedirectLayer(next, redirect_limit, forwardheaders) - -function Layers.request(layer::RedirectLayer, method::String, url::URI, headers, body) +function Layers.request(layer::RedirectLayer, ctx, method, url, headers, body) redirect_limit = layer.redirect_limit + if !layer.redirect || layer.redirect_limit == 0 + # no redirecting + return Layers.request(layer.next, ctx, method, url, headers, body) + end + forwardheaders = layer.forwardheaders count = 0 while true @@ -36,8 +38,10 @@ function Layers.request(layer::RedirectLayer, method::String, url::URI, headers, # Verify the url before making the request. Verification is done in # the redirect loop to also catch bad redirect URLs. verify_url(url) - # FIXME: can't pass keywords to other layers? - res = Layers.request(layer.next, method, url, headers, body; reached_redirect_limit=(count == redirect_limit), kw...) + if count == redirect_limit + ctx[:redirectlimitreached] = true + end + res = Layers.request(layer.next, ctx, method, url, headers, body) if (count == redirect_limit || !isredirect(res) @@ -45,18 +49,16 @@ function Layers.request(layer::RedirectLayer, method::String, url::URI, headers, return res end - - kw = merge(merge(NamedTuple(), kw), (parent = res,)) + # follow redirect + ctx[:parentrequest] = res.request oldurl = url - url = resolvereference(url, location) + url = resolvereference(oldurl, location) if forwardheaders - headers = filter(headers) do h + headers = filter(headers) do (header, _) # false return values are filtered out - header, value = h if header == "Host" return false - elseif (header in SENSITIVE_HEADERS - && !isdomainorsubdomain(url.host, oldurl.host)) + elseif (header in SENSITIVE_HEADERS && !isdomainorsubdomain(url.host, url.host)) return false else return true diff --git a/src/RetryRequest.jl b/src/RetryRequest.jl index c29e599f5..53eef8dd2 100644 --- a/src/RetryRequest.jl +++ b/src/RetryRequest.jl @@ -23,20 +23,23 @@ e.g. `HTTP.IOError`, `Sockets.DNSError`, `Base.EOFError` and `HTTP.StatusError` """ struct RetryLayer{Next <: Layer} <: RequestLayer next::Next + retry::Bool retries::Int retry_non_idempotent::Bool end export RetryLayer -Layers.keywordforlayer(::Val{:retry}) = RetryLayer -RetryLayer(next; retry::Bool=true, retries::int=4, retry_non_idempotent=false, kw...) = - retry ? RetryLayer(next, retries, retry_non_idempotent) : nothing - -function Layers.request(layer::RetryLayer, url, req, body) +RetryLayer(next; retry::Bool=true, retries::Int=4, retry_non_idempotent=false, kw...) = + RetryLayer(next, retry, retries, retry_non_idempotent) +function Layers.request(layer::RetryLayer, ctx, req::Request, body) + if !layer.retry || layer.retries == 0 + # no retry + return Layers.request(layer.next, ctx, req, body) + end retry_request = Base.retry(Layers.request, - delays=ExponentialBackOff(n = retries), + delays=ExponentialBackOff(n = layer.retries), check=(s,ex)->begin - retry = isrecoverable(ex, req, retry_non_idempotent) + retry = isrecoverable(ex, req, layer.retry_non_idempotent, get(ctx, :retrycount, 0)) if retry @debug 1 "🔄 Retry $ex: $(sprintcompact(req))" reset!(req.response) @@ -46,7 +49,7 @@ function Layers.request(layer::RetryLayer, url, req, body) return s, retry end) - retry_request(layer.next, url, req, body) + return retry_request(layer.next, ctx, req, body) end isrecoverable(e) = false @@ -56,11 +59,11 @@ isrecoverable(e::HTTP.StatusError) = e.status == 403 || # Forbidden e.status == 408 || # Timeout e.status >= 500 # Server Error -isrecoverable(e, req, retry_non_idempotent) = +isrecoverable(e, req, retry_non_idempotent, retrycount) = isrecoverable(e) && !(req.body === body_was_streamed) && !(req.response.body === body_was_streamed) && - (retry_non_idempotent || req.txcount == 0 || isidempotent(req)) + (retry_non_idempotent || retrycount == 0 || isidempotent(req)) # "MUST NOT automatically retry a request with a non-idempotent method" # https://tools.ietf.org/html/rfc7230#section-6.3.1 diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index c47150d32..d8c649641 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -21,21 +21,18 @@ indicates that the server does not wish to receive the message body. """ struct StreamLayer{Next <: Layer} <: ConnectionLayer next::Next - reached_redirect_limit::Bool response_stream iofunction verbose end export StreamLayer StreamLayer(next; - reached_redirect_limit=false, response_stream=nothing, iofunction=nothing, verbose::Int=0, - kw...) = StreamLayer(next, reached_redirect_limit, response_stream, iofunction, verbose) + kw...) = StreamLayer(next, response_stream, iofunction, verbose) -function Layers.request(layer::StreamLayer, io::IO, req::Request, body)::Response - reached_redirect_limit = layer.reached_redirect_limit +function Layers.request(layer::StreamLayer, ctx, io::IO, req::Request, body)::Response response_stream = layer.response_stream iofunction = layer.iofunction verbose = layer.verbose @@ -47,6 +44,7 @@ function Layers.request(layer::StreamLayer, io::IO, req::Request, body)::Respons startwrite(http) if verbose == 2 + println("printing req") println(req) if iofunction === nothing && req.body === body_is_a_stream println("$(typeof(req)).body: $(sprintcompact(body))") @@ -58,7 +56,7 @@ function Layers.request(layer::StreamLayer, io::IO, req::Request, body)::Respons @sync begin if iofunction === nothing @async try - writebody(http, req, body) + writebody(http, ctx, req, body) @debug 2 "client closewrite" closewrite(http) catch e @@ -67,7 +65,7 @@ function Layers.request(layer::StreamLayer, io::IO, req::Request, body)::Respons end @debug 2 "client startread" startread(http) - readbody(http, response, response_stream, reached_redirect_limit) + readbody(http, response, response_stream, get(ctx, :redirectlimitreached, false)) else iofunction(http) end @@ -94,10 +92,10 @@ function Layers.request(layer::StreamLayer, io::IO, req::Request, body)::Respons verbose == 1 && printlncompact(response) verbose == 2 && println(response) - return Layers.request(layer.next, response) + return Layers.request(layer.next, ctx, response) end -function writebody(http::Stream, req::Request, body) +function writebody(http::Stream, ctx, req::Request, body) if req.body === body_is_a_stream writebodystream(http, req, body) @@ -105,8 +103,7 @@ function writebody(http::Stream, req::Request, body) else write(http, req.body) end - - req.txcount += 1 + ctx[:retrycount] = get(ctx, :retrycount, 0) + 1 return end @@ -124,11 +121,11 @@ end writechunk(http, req, body::IO) = writebodystream(http, req, body) writechunk(http, req, body) = write(http, body) -function readbody(http::Stream, res::Response, response_stream, reached_redirect_limit) +function readbody(http::Stream, res::Response, response_stream, redirectlimitreached) if response_stream === nothing res.body = read(http) else - if reached_redirect_limit || !isredirect(res) + if redirectlimitreached || !isredirect(res) res.body = body_was_streamed write(response_stream, http) end diff --git a/src/TimeoutRequest.jl b/src/TimeoutRequest.jl index b3cd3980e..258c260e3 100644 --- a/src/TimeoutRequest.jl +++ b/src/TimeoutRequest.jl @@ -22,12 +22,14 @@ struct TimeoutLayer{Next <: Layer} <: ConnectionLayer readtimeout::Int end export TimeoutLayer -Layers.keywordforlayer(::Val{:readtimeout}) = TimeoutLayer -Layers.shouldinclude(::Type{<:TimeoutLayer}; readtimeout::Int=0, kw...) = readtimeout > 0 TimeoutLayer(next; readtimeout::Int=0, kw...) = TimeoutLayer(next, readtimeout) -function Layers.request(layer::TimeoutLayer, io::IO, req, body) +function Layers.request(layer::TimeoutLayer, ctx, io::IO, req, body) readtimeout = layer.readtimeout + if readtimeout <= 0 + # skip + return Layers.request(layer.next, ctx, io, req, body) + end wait_for_timeout = Ref{Bool}(true) timedout = Ref{Bool}(false) @@ -42,7 +44,7 @@ function Layers.request(layer::TimeoutLayer, io::IO, req, body) end try - return Layers.request(layer.next, io, req, body) + return Layers.request(layer.next, ctx, io, req, body) catch e if timedout[] throw(ReadTimeoutError(readtimeout)) diff --git a/src/layers.jl b/src/layers.jl index 8603b8289..08456b5b3 100644 --- a/src/layers.jl +++ b/src/layers.jl @@ -1,33 +1,160 @@ -module Layers -export Layer, InitialLayer, RequestLayer, ConnectionLayer, ResponseLayer +""" +The Layers module in the HTTP.jl package contains the internal machinery for how http client requests are actually made. + +It exposes the concept of "layers" which are single components each responsible for handling +one "piece" of an http client request. Layers are linked together by each being required to +have a dedicated field to store the "next" layer in the stack to form a linked-list +where each layer has a window of execution when control is "passed" to it from the previous +layer in the stack. + +Builtin to the HTTP.jl package, there are 4 "kinds" of layers that are characterized by +"where" they live in stack and the corresponding arguments they have access to when control is passed to them: + * [`Layers.InitialLayer`](@ref): the "outermost" layers that receive arguments almost as-is provided from the calling user; + arguments include the http `method`, `url`, `headers`, and `body`. The HTTP.jl-internal layer `MessageLayer` comes after the + the last `Layers.InitialLayer` layer and transitions to the next layer kind + * [`Layers.RequestLayer`](@ref): the `MessageLayer` took the `method`, `url`, and `headers` arguments and formed a full `HTTP.Request` + object that layers in this "kind" now have access to; the `ConnectionPoolLayer` follows the last `RequestLayer` and opens a live + connection to the remote server, transitioning us to the next layer kind + * [`Layers.ConnectionLayer`](@ref): in addition to the `Request` object, we now also have access to the live/open `Connection` object + which is connected to the remote; the `StreamLayer` follows the last `ConnectionLayer` layer to execute the actual request and read the response + * [`Layers.ResponseLayer`](@ref): these are the "deepest" layers in the stack because they are only called after the request + has been sent and a response has been received + +So the rough flow of what actually happens when a user makes a call like `HTTP.get("https://google.com")` is as follows: + * A "stack" of layers is made, starting with `ResponseLayer`s, then wrapping those in `ConnectionLayer`s, and so on to the outermost `InitialLayer`s + * A little argument processing happens, but the request really begins execution with the first call to `Layers.request(layers, ctx, method, url, headers, body)` + * This passes control to the outermost layer's `Layers.request` method, where it's responsible for, at a minimum, passing control on by calling `Layers.request(layer.next, ctx, method, url, headers, body)` + * Conrol continues to pass down through the stack of layers until the `StreamLayer`, which physically sends the request and receives the response + * Control then goes "back up" the stack starting with the `ResponseLayer`s all the way back to the outermost "first" `InitialLayer` layer before actually returning to the user -struct LayerNotFoundException <: Exception - var::String +Ok, so why is all this important? Well, in addition to having a better understanding of what actually happens when you make a request, +this also provides necessary context for users who desire to _extend_ or _customize_ the request process. +Some examples of ways users may want to cusotmize: + * Compute and add a required authentication header to every request made to a specific service/host + * Provide configurable response "caching" given certain request inputs + * Act as a "load balancer" where service names are mapped to an internal registry of physical IP addresses + +We've already hinted in the explanations above about the requirements for implementing a proper layer, +so let's spell the interface out explicitly here: + * Create a custom layer struct that subtypes one of 4 layer "kind" types: + * `Layers.InitialLayer` + * `Layers.RequestLayer` + * `Layers.ConnectionLayer` + * `Layers.ResponseLayer` + * The custom layer struct MUST HAVE a dedicated field for storing the "next" layer in the stack; this usually looks something like: + +``` +struct CustomLayer{Next <: Layers.Layer} <: Layers.InitialLayer + next::Next + # additional fields... end +``` + * There must be a constructor method of the form: `Layer(next; kw...)` where the `next` argument is some `Layers.Layer` subtype and must be stored in the above-mentioned required field + * The custom layer must then overload the `Layers.request` method that corresponds to the layer "kind" they subtype: + * `Layers.InitialLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, method, url, headers, body)` + * `Layers.RequestLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, request, body)` + * `Layers.ConnectionLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, io, request, body)` + * `Layers.ResponseLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, response)` + * The final requirement is that IN THE `Layers.request` overload, control MUST BE passed on to the next layer in the stack by, at some point, calling `Layers.request(layer.next, args...)`, + where `layer.next` refers to the above-mentioned required field storing the "next" layer in the stack, and `args` are the SAME ARGUMENTS that were overloaded in the custom layer's `Layer.request` + overloaded method. + +Ok great, I think I've got a handle on how to go about creating my own custom layer (I can also poke around the many examples in +the HTTP.jl package itself, since they all implement this exact machinery). But once I have a custom layer, how do I USE IT? Or in +other words, how do I get it included in the request stack? + +HTTP.jl provides the `HTTP.stack(layers...; kw...)` function that takes any number of custom layers as initial positional arguments, +along with _all_ keyword arguments passed from users, and returns the request stack that will immediately be passed to `Layers.request`. +So manually, if I had my `CustomLayer` all setup and defined, I could "include" it by doing something like: +``` +resp = HTTP.request(HTTP.stack(CustomLayer), "GET", "https://google.com") +``` + +Ok, not terrible, but can we make it a little more convenient? We can. HTTP.jl provides a convenience macro that +will automatically define your own set of "user-facing request" methods, but with any specified custom layers +automatically included in the stack. Wait, this sounds magical; show me? + +``` +module MyClient + +using HTTP + +include("customlayer_definitions.jl") +HTTP.@client CustomLayer -function Base.showerror(io::IO, e::LayerNotFoundException) - println(io, typeof(e), ": ", e.var) +end +``` + +Ok, so what we defined here is a module called `MyClient`, which included a custom layer implementation (not fully shown, just +`include`ed) and then the macro invocation of `HTTP.@client CustomerLayer`. The macro expands to define our very own +`MyClient.get`, `MyClient.post`, `MyClient.put`, `MyClient.delete`, etc. methods, but which each include `CustomLayer` +in the constructed request stack. Neat! So now users can just call: + +``` +using MyClient +resp = MyClient.get("https://google.com") +``` + +And they're using your customized http client request stack with the `CustomLayer` functionality! Cool! + +""" +module Layers + +export Layer, InitialLayer, RequestLayer, ConnectionLayer, ResponseLayer, BottomLayer + +""" + Layers.request(layer::L, args...) + +HTTP.jl internal method for executing the stack of layers +that make up a client request. Layers form a linked list +and must explicitly pass control to the next layer to ensure +each layer has a chance to execute its part of the request. +Each layer overloads `Layers.request` for their specific layer +type and the `args` to overload depend on which layer "kind" +they subtype: + * [`Layers.InitialLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, method, url, headers, body)`; this is the top-most layer type + * [`Layers.RequestLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, request, body)`; the `method`, `url`, and `headers` of the `InitialLayer` have been bundled together into a single `request::Request` argument + * [`Layers.ConnectionLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, io, request, body)`; a connection has now been opened to the remote and is available in the `io` argument + * [`Layers.ResponseLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, resp)`; the request has been sent and a response has been received in the form of the `resp` argument + +Note that _every_ `Layers.request` overloads has a `ctx::Dict{Symbol, Any}` argument available if +any state needs to be shared between layers during the life of the request. + +See docs for the [`Layers`](@ref) module for a broader discussion on extending/customizing the client request stack. +""" +function request end + +""" + Layers.Layer + +Abstract type that all client request layer "kinds" must subtype. These currently include: + * [`Layers.InitialLayer`](@ref): top-most layers + * [`Layers.RequestLayer`](@ref): initial `Request` object has been formed + * [`Layers.ConnectionLayer`](@ref): a connection has been opened to the remote + * [`Layers.ResponseLayer`](@ref): the `Request` has been written on the connection and a response received + +Custom layers should subtype one of these layer "kinds" instead of subtyping `Layers.Layer` directly. +""" +abstract type Layer#{Next <: Layer} + # next::Next end -abstract type Layer end abstract type InitialLayer <: Layer end abstract type RequestLayer <: Layer end abstract type ConnectionLayer <: Layer end abstract type ResponseLayer <: Layer end -const LAYERS = Dict +# start the stack off w/ BottomLayer +struct BottomLayer <: ResponseLayer end -function shouldinclude end +# bottom layer just returns the response +request(::BottomLayer, ctx, resp) = resp -shouldinclude(T; kw...) = true # custom layers must subtype one of above -# must register a keyword arg for layer # must have a layer constructor like: Layer(next; kw...) # must have a field to store `next` layer # must overload: request(layer::MyLayer, args...; kw...) # in `request` overload, must call: request(layer.next, args...; kw...) -function request end - end diff --git a/test/client.jl b/test/client.jl index ec7d8bd23..0eb0fc3b9 100644 --- a/test/client.jl +++ b/test/client.jl @@ -9,7 +9,7 @@ status(r) = r.status @testset "Custom HTTP Stack" begin @testset "Low-level Request" begin wasincluded = Ref(false) - result = HTTP.request("GET", "https://httpbin.org/ip"; httptestlayer=wasincluded) + result = TestRequest.get("https://httpbin.org/ip"; httptestlayer=wasincluded) @test status(result) == 200 @test wasincluded[] end @@ -377,7 +377,7 @@ import NetworkOptions, MbedTLS # Set up server with self-signed cert server = listen(IPv4(0), 8443) try - cert, key = joinpath.(@__DIR__, "resources", ("cert.pem", "key.pem")) + cert, key = joinpath.(dirname(pathof(HTTP)), "../test", "resources", ("cert.pem", "key.pem")) sslconfig = MbedTLS.SSLConfig(cert, key) tsk = @async HTTP.listen("0.0.0.0", 8443; server=server, sslconfig=sslconfig) do http HTTP.setstatus(http, 200) diff --git a/test/resources/TestRequest.jl b/test/resources/TestRequest.jl index 46fae4c18..be14d4517 100644 --- a/test/resources/TestRequest.jl +++ b/test/resources/TestRequest.jl @@ -8,25 +8,26 @@ struct TestLayer{Next <: Layer} <: InitialLayer next::Next wasincluded::Ref{Bool} end -Layers.keywordforlayer(::Val{:httptestlayer}) = TestLayer TestLayer(next; httptestlayer=Ref(false), kw...) = TestLayer(next, httptestlayer) -function Layers.request(layer::TestLayer, meth, url, headers, body; kw...) +function Layers.request(layer::TestLayer, ctx, meth, url, headers, body) layer.wasincluded[] = true - return Layers.request(layer.next, meth, url, headers, body; kw...) + return Layers.request(layer.next, ctx, meth, url, headers, body) end -struct LastLayer{Next <: Layer} <: ConnectionLayer - next::Next - wasincluded::Ref{Bool} -end -Layers.keywordforlayer(::Val{:httplastlayer}) = LastLayer -LastLayer(next; httplastlayer=Ref(false), kw...) = LastLayer(next, httplastlayer) +HTTP.@client TestLayer -function Layers.request(layer::LastLayer, io::IO, req, body; kw...) - resp = Layers.request(layer.next, io, req, body; kw...) - layer.wasincluded[] = true - return resp -end +# struct LastLayer{Next <: Layer} <: ConnectionLayer +# next::Next +# wasincluded::Ref{Bool} +# end +# Layers.keywordforlayer(::Val{:httplastlayer}) = LastLayer +# LastLayer(next; httplastlayer=Ref(false), kw...) = LastLayer(next, httplastlayer) + +# function Layers.request(layer::LastLayer, io::IO, req, body; kw...) +# resp = Layers.request(layer.next, io, req, body; kw...) +# layer.wasincluded[] = true +# return resp +# end end From ede23919427e0966f3fefe8b021bf15cb556a931 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 15 Mar 2022 21:03:54 -0600 Subject: [PATCH 08/19] work --- src/RedirectRequest.jl | 2 +- src/layers.jl | 6 ------ 2 files changed, 1 insertion(+), 7 deletions(-) diff --git a/src/RedirectRequest.jl b/src/RedirectRequest.jl index 5b9ff8d6f..f5d4deb1c 100644 --- a/src/RedirectRequest.jl +++ b/src/RedirectRequest.jl @@ -50,7 +50,7 @@ function Layers.request(layer::RedirectLayer, ctx, method, url, headers, body) end # follow redirect - ctx[:parentrequest] = res.request + ctx[:parentrequest] = res oldurl = url url = resolvereference(oldurl, location) if forwardheaders diff --git a/src/layers.jl b/src/layers.jl index 08456b5b3..4790fda55 100644 --- a/src/layers.jl +++ b/src/layers.jl @@ -151,10 +151,4 @@ struct BottomLayer <: ResponseLayer end # bottom layer just returns the response request(::BottomLayer, ctx, resp) = resp -# custom layers must subtype one of above -# must have a layer constructor like: Layer(next; kw...) -# must have a field to store `next` layer -# must overload: request(layer::MyLayer, args...; kw...) -# in `request` overload, must call: request(layer.next, args...; kw...) - end From 4e9b8fe3cb11be058df518763eee97ab933aa55a Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Wed, 16 Mar 2022 20:47:24 -0600 Subject: [PATCH 09/19] more work --- src/BasicAuthRequest.jl | 27 ++--- src/CanonicalizeRequest.jl | 31 +++--- src/ConnectionRequest.jl | 104 +++++++++---------- src/ContentTypeRequest.jl | 29 +++--- src/CookieRequest.jl | 60 +++++------ src/DebugRequest.jl | 34 ++---- src/ExceptionRequest.jl | 26 ++--- src/HTTP.jl | 189 +++++----------------------------- src/IOExtras.jl | 4 +- src/MessageRequest.jl | 86 +++++++--------- src/Messages.jl | 47 ++++----- src/RedirectRequest.jl | 100 ++++++++---------- src/RetryRequest.jl | 51 ++++----- src/StreamRequest.jl | 84 ++++++--------- src/Streams.jl | 22 ++-- src/TimeoutRequest.jl | 65 ++++++------ src/layers.jl | 154 --------------------------- test/chunking.jl | 11 +- test/client.jl | 7 +- test/resources/TestRequest.jl | 41 ++++---- test/server.jl | 10 +- 21 files changed, 398 insertions(+), 784 deletions(-) delete mode 100644 src/layers.jl diff --git a/src/BasicAuthRequest.jl b/src/BasicAuthRequest.jl index 2eb0de099..de85114cf 100644 --- a/src/BasicAuthRequest.jl +++ b/src/BasicAuthRequest.jl @@ -1,32 +1,27 @@ module BasicAuthRequest using ..Base64 -using ..Layers using URIs using ..Pairs: getkv, setkv import ..@debug, ..DEBUG_LEVEL +export basicauthlayer """ - Layers.request(BasicAuthLayer, method, ::URI, headers, body) -> HTTP.Response + basicauthlayer(ctx, method, ::URI, headers, body) -> HTTP.Response Add `Authorization: Basic` header using credentials from url userinfo. """ -struct BasicAuthLayer{Next <: Layer} <: InitialLayer - next::Next - basicauth::Bool -end -export BasicAuthLayer -BasicAuthLayer(next; basicauth::Bool=true, kw...) = BasicAuthLayer(next, basicauth) - -function Layers.request(layer::BasicAuthLayer, ctx, method::String, url::URI, headers, body) - if layer.basicauth - userinfo = unescapeuri(url.userinfo) - if !isempty(userinfo) && getkv(headers, "Authorization", "") == "" - @debug 1 "Adding Authorization: Basic header." - setkv(headers, "Authorization", "Basic $(base64encode(userinfo))") +function basicauthlayer(handler) + return function(ctx, method, url, headers, body; basicauth::Bool=true, kw...) + if basicauth + userinfo = unescapeuri(url.userinfo) + if !isempty(userinfo) && getkv(headers, "Authorization", "") == "" + @debug 1 "Adding Authorization: Basic header." + setkv(headers, "Authorization", "Basic $(base64encode(userinfo))") + end end + return handler(ctx, method, url, headers, body; kw...) end - return Layers.request(layer.next, ctx, method, url, headers, body) end end # module BasicAuthRequest diff --git a/src/CanonicalizeRequest.jl b/src/CanonicalizeRequest.jl index ec0d408d9..7d557233d 100644 --- a/src/CanonicalizeRequest.jl +++ b/src/CanonicalizeRequest.jl @@ -1,31 +1,26 @@ module CanonicalizeRequest -using ..Layers using ..Messages using ..Strings: tocameldash +export canonicalizelayer + """ - Layers.request(CanonicalizeLayer, method, ::URI, headers, body) -> HTTP.Response + canonicalizelayer(ctx, method, ::URI, headers, body) -> HTTP.Response Rewrite request and response headers in Canonical-Camel-Dash-Format. """ -struct CanonicalizeLayer{Next <: Layer} <: InitialLayer - next::Next - canonicalize_headers::Bool -end -export CanonicalizeLayer -CanonicalizeLayer(next; canonicalize_headers::Bool=false, kw...) = CanonicalizeLayer(next, canonicalize_headers) - -function Layers.request(layer::CanonicalizeLayer, ctx, method::String, url, headers, body) - - if layer.canonicalize_headers - headers = canonicalizeheaders(headers) - end - res = Layers.request(layer.next, ctx, method, url, headers, body) - if layer.canonicalize_headers - res.headers = canonicalizeheaders(res.headers) +function canonicalizelayer(handler) + return function(ctx, method, url, headers, body; canonicalize_headers::Bool=false, kw...) + if canonicalize_headers + headers = canonicalizeheaders(headers) + end + res = handler(ctx, method, url, headers, body; kw...) + if canonicalize_headers + res.headers = canonicalizeheaders(res.headers) + end + return res end - return res end canonicalizeheaders(h::T) where {T} = T([tocameldash(k) => v for (k,v) in h]) diff --git a/src/ConnectionRequest.jl b/src/ConnectionRequest.jl index 2939e125e..ea10dd1c7 100644 --- a/src/ConnectionRequest.jl +++ b/src/ConnectionRequest.jl @@ -1,6 +1,5 @@ module ConnectionRequest -using ..Layers using URIs, ..Sockets using ..Messages using ..IOExtras @@ -9,6 +8,7 @@ using MbedTLS: SSLContext, SSLConfig using ..Pairs: getkv, setkv using Base64: base64encode import ..@debug, ..DEBUG_LEVEL +import ..Streams: Stream const nosslconfig = SSLConfig() @@ -48,8 +48,10 @@ function getproxy(scheme, host) return nothing end +export connectionlayer + """ - Layers.request(ConnectionPoolLayer, ::URI, ::Request, body) -> HTTP.Response + connectionlayer(ctx, ::Request, body) -> HTTP.Response Retrieve an `IO` connection from the [`ConnectionPool`](@ref). @@ -59,71 +61,59 @@ Otherwise leave it open so that it can be reused. `IO` related exceptions from `Base` are wrapped in `HTTP.IOError`. See [`isioerror`](@ref). """ -struct ConnectionPoolLayer{Next <: Layer} <: RequestLayer - next::Next - proxy::String - socket_type::Any - kw -end -export ConnectionPoolLayer -ConnectionPoolLayer(next; - proxy="", - socket_type::Type=TCPSocket, - kw...) = ConnectionPoolLayer(next, proxy, socket_type, kw) - -function Layers.request(layer::ConnectionPoolLayer, ctx, req, body) - proxy = layer.proxy != "" ? layer.proxy : getproxy(req.url.scheme, req.url.host) - socket_type = layer.socket_type - if proxy !== nothing - target_url = req.url - url = URI(proxy) - if target_url.scheme == "http" - req.target = string(target_url) +function connectionlayer(handler) + return function(ctx, req; proxy=getproxy(req.url.scheme, req.url.host), socket_type::Type=TCPSocket, kw...) + if proxy !== nothing + target_url = req.url + url = URI(proxy) + if target_url.scheme == "http" + req.target = string(target_url) + end + + userinfo = unescapeuri(url.userinfo) + if !isempty(userinfo) && getkv(req.headers, "Proxy-Authorization", "") == "" + @debug 1 "Adding Proxy-Authorization: Basic header." + setkv(req.headers, "Proxy-Authorization", "Basic $(base64encode(userinfo))") + end + else + url = req.url end - userinfo = unescapeuri(url.userinfo) - if !isempty(userinfo) && getkv(req.headers, "Proxy-Authorization", "") == "" - @debug 1 "Adding Proxy-Authorization: Basic header." - setkv(req.headers, "Proxy-Authorization", "Basic $(base64encode(userinfo))") + IOType = sockettype(url, socket_type) + local io + try + io = newconnection(IOType, url.host, url.port; kw...) + catch e + rethrow(isioerror(e) ? IOError(e, "during request($url)") : e) end - else - url = req.url - end - IOType = sockettype(url, socket_type) - local io - try - io = newconnection(IOType, url.host, url.port; layer.kw...) - catch e - rethrow(isioerror(e) ? IOError(e, "during request($url)") : e) - end + try + if proxy !== nothing && target_url.scheme == "https" + # tunnel request + target_url = URI(target_url, port=443) + r = connect_tunnel(io, target_url, req) + if r.status != 200 + close(io) + return r + end + io = ConnectionPool.sslupgrade(io, target_url.host; kw...) + req.headers = filter(x->x.first != "Proxy-Authorization", req.headers) + end + + stream = Stream(req.response, io) + resp = handler(ctx, stream; kw...) - try - if proxy !== nothing && target_url.scheme == "https" - # tunnel request - target_url = URI(target_url, port=443) - r = connect_tunnel(io, target_url, req) - if r.status != 200 + if proxy !== nothing && target_url.scheme == "https" close(io) - return r end - io = ConnectionPool.sslupgrade(io, target_url.host; layer.kw...) - req.headers = filter(x->x.first != "Proxy-Authorization", req.headers) - end - r = Layers.request(layer.next, ctx, io, req, body) - - if proxy !== nothing && target_url.scheme == "https" - close(io) + return resp + catch e + @debug 1 "❗️ ConnectionLayer $e. Closing: $io" + try; close(io); catch; end + rethrow(isioerror(e) ? IOError(e, "during request($url)") : e) end - - return r - catch e - @debug 1 "❗️ ConnectionLayer $e. Closing: $io" - try; close(io); catch; end - rethrow(isioerror(e) ? IOError(e, "during request($url)") : e) end - end sockettype(url::URI, default) = url.scheme in ("wss", "https") ? SSLContext : default diff --git a/src/ContentTypeRequest.jl b/src/ContentTypeRequest.jl index 186d5ef73..14258f001 100644 --- a/src/ContentTypeRequest.jl +++ b/src/ContentTypeRequest.jl @@ -1,6 +1,5 @@ module ContentTypeDetection -using ..Layers using URIs using ..Pairs: getkv, setkv import ..sniff @@ -9,25 +8,21 @@ using ..Messages import ..MessageRequest: bodylength, bodybytes import ..@debug, ..DEBUG_LEVEL -struct ContentTypeDetectionLayer{Next <: Layer} <: InitialLayer - next::Next - detect_content_type::Bool -end -export ContentTypeDetectionLayer -ContentTypeDetectionLayer(next; detect_content_type::Bool=false, kw...) = ContentTypeDetectionLayer(next, detect_content_type) - -function Layers.request(layer::ContentTypeDetectionLayer, ctx, method::String, url::URI, headers, body) +export contenttypedetectionlayer - if layer.detect_content_type && (getkv(headers, "Content-Type", "") == "" - && !isa(body, Form) - && bodylength(body) != unknown_length - && bodylength(body) > 0) +function contenttypedetectionlayer(handler) + return function(ctx, method, url, headers, body; detect_content_type::Bool=false, kw...) + if detect_content_type && (getkv(headers, "Content-Type", "") == "" + && !isa(body, Form) + && bodylength(body) != unknown_length + && bodylength(body) > 0) - sn = sniff(bodybytes(body)) - setkv(headers, "Content-Type", sn) - @debug 1 "setting Content-Type header to: $sn" + sn = sniff(bodybytes(body)) + setkv(headers, "Content-Type", sn) + @debug 1 "setting Content-Type header to: $sn" + end + return handler(ctx, method, url, headers, body; kw...) end - return Layers.request(layer.next, ctx, method, url, headers, body) end end # module diff --git a/src/CookieRequest.jl b/src/CookieRequest.jl index 5409591d2..2ab43b965 100644 --- a/src/CookieRequest.jl +++ b/src/CookieRequest.jl @@ -1,10 +1,9 @@ module CookieRequest import ..Dates -using ..Layers using URIs using ..Cookies -using ..Messages: ascii_lc_isequal +using ..Messages: Request, ascii_lc_isequal using ..Pairs: getkv, setkv import ..@debug, ..DEBUG_LEVEL, ..access_threaded @@ -15,45 +14,38 @@ function __init__() return end +export cookielayer + """ - Layers.request(CookieLayer, method, ::URI, headers, body) -> HTTP.Response + cookielayer(ctx, method, ::URI, headers, body) -> HTTP.Response Add locally stored Cookies to the request headers. Store new Cookies found in the response headers. """ -struct CookieLayer{Next <: Layer} <: InitialLayer - next::Next - cookies - cookiejar::Dict{String, Set{Cookie}} -end -export CookieLayer -CookieLayer(next; - cookies=true, - cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) = - CookieLayer(next, cookies, cookiejar) - -function Layers.request(layer::CookieLayer, ctx, method::String, url::URI, headers, body) - cookies = layer.cookies - if cookies === true || (cookies isa AbstractDict && !isempty(cookies)) - cookiejar = layer.cookiejar - hostcookies = get!(cookiejar, url.host, Set{Cookie}()) - - cookiestosend = getcookies(hostcookies, url) - if !(cookies isa Bool) - for (name, value) in cookies - push!(cookiestosend, Cookie(name, value)) +function cookielayer(handler) + return function(ctx, req::Request; cookies=true, cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) + println("cookielayer") + if cookies === true || (cookies isa AbstractDict && !isempty(cookies)) + url = req.url + hostcookies = get!(cookiejar, url.host, Set{Cookie}()) + cookiestosend = getcookies(hostcookies, url) + if !(cookies isa Bool) + for (name, value) in cookies + push!(cookiestosend, Cookie(name, value)) + end end + if !isempty(cookiestosend) + setkv(req.headers, "Cookie", stringify(getkv(req.headers, "Cookie", ""), cookiestosend)) + end + @show cookiestosend + res = handler(ctx, req; kw...) + setcookies(hostcookies, url.host, res.headers) + @show hostcookies + return res + else + # skip + return handler(ctx, req; kw...) end - if !isempty(cookiestosend) - setkv(headers, "Cookie", stringify(getkv(headers, "Cookie", ""), cookiestosend)) - end - - res = Layers.request(layer.next, ctx, method, url, headers, body) - setcookies(hostcookies, url.host, res.headers) - return res - else - # skip - return Layers.request(layer.next, ctx, method, url, headers, body) end end diff --git a/src/DebugRequest.jl b/src/DebugRequest.jl index 3f8d2ea6f..aab1a59bc 100644 --- a/src/DebugRequest.jl +++ b/src/DebugRequest.jl @@ -1,39 +1,25 @@ module DebugRequest -using ..Layers import ..DEBUG_LEVEL using ..IOExtras - -const live_mode = true +import ..Streams: Stream include("IODebug.jl") +export debuglayer + """ - Layers.request(DebugLayer, ::IO, ::Request, body) -> HTTP.Response + debuglayer(ctx, stream::Stream) -> HTTP.Response Wrap the `IO` stream in an `IODebug` stream and print Message data. """ -struct DebugLayer{Next <:Layer} <: ConnectionLayer - next::Next - verbose::Int -end -export DebugLayer -DebugLayer(next; verbose=0, kw...) = DebugLayer(next, verbose) - -function Layers.request(layer::DebugLayer, ctx, io::IO, req, body) - # if not debugging, just call to next layer - if !(layer.verbose >= 3 || DEBUG_LEVEL[] >= 3) - return Layers.request(layer.next, ctx, io, req, body) - end - @static if live_mode - return Layers.request(layer.next, ctx, IODebug(io), req, body) - else - iod = IODebug(io) - try - return Layers.request(layer.next, ctx, iod, req, body) - finally - show_log(stdout, iod) +function debuglayer(handler) + return function(ctx, stream::Stream; verbose::Int=0, kw...) + # if debugging, wrap stream.stream in IODebug + if verbose >= 3 || DEBUG_LEVEL[] >= 3 + stream = Stream(stream.message, IODebug(stream.stream)) end + return handler(ctx, stream; verbose=verbose, kw...) end end diff --git a/src/ExceptionRequest.jl b/src/ExceptionRequest.jl index b36edaa5d..2662d5f07 100644 --- a/src/ExceptionRequest.jl +++ b/src/ExceptionRequest.jl @@ -2,28 +2,24 @@ module ExceptionRequest export StatusError -using ..Layers import ..HTTP using ..Messages: iserror +export exceptionlayer + """ - Layers.request(ExceptionLayer, ::Response) -> HTTP.Response + exceptionlayer(ctx, stream) -> HTTP.Response Throw a `StatusError` if the request returns an error response status. """ -struct ExceptionLayer{Next <: Layer} <: ResponseLayer - next::Next - status_exception::Bool -end -export ExceptionLayer -ExceptionLayer(next; status_exception::Bool=true) = ExceptionLayer(next, status_exception) - -function Layers.request(layer::ExceptionLayer, ctx, resp) - res = Layers.request(layer.next, ctx, resp) - if layer.status_exception && iserror(res) - throw(StatusError(res.status, res.request.method, res.request.target, res)) - else - return res +function exceptionlayer(handler) + return function(ctx, stream; status_exception::Bool=true, kw...) + res = handler(ctx, stream; kw...) + if status_exception && iserror(res) + throw(StatusError(res.status, res.request.method, res.request.target, res)) + else + return res + end end end diff --git a/src/HTTP.jl b/src/HTTP.jl index 5e5b76995..350551b82 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -37,7 +37,6 @@ include("ConnectionPool.jl") include("Messages.jl") ;using .Messages include("cookies.jl") ;using .Cookies include("Streams.jl") ;using .Streams -include("layers.jl") ;using .Layers const nobody = UInt8[] @@ -308,17 +307,38 @@ end """ function request(method, url, h=Header[], b=nobody; headers=h, body=b, query=nothing, kw...)::Response - return request(HTTP.stack(; kw...), method, url, headers, body, query) + return request(HTTP.stack(), method, url, headers, body, query; kw...) end const Context = Dict{Symbol, Any} -function request(stack::Layers.Layer, method, url, h=Header[], b=nobody, q=nothing; - headers=h, body=b, query=q)::Response - return Layers.request(stack, Context(), string(method), request_uri(url, query), mkheaders(headers), body) +struct Stack + stack end -macro client(layertypes...) +function stack( + # custom layers + initiallayers=(), + requestlayers=(), + streamlayers=()) + + # stream layers + slayers = (timeoutlayer, exceptionlayer, debuglayer, streamlayers...) + layers = foldl((x, y) -> y(x), slayers, init=streamlayer) + # transition to stream and request layers + rlayers = (cookielayer, retrylayer, requestlayers...) + layers2 = foldl((x, y) -> y(x), rlayers; init=connectionlayer(layers)) + # transition to request and initial layers + ilayers = (redirectlayer, basicauthlayer, contenttypedetectionlayer, canonicalizelayer, initiallayers...) + return Stack(foldl((x, y) -> y(x), ilayers; init=messagelayer(layers2))) +end + +function request(stack::Stack, method, url, h=Header[], b=nobody, q=nothing; + headers=h, body=b, query=q, kw...)::Response + return stack.stack(Context(), string(method), request_uri(url, query), mkheaders(headers), body; kw...) +end + +macro client(initiallayers, requestlayers, streamlayers) esc(quote get(a...; kw...) = request("GET", a...; kw...) put(a...; kw...) = request("PUT", a...; kw...) @@ -327,7 +347,7 @@ macro client(layertypes...) head(u; kw...) = request("HEAD", u; kw...) delete(a...; kw...) = request("DELETE", a...; kw...) request(method, url, h=HTTP.Header[], b=HTTP.nobody; headers=h, body=b, query=nothing, kw...)::HTTP.Response = - HTTP.request(HTTP.stack($(layertypes...); kw...), method, url, headers, body, query) + HTTP.request(HTTP.stack($initiallayers, $requestlayers, $streamlayers), method, url, headers, body, query; kw...) end) end @@ -450,161 +470,6 @@ include("DebugRequest.jl"); using .DebugRequest include("StreamRequest.jl"); using .StreamRequest include("ContentTypeRequest.jl"); using .ContentTypeDetection -""" -The `stack()` function returns the default HTTP Layer-stack type. -This type is passed as the first parameter to the [`HTTP.request`](@ref) function. - -`stack()` accepts optional keyword arguments to enable/disable specific layers -in the stack: -`request(method, args...; kw...) request(stack(; kw...), args...; kw...)` - - -The minimal request execution stack is: - -```julia -stack = MessageLayer{ConnectionPoolLayer{StreamLayer}} -``` - -The figure below illustrates the full request execution stack and its -relationship with [`HTTP.Response`](@ref), [`HTTP.Parsers`](@ref), -[`HTTP.Stream`](@ref) and the [`HTTP.ConnectionPool`](@ref). - -``` - ┌────────────────────────────────────────────────────────────────────────────┐ - │ ┌───────────────────┐ │ - │ HTTP.jl Request Execution Stack │ HTTP.ParsingError ├ ─ ─ ─ ─ ┐ │ - │ └───────────────────┘ │ - │ ┌───────────────────┐ │ │ - │ │ HTTP.IOError ├ ─ ─ ─ │ - │ └───────────────────┘ │ │ │ - │ ┌───────────────────┐ │ - │ │ HTTP.StatusError │─ ─ │ │ │ - │ └───────────────────┘ │ │ - │ ┌───────────────────┐ │ │ │ - │ request(method, url, headers, body) -> │ HTTP.Response │ │ │ - │ ────────────────────────── └─────────▲─────────┘ │ │ │ - │ ║ ║ │ │ - │ ┌────────────────────────────────────────────────────────────┐ │ │ │ - │ │ request(TopLayer, method, ::URI, ::Headers, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(BasicAuthLayer, method, ::URI, ::Headers, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(BasicAuthLayer, method, ::URI, ::Headers, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(CookieLayer, method, ::URI, ::Headers, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(CanonicalizeLayer, method, ::URI, ::Headers, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(MessageLayer, method, ::URI, ::Headers, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(RetryLayer, ::URI, ::Request, body) │ │ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ - │ │ request(ExceptionLayer, ::URI, ::Request, body) ├ ─ ┘ │ - │ ├────────────────────────────────────────────────────────────┤ │ │ │ -┌┼───┤ request(ConnectionPoolLayer, ::URI, ::Request, body) ├ ─ ─ ─ │ -││ ├────────────────────────────────────────────────────────────┤ │ │ -││ │ request(DebugLayer, ::IO, ::Request, body) │ │ -││ ├────────────────────────────────────────────────────────────┤ │ │ -││ │ request(TimeoutLayer, ::IO, ::Request, body) │ │ -││ ├────────────────────────────────────────────────────────────┤ │ │ -││ │ request(StreamLayer, ::IO, ::Request, body) │ │ -││ └──────────────┬───────────────────┬─────────────────────────┘ │ │ -│└──────────────────┼────────║──────────┼───────────────║─────────────────────┘ -│ │ ║ │ ║ │ -│┌──────────────────▼───────────────┐ │ ┌──────────────────────────────────┐ -││ HTTP.Request │ │ │ HTTP.Response │ │ -││ │ │ │ │ -││ method::String ◀───┼──▶ status::Int │ │ -││ target::String │ │ │ headers::Vector{Pair} │ -││ headers::Vector{Pair} │ │ │ body::Vector{UInt8} │ │ -││ body::Vector{UInt8} │ │ │ │ -│└──────────────────▲───────────────┘ │ └───────────────▲────────────────┼─┘ -│┌──────────────────┴────────║──────────▼───────────────║──┴──────────────────┐ -││ HTTP.Stream <:IO ║ ╔══════╗ ║ │ │ -││ ┌───────────────────────────┐ ║ ┌──▼─────────────────────────┐ │ -││ │ startwrite(::Stream) │ ║ │ startread(::Stream) │ │ │ -││ │ write(::Stream, body) │ ║ │ read(::Stream) -> body │ │ -││ │ ... │ ║ │ ... │ │ │ -││ │ closewrite(::Stream) │ ║ │ closeread(::Stream) │ │ -││ └───────────────────────────┘ ║ └────────────────────────────┘ │ │ -│└───────────────────────────║────────┬──║──────║───────║──┬──────────────────┘ -│┌──────────────────────────────────┐ │ ║ ┌────▼───────║──▼────────────────┴─┐ -││ HTTP.Messages │ │ ║ │ HTTP.Parsers │ -││ │ │ ║ │ │ -││ writestartline(::IO, ::Request) │ │ ║ │ parse_status_line(bytes, ::Req') │ -││ writeheaders(::IO, ::Request) │ │ ║ │ parse_header_field(bytes, ::Req')│ -│└──────────────────────────────────┘ │ ║ └──────────────────────────────────┘ -│ ║ │ ║ -│┌───────────────────────────║────────┼──║────────────────────────────────────┐ -└▶ HTTP.ConnectionPool ║ │ ║ │ - │ ┌──────────────▼────────┐ ┌───────────────────────┐ │ - │ getconnection() -> │ HTTP.Connection <:IO │ │ HTTP.Connection <:IO │ │ - │ └───────────────────────┘ └───────────────────────┘ │ - │ ║ ╲│╱ ║ ╲│╱ │ - │ ║ │ ║ │ │ - │ ┌───────────▼───────────┐ ┌───────────▼───────────┐ │ - │ pool: [│ HTTP.Connection │,│ HTTP.Connection │...]│ - │ └───────────┬───────────┘ └───────────┬───────────┘ │ - │ ║ │ ║ │ │ - │ ┌───────────▼───────────┐ ┌───────────▼───────────┐ │ - │ │ Base.TCPSocket <:IO │ │MbedTLS.SSLContext <:IO│ │ - │ └───────────────────────┘ └───────────┬───────────┘ │ - │ ║ ║ │ │ - │ ║ ║ ┌───────────▼───────────┐ │ - │ ║ ║ │ Base.TCPSocket <:IO │ │ - │ ║ ║ └───────────────────────┘ │ - └───────────────────────────║───────────║────────────────────────────────────┘ - ║ ║ - ┌───────────────────────────║───────────║──────────────┐ ┏━━━━━━━━━━━━━━━━━━┓ - │ HTTP Server ▼ │ ┃ data flow: ════▶ ┃ - │ Request Response │ ┃ reference: ────▶ ┃ - └──────────────────────────────────────────────────────┘ ┗━━━━━━━━━━━━━━━━━━┛ -``` -*See `docs/src/layers`[`.monopic`](http://monodraw.helftone.com).* -""" -function stack(layertypes::Type{<:Layers.Layer}...; - # default keyword arg values - basicauth=true, - cookies=false, - canonicalize_headers=false, - retry=true, - status_exception=true, - readtimeout=0, - detect_content_type=false, - verbose=0, - redirect=true, - kw...) - - # ResponseLayers - layers = ExceptionLayer(BottomLayer(); status_exception=status_exception) - layers = stacklayertypes(Layers.ResponseLayer, layers, layertypes; kw...) - # transition ConnectionLayer => ResponseLayer - layers = StreamLayer(layers; verbose=verbose, kw...) - # ConnectionLayers - layers = DebugLayer(TimeoutLayer(layers; readtimeout=readtimeout); verbose=verbose) - layers = stacklayertypes(Layers.ConnectionLayer, layers, layertypes; kw...) - # transition RequestLayer => ConnectionLayer - layers = ConnectionPoolLayer(layers; kw...) - # RequestLayers - layers = RetryLayer(layers; retry=retry, kw...) - layers = stacklayertypes(Layers.RequestLayer, layers, layertypes; kw...) - # transition InitialLayer => RequestLayer - layers = MessageLayer(layers; kw...) - layers = BasicAuthLayer(CanonicalizeLayer(ContentTypeDetectionLayer(RedirectLayer(CookieLayer(layers; - cookies=cookies, kw...); redirect=redirect, kw...); detect_content_type=detect_content_type, kw...); - canonicalize_headers=canonicalize_headers, kw...); basicauth=basicauth, kw...) - return stacklayertypes(Layers.InitialLayer, layers, layertypes; kw...) -end - -function stacklayertypes(::Type{T}, layers::Layers.Layer, layertypes; kw...) where {T} - for LayerType in layertypes - if LayerType <: T - layers = LayerType(layers; kw...) - end - end - return layers -end - include("download.jl") include("Servers.jl") ;using .Servers; using .Servers: listen include("Handlers.jl") ;using .Handlers; using .Handlers: serve diff --git a/src/IOExtras.jl b/src/IOExtras.jl index 314292293..4e1894390 100644 --- a/src/IOExtras.jl +++ b/src/IOExtras.jl @@ -10,7 +10,7 @@ module IOExtras using ..Sockets using MbedTLS: MbedException -export bytes, ByteView, nobytes, CodeUnits, IOError, isioerror, +export bytes, isbytes, ByteView, nobytes, CodeUnits, IOError, isioerror, startwrite, closewrite, startread, closeread, tcpsocket, localport, safe_getpeername @@ -30,6 +30,8 @@ bytes(s::SubString{String}) = codeunits(s) bytes(s::Vector{UInt8}) = s +isbytes(x) = x isa AbstractVector{UInt8} + """ isioerror(exception) diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index 2b12079a2..6c295ec7f 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -1,9 +1,8 @@ module MessageRequest -export body_is_a_stream, body_was_streamed, setuseragent!, resource +export setuseragent!, resource using ..Base64 -using ..Layers using ..IOExtras using URIs using ..Messages @@ -11,55 +10,47 @@ import ..Messages: bodylength import ..Headers import ..Form, ..content_type +export messagelayer + """ - Layers.request(MessageLayer, method, ::URI, headers, body) -> HTTP.Response + messagelayer(ctx, method, ::URI, headers, body) -> HTTP.Response Construct a [`Request`](@ref) object and set mandatory headers. """ -struct MessageLayer{Next <: Layer} <: RequestLayer - next::Next - http_version::VersionNumber - iofunction -end -export MessageLayer -MessageLayer(next; - http_version=v"1.1", - iofunction=nothing, - kw...) = MessageLayer(next, http_version, iofunction) - -function Layers.request(layer::MessageLayer, ctx, method::String, url::URI, headers::Headers, body) - - if isempty(url.port) || - (url.scheme == "http" && url.port == "80") || - (url.scheme == "https" && url.port == "443") - hostheader = url.host - else - hostheader = url.host * ":" * url.port - end - defaultheader!(headers, "Host" => hostheader) - defaultheader!(headers, "Accept" => "*/*") - if USER_AGENT[] !== nothing - defaultheader!(headers, "User-Agent" => USER_AGENT[]) - end +function messagelayer(handler) + return function(ctx, method::String, url::URI, headers::Headers, body; iofunction=nothing, response_stream=nothing, http_version=v"1.1", kw...) + if isempty(url.port) || + (url.scheme == "http" && url.port == "80") || + (url.scheme == "https" && url.port == "443") + hostheader = url.host + else + hostheader = url.host * ":" * url.port + end + defaultheader!(headers, "Host" => hostheader) + defaultheader!(headers, "Accept" => "*/*") + if USER_AGENT[] !== nothing + defaultheader!(headers, "User-Agent" => USER_AGENT[]) + end - if !hasheader(headers, "Content-Length") && - !hasheader(headers, "Transfer-Encoding") && - !hasheader(headers, "Upgrade") - l = bodylength(body) - if l != unknown_length - setheader(headers, "Content-Length" => string(l)) - elseif method == "GET" && layer.iofunction isa Function - setheader(headers, "Content-Length" => "0") + if !hasheader(headers, "Content-Length") && + !hasheader(headers, "Transfer-Encoding") && + !hasheader(headers, "Upgrade") + l = bodylength(body) + if l != unknown_length + setheader(headers, "Content-Length" => string(l)) + elseif method == "GET" && iofunction isa Function + setheader(headers, "Content-Length" => "0") + end end - end - if !hasheader(headers, "Content-Type") && body isa Form && method in ("POST", "PUT") - # "Content-Type" => "multipart/form-data; boundary=..." - setheader(headers, content_type(body)) - end - parent = get(ctx, :parentrequest, nothing) - req = Request(method, resource(url), headers, bodybytes(body); url=url, version=layer.http_version, parent=parent) + if !hasheader(headers, "Content-Type") && body isa Form && method in ("POST", "PUT") + # "Content-Type" => "multipart/form-data; boundary=..." + setheader(headers, content_type(body)) + end + parent = get(ctx, :parentrequest, nothing) + req = Request(method, resource(url), headers, bodybytes(body); url=url, version=http_version, responsebody=response_stream, parent=parent) - return Layers.request(layer.next, ctx, req, body) + return handler(ctx, req; iofunction=iofunction, kw...) + end end const USER_AGENT = Ref{Union{String, Nothing}}("HTTP.jl/$VERSION") @@ -85,14 +76,11 @@ bodylength(body::Vector{T}) where T <: AbstractArray{UInt8,1} = sum(length, body bodylength(body::IOBuffer) = bytesavailable(body) bodylength(body::Vector{IOBuffer}) = sum(bytesavailable, body) -const body_is_a_stream = UInt8[] -const body_was_streamed = bytes("[Message Body was streamed]") -bodybytes(body) = body_is_a_stream +bodybytes(body) = body bodybytes(body::Vector{UInt8}) = body bodybytes(body::IOBuffer) = read(body) bodybytes(body::AbstractVector{UInt8}) = Vector{UInt8}(body) bodybytes(body::AbstractString) = bytes(body) -bodybytes(body::Vector) = length(body) == 1 ? bodybytes(body[1]) : - body_is_a_stream +bodybytes(body::Vector) = length(body) == 1 ? bodybytes(body[1]) : body end # module MessageRequest diff --git a/src/Messages.jl b/src/Messages.jl index 1f8028566..473c5d3db 100644 --- a/src/Messages.jl +++ b/src/Messages.jl @@ -75,6 +75,7 @@ import ..bytes include("ascii.jl") +const nobody = UInt8[] const unknown_length = typemax(Int) abstract type Message end @@ -104,26 +105,25 @@ Represents a HTTP Response Message. You can get each data with [`HTTP.status`](@ref), [`HTTP.headers`](@ref), and [`HTTP.body`](@ref). """ -mutable struct Response <: Message +mutable struct Response{T} <: Message version::VersionNumber status::Int16 headers::Headers - body::Vector{UInt8} - request::Message + body::T # Vector{UInt8} or IO + request::Union{Message, Nothing} @doc """ Response(status::Int, headers=[]; body=UInt8[], request=nothing) -> HTTP.Response """ - function Response(status::Integer, headers=[]; body=UInt8[], request=nothing) - r = new() - r.version = v"1.1" - r.status = status - r.headers = mkheaders(headers) - r.body = bytes(body) - if request !== nothing - r.request = request - end - return r + function Response(status::Integer, headers=[]; body=nobody, request=nothing) + b = body isa IO ? body : bytes(body) + return new{typeof(b)}( + v"1.1", + status, + mkheaders(headers), + b, + request + ) end end @@ -145,7 +145,7 @@ Response(s::Int, body::AbstractString) = Response(s, bytes(body)) Response(body) = Response(200, body) -Base.convert(::Type{Response},s::AbstractString) = Response(s) +Base.convert(::Type{Response}, s::AbstractString) = Response(s) function reset!(r::Response) r.version = v"1.1" @@ -153,7 +153,7 @@ function reset!(r::Response) if !isempty(r.headers) empty!(r.headers) end - if !isempty(r.body) + if r.body isa Vector{UInt8} && !isempty(r.body) empty!(r.body) end end @@ -198,7 +198,7 @@ Represents a HTTP Request Message. - `headers::Vector{Pair{String,String}}` [RFC7230 3.2](https://tools.ietf.org/html/rfc7230#section-3.2) -- `body::Vector{UInt8}` +- `body::Union{Vector{UInt8}, IO}` [RFC7230 3.3](https://tools.ietf.org/html/rfc7230#section-3.3) - `response`, the `Response` to this `Request` @@ -210,12 +210,12 @@ Represents a HTTP Request Message. You can get each data with [`HTTP.method`](@ref), [`HTTP.headers`](@ref), [`HTTP.uri`](@ref), and [`HTTP.body`](@ref). """ -mutable struct Request <: Message +mutable struct Request{T} <: Message method::String target::String version::VersionNumber headers::Headers - body::Vector{UInt8} + body::T # Vector{UInt8} or some kind of IO response::Response url::URI parent @@ -229,14 +229,15 @@ Request() = Request("", "") Constructor for `HTTP.Request`. For daily use, see [`HTTP.request`](@ref). """ -function Request(method::String, target, headers=[], body=UInt8[]; - version=v"1.1", url::URI=URI(), parent=nothing) - r = Request(method, +function Request(method::String, target, headers=[], body=nobody; + version=v"1.1", url::URI=URI(), responsebody=nothing, parent=nothing) + b = body isa IO ? body : bytes(something(body, nobody)) + r = Request{typeof(b)}(method, target == "" ? "/" : target, version, mkheaders(headers), - bytes(body), - Response(0), + b, + Response(0; body=something(responsebody, nobody)), url, parent) r.response.request = r diff --git a/src/RedirectRequest.jl b/src/RedirectRequest.jl index f5d4deb1c..376d48b43 100644 --- a/src/RedirectRequest.jl +++ b/src/RedirectRequest.jl @@ -1,79 +1,65 @@ module RedirectRequest -using ..Layers using URIs using ..Messages using ..Pairs: setkv import ..Header import ..@debug, ..DEBUG_LEVEL +export redirectlayer + """ - Layers.request(RedirectLayer, method, ::URI, headers, body) -> HTTP.Response + redirectlayer(ctx, method, ::URI, headers, body) -> HTTP.Response Redirects the request in the case of 3xx response status. """ -struct RedirectLayer{Next <: Layer} <: InitialLayer - next::Next - redirect::Bool - redirect_limit::Int - forwardheaders::Bool -end - -export RedirectLayer - -RedirectLayer(next; redirect::Bool=true, redirect_limit=3, forwardheaders=true, kw...) = - RedirectLayer(next, redirect, redirect_limit, forwardheaders) - -function Layers.request(layer::RedirectLayer, ctx, method, url, headers, body) - redirect_limit = layer.redirect_limit - if !layer.redirect || layer.redirect_limit == 0 - # no redirecting - return Layers.request(layer.next, ctx, method, url, headers, body) - end - - forwardheaders = layer.forwardheaders - count = 0 - while true - - # Verify the url before making the request. Verification is done in - # the redirect loop to also catch bad redirect URLs. - verify_url(url) - if count == redirect_limit - ctx[:redirectlimitreached] = true +function redirectlayer(handler) + return function(ctx, method, url, headers, body; redirect::Bool=true, redirect_limit::Int=3, forwardheaders::Bool=true, kw...) + println("redirectlayer") + if !redirect || redirect_limit == 0 + # no redirecting + return handler(ctx, method, url, headers, body; kw...) end - res = Layers.request(layer.next, ctx, method, url, headers, body) - if (count == redirect_limit - || !isredirect(res) - || (location = header(res, "Location")) == "") - return res - end + count = 0 + while true + # Verify the url before making the request. Verification is done in + # the redirect loop to also catch bad redirect URLs. + verify_url(url) + if count == redirect_limit + ctx[:redirectlimitreached] = true + end + res = handler(ctx, method, url, headers, body; kw...) + + if (count == redirect_limit || !isredirect(res) + || (location = header(res, "Location")) == "") + return res + end - # follow redirect - ctx[:parentrequest] = res - oldurl = url - url = resolvereference(oldurl, location) - if forwardheaders - headers = filter(headers) do (header, _) - # false return values are filtered out - if header == "Host" - return false - elseif (header in SENSITIVE_HEADERS && !isdomainorsubdomain(url.host, url.host)) - return false - else - return true + # follow redirect + ctx[:parentrequest] = res + oldurl = url + url = resolvereference(oldurl, location) + if forwardheaders + headers = filter(headers) do (header, _) + # false return values are filtered out + if header == "Host" + return false + elseif (header in SENSITIVE_HEADERS && !isdomainorsubdomain(url.host, oldurl.host)) + return false + else + return true + end end + else + headers = Header[] end - else - headers = Header[] + @show 1 "➡️ Redirect: $url" + @show headers + count += 1 end - - @debug 1 "➡️ Redirect: $url" - - count += 1 + @assert false "Unreachable!" end - - @assert false "Unreachable!" end const SENSITIVE_HEADERS = Set([ diff --git a/src/RetryRequest.jl b/src/RetryRequest.jl index 53eef8dd2..a39557811 100644 --- a/src/RetryRequest.jl +++ b/src/RetryRequest.jl @@ -1,15 +1,16 @@ module RetryRequest import ..HTTP -using ..Layers using ..Sockets using ..IOExtras using ..MessageRequest using ..Messages import ..@debug, ..DEBUG_LEVEL, ..sprintcompact +export retrylayer + """ - Layers.request(RetryLayer, ::URI, ::Request, body) -> HTTP.Response + retrylayer(ctx, req) -> HTTP.Response Retry the request if it throws a recoverable exception. @@ -21,35 +22,27 @@ Methods of `isrecoverable(e)` define which exception types lead to a retry. e.g. `HTTP.IOError`, `Sockets.DNSError`, `Base.EOFError` and `HTTP.StatusError` (if status is ``5xx`). """ -struct RetryLayer{Next <: Layer} <: RequestLayer - next::Next - retry::Bool - retries::Int - retry_non_idempotent::Bool -end -export RetryLayer -RetryLayer(next; retry::Bool=true, retries::Int=4, retry_non_idempotent=false, kw...) = - RetryLayer(next, retry, retries, retry_non_idempotent) +function retrylayer(handler) + return function(ctx, req::Request; retry::Bool=true, retries::Int=4, retry_non_idempotent::Bool=false, kw...) + if !retry || retries == 0 + # no retry + return handler(ctx, req; kw...) + end + retry_request = Base.retry(handler, + delays=ExponentialBackOff(n = retries), + check=(s, ex)->begin + retry = isrecoverable(ex, req, retry_non_idempotent, get(ctx, :retrycount, 0)) + if retry + @debug 1 "🔄 Retry $ex: $(sprintcompact(req))" + reset!(req.response) + else + @debug 1 "🚷 No Retry: $(no_retry_reason(ex, req))" + end + return s, retry + end) -function Layers.request(layer::RetryLayer, ctx, req::Request, body) - if !layer.retry || layer.retries == 0 - # no retry - return Layers.request(layer.next, ctx, req, body) + return retry_request(ctx, req; kw...) end - retry_request = Base.retry(Layers.request, - delays=ExponentialBackOff(n = layer.retries), - check=(s,ex)->begin - retry = isrecoverable(ex, req, layer.retry_non_idempotent, get(ctx, :retrycount, 0)) - if retry - @debug 1 "🔄 Retry $ex: $(sprintcompact(req))" - reset!(req.response) - else - @debug 1 "🚷 No Retry: $(no_retry_reason(ex, req))" - end - return s, retry - end) - - return retry_request(layer.next, ctx, req, body) end isrecoverable(e) = false diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index d8c649641..8cfec3e3f 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -1,6 +1,5 @@ module StreamRequest -using ..Layers using ..IOExtras using ..Messages using ..Streams @@ -8,8 +7,10 @@ import ..ConnectionPool using ..MessageRequest import ..@debug, ..DEBUG_LEVEL, ..printlncompact, ..sprintcompact +export streamlayer + """ - Layers.request(StreamLayer, ::IO, ::Request, body) -> HTTP.Response + streamlayer(ctx, stream) -> HTTP.Response Create a [`Stream`](@ref) to send a `Request` and `body` to an `IO` stream and read the response. @@ -19,35 +20,18 @@ immediately so that the transmission can be aborted if the `Response` status indicates that the server does not wish to receive the message body. [RFC7230 6.5](https://tools.ietf.org/html/rfc7230#section-6.5). """ -struct StreamLayer{Next <: Layer} <: ConnectionLayer - next::Next - response_stream - iofunction - verbose -end -export StreamLayer -StreamLayer(next; - response_stream=nothing, - iofunction=nothing, - verbose::Int=0, - kw...) = StreamLayer(next, response_stream, iofunction, verbose) - -function Layers.request(layer::StreamLayer, ctx, io::IO, req::Request, body)::Response - response_stream = layer.response_stream - iofunction = layer.iofunction - verbose = layer.verbose +function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...)::Response + response = stream.message + req = response.request verbose == 1 && printlncompact(req) - - response = req.response - http = Stream(response, io) @debug 2 "client startwrite" - startwrite(http) + startwrite(stream) if verbose == 2 println("printing req") println(req) - if iofunction === nothing && req.body === body_is_a_stream - println("$(typeof(req)).body: $(sprintcompact(body))") + if iofunction === nothing && req.body isa IO + println("$(typeof(req)).body: $(sprintcompact(req.body))") end end @@ -56,21 +40,21 @@ function Layers.request(layer::StreamLayer, ctx, io::IO, req::Request, body)::Re @sync begin if iofunction === nothing @async try - writebody(http, ctx, req, body) + writebody(stream, ctx, req) @debug 2 "client closewrite" - closewrite(http) + closewrite(stream) catch e write_error = e isopen(io) && try; close(io); catch; end end @debug 2 "client startread" - startread(http) - readbody(http, response, response_stream, get(ctx, :redirectlimitreached, false)) + startread(stream) + readbody(stream, response, get(ctx, :redirectlimitreached, false)) else - iofunction(http) + iofunction(stream) end - if isaborted(http) + if isaborted(stream) # The server may have closed the connection. # Don't propagate such errors. try; close(io); catch; end @@ -85,49 +69,47 @@ function Layers.request(layer::StreamLayer, ctx, io::IO, req::Request, body)::Re end @debug 2 "client closewrite" - closewrite(http) + closewrite(stream) @debug 2 "client closeread" - closeread(http) + closeread(stream) verbose == 1 && printlncompact(response) verbose == 2 && println(response) - return Layers.request(layer.next, ctx, response) + return response end -function writebody(http::Stream, ctx, req::Request, body) +function writebody(stream::Stream, ctx, req::Request) - if req.body === body_is_a_stream - writebodystream(http, req, body) - closebody(http) + if !isbytes(req.body) + writebodystream(stream, req.body) + closebody(stream) else - write(http, req.body) + write(stream, req.body) end ctx[:retrycount] = get(ctx, :retrycount, 0) + 1 return end -function writebodystream(http, req, body) +function writebodystream(stream, body) for chunk in body - writechunk(http, req, chunk) + writechunk(stream, chunk) end end -function writebodystream(http, req, body::IO) - req.body = body_was_streamed - write(http, body) +function writebodystream(stream, body::IO) + write(stream, body) end -writechunk(http, req, body::IO) = writebodystream(http, req, body) -writechunk(http, req, body) = write(http, body) +writechunk(stream, body::IO) = writebodystream(stream, body) +writechunk(stream, body) = write(stream, body) -function readbody(http::Stream, res::Response, response_stream, redirectlimitreached) - if response_stream === nothing - res.body = read(http) +function readbody(stream::Stream, res::Response, redirectlimitreached) + if isbytes(res.body) + res.body = read(stream) else if redirectlimitreached || !isredirect(res) - res.body = body_was_streamed - write(response_stream, http) + write(res.body, stream) end end end diff --git a/src/Streams.jl b/src/Streams.jl index e763a427d..a75c4a56c 100644 --- a/src/Streams.jl +++ b/src/Streams.jl @@ -45,7 +45,7 @@ Creates a `HTTP.Stream` that wraps an existing `IO` stream. for reuse. If a complete response has not been received, `closeread` throws `EOFError`. """ -Stream(r::M, io::S) where {M, S} = Stream{M,S}(r, io, false, false, true, 0, 0) +Stream(r::M, io::S) where {M, S} = Stream{M, S}(r, io, false, false, true, 0, 0) header(http::Stream, a...) = header(http.message, a...) setstatus(http::Stream, status) = (http.message.response.status = status) @@ -65,8 +65,8 @@ IOExtras.isopen(http::Stream) = isopen(http.stream) # Writing HTTP Messages -messagetowrite(http::Stream{Response}) = http.message.request -messagetowrite(http::Stream{Request}) = http.message.response +messagetowrite(http::Stream{<:Response}) = http.message.request +messagetowrite(http::Stream{<:Request}) = http.message.response IOExtras.iswritable(http::Stream) = iswritable(http.stream) @@ -124,7 +124,7 @@ function closebody(http::Stream) end end -function IOExtras.closewrite(http::Stream{Response}) +function IOExtras.closewrite(http::Stream{<:Response}) if !iswritable(http) return end @@ -132,7 +132,7 @@ function IOExtras.closewrite(http::Stream{Response}) closewrite(http.stream) end -function IOExtras.closewrite(http::Stream{Request}) +function IOExtras.closewrite(http::Stream{<:Request}) if iswritable(http) closebody(http) @@ -176,14 +176,14 @@ end https://tools.ietf.org/html/rfc7230#section-5.6 https://tools.ietf.org/html/rfc7231#section-6.2.1 """ -function handle_continue(http::Stream{Response}) +function handle_continue(http::Stream{<:Response}) if http.message.status == 100 @debug 1 "✅ Continue: $(http.stream)" readheaders(http.stream, http.message) end end -function handle_continue(http::Stream{Request}) +function handle_continue(http::Stream{<:Request}) if hasheader(http.message, "Expect", "100-continue") if !iswritable(http.stream) startwrite(http.stream) @@ -316,7 +316,7 @@ function Base.read(http::Stream) end """ - isaborted(::Stream{Response}) + isaborted(::Stream{<:Response}) Has the server signaled that it does not wish to receive the message body? @@ -325,7 +325,7 @@ Has the server signaled that it does not wish to receive the message body? immediately cease transmitting the body and close the connection." [RFC7230, 6.5](https://tools.ietf.org/html/rfc7230#section-6.5) """ -function isaborted(http::Stream{Response}) +function isaborted(http::Stream{<:Response}) if iswritable(http.stream) && iserror(http.message) && @@ -341,7 +341,7 @@ end incomplete(http::Stream) = http.ntoread > 0 && (http.readchunked || http.ntoread != unknown_length) -function IOExtras.closeread(http::Stream{Response}) +function IOExtras.closeread(http::Stream{<:Response}) if hasheader(http.message, "Connection", "close") # Close conncetion if server sent "Connection: close"... @@ -368,7 +368,7 @@ function IOExtras.closeread(http::Stream{Response}) return http.message end -function IOExtras.closeread(http::Stream{Request}) +function IOExtras.closeread(http::Stream{<:Request}) if incomplete(http) # Error if Message is not complete... close(http.stream) diff --git a/src/TimeoutRequest.jl b/src/TimeoutRequest.jl index 258c260e3..54e13f63e 100644 --- a/src/TimeoutRequest.jl +++ b/src/TimeoutRequest.jl @@ -1,8 +1,8 @@ module TimeoutRequest -using ..Layers using ..ConnectionPool import ..@debug, ..DEBUG_LEVEL +import ..Streams: Stream struct ReadTimeoutError <:Exception readtimeout::Int @@ -12,46 +12,43 @@ function Base.showerror(io::IO, e::ReadTimeoutError) print(io, "ReadTimeoutError: Connection closed after $(e.readtimeout) seconds") end +export timeoutlayer + """ - Layers.request(TimeoutLayer, ::IO, ::Request, body) -> HTTP.Response + timeoutlayer(ctx, stream) -> HTTP.Response Close `IO` if no data has been received for `timeout` seconds. """ -struct TimeoutLayer{Next <: Layer} <: ConnectionLayer - next::Next - readtimeout::Int -end -export TimeoutLayer -TimeoutLayer(next; readtimeout::Int=0, kw...) = TimeoutLayer(next, readtimeout) - -function Layers.request(layer::TimeoutLayer, ctx, io::IO, req, body) - readtimeout = layer.readtimeout - if readtimeout <= 0 - # skip - return Layers.request(layer.next, ctx, io, req, body) - end - wait_for_timeout = Ref{Bool}(true) - timedout = Ref{Bool}(false) - - @async while wait_for_timeout[] - if isreadable(io) && inactiveseconds(io) > readtimeout - timedout[] = true - close(io) - @debug 1 "💥 Read inactive > $(readtimeout)s: $io" - break +function timeoutlayer(handler) + return function(ctx, stream::Stream; readtimeout::Int=0, kw...) + if readtimeout <= 0 + # skip + return handler(ctx, stream; kw...) + end + io = stream.stream + wait_for_timeout = Ref{Bool}(true) + timedout = Ref{Bool}(false) + + @async while wait_for_timeout[] + if isreadable(io) && inactiveseconds(io) > readtimeout + timedout[] = true + close(io) + @debug 1 "💥 Read inactive > $(readtimeout)s: $io" + break + end + sleep(readtimeout / 10) end - sleep(readtimeout / 10) - end - try - return Layers.request(layer.next, ctx, io, req, body) - catch e - if timedout[] - throw(ReadTimeoutError(readtimeout)) + try + return handler(ctx, stream; kw...) + catch e + if timedout[] + throw(ReadTimeoutError(readtimeout)) + end + rethrow(e) + finally + wait_for_timeout[] = false end - rethrow(e) - finally - wait_for_timeout[] = false end end diff --git a/src/layers.jl b/src/layers.jl deleted file mode 100644 index 4790fda55..000000000 --- a/src/layers.jl +++ /dev/null @@ -1,154 +0,0 @@ -""" -The Layers module in the HTTP.jl package contains the internal machinery for how http client requests are actually made. - -It exposes the concept of "layers" which are single components each responsible for handling -one "piece" of an http client request. Layers are linked together by each being required to -have a dedicated field to store the "next" layer in the stack to form a linked-list -where each layer has a window of execution when control is "passed" to it from the previous -layer in the stack. - -Builtin to the HTTP.jl package, there are 4 "kinds" of layers that are characterized by -"where" they live in stack and the corresponding arguments they have access to when control is passed to them: - * [`Layers.InitialLayer`](@ref): the "outermost" layers that receive arguments almost as-is provided from the calling user; - arguments include the http `method`, `url`, `headers`, and `body`. The HTTP.jl-internal layer `MessageLayer` comes after the - the last `Layers.InitialLayer` layer and transitions to the next layer kind - * [`Layers.RequestLayer`](@ref): the `MessageLayer` took the `method`, `url`, and `headers` arguments and formed a full `HTTP.Request` - object that layers in this "kind" now have access to; the `ConnectionPoolLayer` follows the last `RequestLayer` and opens a live - connection to the remote server, transitioning us to the next layer kind - * [`Layers.ConnectionLayer`](@ref): in addition to the `Request` object, we now also have access to the live/open `Connection` object - which is connected to the remote; the `StreamLayer` follows the last `ConnectionLayer` layer to execute the actual request and read the response - * [`Layers.ResponseLayer`](@ref): these are the "deepest" layers in the stack because they are only called after the request - has been sent and a response has been received - -So the rough flow of what actually happens when a user makes a call like `HTTP.get("https://google.com")` is as follows: - * A "stack" of layers is made, starting with `ResponseLayer`s, then wrapping those in `ConnectionLayer`s, and so on to the outermost `InitialLayer`s - * A little argument processing happens, but the request really begins execution with the first call to `Layers.request(layers, ctx, method, url, headers, body)` - * This passes control to the outermost layer's `Layers.request` method, where it's responsible for, at a minimum, passing control on by calling `Layers.request(layer.next, ctx, method, url, headers, body)` - * Conrol continues to pass down through the stack of layers until the `StreamLayer`, which physically sends the request and receives the response - * Control then goes "back up" the stack starting with the `ResponseLayer`s all the way back to the outermost "first" `InitialLayer` layer before actually returning to the user - -Ok, so why is all this important? Well, in addition to having a better understanding of what actually happens when you make a request, -this also provides necessary context for users who desire to _extend_ or _customize_ the request process. -Some examples of ways users may want to cusotmize: - * Compute and add a required authentication header to every request made to a specific service/host - * Provide configurable response "caching" given certain request inputs - * Act as a "load balancer" where service names are mapped to an internal registry of physical IP addresses - -We've already hinted in the explanations above about the requirements for implementing a proper layer, -so let's spell the interface out explicitly here: - * Create a custom layer struct that subtypes one of 4 layer "kind" types: - * `Layers.InitialLayer` - * `Layers.RequestLayer` - * `Layers.ConnectionLayer` - * `Layers.ResponseLayer` - * The custom layer struct MUST HAVE a dedicated field for storing the "next" layer in the stack; this usually looks something like: - -``` -struct CustomLayer{Next <: Layers.Layer} <: Layers.InitialLayer - next::Next - # additional fields... -end -``` - * There must be a constructor method of the form: `Layer(next; kw...)` where the `next` argument is some `Layers.Layer` subtype and must be stored in the above-mentioned required field - * The custom layer must then overload the `Layers.request` method that corresponds to the layer "kind" they subtype: - * `Layers.InitialLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, method, url, headers, body)` - * `Layers.RequestLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, request, body)` - * `Layers.ConnectionLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, io, request, body)` - * `Layers.ResponseLayer` => must overload: `Layers.request(layer::CustomLayer, ctx, response)` - * The final requirement is that IN THE `Layers.request` overload, control MUST BE passed on to the next layer in the stack by, at some point, calling `Layers.request(layer.next, args...)`, - where `layer.next` refers to the above-mentioned required field storing the "next" layer in the stack, and `args` are the SAME ARGUMENTS that were overloaded in the custom layer's `Layer.request` - overloaded method. - -Ok great, I think I've got a handle on how to go about creating my own custom layer (I can also poke around the many examples in -the HTTP.jl package itself, since they all implement this exact machinery). But once I have a custom layer, how do I USE IT? Or in -other words, how do I get it included in the request stack? - -HTTP.jl provides the `HTTP.stack(layers...; kw...)` function that takes any number of custom layers as initial positional arguments, -along with _all_ keyword arguments passed from users, and returns the request stack that will immediately be passed to `Layers.request`. -So manually, if I had my `CustomLayer` all setup and defined, I could "include" it by doing something like: -``` -resp = HTTP.request(HTTP.stack(CustomLayer), "GET", "https://google.com") -``` - -Ok, not terrible, but can we make it a little more convenient? We can. HTTP.jl provides a convenience macro that -will automatically define your own set of "user-facing request" methods, but with any specified custom layers -automatically included in the stack. Wait, this sounds magical; show me? - -``` -module MyClient - -using HTTP - -include("customlayer_definitions.jl") -HTTP.@client CustomLayer - -end -``` - -Ok, so what we defined here is a module called `MyClient`, which included a custom layer implementation (not fully shown, just -`include`ed) and then the macro invocation of `HTTP.@client CustomerLayer`. The macro expands to define our very own -`MyClient.get`, `MyClient.post`, `MyClient.put`, `MyClient.delete`, etc. methods, but which each include `CustomLayer` -in the constructed request stack. Neat! So now users can just call: - -``` -using MyClient -resp = MyClient.get("https://google.com") -``` - -And they're using your customized http client request stack with the `CustomLayer` functionality! Cool! - -""" -module Layers - -export Layer, InitialLayer, RequestLayer, ConnectionLayer, ResponseLayer, BottomLayer - -""" - Layers.request(layer::L, args...) - -HTTP.jl internal method for executing the stack of layers -that make up a client request. Layers form a linked list -and must explicitly pass control to the next layer to ensure -each layer has a chance to execute its part of the request. -Each layer overloads `Layers.request` for their specific layer -type and the `args` to overload depend on which layer "kind" -they subtype: - * [`Layers.InitialLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, method, url, headers, body)`; this is the top-most layer type - * [`Layers.RequestLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, request, body)`; the `method`, `url`, and `headers` of the `InitialLayer` have been bundled together into a single `request::Request` argument - * [`Layers.ConnectionLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, io, request, body)`; a connection has now been opened to the remote and is available in the `io` argument - * [`Layers.ResponseLayer`](@ref): overloads `Layers.request(layer::TestLayer, ctx, resp)`; the request has been sent and a response has been received in the form of the `resp` argument - -Note that _every_ `Layers.request` overloads has a `ctx::Dict{Symbol, Any}` argument available if -any state needs to be shared between layers during the life of the request. - -See docs for the [`Layers`](@ref) module for a broader discussion on extending/customizing the client request stack. -""" -function request end - -""" - Layers.Layer - -Abstract type that all client request layer "kinds" must subtype. These currently include: - * [`Layers.InitialLayer`](@ref): top-most layers - * [`Layers.RequestLayer`](@ref): initial `Request` object has been formed - * [`Layers.ConnectionLayer`](@ref): a connection has been opened to the remote - * [`Layers.ResponseLayer`](@ref): the `Request` has been written on the connection and a response received - -Custom layers should subtype one of these layer "kinds" instead of subtyping `Layers.Layer` directly. -""" -abstract type Layer#{Next <: Layer} - # next::Next -end - - -abstract type InitialLayer <: Layer end -abstract type RequestLayer <: Layer end -abstract type ConnectionLayer <: Layer end -abstract type ResponseLayer <: Layer end - -# start the stack off w/ BottomLayer -struct BottomLayer <: ResponseLayer end - -# bottom layer just returns the response -request(::BottomLayer, ctx, resp) = resp - -end diff --git a/test/chunking.jl b/test/chunking.jl index 08d027b53..bc25d5d17 100644 --- a/test/chunking.jl +++ b/test/chunking.jl @@ -1,4 +1,6 @@ -using Test +module TestChunking + +using Test, Sockets using HTTP, HTTP.IOExtras using BufferedStreams @@ -15,8 +17,8 @@ using BufferedStreams "data: 3$(repeat("x", sz))\n\n" split1 = 106 split2 = 300 - - t = @async HTTP.listen("127.0.0.1", port) do http + server = Sockets.listen(ip"127.0.0.1", port) + t = @async HTTP.listen("127.0.0.1", port; server=server) do http startwrite(http) tcp = http.stream.io @@ -59,4 +61,7 @@ using BufferedStreams @test r == decoded_data end + close(server) end + +end # module \ No newline at end of file diff --git a/test/client.jl b/test/client.jl index 0eb0fc3b9..177ad5dc6 100644 --- a/test/client.jl +++ b/test/client.jl @@ -1,5 +1,8 @@ -using ..TestRequest +module TestClient + using HTTP +include(joinpath(dirname(pathof(HTTP)), "../test/resources/TestRequest.jl")) +using .TestRequest using Sockets using JSON using Test @@ -489,3 +492,5 @@ end end end end + +end # module \ No newline at end of file diff --git a/test/resources/TestRequest.jl b/test/resources/TestRequest.jl index be14d4517..928d8a453 100644 --- a/test/resources/TestRequest.jl +++ b/test/resources/TestRequest.jl @@ -1,33 +1,28 @@ module TestRequest -export TestLayer, LastLayer +using HTTP -using HTTP, HTTP.Layers - -struct TestLayer{Next <: Layer} <: InitialLayer - next::Next - wasincluded::Ref{Bool} +function testinitiallayer(handler) + return function(ctx, m, url, h, b; httptestlayer=Ref(false), kw...) + httptestlayer[] = true + return handler(ctx, m, url, h, b; kw...) + end end -TestLayer(next; httptestlayer=Ref(false), kw...) = TestLayer(next, httptestlayer) -function Layers.request(layer::TestLayer, ctx, meth, url, headers, body) - layer.wasincluded[] = true - return Layers.request(layer.next, ctx, meth, url, headers, body) +function testrequestlayer(handler) + return function(ctx, req; httptestlayer=Ref(false), kw...) + httptestlayer[] = true + return handler(ctx, req; kw...) + end end -HTTP.@client TestLayer - -# struct LastLayer{Next <: Layer} <: ConnectionLayer -# next::Next -# wasincluded::Ref{Bool} -# end -# Layers.keywordforlayer(::Val{:httplastlayer}) = LastLayer -# LastLayer(next; httplastlayer=Ref(false), kw...) = LastLayer(next, httplastlayer) +function teststreamlayer(handler) + return function(ctx, stream; httptestlayer=Ref(false), kw...) + httptestlayer[] = true + return handler(ctx, stream; kw...) + end +end -# function Layers.request(layer::LastLayer, io::IO, req, body; kw...) -# resp = Layers.request(layer.next, io, req, body; kw...) -# layer.wasincluded[] = true -# return resp -# end +HTTP.@client (testinitiallayer,) (testrequestlayer,) (teststreamlayer,) end diff --git a/test/server.jl b/test/server.jl index 5c38f3e20..6c49aec4a 100644 --- a/test/server.jl +++ b/test/server.jl @@ -27,7 +27,7 @@ end write(http, request.response.body) end - server = Sockets.listen(Sockets.InetAddr(parse(IPAddr, "127.0.0.1"), port)) + server = Sockets.listen(ip"127.0.0.1", port) tsk = @async HTTP.listen(handler, "127.0.0.1", port; server=server) sleep(3.0) @test !istaskdone(tsk) @@ -37,12 +37,12 @@ end sleep(0.5) @test istaskdone(tsk) - server = Sockets.listen(Sockets.InetAddr(parse(IPAddr, "127.0.0.1"), port)) + server = Sockets.listen(ip"127.0.0.1", port) tsk = @async HTTP.listen(handler, "127.0.0.1", port; server=server) handler2 = HTTP.Handlers.RequestHandlerFunction(req->HTTP.Response(200, req.body)) - server2 = Sockets.listen(Sockets.InetAddr(parse(IPAddr, "127.0.0.1"), port+100)) + server2 = Sockets.listen(ip"127.0.0.1", port+100) tsk2 = @async HTTP.serve(handler2, "127.0.0.1", port+100; server=server2) sleep(0.5) @test !istaskdone(tsk) @@ -129,7 +129,7 @@ end # keep-alive vs. close: issue #81 port += 1 - server = Sockets.listen(Sockets.InetAddr(parse(IPAddr, "127.0.0.1"), port)) + server = Sockets.listen(ip"127.0.0.1", port) tsk = @async HTTP.listen(hello, "127.0.0.1", port; server=server, verbose=true) sleep(0.5) @test !istaskdone(tsk) @@ -267,7 +267,7 @@ end @testset "on_shutdown" begin @test HTTP.Servers.shutdown(nothing) === nothing - IOserver = Sockets.listen(Sockets.InetAddr(parse(IPAddr, "127.0.0.1"), 8052)) + IOserver = Sockets.listen(ip"127.0.0.1", 8052) # Shutdown adds 1 TEST_COUNT = Ref(0) From 60d4d73168333d61522060603020f2baa3510997 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 17 Mar 2022 18:24:23 -0600 Subject: [PATCH 10/19] Another refactor to move layers to a more functional architecture and drastically reduce the interface requirements --- src/ContentTypeRequest.jl | 9 +++---- src/CookieRequest.jl | 16 +++++++++--- src/IOExtras.jl | 32 +++++++++++++++-------- src/MessageRequest.jl | 23 +++------------- src/Messages.jl | 18 ++++++------- src/RedirectRequest.jl | 4 +-- src/RetryRequest.jl | 8 +++--- src/StreamRequest.jl | 2 +- src/multipart.jl | 1 + src/sniff.jl | 2 ++ test/client.jl | 4 ++- test/cookies.jl | 7 +++++ test/loopback.jl | 19 +++++++------- test/messages.jl | 55 ++++++++++++++++++--------------------- test/server.jl | 1 - 15 files changed, 103 insertions(+), 98 deletions(-) diff --git a/src/ContentTypeRequest.jl b/src/ContentTypeRequest.jl index 14258f001..e5a6ac6d4 100644 --- a/src/ContentTypeRequest.jl +++ b/src/ContentTypeRequest.jl @@ -5,19 +5,18 @@ using ..Pairs: getkv, setkv import ..sniff import ..Form using ..Messages -import ..MessageRequest: bodylength, bodybytes +import ..IOExtras import ..@debug, ..DEBUG_LEVEL export contenttypedetectionlayer - +# f(::Handler) -> Handler function contenttypedetectionlayer(handler) return function(ctx, method, url, headers, body; detect_content_type::Bool=false, kw...) if detect_content_type && (getkv(headers, "Content-Type", "") == "" && !isa(body, Form) - && bodylength(body) != unknown_length - && bodylength(body) > 0) + && isbytes(body)) - sn = sniff(bodybytes(body)) + sn = sniff(bytes(body)) setkv(headers, "Content-Type", sn) @debug 1 "setting Content-Type header to: $sn" end diff --git a/src/CookieRequest.jl b/src/CookieRequest.jl index 2ab43b965..11dc1ffb4 100644 --- a/src/CookieRequest.jl +++ b/src/CookieRequest.jl @@ -24,7 +24,6 @@ Store new Cookies found in the response headers. """ function cookielayer(handler) return function(ctx, req::Request; cookies=true, cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) - println("cookielayer") if cookies === true || (cookies isa AbstractDict && !isempty(cookies)) url = req.url hostcookies = get!(cookiejar, url.host, Set{Cookie}()) @@ -35,12 +34,21 @@ function cookielayer(handler) end end if !isempty(cookiestosend) - setkv(req.headers, "Cookie", stringify(getkv(req.headers, "Cookie", ""), cookiestosend)) + existingcookie = getkv(req.headers, "Cookie", "") + if existingcookie != "" && get(ctx, :includedCookies, nothing) !== nothing + # this is a redirect where we previously included cookies + # we want to filter those out to avoid duplicate cookie sending + # and the case where a cookie was set to expire from the 1st request + previouscookies = Cookies.readcookies(req.headers, "") + previouslyincluded = ctx[:includedCookies] + filtered = filter(x -> !(x.name in previouslyincluded), previouscookies) + existingcookie = stringify("", filtered) + end + setkv(req.headers, "Cookie", stringify(existingcookie, cookiestosend)) + ctx[:includedCookies] = map(x -> x.name, cookiestosend) end - @show cookiestosend res = handler(ctx, req; kw...) setcookies(hostcookies, url.host, res.headers) - @show hostcookies return res else # skip diff --git a/src/IOExtras.jl b/src/IOExtras.jl index 4e1894390..8e44a8f59 100644 --- a/src/IOExtras.jl +++ b/src/IOExtras.jl @@ -10,27 +10,37 @@ module IOExtras using ..Sockets using MbedTLS: MbedException -export bytes, isbytes, ByteView, nobytes, CodeUnits, IOError, isioerror, +export bytes, isbytes, nbytes, ByteView, nobytes, IOError, isioerror, startwrite, closewrite, startread, closeread, tcpsocket, localport, safe_getpeername - """ - bytes(s::String) + bytes(x) -Get a `Vector{UInt8}`, a vector of bytes of a string. +If `x` is "castable" to an `AbstractVector{UInt8}`, then an +`AbstractVector{UInt8}` is returned; otherwise `x` is returned. """ function bytes end -bytes(s::SubArray{UInt8}) = unsafe_wrap(Array, pointer(s), length(s)) +bytes(s::AbstractVector{UInt8}) = s +bytes(s::AbstractString) = codeunits(s) +bytes(x) = x -const CodeUnits = Union{Vector{UInt8}, Base.CodeUnits} -bytes(s::Base.CodeUnits) = bytes(String(s)) -bytes(s::String) = codeunits(s) -bytes(s::SubString{String}) = codeunits(s) +"""whether `x` is "castable" to an `AbstractVector{UInt8}`; i.e. you can call `bytes(x)` if `isbytes(x)` === true""" +isbytes(x) = x isa AbstractVector{UInt8} || x isa AbstractString -bytes(s::Vector{UInt8}) = s +""" + nbytes(x) -> Int -isbytes(x) = x isa AbstractVector{UInt8} +Length in bytes of `x` if `x` is `isbytes(x)`. +""" +function nbytes end +nbytes(x) = nothing +nbytes(x::AbstractVector{UInt8}) = length(x) +nbytes(x::AbstractString) = sizeof(x) +nbytes(x::Vector{T}) where T <: AbstractString = sum(sizeof, x) +nbytes(x::Vector{T}) where T <: AbstractVector{UInt8} = sum(length, x) +nbytes(x::IOBuffer) = bytesavailable(x) +nbytes(x::Vector{IOBuffer}) = sum(bytesavailable, x) """ isioerror(exception) diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index 6c295ec7f..a7d0a4460 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -6,7 +6,6 @@ using ..Base64 using ..IOExtras using URIs using ..Messages -import ..Messages: bodylength import ..Headers import ..Form, ..content_type @@ -35,8 +34,8 @@ function messagelayer(handler) if !hasheader(headers, "Content-Length") && !hasheader(headers, "Transfer-Encoding") && !hasheader(headers, "Upgrade") - l = bodylength(body) - if l != unknown_length + l = nbytes(body) + if l !== nothing setheader(headers, "Content-Length" => string(l)) elseif method == "GET" && iofunction isa Function setheader(headers, "Content-Length" => "0") @@ -47,7 +46,7 @@ function messagelayer(handler) setheader(headers, content_type(body)) end parent = get(ctx, :parentrequest, nothing) - req = Request(method, resource(url), headers, bodybytes(body); url=url, version=http_version, responsebody=response_stream, parent=parent) + req = Request(method, resource(url), headers, body; url=url, version=http_version, responsebody=response_stream, parent=parent) return handler(ctx, req; iofunction=iofunction, kw...) end @@ -67,20 +66,4 @@ function setuseragent!(x::Union{String, Nothing}) return end -bodylength(body) = unknown_length -bodylength(body::AbstractVector{UInt8}) = length(body) -bodylength(body::AbstractString) = sizeof(body) -bodylength(body::Form) = length(body) -bodylength(body::Vector{T}) where T <: AbstractString = sum(sizeof, body) -bodylength(body::Vector{T}) where T <: AbstractArray{UInt8,1} = sum(length, body) -bodylength(body::IOBuffer) = bytesavailable(body) -bodylength(body::Vector{IOBuffer}) = sum(bytesavailable, body) - -bodybytes(body) = body -bodybytes(body::Vector{UInt8}) = body -bodybytes(body::IOBuffer) = read(body) -bodybytes(body::AbstractVector{UInt8}) = Vector{UInt8}(body) -bodybytes(body::AbstractString) = bytes(body) -bodybytes(body::Vector) = length(body) == 1 ? bodybytes(body[1]) : body - end # module MessageRequest diff --git a/src/Messages.jl b/src/Messages.jl index 473c5d3db..7114a1d6e 100644 --- a/src/Messages.jl +++ b/src/Messages.jl @@ -115,8 +115,8 @@ mutable struct Response{T} <: Message @doc """ Response(status::Int, headers=[]; body=UInt8[], request=nothing) -> HTTP.Response """ - function Response(status::Integer, headers=[]; body=nobody, request=nothing) - b = body isa IO ? body : bytes(body) + function Response(status::Integer, headers=[]; body=nothing, request=nothing) + b = isbytes(body) ? bytes(body) : something(body, nobody) return new{typeof(b)}( v"1.1", status, @@ -141,9 +141,9 @@ HTTP.Response(200, headers; body = "Hello") Response() = Request().response Response(s::Int, body::AbstractVector{UInt8}) = Response(s; body=body) -Response(s::Int, body::AbstractString) = Response(s, bytes(body)) +Response(s::Int, body::AbstractString) = Response(s; body=bytes(body)) -Response(body) = Response(200, body) +Response(body) = Response(200; body=body) Base.convert(::Type{Response}, s::AbstractString) = Response(s) @@ -231,13 +231,13 @@ For daily use, see [`HTTP.request`](@ref). """ function Request(method::String, target, headers=[], body=nobody; version=v"1.1", url::URI=URI(), responsebody=nothing, parent=nothing) - b = body isa IO ? body : bytes(something(body, nobody)) + b = isbytes(body) ? bytes(body) : something(body, nobody) r = Request{typeof(b)}(method, target == "" ? "/" : target, version, mkheaders(headers), b, - Response(0; body=something(responsebody, nobody)), + Response(0; body=responsebody), url, parent) r.response.request = r @@ -591,7 +591,7 @@ body_show_max = 1000 The first chunk of the Message Body (for display purposes). """ -bodysummary(bytes) = view(bytes, 1:min(length(bytes), body_show_max)) +bodysummary(body) = isbytes(body) ? view(bytes(body), 1:min(nbytes(body), body_show_max)) : "[Message Body was streamed]" function compactstartline(m::Message) b = IOBuffer() @@ -617,8 +617,8 @@ function Base.show(io::IO, m::Message) summary = bodysummary(m.body) validsummary = isvalidstr(summary) validsummary && write(io, summary) - if !validsummary || length(m.body) > length(summary) - println(io, "\n⋮\n$(length(m.body))-byte body") + if !validsummary || something(nbytes(m.body), 0) > length(summary) + println(io, "\n⋮\n$(nbytes(m.body))-byte body") end print(io, "\"\"\"") return diff --git a/src/RedirectRequest.jl b/src/RedirectRequest.jl index 376d48b43..c337575de 100644 --- a/src/RedirectRequest.jl +++ b/src/RedirectRequest.jl @@ -15,7 +15,6 @@ Redirects the request in the case of 3xx response status. """ function redirectlayer(handler) return function(ctx, method, url, headers, body; redirect::Bool=true, redirect_limit::Int=3, forwardheaders::Bool=true, kw...) - println("redirectlayer") if !redirect || redirect_limit == 0 # no redirecting return handler(ctx, method, url, headers, body; kw...) @@ -54,8 +53,7 @@ function redirectlayer(handler) else headers = Header[] end - @show 1 "➡️ Redirect: $url" - @show headers + @debug 1 "➡️ Redirect: $url" count += 1 end @assert false "Unreachable!" diff --git a/src/RetryRequest.jl b/src/RetryRequest.jl index a39557811..1764864f9 100644 --- a/src/RetryRequest.jl +++ b/src/RetryRequest.jl @@ -54,8 +54,8 @@ isrecoverable(e::HTTP.StatusError) = e.status == 403 || # Forbidden isrecoverable(e, req, retry_non_idempotent, retrycount) = isrecoverable(e) && - !(req.body === body_was_streamed) && - !(req.response.body === body_was_streamed) && + isbytes(req.body) && + isbytes(req.response.body) && (retry_non_idempotent || retrycount == 0 || isidempotent(req)) # "MUST NOT automatically retry a request with a non-idempotent method" # https://tools.ietf.org/html/rfc7230#section-6.3.1 @@ -67,8 +67,8 @@ function no_retry_reason(ex, req) print(buf, ", ", ex isa HTTP.StatusError ? "HTTP $(ex.status): " : !isrecoverable(ex) ? "$ex not recoverable, " : "", - (req.body === body_was_streamed) ? "request streamed, " : "", - (req.response.body === body_was_streamed) ? "response streamed, " : "", + !isbytes(req.body) ? "request streamed, " : "", + !isbytes(req.response.body) ? "response streamed, " : "", !isidempotent(req) ? "$(req.method) non-idempotent" : "") return String(take!(buf)) end diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index 8cfec3e3f..ba54bc503 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -23,6 +23,7 @@ indicates that the server does not wish to receive the message body. function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...)::Response response = stream.message req = response.request + io = stream.stream verbose == 1 && printlncompact(req) @debug 2 "client startwrite" startwrite(stream) @@ -53,7 +54,6 @@ function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...): else iofunction(stream) end - if isaborted(stream) # The server may have closed the connection. # Don't propagate such errors. diff --git a/src/multipart.jl b/src/multipart.jl index a6071d65b..fe8671a47 100644 --- a/src/multipart.jl +++ b/src/multipart.jl @@ -10,6 +10,7 @@ Base.eof(f::Form) = f.index > length(f.data) Base.isopen(f::Form) = false Base.close(f::Form) = nothing Base.length(f::Form) = sum(x->isa(x, IOStream) ? filesize(x) - position(x) : bytesavailable(x), f.data) +IOExtras.nbytes(x::Form) = length(x) function Base.position(f::Form) index = f.index foreach(mark, f.data) diff --git a/src/sniff.jl b/src/sniff.jl index c0cc9abd8..62daea4a2 100644 --- a/src/sniff.jl +++ b/src/sniff.jl @@ -1,3 +1,5 @@ +const CodeUnits = Union{Vector{UInt8}, Base.CodeUnits} + # compression detection const ZIP = UInt8[0x50, 0x4b, 0x03, 0x04] const GZIP = UInt8[0x1f, 0x8b, 0x08] diff --git a/test/client.jl b/test/client.jl index 177ad5dc6..b8fab8780 100644 --- a/test/client.jl +++ b/test/client.jl @@ -54,7 +54,9 @@ end @test replace(replace(body, " "=>""), "\n"=>"") == "{\"cookies\":{\"foo\":\"bar\",\"hey\":\"sailor\"}}" r = HTTP.get("$sch://httpbin.org/cookies/delete?hey") - @test isempty(JSON.parse(String(r.body))["cookies"]) + cookies = JSON.parse(String(r.body))["cookies"] + @test length(cookies) == 1 + @test cookies["foo"] == "bar" end @testset "Client Streaming Test" begin diff --git a/test/cookies.jl b/test/cookies.jl index c8215d936..70ab80839 100644 --- a/test/cookies.jl +++ b/test/cookies.jl @@ -1,3 +1,8 @@ +module TestCookies + +using HTTP +using Sockets, Test + @testset "Cookies" begin c = HTTP.Cookies.Cookie() @test c.name == "" @@ -173,3 +178,5 @@ @test istaskdone(tsk) end end + +end # module \ No newline at end of file diff --git a/test/loopback.jl b/test/loopback.jl index f4a1e5412..8aa7bdad7 100644 --- a/test/loopback.jl +++ b/test/loopback.jl @@ -1,10 +1,11 @@ +module TestLoopback + using Test using HTTP using HTTP.IOExtras using HTTP.Parsers using HTTP.Messages using HTTP.Sockets -using HTTP.MessageRequest: bodylength mutable struct FunctionIO <: IO f::Function @@ -289,21 +290,20 @@ end FunctionIO(()->(sleep(0.1); " World!"))]) @test String(r.body) == "Hello World!" - hello_sent = false - world_sent = false + hello_sent = Ref(false) + world_sent = Ref(false) @test_throws HTTP.StatusError begin r = lbreq("abort", [], [ - FunctionIO(()->(hello_sent = true; sleep(0.1); "Hello")), - FunctionIO(()->(world_sent = true; " World!"))]) + FunctionIO(()->(println("hello_sent"); hello_sent[] = true; sleep(0.5); "Hello")), + FunctionIO(()->(println("world_sent"); world_sent[] = true; " World!"))]) end - @test hello_sent - @test !world_sent + @test hello_sent[] + @test !world_sent[] end @testset "ASync - Pipeline limit = 0" begin server_events = [] t = async_test(;pipeline_limit=0) - @show t if haskey(ENV, "HTTP_JL_TEST_TIMING_SENSITIVE") @test server_events == [ "Request: GET /delay1 HTTP/1.1", @@ -322,7 +322,6 @@ end @testset "ASync - " begin server_events = [] t = async_test() - @show t if haskey(ENV, "HTTP_JL_TEST_TIMING_SENSITIVE") @test server_events == [ "Request: GET /delay1 HTTP/1.1", @@ -350,3 +349,5 @@ end "Response: HTTP/1.1 200 OK <= (POST /delay1 HTTP/1.1)"] end end + +end # module \ No newline at end of file diff --git a/test/messages.jl b/test/messages.jl index e6d0cfa0e..4a01478ca 100644 --- a/test/messages.jl +++ b/test/messages.jl @@ -1,17 +1,12 @@ -using ..Test +using Test using HTTP.Messages import HTTP.Messages.appendheader import HTTP.URI import HTTP.request -import HTTP: bytes - +import HTTP: bytes, nbytes using HTTP: StatusError -using HTTP.MessageRequest: bodylength -using HTTP.MessageRequest: bodybytes -using HTTP.MessageRequest: unknown_length - using JSON @testset "HTTP.Messages" begin @@ -23,29 +18,29 @@ using JSON http_writes = ["POST", "PUT", "DELETE", "PATCH"] @testset "Body Length" begin - @test bodylength(7) == unknown_length - @test bodylength(UInt8[1,2,3]) == 3 - @test bodylength(view(UInt8[1,2,3], 1:2)) == 2 - @test bodylength("Hello") == 5 - @test bodylength(SubString("World!",1,5)) == 5 - @test bodylength(["Hello", " ", "World!"]) == 12 - @test bodylength(["Hello", " ", SubString("World!",1,5)]) == 11 - @test bodylength([SubString("Hello", 1,5), " ", SubString("World!",1,5)]) == 11 - @test bodylength([UInt8[1,2,3], UInt8[4,5,6]]) == 6 - @test bodylength([UInt8[1,2,3], view(UInt8[4,5,6],1:2)]) == 5 - @test bodylength([view(UInt8[1,2,3],1:2), view(UInt8[4,5,6],1:2)]) == 4 - @test bodylength(IOBuffer("foo")) == 3 - @test bodylength([IOBuffer("foo"), IOBuffer("bar")]) == 6 + @test nbytes(7) === nothing + @test nbytes(UInt8[1,2,3]) == 3 + @test nbytes(view(UInt8[1,2,3], 1:2)) == 2 + @test nbytes("Hello") == 5 + @test nbytes(SubString("World!",1,5)) == 5 + @test nbytes(["Hello", " ", "World!"]) == 12 + @test nbytes(["Hello", " ", SubString("World!",1,5)]) == 11 + @test nbytes([SubString("Hello", 1,5), " ", SubString("World!",1,5)]) == 11 + @test nbytes([UInt8[1,2,3], UInt8[4,5,6]]) == 6 + @test nbytes([UInt8[1,2,3], view(UInt8[4,5,6],1:2)]) == 5 + @test nbytes([view(UInt8[1,2,3],1:2), view(UInt8[4,5,6],1:2)]) == 4 + @test nbytes(IOBuffer("foo")) == 3 + @test nbytes([IOBuffer("foo"), IOBuffer("bar")]) == 6 end @testset "Body Bytes" begin - @test bodybytes(7) == UInt8[] - @test bodybytes(UInt8[1,2,3]) == UInt8[1,2,3] - @test bodybytes(view(UInt8[1,2,3], 1:2)) == UInt8[1,2] - @test bodybytes("Hello") == bytes("Hello") - @test bodybytes(SubString("World!",1,5)) == bytes("World") - @test bodybytes(["Hello", " ", "World!"]) == UInt8[] - @test bodybytes([UInt8[1,2,3], UInt8[4,5,6]]) == UInt8[] + @test bytes(7) == 7 + @test bytes(UInt8[1,2,3]) == UInt8[1,2,3] + @test bytes(view(UInt8[1,2,3], 1:2)) == UInt8[1,2] + @test bytes("Hello") == codeunits("Hello") + @test bytes(SubString("World!",1,5)) == codeunits("World") + @test bytes(["Hello", " ", "World!"]) == ["Hello", " ", "World!"] + @test bytes([UInt8[1,2,3], UInt8[4,5,6]]) == [UInt8[1,2,3], UInt8[4,5,6]] end @testset "Request" begin @@ -176,15 +171,15 @@ using JSON end @testset "Display" begin - @test repr(Response(200, []; body="Hello world.")) == "Response:\n\"\"\"\nHTTP/1.1 200 OK\r\n\r\nHello world.\"\"\"" + @test repr(Response(200, []; body="Hello world.")) == "Response{Base.CodeUnits{UInt8, String}}:\n\"\"\"\nHTTP/1.1 200 OK\r\n\r\nHello world.\"\"\"" # truncation of long bodies for body_show_max in (Messages.body_show_max, 100) Messages.set_show_max(body_show_max) - @test repr(Response(200, []; body="Hello world.\n"*'x'^10000)) == "Response:\n\"\"\"\nHTTP/1.1 200 OK\r\n\r\nHello world.\n"*'x'^(body_show_max-13)*"\n⋮\n10013-byte body\n\"\"\"" + @test repr(Response(200, []; body="Hello world.\n"*'x'^10000)) == "Response{Base.CodeUnits{UInt8, String}}:\n\"\"\"\nHTTP/1.1 200 OK\r\n\r\nHello world.\n"*'x'^(body_show_max-13)*"\n⋮\n10013-byte body\n\"\"\"" end # don't display raw binary (non-Unicode) data: - @test repr(Response(200, []; body=String([0xde,0xad,0xc1,0x71,0x1c]))) == "Response:\n\"\"\"\nHTTP/1.1 200 OK\r\n\r\n\n⋮\n5-byte body\n\"\"\"" + @test repr(Response(200, []; body=String([0xde,0xad,0xc1,0x71,0x1c]))) == "Response{Base.CodeUnits{UInt8, String}}:\n\"\"\"\nHTTP/1.1 200 OK\r\n\r\n\n⋮\n5-byte body\n\"\"\"" end end diff --git a/test/server.jl b/test/server.jl index 6c49aec4a..5b75f79aa 100644 --- a/test/server.jl +++ b/test/server.jl @@ -66,7 +66,6 @@ end # large headers tcp = Sockets.connect(ip"127.0.0.1", port) x = "GET / HTTP/1.1\r\n$(repeat("Foo: Bar\r\n", 10000))\r\n"; - @show length(x) write(tcp, "GET / HTTP/1.1\r\n$(repeat("Foo: Bar\r\n", 10000))\r\n") sleep(0.1) try From a9473b44a72c0cdf152867a1b4e46e821576c39a Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 17 Mar 2022 23:23:54 -0600 Subject: [PATCH 11/19] more cleanup --- src/BasicAuthRequest.jl | 14 ++++---- src/CanonicalizeRequest.jl | 8 ++--- src/ConnectionRequest.jl | 9 +++-- src/ContentTypeRequest.jl | 17 +++++----- src/CookieRequest.jl | 21 ++++++------ src/DebugRequest.jl | 6 ++-- src/DefaultHeadersRequest.jl | 63 +++++++++++++++++++++++++++++++++++ src/ExceptionRequest.jl | 4 +-- src/HTTP.jl | 30 ++++++----------- src/MessageRequest.jl | 60 ++++----------------------------- src/Messages.jl | 11 ++++-- src/RedirectRequest.jl | 30 +++++++++-------- src/RetryRequest.jl | 10 +++--- src/StreamRequest.jl | 13 ++++---- src/TimeoutRequest.jl | 8 ++--- test/client.jl | 4 +-- test/loopback.jl | 37 +++++++++++++------- test/resources/TestRequest.jl | 17 +++------- test/server.jl | 2 +- 19 files changed, 192 insertions(+), 172 deletions(-) create mode 100644 src/DefaultHeadersRequest.jl diff --git a/src/BasicAuthRequest.jl b/src/BasicAuthRequest.jl index de85114cf..f8c72cb47 100644 --- a/src/BasicAuthRequest.jl +++ b/src/BasicAuthRequest.jl @@ -2,25 +2,25 @@ module BasicAuthRequest using ..Base64 using URIs -using ..Pairs: getkv, setkv +import ..Messages: setheader, hasheader import ..@debug, ..DEBUG_LEVEL export basicauthlayer """ - basicauthlayer(ctx, method, ::URI, headers, body) -> HTTP.Response + basicauthlayer(req) -> HTTP.Response Add `Authorization: Basic` header using credentials from url userinfo. """ function basicauthlayer(handler) - return function(ctx, method, url, headers, body; basicauth::Bool=true, kw...) + return function(req; basicauth::Bool=true, kw...) if basicauth - userinfo = unescapeuri(url.userinfo) - if !isempty(userinfo) && getkv(headers, "Authorization", "") == "" + userinfo = unescapeuri(req.url.userinfo) + if !isempty(userinfo) && !hasheader(req.headers, "Authorization") @debug 1 "Adding Authorization: Basic header." - setkv(headers, "Authorization", "Basic $(base64encode(userinfo))") + setheader(req.headers, "Authorization" => "Basic $(base64encode(userinfo))") end end - return handler(ctx, method, url, headers, body; kw...) + return handler(req; kw...) end end diff --git a/src/CanonicalizeRequest.jl b/src/CanonicalizeRequest.jl index 7d557233d..4985dd7e6 100644 --- a/src/CanonicalizeRequest.jl +++ b/src/CanonicalizeRequest.jl @@ -6,16 +6,16 @@ using ..Strings: tocameldash export canonicalizelayer """ - canonicalizelayer(ctx, method, ::URI, headers, body) -> HTTP.Response + canonicalizelayer(req) -> HTTP.Response Rewrite request and response headers in Canonical-Camel-Dash-Format. """ function canonicalizelayer(handler) - return function(ctx, method, url, headers, body; canonicalize_headers::Bool=false, kw...) + return function(req; canonicalize_headers::Bool=false, kw...) if canonicalize_headers - headers = canonicalizeheaders(headers) + req.headers = canonicalizeheaders(req.headers) end - res = handler(ctx, method, url, headers, body; kw...) + res = handler(req; kw...) if canonicalize_headers res.headers = canonicalizeheaders(res.headers) end diff --git a/src/ConnectionRequest.jl b/src/ConnectionRequest.jl index ea10dd1c7..0a290711e 100644 --- a/src/ConnectionRequest.jl +++ b/src/ConnectionRequest.jl @@ -5,7 +5,6 @@ using ..Messages using ..IOExtras using ..ConnectionPool using MbedTLS: SSLContext, SSLConfig -using ..Pairs: getkv, setkv using Base64: base64encode import ..@debug, ..DEBUG_LEVEL import ..Streams: Stream @@ -62,7 +61,7 @@ Otherwise leave it open so that it can be reused. See [`isioerror`](@ref). """ function connectionlayer(handler) - return function(ctx, req; proxy=getproxy(req.url.scheme, req.url.host), socket_type::Type=TCPSocket, kw...) + return function(req; proxy=getproxy(req.url.scheme, req.url.host), socket_type::Type=TCPSocket, kw...) if proxy !== nothing target_url = req.url url = URI(proxy) @@ -71,9 +70,9 @@ function connectionlayer(handler) end userinfo = unescapeuri(url.userinfo) - if !isempty(userinfo) && getkv(req.headers, "Proxy-Authorization", "") == "" + if !isempty(userinfo) && !hasheader(req.headers, "Proxy-Authorization") @debug 1 "Adding Proxy-Authorization: Basic header." - setkv(req.headers, "Proxy-Authorization", "Basic $(base64encode(userinfo))") + setheader(req.headers, "Proxy-Authorization" => "Basic $(base64encode(userinfo))") end else url = req.url @@ -101,7 +100,7 @@ function connectionlayer(handler) end stream = Stream(req.response, io) - resp = handler(ctx, stream; kw...) + resp = handler(stream; kw...) if proxy !== nothing && target_url.scheme == "https" close(io) diff --git a/src/ContentTypeRequest.jl b/src/ContentTypeRequest.jl index e5a6ac6d4..64495902c 100644 --- a/src/ContentTypeRequest.jl +++ b/src/ContentTypeRequest.jl @@ -1,7 +1,6 @@ module ContentTypeDetection using URIs -using ..Pairs: getkv, setkv import ..sniff import ..Form using ..Messages @@ -9,18 +8,18 @@ import ..IOExtras import ..@debug, ..DEBUG_LEVEL export contenttypedetectionlayer -# f(::Handler) -> Handler + function contenttypedetectionlayer(handler) - return function(ctx, method, url, headers, body; detect_content_type::Bool=false, kw...) - if detect_content_type && (getkv(headers, "Content-Type", "") == "" - && !isa(body, Form) - && isbytes(body)) + return function(req; detect_content_type::Bool=false, kw...) + if detect_content_type && (!hasheader(req.headers, "Content-Type") + && !isa(req.body, Form) + && isbytes(req.body)) - sn = sniff(bytes(body)) - setkv(headers, "Content-Type", sn) + sn = sniff(bytes(req.body)) + setheader(req.headers, "Content-Type" => sn) @debug 1 "setting Content-Type header to: $sn" end - return handler(ctx, method, url, headers, body; kw...) + return handler(req; kw...) end end diff --git a/src/CookieRequest.jl b/src/CookieRequest.jl index 11dc1ffb4..895653ac4 100644 --- a/src/CookieRequest.jl +++ b/src/CookieRequest.jl @@ -3,8 +3,7 @@ module CookieRequest import ..Dates using URIs using ..Cookies -using ..Messages: Request, ascii_lc_isequal -using ..Pairs: getkv, setkv +using ..Messages: Request, ascii_lc_isequal, header, setheader import ..@debug, ..DEBUG_LEVEL, ..access_threaded const default_cookiejar = Dict{String, Set{Cookie}}[] @@ -17,13 +16,13 @@ end export cookielayer """ - cookielayer(ctx, method, ::URI, headers, body) -> HTTP.Response + cookielayer(req) -> HTTP.Response Add locally stored Cookies to the request headers. Store new Cookies found in the response headers. """ function cookielayer(handler) - return function(ctx, req::Request; cookies=true, cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) + return function(req::Request; cookies=true, cookiejar::Dict{String, Set{Cookie}}=access_threaded(Dict{String, Set{Cookie}}, default_cookiejar), kw...) if cookies === true || (cookies isa AbstractDict && !isempty(cookies)) url = req.url hostcookies = get!(cookiejar, url.host, Set{Cookie}()) @@ -34,25 +33,25 @@ function cookielayer(handler) end end if !isempty(cookiestosend) - existingcookie = getkv(req.headers, "Cookie", "") - if existingcookie != "" && get(ctx, :includedCookies, nothing) !== nothing + existingcookie = header(req.headers, "Cookie") + if existingcookie != "" && get(req.context, :includedCookies, nothing) !== nothing # this is a redirect where we previously included cookies # we want to filter those out to avoid duplicate cookie sending # and the case where a cookie was set to expire from the 1st request previouscookies = Cookies.readcookies(req.headers, "") - previouslyincluded = ctx[:includedCookies] + previouslyincluded = req.context[:includedCookies] filtered = filter(x -> !(x.name in previouslyincluded), previouscookies) existingcookie = stringify("", filtered) end - setkv(req.headers, "Cookie", stringify(existingcookie, cookiestosend)) - ctx[:includedCookies] = map(x -> x.name, cookiestosend) + setheader(req.headers, "Cookie" => stringify(existingcookie, cookiestosend)) + req.context[:includedCookies] = map(x -> x.name, cookiestosend) end - res = handler(ctx, req; kw...) + res = handler(req; kw...) setcookies(hostcookies, url.host, res.headers) return res else # skip - return handler(ctx, req; kw...) + return handler(req; kw...) end end end diff --git a/src/DebugRequest.jl b/src/DebugRequest.jl index aab1a59bc..598332a27 100644 --- a/src/DebugRequest.jl +++ b/src/DebugRequest.jl @@ -9,17 +9,17 @@ include("IODebug.jl") export debuglayer """ - debuglayer(ctx, stream::Stream) -> HTTP.Response + debuglayer(stream::Stream) -> HTTP.Response Wrap the `IO` stream in an `IODebug` stream and print Message data. """ function debuglayer(handler) - return function(ctx, stream::Stream; verbose::Int=0, kw...) + return function(stream::Stream; verbose::Int=0, kw...) # if debugging, wrap stream.stream in IODebug if verbose >= 3 || DEBUG_LEVEL[] >= 3 stream = Stream(stream.message, IODebug(stream.stream)) end - return handler(ctx, stream; verbose=verbose, kw...) + return handler(stream; verbose=verbose, kw...) end end diff --git a/src/DefaultHeadersRequest.jl b/src/DefaultHeadersRequest.jl new file mode 100644 index 000000000..f29bfe4cd --- /dev/null +++ b/src/DefaultHeadersRequest.jl @@ -0,0 +1,63 @@ +module DefaultHeadersRequest + +export defaultheaderslayer, setuseragent! + +using ..Messages +using ..IOExtras +import ..Form, ..content_type + +""" + defaultheaderslayer(req) -> Response + +Sets default expected headers. +""" +function defaultheaderslayer(handler) + return function(req; iofunction=nothing, kw...) + headers = req.headers + if isempty(req.url.port) || + (req.url.scheme == "http" && req.url.port == "80") || + (req.url.scheme == "https" && req.url.port == "443") + hostheader = req.url.host + else + hostheader = req.url.host * ":" * req.url.port + end + setheader(headers, "Host" => hostheader) + defaultheader!(headers, "Accept" => "*/*") + if USER_AGENT[] !== nothing + defaultheader!(headers, "User-Agent" => USER_AGENT[]) + end + + if !hasheader(headers, "Content-Length") && + !hasheader(headers, "Transfer-Encoding") && + !hasheader(headers, "Upgrade") + l = nbytes(req.body) + @show nbytes(req.body) + if l !== nothing + setheader(headers, "Content-Length" => string(l)) + elseif req.method == "GET" && iofunction isa Function + setheader(headers, "Content-Length" => "0") + end + end + if !hasheader(headers, "Content-Type") && req.body isa Form && req.method in ("POST", "PUT") + # "Content-Type" => "multipart/form-data; boundary=..." + setheader(headers, content_type(req.body)) + end + return handler(req; iofunction=iofunction, kw...) + end +end + +const USER_AGENT = Ref{Union{String, Nothing}}("HTTP.jl/$VERSION") + +""" + setuseragent!(x::Union{String, Nothing}) + +Set the default User-Agent string to be used in each HTTP request. +Can be manually overridden by passing an explicit `User-Agent` header. +Setting `nothing` will prevent the default `User-Agent` header from being passed. +""" +function setuseragent!(x::Union{String, Nothing}) + USER_AGENT[] = x + return +end + +end # module \ No newline at end of file diff --git a/src/ExceptionRequest.jl b/src/ExceptionRequest.jl index 2662d5f07..9b92ed7b9 100644 --- a/src/ExceptionRequest.jl +++ b/src/ExceptionRequest.jl @@ -13,8 +13,8 @@ export exceptionlayer Throw a `StatusError` if the request returns an error response status. """ function exceptionlayer(handler) - return function(ctx, stream; status_exception::Bool=true, kw...) - res = handler(ctx, stream; kw...) + return function(stream; status_exception::Bool=true, kw...) + res = handler(stream; kw...) if status_exception && iserror(res) throw(StatusError(res.status, res.request.method, res.request.target, res)) else diff --git a/src/HTTP.jl b/src/HTTP.jl index 350551b82..25a03a156 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -310,35 +310,26 @@ function request(method, url, h=Header[], b=nobody; return request(HTTP.stack(), method, url, headers, body, query; kw...) end -const Context = Dict{Symbol, Any} - -struct Stack - stack -end - function stack( # custom layers - initiallayers=(), requestlayers=(), streamlayers=()) # stream layers slayers = (timeoutlayer, exceptionlayer, debuglayer, streamlayers...) - layers = foldl((x, y) -> y(x), slayers, init=streamlayer) - # transition to stream and request layers - rlayers = (cookielayer, retrylayer, requestlayers...) - layers2 = foldl((x, y) -> y(x), rlayers; init=connectionlayer(layers)) - # transition to request and initial layers - ilayers = (redirectlayer, basicauthlayer, contenttypedetectionlayer, canonicalizelayer, initiallayers...) - return Stack(foldl((x, y) -> y(x), ilayers; init=messagelayer(layers2))) + layers = foldr((x, y) -> x(y), slayers, init=streamlayer) + # request layers + # messagelayer must be the 1st/outermost layer to convert initial args to Request + rlayers = (messagelayer, redirectlayer, defaultheaderslayer, basicauthlayer, contenttypedetectionlayer, cookielayer, retrylayer, canonicalizelayer, requestlayers...) + return foldr((x, y) -> x(y), rlayers; init=connectionlayer(layers)) end -function request(stack::Stack, method, url, h=Header[], b=nobody, q=nothing; +function request(stack::Base.Callable, method, url, h=Header[], b=nobody, q=nothing; headers=h, body=b, query=q, kw...)::Response - return stack.stack(Context(), string(method), request_uri(url, query), mkheaders(headers), body; kw...) + return stack(string(method), request_uri(url, query), mkheaders(headers), body; kw...) end -macro client(initiallayers, requestlayers, streamlayers) +macro client(requestlayers, streamlayers=[]) esc(quote get(a...; kw...) = request("GET", a...; kw...) put(a...; kw...) = request("PUT", a...; kw...) @@ -347,7 +338,7 @@ macro client(initiallayers, requestlayers, streamlayers) head(u; kw...) = request("HEAD", u; kw...) delete(a...; kw...) = request("DELETE", a...; kw...) request(method, url, h=HTTP.Header[], b=HTTP.nobody; headers=h, body=b, query=nothing, kw...)::HTTP.Response = - HTTP.request(HTTP.stack($initiallayers, $requestlayers, $streamlayers), method, url, headers, body, query; kw...) + HTTP.request(HTTP.stack($requestlayers, $streamlayers), method, url, headers, body, query; kw...) end) end @@ -456,12 +447,13 @@ function openraw(method::Union{String,Symbol}, url, headers=Header[]; kw...)::Tu take!(socketready) end +include("MessageRequest.jl"); using .MessageRequest include("RedirectRequest.jl"); using .RedirectRequest +include("DefaultHeadersRequest.jl"); using .DefaultHeadersRequest include("BasicAuthRequest.jl"); using .BasicAuthRequest include("CookieRequest.jl"); using .CookieRequest include("CanonicalizeRequest.jl"); using .CanonicalizeRequest include("TimeoutRequest.jl"); using .TimeoutRequest -include("MessageRequest.jl"); using .MessageRequest include("ExceptionRequest.jl"); using .ExceptionRequest import .ExceptionRequest.StatusError include("RetryRequest.jl"); using .RetryRequest diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index a7d0a4460..1ca335e11 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -1,69 +1,23 @@ module MessageRequest -export setuseragent!, resource +export setuseragent! -using ..Base64 -using ..IOExtras using URIs -using ..Messages import ..Headers -import ..Form, ..content_type +using ..Messages export messagelayer """ - messagelayer(ctx, method, ::URI, headers, body) -> HTTP.Response + messagelayer(method, ::URI, headers, body) -> HTTP.Response -Construct a [`Request`](@ref) object and set mandatory headers. +Construct a [`Request`](@ref) object. """ function messagelayer(handler) - return function(ctx, method::String, url::URI, headers::Headers, body; iofunction=nothing, response_stream=nothing, http_version=v"1.1", kw...) - if isempty(url.port) || - (url.scheme == "http" && url.port == "80") || - (url.scheme == "https" && url.port == "443") - hostheader = url.host - else - hostheader = url.host * ":" * url.port - end - defaultheader!(headers, "Host" => hostheader) - defaultheader!(headers, "Accept" => "*/*") - if USER_AGENT[] !== nothing - defaultheader!(headers, "User-Agent" => USER_AGENT[]) - end - - if !hasheader(headers, "Content-Length") && - !hasheader(headers, "Transfer-Encoding") && - !hasheader(headers, "Upgrade") - l = nbytes(body) - if l !== nothing - setheader(headers, "Content-Length" => string(l)) - elseif method == "GET" && iofunction isa Function - setheader(headers, "Content-Length" => "0") - end - end - if !hasheader(headers, "Content-Type") && body isa Form && method in ("POST", "PUT") - # "Content-Type" => "multipart/form-data; boundary=..." - setheader(headers, content_type(body)) - end - parent = get(ctx, :parentrequest, nothing) - req = Request(method, resource(url), headers, body; url=url, version=http_version, responsebody=response_stream, parent=parent) - - return handler(ctx, req; iofunction=iofunction, kw...) + return function(method::String, url::URI, headers::Headers, body; response_stream=nothing, http_version=v"1.1", kw...) + req = Request(method, resource(url), headers, body; url=url, version=http_version, responsebody=response_stream) + return handler(req; response_stream=response_stream, kw...) end end -const USER_AGENT = Ref{Union{String, Nothing}}("HTTP.jl/$VERSION") - -""" - setuseragent!(x::Union{String, Nothing}) - -Set the default User-Agent string to be used in each HTTP request. -Can be manually overridden by passing an explicit `User-Agent` header. -Setting `nothing` will prevent the default `User-Agent` header from being passed. -""" -function setuseragent!(x::Union{String, Nothing}) - USER_AGENT[] = x - return -end - end # module MessageRequest diff --git a/src/Messages.jl b/src/Messages.jl index 7114a1d6e..7bdc510a6 100644 --- a/src/Messages.jl +++ b/src/Messages.jl @@ -180,6 +180,7 @@ Get body from a response. body(r::Response) = getfield(r, :body) # HTTP Request +const Context = Dict{Symbol, Any} """ Request <: Message @@ -207,6 +208,8 @@ Represents a HTTP Request Message. (e.g. in the case of a redirect). [RFC7230 6.4](https://tools.ietf.org/html/rfc7231#section-6.4) +- `context`, a `Dict{Symbol, Any}` store used by middleware to share state + You can get each data with [`HTTP.method`](@ref), [`HTTP.headers`](@ref), [`HTTP.uri`](@ref), and [`HTTP.body`](@ref). """ @@ -218,7 +221,8 @@ mutable struct Request{T} <: Message body::T # Vector{UInt8} or some kind of IO response::Response url::URI - parent + parent::Union{Response, Nothing} + context::Context end Request() = Request("", "") @@ -230,7 +234,7 @@ Constructor for `HTTP.Request`. For daily use, see [`HTTP.request`](@ref). """ function Request(method::String, target, headers=[], body=nobody; - version=v"1.1", url::URI=URI(), responsebody=nothing, parent=nothing) + version=v"1.1", url::URI=URI(), responsebody=nothing, parent=nothing, context=Context()) b = isbytes(body) ? bytes(body) : something(body, nobody) r = Request{typeof(b)}(method, target == "" ? "/" : target, @@ -239,7 +243,8 @@ function Request(method::String, target, headers=[], body=nobody; b, Response(0; body=responsebody), url, - parent) + parent, + context) r.response.request = r return r end diff --git a/src/RedirectRequest.jl b/src/RedirectRequest.jl index c337575de..50fb755cf 100644 --- a/src/RedirectRequest.jl +++ b/src/RedirectRequest.jl @@ -6,29 +6,26 @@ using ..Pairs: setkv import ..Header import ..@debug, ..DEBUG_LEVEL -export redirectlayer +export redirectlayer, nredirects """ - redirectlayer(ctx, method, ::URI, headers, body) -> HTTP.Response + redirectlayer(req) -> HTTP.Response Redirects the request in the case of 3xx response status. """ function redirectlayer(handler) - return function(ctx, method, url, headers, body; redirect::Bool=true, redirect_limit::Int=3, forwardheaders::Bool=true, kw...) + return function(req; redirect::Bool=true, redirect_limit::Int=3, forwardheaders::Bool=true, response_stream=nothing, kw...) if !redirect || redirect_limit == 0 # no redirecting - return handler(ctx, method, url, headers, body; kw...) + return handler(req; redirect_limit=redirect_limit, kw...) end count = 0 while true # Verify the url before making the request. Verification is done in # the redirect loop to also catch bad redirect URLs. - verify_url(url) - if count == redirect_limit - ctx[:redirectlimitreached] = true - end - res = handler(ctx, method, url, headers, body; kw...) + verify_url(req.url) + res = handler(req; redirect_limit=redirect_limit, kw...) if (count == redirect_limit || !isredirect(res) || (location = header(res, "Location")) == "") @@ -36,11 +33,12 @@ function redirectlayer(handler) end # follow redirect - ctx[:parentrequest] = res - oldurl = url - url = resolvereference(oldurl, location) + oldurl = req.url + url = resolvereference(req.url, location) + req = Request(req.method, resource(url), copy(req.headers), req.body; + url=url, version=req.version, responsebody=response_stream, parent=res, context=req.context) if forwardheaders - headers = filter(headers) do (header, _) + req.headers = filter(req.headers) do (header, _) # false return values are filtered out if header == "Host" return false @@ -51,7 +49,7 @@ function redirectlayer(handler) end end else - headers = Header[] + req.headers = Header[] end @debug 1 "➡️ Redirect: $url" count += 1 @@ -60,6 +58,10 @@ function redirectlayer(handler) end end +function nredirects(req) + return req.parent === nothing ? 0 : (1 + nredirects(req.parent.request)) +end + const SENSITIVE_HEADERS = Set([ "Authorization", "Www-Authenticate", diff --git a/src/RetryRequest.jl b/src/RetryRequest.jl index 1764864f9..1c495e635 100644 --- a/src/RetryRequest.jl +++ b/src/RetryRequest.jl @@ -10,7 +10,7 @@ import ..@debug, ..DEBUG_LEVEL, ..sprintcompact export retrylayer """ - retrylayer(ctx, req) -> HTTP.Response + retrylayer(req) -> HTTP.Response Retry the request if it throws a recoverable exception. @@ -23,15 +23,15 @@ e.g. `HTTP.IOError`, `Sockets.DNSError`, `Base.EOFError` and `HTTP.StatusError` (if status is ``5xx`). """ function retrylayer(handler) - return function(ctx, req::Request; retry::Bool=true, retries::Int=4, retry_non_idempotent::Bool=false, kw...) + return function(req::Request; retry::Bool=true, retries::Int=4, retry_non_idempotent::Bool=false, kw...) if !retry || retries == 0 # no retry - return handler(ctx, req; kw...) + return handler(req; kw...) end retry_request = Base.retry(handler, delays=ExponentialBackOff(n = retries), check=(s, ex)->begin - retry = isrecoverable(ex, req, retry_non_idempotent, get(ctx, :retrycount, 0)) + retry = isrecoverable(ex, req, retry_non_idempotent, get(req.context, :retrycount, 0)) if retry @debug 1 "🔄 Retry $ex: $(sprintcompact(req))" reset!(req.response) @@ -41,7 +41,7 @@ function retrylayer(handler) return s, retry end) - return retry_request(ctx, req; kw...) + return retry_request(req; kw...) end end diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index ba54bc503..35a5718fa 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -5,6 +5,7 @@ using ..Messages using ..Streams import ..ConnectionPool using ..MessageRequest +import ..RedirectRequest: nredirects import ..@debug, ..DEBUG_LEVEL, ..printlncompact, ..sprintcompact export streamlayer @@ -20,7 +21,8 @@ immediately so that the transmission can be aborted if the `Response` status indicates that the server does not wish to receive the message body. [RFC7230 6.5](https://tools.ietf.org/html/rfc7230#section-6.5). """ -function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...)::Response +function streamlayer(stream::Stream; iofunction=nothing, verbose=0, redirect_limit::Int=3, kw...)::Response + @show iofunction response = stream.message req = response.request io = stream.stream @@ -29,7 +31,6 @@ function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...): startwrite(stream) if verbose == 2 - println("printing req") println(req) if iofunction === nothing && req.body isa IO println("$(typeof(req)).body: $(sprintcompact(req.body))") @@ -41,7 +42,7 @@ function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...): @sync begin if iofunction === nothing @async try - writebody(stream, ctx, req) + writebody(stream, req) @debug 2 "client closewrite" closewrite(stream) catch e @@ -50,7 +51,7 @@ function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...): end @debug 2 "client startread" startread(stream) - readbody(stream, response, get(ctx, :redirectlimitreached, false)) + readbody(stream, response, redirect_limit == nredirects(req)) else iofunction(stream) end @@ -79,7 +80,7 @@ function streamlayer(ctx, stream::Stream; iofunction=nothing, verbose=0, kw...): return response end -function writebody(stream::Stream, ctx, req::Request) +function writebody(stream::Stream, req::Request) if !isbytes(req.body) writebodystream(stream, req.body) @@ -87,7 +88,7 @@ function writebody(stream::Stream, ctx, req::Request) else write(stream, req.body) end - ctx[:retrycount] = get(ctx, :retrycount, 0) + 1 + req.context[:retrycount] = get(req.context, :retrycount, 0) + 1 return end diff --git a/src/TimeoutRequest.jl b/src/TimeoutRequest.jl index 54e13f63e..1dbb3226a 100644 --- a/src/TimeoutRequest.jl +++ b/src/TimeoutRequest.jl @@ -15,15 +15,15 @@ end export timeoutlayer """ - timeoutlayer(ctx, stream) -> HTTP.Response + timeoutlayer(stream) -> HTTP.Response Close `IO` if no data has been received for `timeout` seconds. """ function timeoutlayer(handler) - return function(ctx, stream::Stream; readtimeout::Int=0, kw...) + return function(stream::Stream; readtimeout::Int=0, kw...) if readtimeout <= 0 # skip - return handler(ctx, stream; kw...) + return handler(stream; kw...) end io = stream.stream wait_for_timeout = Ref{Bool}(true) @@ -40,7 +40,7 @@ function timeoutlayer(handler) end try - return handler(ctx, stream; kw...) + return handler(stream; kw...) catch e if timedout[] throw(ReadTimeoutError(readtimeout)) diff --git a/test/client.jl b/test/client.jl index b8fab8780..0a040aa09 100644 --- a/test/client.jl +++ b/test/client.jl @@ -53,7 +53,7 @@ end body = String(r.body) @test replace(replace(body, " "=>""), "\n"=>"") == "{\"cookies\":{\"foo\":\"bar\",\"hey\":\"sailor\"}}" - r = HTTP.get("$sch://httpbin.org/cookies/delete?hey") + r = HTTP.get("$sch://httpbin.org/cookies/delete?hey"; verbose=2) cookies = JSON.parse(String(r.body))["cookies"] @test length(cookies) == 1 @test cookies["foo"] == "bar" @@ -350,7 +350,7 @@ end HTTP.startwrite(http) HTTP.write(http, sprint(JSON.print, data)) end - old_user_agent = HTTP.MessageRequest.USER_AGENT[] + old_user_agent = HTTP.DefaultHeadersRequest.USER_AGENT[] default_user_agent = "HTTP.jl/$VERSION" # Default values HTTP.setuseragent!(default_user_agent) diff --git a/test/loopback.jl b/test/loopback.jl index 8aa7bdad7..cf16ec3aa 100644 --- a/test/loopback.jl +++ b/test/loopback.jl @@ -94,6 +94,7 @@ function on_body(f::Function, lb::Loopback) req = nothing try + @show s req = parse(HTTP.Request, s) catch e if !(e isa EOFError || e isa HTTP.ParseError) @@ -137,14 +138,20 @@ function Base.unsafe_write(lb::Loopback, p::Ptr{UInt8}, n::UInt) l = length(req.body) response = HTTP.Response(200, ["Content-Length" => l], body = req.body; request=req) + @show req + @show response if req.target == "/echo" push!(server_events, "Response: $(HTTP.sprintcompact(response))") write(lb.io, response) elseif (m = match(r"^/delay([0-9]*)$", req.target)) !== nothing t = parse(Int, first(m.captures)) + println("sleeping") sleep(t/10) + println("done sleeping") push!(server_events, "Response: $(HTTP.sprintcompact(response))") + println("writing response") write(lb.io, response) + println("done writing response") else response = HTTP.Response(403, ["Connection" => "close", @@ -157,6 +164,13 @@ function Base.unsafe_write(lb::Loopback, p::Ptr{UInt8}, n::UInt) return n end +function HTTP.ConnectionPool.getconnection(::Type{Loopback}, + host::AbstractString, + port::AbstractString; + kw...)::Loopback + return Loopback() +end + function async_test(m=["GET","GET","GET","GET","GET"];kw...) r1 = r2 = r3 = r4 = r5 = nothing t1 = time() @@ -188,13 +202,6 @@ function async_test(m=["GET","GET","GET","GET","GET"];kw...) return t2 - t1 end -function HTTP.ConnectionPool.getconnection(::Type{Loopback}, - host::AbstractString, - port::AbstractString; - kw...)::Loopback - return Loopback() -end - @testset "loopback" begin global server_events @@ -233,21 +240,27 @@ end end @testset "lbopen - Body - Delay" begin - body = nothing - body_sent = false + body = Ref{Any}(nothing) + body_sent = Ref(false) r = lbopen("delay10", []) do http @sync begin @async begin + println("writing") write(http, "Hello World!") + println("done writing") closewrite(http) - body_sent = true + println("setting body_sent") + body_sent[] = true end + println("startread") startread(http) - body = read(http) + println("calling read") + body[] = read(http) + println("done reading") closeread(http) end end - @test String(body) == "Hello World!" + @test String(body[]) == "Hello World!" end # "If [the response] indicates the server does not wish to receive the diff --git a/test/resources/TestRequest.jl b/test/resources/TestRequest.jl index 928d8a453..beda0392f 100644 --- a/test/resources/TestRequest.jl +++ b/test/resources/TestRequest.jl @@ -2,27 +2,20 @@ module TestRequest using HTTP -function testinitiallayer(handler) - return function(ctx, m, url, h, b; httptestlayer=Ref(false), kw...) - httptestlayer[] = true - return handler(ctx, m, url, h, b; kw...) - end -end - function testrequestlayer(handler) - return function(ctx, req; httptestlayer=Ref(false), kw...) + return function(req; httptestlayer=Ref(false), kw...) httptestlayer[] = true - return handler(ctx, req; kw...) + return handler(req; kw...) end end function teststreamlayer(handler) - return function(ctx, stream; httptestlayer=Ref(false), kw...) + return function(stream; httptestlayer=Ref(false), kw...) httptestlayer[] = true - return handler(ctx, stream; kw...) + return handler(stream; kw...) end end -HTTP.@client (testinitiallayer,) (testrequestlayer,) (teststreamlayer,) +HTTP.@client (testrequestlayer,) (teststreamlayer,) end diff --git a/test/server.jl b/test/server.jl index 5b75f79aa..091d0b07a 100644 --- a/test/server.jl +++ b/test/server.jl @@ -348,7 +348,7 @@ end # @testset logs = with_testserver(combined_logfmt) do HTTP.get("http://localhost:32612", ["Referer" => "julialang.org"]) HTTP.get("http://localhost:32612/index.html") - useragent = HTTP.MessageRequest.USER_AGENT[] + useragent = HTTP.DefaultHeadersRequest.USER_AGENT[] HTTP.setuseragent!(nothing) HTTP.get("http://localhost:32612/index.html?a=b") HTTP.setuseragent!(useragent) From acc07279129eb83975550f656640ca821aa26899 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Fri, 18 Mar 2022 00:09:16 -0600 Subject: [PATCH 12/19] fix failing tests --- src/DefaultHeadersRequest.jl | 3 +-- src/Messages.jl | 6 +++--- src/StreamRequest.jl | 1 - test/client.jl | 2 +- test/loopback.jl | 17 ++--------------- 5 files changed, 7 insertions(+), 22 deletions(-) diff --git a/src/DefaultHeadersRequest.jl b/src/DefaultHeadersRequest.jl index f29bfe4cd..c0b006db6 100644 --- a/src/DefaultHeadersRequest.jl +++ b/src/DefaultHeadersRequest.jl @@ -21,7 +21,7 @@ function defaultheaderslayer(handler) else hostheader = req.url.host * ":" * req.url.port end - setheader(headers, "Host" => hostheader) + defaultheader!(headers, "Host" => hostheader) defaultheader!(headers, "Accept" => "*/*") if USER_AGENT[] !== nothing defaultheader!(headers, "User-Agent" => USER_AGENT[]) @@ -31,7 +31,6 @@ function defaultheaderslayer(handler) !hasheader(headers, "Transfer-Encoding") && !hasheader(headers, "Upgrade") l = nbytes(req.body) - @show nbytes(req.body) if l !== nothing setheader(headers, "Content-Length" => string(l)) elseif req.method == "GET" && iofunction isa Function diff --git a/src/Messages.jl b/src/Messages.jl index 7bdc510a6..b67b17671 100644 --- a/src/Messages.jl +++ b/src/Messages.jl @@ -115,7 +115,7 @@ mutable struct Response{T} <: Message @doc """ Response(status::Int, headers=[]; body=UInt8[], request=nothing) -> HTTP.Response """ - function Response(status::Integer, headers=[]; body=nothing, request=nothing) + function Response(status::Integer, headers=[]; body=nobody, request=nothing) b = isbytes(body) ? bytes(body) : something(body, nobody) return new{typeof(b)}( v"1.1", @@ -235,8 +235,8 @@ For daily use, see [`HTTP.request`](@ref). """ function Request(method::String, target, headers=[], body=nobody; version=v"1.1", url::URI=URI(), responsebody=nothing, parent=nothing, context=Context()) - b = isbytes(body) ? bytes(body) : something(body, nobody) - r = Request{typeof(b)}(method, + b = isbytes(body) ? bytes(body) : body + r = Request{b === nothing ? Any : typeof(b)}(method, target == "" ? "/" : target, version, mkheaders(headers), diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index 35a5718fa..a150b6c7f 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -22,7 +22,6 @@ indicates that the server does not wish to receive the message body. [RFC7230 6.5](https://tools.ietf.org/html/rfc7230#section-6.5). """ function streamlayer(stream::Stream; iofunction=nothing, verbose=0, redirect_limit::Int=3, kw...)::Response - @show iofunction response = stream.message req = response.request io = stream.stream diff --git a/test/client.jl b/test/client.jl index 0a040aa09..c77d889e9 100644 --- a/test/client.jl +++ b/test/client.jl @@ -53,7 +53,7 @@ end body = String(r.body) @test replace(replace(body, " "=>""), "\n"=>"") == "{\"cookies\":{\"foo\":\"bar\",\"hey\":\"sailor\"}}" - r = HTTP.get("$sch://httpbin.org/cookies/delete?hey"; verbose=2) + r = HTTP.get("$sch://httpbin.org/cookies/delete?hey") cookies = JSON.parse(String(r.body))["cookies"] @test length(cookies) == 1 @test cookies["foo"] == "bar" diff --git a/test/loopback.jl b/test/loopback.jl index cf16ec3aa..79842f27e 100644 --- a/test/loopback.jl +++ b/test/loopback.jl @@ -94,7 +94,6 @@ function on_body(f::Function, lb::Loopback) req = nothing try - @show s req = parse(HTTP.Request, s) catch e if !(e isa EOFError || e isa HTTP.ParseError) @@ -138,20 +137,14 @@ function Base.unsafe_write(lb::Loopback, p::Ptr{UInt8}, n::UInt) l = length(req.body) response = HTTP.Response(200, ["Content-Length" => l], body = req.body; request=req) - @show req - @show response if req.target == "/echo" push!(server_events, "Response: $(HTTP.sprintcompact(response))") write(lb.io, response) elseif (m = match(r"^/delay([0-9]*)$", req.target)) !== nothing t = parse(Int, first(m.captures)) - println("sleeping") sleep(t/10) - println("done sleeping") push!(server_events, "Response: $(HTTP.sprintcompact(response))") - println("writing response") write(lb.io, response) - println("done writing response") else response = HTTP.Response(403, ["Connection" => "close", @@ -245,18 +238,12 @@ end r = lbopen("delay10", []) do http @sync begin @async begin - println("writing") write(http, "Hello World!") - println("done writing") closewrite(http) - println("setting body_sent") body_sent[] = true end - println("startread") startread(http) - println("calling read") body[] = read(http) - println("done reading") closeread(http) end end @@ -307,8 +294,8 @@ end world_sent = Ref(false) @test_throws HTTP.StatusError begin r = lbreq("abort", [], [ - FunctionIO(()->(println("hello_sent"); hello_sent[] = true; sleep(0.5); "Hello")), - FunctionIO(()->(println("world_sent"); world_sent[] = true; " World!"))]) + FunctionIO(()->(hello_sent[] = true; sleep(0.5); "Hello")), + FunctionIO(()->(world_sent[] = true; " World!"))]) end @test hello_sent[] @test !world_sent[] From a0e904c947a93b298a3ef31452d55407b8f09f9e Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Fri, 18 Mar 2022 00:27:15 -0600 Subject: [PATCH 13/19] add global stack state w/ modifiers --- src/HTTP.jl | 57 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 25 deletions(-) diff --git a/src/HTTP.jl b/src/HTTP.jl index 25a03a156..724222336 100644 --- a/src/HTTP.jl +++ b/src/HTTP.jl @@ -37,6 +37,26 @@ include("ConnectionPool.jl") include("Messages.jl") ;using .Messages include("cookies.jl") ;using .Cookies include("Streams.jl") ;using .Streams +include("MessageRequest.jl"); using .MessageRequest +include("RedirectRequest.jl"); using .RedirectRequest +include("DefaultHeadersRequest.jl"); using .DefaultHeadersRequest +include("BasicAuthRequest.jl"); using .BasicAuthRequest +include("CookieRequest.jl"); using .CookieRequest +include("CanonicalizeRequest.jl"); using .CanonicalizeRequest +include("TimeoutRequest.jl"); using .TimeoutRequest +include("ExceptionRequest.jl"); using .ExceptionRequest + import .ExceptionRequest.StatusError +include("RetryRequest.jl"); using .RetryRequest +include("ConnectionRequest.jl"); using .ConnectionRequest +include("DebugRequest.jl"); using .DebugRequest +include("StreamRequest.jl"); using .StreamRequest +include("ContentTypeRequest.jl"); using .ContentTypeDetection + +include("download.jl") +include("Servers.jl") ;using .Servers; using .Servers: listen +include("Handlers.jl") ;using .Handlers; using .Handlers: serve +include("parsemultipart.jl") ;using .MultiPartParsing: parse_multipart_form +include("WebSockets.jl") ;using .WebSockets const nobody = UInt8[] @@ -310,18 +330,26 @@ function request(method, url, h=Header[], b=nobody; return request(HTTP.stack(), method, url, headers, body, query; kw...) end +const STREAM_LAYERS = [timeoutlayer, exceptionlayer, debuglayer] +const REQUEST_LAYERS = [messagelayer, redirectlayer, defaultheaderslayer, basicauthlayer, contenttypedetectionlayer, cookielayer, retrylayer, canonicalizelayer] + +pushlayer!(layer; request::Bool=true) = push!(request ? REQUEST_LAYERS : STREAM_LAYERS, layer) +pushfirstlayer!(layer; request::Bool=true) = pushfirst!(request ? REQUEST_LAYERS : STREAM_LAYERS, layer) +poplayer!(; request::Bool=true) = pop!(request ? REQUEST_LAYERS : STREAM_LAYERS) +popfirstlayer!(; request::Bool=true) = popfirst!(request ? REQUEST_LAYERS : STREAM_LAYERS) + function stack( # custom layers requestlayers=(), streamlayers=()) # stream layers - slayers = (timeoutlayer, exceptionlayer, debuglayer, streamlayers...) - layers = foldr((x, y) -> x(y), slayers, init=streamlayer) + layers = foldr((x, y) -> x(y), streamlayers, init=streamlayer) + layers2 = foldr((x, y) -> x(y), STREAM_LAYERS, init=layers) # request layers # messagelayer must be the 1st/outermost layer to convert initial args to Request - rlayers = (messagelayer, redirectlayer, defaultheaderslayer, basicauthlayer, contenttypedetectionlayer, cookielayer, retrylayer, canonicalizelayer, requestlayers...) - return foldr((x, y) -> x(y), rlayers; init=connectionlayer(layers)) + layers3 = foldr((x, y) -> x(y), requestlayers; init=connectionlayer(layers2)) + return foldr((x, y) -> x(y), REQUEST_LAYERS; init=layers3) end function request(stack::Base.Callable, method, url, h=Header[], b=nobody, q=nothing; @@ -447,27 +475,6 @@ function openraw(method::Union{String,Symbol}, url, headers=Header[]; kw...)::Tu take!(socketready) end -include("MessageRequest.jl"); using .MessageRequest -include("RedirectRequest.jl"); using .RedirectRequest -include("DefaultHeadersRequest.jl"); using .DefaultHeadersRequest -include("BasicAuthRequest.jl"); using .BasicAuthRequest -include("CookieRequest.jl"); using .CookieRequest -include("CanonicalizeRequest.jl"); using .CanonicalizeRequest -include("TimeoutRequest.jl"); using .TimeoutRequest -include("ExceptionRequest.jl"); using .ExceptionRequest - import .ExceptionRequest.StatusError -include("RetryRequest.jl"); using .RetryRequest -include("ConnectionRequest.jl"); using .ConnectionRequest -include("DebugRequest.jl"); using .DebugRequest -include("StreamRequest.jl"); using .StreamRequest -include("ContentTypeRequest.jl"); using .ContentTypeDetection - -include("download.jl") -include("Servers.jl") ;using .Servers; using .Servers: listen -include("Handlers.jl") ;using .Handlers; using .Handlers: serve -include("parsemultipart.jl") ;using .MultiPartParsing: parse_multipart_form -include("WebSockets.jl") ;using .WebSockets - import .ConnectionPool: Connection function Base.parse(::Type{T}, str::AbstractString)::T where T <: Message From 9aebfea4fa80d6d410d0a81c14649a4b6363cac9 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Fri, 18 Mar 2022 01:18:00 -0600 Subject: [PATCH 14/19] a little more cleanup --- src/ConnectionRequest.jl | 4 +--- src/ExceptionRequest.jl | 2 +- src/MessageRequest.jl | 2 -- src/StreamRequest.jl | 4 ++-- 4 files changed, 4 insertions(+), 8 deletions(-) diff --git a/src/ConnectionRequest.jl b/src/ConnectionRequest.jl index 0a290711e..e92693746 100644 --- a/src/ConnectionRequest.jl +++ b/src/ConnectionRequest.jl @@ -9,8 +9,6 @@ using Base64: base64encode import ..@debug, ..DEBUG_LEVEL import ..Streams: Stream -const nosslconfig = SSLConfig() - # hasdotsuffix reports whether s ends in "."+suffix. hasdotsuffix(s, suffix) = endswith(s, "." * suffix) @@ -50,7 +48,7 @@ end export connectionlayer """ - connectionlayer(ctx, ::Request, body) -> HTTP.Response + connectionlayer(req) -> HTTP.Response Retrieve an `IO` connection from the [`ConnectionPool`](@ref). diff --git a/src/ExceptionRequest.jl b/src/ExceptionRequest.jl index 9b92ed7b9..749300e00 100644 --- a/src/ExceptionRequest.jl +++ b/src/ExceptionRequest.jl @@ -8,7 +8,7 @@ using ..Messages: iserror export exceptionlayer """ - exceptionlayer(ctx, stream) -> HTTP.Response + exceptionlayer(stream) -> HTTP.Response Throw a `StatusError` if the request returns an error response status. """ diff --git a/src/MessageRequest.jl b/src/MessageRequest.jl index 1ca335e11..ff18ad69b 100644 --- a/src/MessageRequest.jl +++ b/src/MessageRequest.jl @@ -1,7 +1,5 @@ module MessageRequest -export setuseragent! - using URIs import ..Headers using ..Messages diff --git a/src/StreamRequest.jl b/src/StreamRequest.jl index a150b6c7f..6061a7be1 100644 --- a/src/StreamRequest.jl +++ b/src/StreamRequest.jl @@ -11,7 +11,7 @@ import ..@debug, ..DEBUG_LEVEL, ..printlncompact, ..sprintcompact export streamlayer """ - streamlayer(ctx, stream) -> HTTP.Response + streamlayer(stream) -> HTTP.Response Create a [`Stream`](@ref) to send a `Request` and `body` to an `IO` stream and read the response. @@ -31,7 +31,7 @@ function streamlayer(stream::Stream; iofunction=nothing, verbose=0, redirect_lim if verbose == 2 println(req) - if iofunction === nothing && req.body isa IO + if iofunction === nothing && !isbytes(req.body) println("$(typeof(req)).body: $(sprintcompact(req.body))") end end From 131e7e572c0c63e17b4208960d76c869b65e287d Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 22 Mar 2022 09:31:19 -0600 Subject: [PATCH 15/19] Update test/loopback.jl Co-authored-by: mattBrzezinski --- test/loopback.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/loopback.jl b/test/loopback.jl index 79842f27e..a7ef9822b 100644 --- a/test/loopback.jl +++ b/test/loopback.jl @@ -350,4 +350,4 @@ end end end -end # module \ No newline at end of file +end # module From 08e8a6c72bf9942195f143ddf18610ce98ec9416 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 22 Mar 2022 09:31:26 -0600 Subject: [PATCH 16/19] Update test/cookies.jl Co-authored-by: mattBrzezinski --- test/cookies.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/cookies.jl b/test/cookies.jl index 70ab80839..729c5af06 100644 --- a/test/cookies.jl +++ b/test/cookies.jl @@ -179,4 +179,4 @@ using Sockets, Test end end -end # module \ No newline at end of file +end # module From a317e438dad8a3fe1eaaa90cf148cc7aadd87cfe Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 22 Mar 2022 09:31:33 -0600 Subject: [PATCH 17/19] Update test/client.jl Co-authored-by: mattBrzezinski --- test/client.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/client.jl b/test/client.jl index c77d889e9..0f120c0f9 100644 --- a/test/client.jl +++ b/test/client.jl @@ -495,4 +495,4 @@ end end end -end # module \ No newline at end of file +end # module From 45e5a5a2bcd48c4b646a3528244cc6f43c33aeff Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 22 Mar 2022 09:31:41 -0600 Subject: [PATCH 18/19] Update test/chunking.jl Co-authored-by: mattBrzezinski --- test/chunking.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/chunking.jl b/test/chunking.jl index bc25d5d17..ce0e12e0b 100644 --- a/test/chunking.jl +++ b/test/chunking.jl @@ -64,4 +64,4 @@ using BufferedStreams close(server) end -end # module \ No newline at end of file +end # module From 3ea2c8ba19dde10a9e7d3603fa5fbcf99e4569e8 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Tue, 22 Mar 2022 09:32:16 -0600 Subject: [PATCH 19/19] Update src/DefaultHeadersRequest.jl Co-authored-by: mattBrzezinski --- src/DefaultHeadersRequest.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/DefaultHeadersRequest.jl b/src/DefaultHeadersRequest.jl index c0b006db6..fb7b0474c 100644 --- a/src/DefaultHeadersRequest.jl +++ b/src/DefaultHeadersRequest.jl @@ -59,4 +59,4 @@ function setuseragent!(x::Union{String, Nothing}) return end -end # module \ No newline at end of file +end # module