From a6d00f50b97562bc147cbc71e42bff96b54302eb Mon Sep 17 00:00:00 2001 From: Matt Date: Sat, 7 Aug 2021 19:27:19 +0100 Subject: [PATCH 1/4] Move json middleware (request and response) from faraday_middleware --- docs/middleware/list.md | 4 + docs/middleware/request/instrumentation.md | 4 +- docs/middleware/request/json.md | 31 ++++++ docs/middleware/request/url_encoded.md | 4 +- docs/middleware/response/json.md | 29 +++++ docs/middleware/response/logger.md | 4 +- lib/faraday.rb | 2 + lib/faraday/request.rb | 13 +-- lib/faraday/request/json.rb | 51 +++++++++ lib/faraday/response.rb | 4 +- lib/faraday/response/json.rb | 52 +++++++++ lib/faraday/response/logger.rb | 6 +- spec/faraday/request/json_spec.rb | 111 +++++++++++++++++++ spec/faraday/response/json_spec.rb | 119 +++++++++++++++++++++ 14 files changed, 417 insertions(+), 17 deletions(-) create mode 100644 docs/middleware/request/json.md create mode 100644 docs/middleware/response/json.md create mode 100644 lib/faraday/request/json.rb create mode 100644 lib/faraday/response/json.rb create mode 100644 spec/faraday/request/json_spec.rb create mode 100644 spec/faraday/response/json_spec.rb diff --git a/docs/middleware/list.md b/docs/middleware/list.md index 59d9ba2ae..0269bccb3 100644 --- a/docs/middleware/list.md +++ b/docs/middleware/list.md @@ -27,6 +27,8 @@ base64 representation. * [`Multipart`][multipart] converts a `Faraday::Request#body` hash of key/value pairs into a multipart form request. * [`UrlEncoded`][url_encoded] converts a `Faraday::Request#body` hash of key/value pairs into a url-encoded request body. +* [`Json Request`][json-request] converts a `Faraday::Request#body` hash of key/value pairs into a json request body. +* [`Json Response`][json-response] parses response body into a hash of key/value pairs. * [`Retry`][retry] automatically retries requests that fail due to intermittent client or server errors (such as network hiccups). * [`Instrumentation`][instrumentation] allows to instrument requests using different tools. @@ -44,7 +46,9 @@ before returning it. [authentication]: ./authentication [multipart]: ./multipart [url_encoded]: ./url-encoded +[json-request]: ./json-request [retry]: ./retry [instrumentation]: ./instrumentation +[json-response]: ./json-response [logger]: ./logger [raise_error]: ./raise-error diff --git a/docs/middleware/request/instrumentation.md b/docs/middleware/request/instrumentation.md index 33fcaae08..86912b62d 100644 --- a/docs/middleware/request/instrumentation.md +++ b/docs/middleware/request/instrumentation.md @@ -5,8 +5,8 @@ permalink: /middleware/instrumentation hide: true prev_name: Retry Middleware prev_link: ./retry -next_name: Logger Middleware -next_link: ./logger +next_name: Json Response Middleware +next_link: ./json-response top_name: Back to Middleware top_link: ./list --- diff --git a/docs/middleware/request/json.md b/docs/middleware/request/json.md new file mode 100644 index 000000000..a3b0dd250 --- /dev/null +++ b/docs/middleware/request/json.md @@ -0,0 +1,31 @@ +--- +layout: documentation +title: "Json Request Middleware" +permalink: /middleware/json-request +hide: true +prev_name: UrlEncoded Middleware +prev_link: ./url-encoded +next_name: Retry Middleware +next_link: ./retry +top_name: Back to Middleware +top_link: ./list +--- + +The `Json Request` middleware converts a `Faraday::Request#body` hash of key/value pairs into a json request body. +The middleware also automatically sets the `Content-Type` header to `application/json`, +processes only requests with matching Content-type or those without a type and +doesn't try to encode bodies that already are in string form. + +### Example Usage + +```ruby +conn = Faraday.new(...) do |f| + f.request :json + ... +end + +conn.post('/', { a: 1, b: 2 }) +# POST with +# Content-Type: application/json +# Body: {"a":1,"b":2} +``` diff --git a/docs/middleware/request/url_encoded.md b/docs/middleware/request/url_encoded.md index 95cdf528c..0336b9c75 100644 --- a/docs/middleware/request/url_encoded.md +++ b/docs/middleware/request/url_encoded.md @@ -5,8 +5,8 @@ permalink: /middleware/url-encoded hide: true prev_name: Multipart Middleware prev_link: ./multipart -next_name: Retry Middleware -next_link: ./retry +next_name: Json Request Middleware +next_link: ./json-request top_name: Back to Middleware top_link: ./list --- diff --git a/docs/middleware/response/json.md b/docs/middleware/response/json.md new file mode 100644 index 000000000..11451e0e9 --- /dev/null +++ b/docs/middleware/response/json.md @@ -0,0 +1,29 @@ +--- +layout: documentation +title: "Json Response Middleware" +permalink: /middleware/json-response +hide: true +prev_name: Instrumentation Middleware +prev_link: ./instrumentation +next_name: Logger Middleware +next_link: ./logger +top_name: Back to Middleware +top_link: ./list +--- + +The `Json Response` middleware parses response body into a hash of key/value pairs. +The behaviour can be customized with the following options: +* **parser_options:** options that will be sent to the JSON.parse method. Defaults to {} +* **content_type:** Single value or Array of response content-types that should be processed. Can be either strings or Regex. Defaults to `/\bjson$/` +* **preserve_raw:** If set to true, the original un-parsed response will be stored in the `response.env[:raw_body]` property. Defaults to `false` + +### Example Usage + +```ruby +conn = Faraday.new('http://httpbingo.org') do |f| + f.response :json, **options +end + +conn.get('json').body +# => {"slideshow"=>{"author"=>"Yours Truly", "date"=>"date of publication", "slides"=>[{"title"=>"Wake up to WonderWidgets!", "type"=>"all"}, {"items"=>["Why WonderWidgets are great", "Who buys WonderWidgets"], "title"=>"Overview", "type"=>"all"}], "title"=>"Sample Slide Show"}} +``` diff --git a/docs/middleware/response/logger.md b/docs/middleware/response/logger.md index fdee39279..7aaa0a104 100644 --- a/docs/middleware/response/logger.md +++ b/docs/middleware/response/logger.md @@ -3,8 +3,8 @@ layout: documentation title: "Logger Middleware" permalink: /middleware/logger hide: true -prev_name: Instrumentation Middleware -prev_link: ./instrumentation +prev_name: Json Response Middleware +prev_link: ./json-response next_name: RaiseError Middleware next_link: ./raise-error top_name: Back to Middleware diff --git a/lib/faraday.rb b/lib/faraday.rb index 182e79197..e8bd243b2 100644 --- a/lib/faraday.rb +++ b/lib/faraday.rb @@ -40,6 +40,8 @@ # conn.get '/' # module Faraday + CONTENT_TYPE = 'Content-Type' + class << self # The root path that Faraday is being loaded from. # diff --git a/lib/faraday/request.rb b/lib/faraday/request.rb index 2ee8d628f..f709bc403 100644 --- a/lib/faraday/request.rb +++ b/lib/faraday/request.rb @@ -46,7 +46,8 @@ class Request < Struct.new( :TokenAuthentication, 'token_authentication' ], - instrumentation: [:Instrumentation, 'instrumentation'] + instrumentation: [:Instrumentation, 'instrumentation'], + json: [:Json, 'json'] # @param request_method [String] # @yield [request] for block customization, if block given @@ -140,11 +141,11 @@ def marshal_dump # @param serialised [Hash] the serialised object. def marshal_load(serialised) self.http_method = serialised[:http_method] - self.body = serialised[:body] - self.headers = serialised[:headers] - self.path = serialised[:path] - self.params = serialised[:params] - self.options = serialised[:options] + self.body = serialised[:body] + self.headers = serialised[:headers] + self.path = serialised[:path] + self.params = serialised[:params] + self.options = serialised[:options] end # @return [Env] the Env for this Request diff --git a/lib/faraday/request/json.rb b/lib/faraday/request/json.rb new file mode 100644 index 000000000..63d80d05a --- /dev/null +++ b/lib/faraday/request/json.rb @@ -0,0 +1,51 @@ +# frozen_string_literal: true + +require 'json' + +module Faraday + class Request + # Request middleware that encodes the body as JSON. + # + # Processes only requests with matching Content-type or those without a type. + # If a request doesn't have a type but has a body, it sets the Content-type + # to JSON MIME-type. + # + # Doesn't try to encode bodies that already are in string form. + class Json < Middleware + MIME_TYPE = 'application/json' + MIME_TYPE_REGEX = %r{^application/(vnd\..+\+)?json$}.freeze + + def on_request(env) + match_content_type(env) do |data| + env[:body] = encode data + end + end + + def encode(data) + ::JSON.generate(data) + end + + def match_content_type(env) + return unless process_request?(env) + + env[:request_headers][CONTENT_TYPE] ||= MIME_TYPE + yield env[:body] unless env[:body].respond_to?(:to_str) + end + + def process_request?(env) + type = request_type(env) + has_body?(env) && (type.empty? || MIME_TYPE_REGEX =~ type) + end + + def has_body?(env) + (body = env[:body]) && !(body.respond_to?(:to_str) && body.empty?) + end + + def request_type(env) + type = env[:request_headers][CONTENT_TYPE].to_s + type = type.split(';', 2).first if type.index(';') + type + end + end + end +end diff --git a/lib/faraday/response.rb b/lib/faraday/response.rb index 3fd17d03d..d58869849 100644 --- a/lib/faraday/response.rb +++ b/lib/faraday/response.rb @@ -22,7 +22,8 @@ def on_complete(env) register_middleware File.expand_path('response', __dir__), raise_error: [:RaiseError, 'raise_error'], - logger: [:Logger, 'logger'] + logger: [:Logger, 'logger'], + json: [:Json, 'json'] def initialize(env = nil) @env = Env.from(env) if env @@ -42,6 +43,7 @@ def reason_phrase def headers finished? ? env.response_headers : {} end + def_delegator :headers, :[] def body diff --git a/lib/faraday/response/json.rb b/lib/faraday/response/json.rb new file mode 100644 index 000000000..ff36da0a1 --- /dev/null +++ b/lib/faraday/response/json.rb @@ -0,0 +1,52 @@ +# frozen_string_literal: true + +require 'json' + +module Faraday + class Response + # Parse response bodies as JSON. + class Json < Middleware + def initialize(app = nil, parser_options: nil, content_type: /\bjson$/, preserve_raw: false) + super(app) + @parser_options = parser_options + @content_types = Array(content_type) + @preserve_raw = preserve_raw + end + + def on_complete(env) + process_response(env) if parse_response?(env) + end + + private + + def process_response(env) + env[:raw_body] = env[:body] if @preserve_raw + env[:body] = parse(env[:body]) + rescue StandardError, SyntaxError => e + raise Faraday::ParsingError.new(e, env[:response]) + end + + def parse(body) + ::JSON.parse(body, @parser_options || {}) unless body.strip.empty? + end + + def parse_response?(env) + process_response_type?(env) && + env[:body].respond_to?(:to_str) + end + + def process_response_type?(env) + type = response_type(env) + @content_types.empty? || @content_types.any? do |pattern| + pattern.is_a?(Regexp) ? type =~ pattern : type == pattern + end + end + + def response_type(env) + type = env[:response_headers][CONTENT_TYPE].to_s + type = type.split(';', 2).first if type.index(';') + type + end + end + end +end diff --git a/lib/faraday/response/logger.rb b/lib/faraday/response/logger.rb index 35aefb860..2ad458716 100644 --- a/lib/faraday/response/logger.rb +++ b/lib/faraday/response/logger.rb @@ -1,6 +1,7 @@ # frozen_string_literal: true require 'forwardable' +require 'logger' require 'faraday/logging/formatter' module Faraday @@ -11,10 +12,7 @@ class Response class Logger < Middleware def initialize(app, logger = nil, options = {}) super(app) - logger ||= begin - require 'logger' - ::Logger.new($stdout) - end + logger ||= ::Logger.new($stdout) formatter_class = options.delete(:formatter) || Logging::Formatter @formatter = formatter_class.new(logger: logger, options: options) yield @formatter if block_given? diff --git a/spec/faraday/request/json_spec.rb b/spec/faraday/request/json_spec.rb new file mode 100644 index 000000000..89949bc2a --- /dev/null +++ b/spec/faraday/request/json_spec.rb @@ -0,0 +1,111 @@ +# frozen_string_literal: true + +RSpec.describe Faraday::Request::Json do + let(:middleware) { described_class.new(->(env) { Faraday::Response.new(env) }) } + + def process(body, content_type = nil) + env = { body: body, request_headers: Faraday::Utils::Headers.new } + env[:request_headers]['content-type'] = content_type if content_type + middleware.call(Faraday::Env.from(env)).env + end + + def result_body + result[:body] + end + + def result_type + result[:request_headers]['content-type'] + end + + context 'no body' do + let(:result) { process(nil) } + + it "doesn't change body" do + expect(result_body).to be_nil + end + + it "doesn't add content type" do + expect(result_type).to be_nil + end + end + + context 'empty body' do + let(:result) { process('') } + + it "doesn't change body" do + expect(result_body).to be_empty + end + + it "doesn't add content type" do + expect(result_type).to be_nil + end + end + + context 'string body' do + let(:result) { process('{"a":1}') } + + it "doesn't change body" do + expect(result_body).to eq('{"a":1}') + end + + it 'adds content type' do + expect(result_type).to eq('application/json') + end + end + + context 'object body' do + let(:result) { process(a: 1) } + + it 'encodes body' do + expect(result_body).to eq('{"a":1}') + end + + it 'adds content type' do + expect(result_type).to eq('application/json') + end + end + + context 'empty object body' do + let(:result) { process({}) } + + it 'encodes body' do + expect(result_body).to eq('{}') + end + end + + context 'object body with json type' do + let(:result) { process({ a: 1 }, 'application/json; charset=utf-8') } + + it 'encodes body' do + expect(result_body).to eq('{"a":1}') + end + + it "doesn't change content type" do + expect(result_type).to eq('application/json; charset=utf-8') + end + end + + context 'object body with vendor json type' do + let(:result) { process({ a: 1 }, 'application/vnd.myapp.v1+json; charset=utf-8') } + + it 'encodes body' do + expect(result_body).to eq('{"a":1}') + end + + it "doesn't change content type" do + expect(result_type).to eq('application/vnd.myapp.v1+json; charset=utf-8') + end + end + + context 'object body with incompatible type' do + let(:result) { process({ a: 1 }, 'application/xml; charset=utf-8') } + + it "doesn't change body" do + expect(result_body).to eq(a: 1) + end + + it "doesn't change content type" do + expect(result_type).to eq('application/xml; charset=utf-8') + end + end +end diff --git a/spec/faraday/response/json_spec.rb b/spec/faraday/response/json_spec.rb new file mode 100644 index 000000000..e8c5b5f54 --- /dev/null +++ b/spec/faraday/response/json_spec.rb @@ -0,0 +1,119 @@ +# frozen_string_literal: true + +RSpec.describe Faraday::Response::Json, type: :response do + let(:options) { {} } + let(:headers) { {} } + let(:middleware) do + described_class.new(lambda { |env| + Faraday::Response.new(env) + }, options) + end + + def process(body, content_type = 'application/json', options = {}) + env = { + body: body, request: options, + request_headers: Faraday::Utils::Headers.new, + response_headers: Faraday::Utils::Headers.new(headers) + } + env[:response_headers]['content-type'] = content_type if content_type + yield(env) if block_given? + middleware.call(Faraday::Env.from(env)) + end + + context 'no type matching' do + it "doesn't change nil body" do + expect(process(nil).body).to be_nil + end + + it 'nullifies empty body' do + expect(process('').body).to be_nil + end + + it 'parses json body' do + response = process('{"a":1}') + expect(response.body).to eq('a' => 1) + expect(response.env[:raw_body]).to be_nil + end + end + + context 'with preserving raw' do + let(:options) { { preserve_raw: true } } + + it 'parses json body' do + response = process('{"a":1}') + expect(response.body).to eq('a' => 1) + expect(response.env[:raw_body]).to eq('{"a":1}') + end + end + + context 'with default regexp type matching' do + it 'parses json body of correct type' do + response = process('{"a":1}', 'application/x-json') + expect(response.body).to eq('a' => 1) + end + + it 'ignores json body of incorrect type' do + response = process('{"a":1}', 'text/json-xml') + expect(response.body).to eq('{"a":1}') + end + end + + context 'with array type matching' do + let(:options) { { content_type: %w[a/b c/d] } } + + it 'parses json body of correct type' do + expect(process('{"a":1}', 'a/b').body).to be_a(Hash) + expect(process('{"a":1}', 'c/d').body).to be_a(Hash) + end + + it 'ignores json body of incorrect type' do + expect(process('{"a":1}', 'a/d').body).not_to be_a(Hash) + end + end + + it 'chokes on invalid json' do + expect { process('{!') }.to raise_error(Faraday::ParsingError) + end + + it 'includes the response on the ParsingError instance' do + begin + process('{') { |env| env[:response] = Faraday::Response.new } + raise 'Parsing should have failed.' + rescue Faraday::ParsingError => e + expect(e.response).to be_a(Faraday::Response) + end + end + + context 'HEAD responses' do + it "nullifies the body if it's only one space" do + response = process(' ') + expect(response.body).to be_nil + end + + it "nullifies the body if it's two spaces" do + response = process(' ') + expect(response.body).to be_nil + end + end + + context 'JSON options' do + let(:body) { '{"a": 1}' } + let(:result) { { a: 1 } } + let(:options) do + { + parser_options: { + symbolize_names: true + } + } + end + + it 'passes relevant options to JSON parse' do + expect(::JSON).to receive(:parse) + .with(body, options[:parser_options]) + .and_return(result) + + response = process(body) + expect(response.body).to eq(result) + end + end +end From ac3266cd986afdada7999c793026d5996090b6bd Mon Sep 17 00:00:00 2001 From: Matt Date: Sat, 7 Aug 2021 19:31:56 +0100 Subject: [PATCH 2/4] Fix rubocop offenses --- lib/faraday/request/json.rb | 8 +++++--- spec/faraday/response/json_spec.rb | 4 ++-- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/faraday/request/json.rb b/lib/faraday/request/json.rb index 63d80d05a..998522751 100644 --- a/lib/faraday/request/json.rb +++ b/lib/faraday/request/json.rb @@ -17,10 +17,12 @@ class Json < Middleware def on_request(env) match_content_type(env) do |data| - env[:body] = encode data + env[:body] = encode(data) end end + private + def encode(data) ::JSON.generate(data) end @@ -34,10 +36,10 @@ def match_content_type(env) def process_request?(env) type = request_type(env) - has_body?(env) && (type.empty? || MIME_TYPE_REGEX =~ type) + body?(env) && (type.empty? || MIME_TYPE_REGEX =~ type) end - def has_body?(env) + def body?(env) (body = env[:body]) && !(body.respond_to?(:to_str) && body.empty?) end diff --git a/spec/faraday/response/json_spec.rb b/spec/faraday/response/json_spec.rb index e8c5b5f54..336ac82f6 100644 --- a/spec/faraday/response/json_spec.rb +++ b/spec/faraday/response/json_spec.rb @@ -109,8 +109,8 @@ def process(body, content_type = 'application/json', options = {}) it 'passes relevant options to JSON parse' do expect(::JSON).to receive(:parse) - .with(body, options[:parser_options]) - .and_return(result) + .with(body, options[:parser_options]) + .and_return(result) response = process(body) expect(response.body).to eq(result) From 275bd90f4b27e4c779768a955f433bb6dd033698 Mon Sep 17 00:00:00 2001 From: Matt Date: Sun, 8 Aug 2021 08:48:26 +0100 Subject: [PATCH 3/4] Fix test middleware initializer --- spec/faraday/response/json_spec.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/spec/faraday/response/json_spec.rb b/spec/faraday/response/json_spec.rb index 336ac82f6..874150801 100644 --- a/spec/faraday/response/json_spec.rb +++ b/spec/faraday/response/json_spec.rb @@ -4,9 +4,10 @@ let(:options) { {} } let(:headers) { {} } let(:middleware) do + puts options described_class.new(lambda { |env| Faraday::Response.new(env) - }, options) + }, **options) end def process(body, content_type = 'application/json', options = {}) From 13c077e8045d858d1a740892724531e8a8600b9f Mon Sep 17 00:00:00 2001 From: Matt Date: Mon, 9 Aug 2021 08:32:06 +0100 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Olle Jonsson --- docs/middleware/list.md | 2 +- docs/middleware/request/instrumentation.md | 2 +- docs/middleware/request/json.md | 6 +++--- docs/middleware/request/url_encoded.md | 2 +- docs/middleware/response/json.md | 2 +- docs/middleware/response/logger.md | 2 +- 6 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/middleware/list.md b/docs/middleware/list.md index 0269bccb3..88f4422cb 100644 --- a/docs/middleware/list.md +++ b/docs/middleware/list.md @@ -27,7 +27,7 @@ base64 representation. * [`Multipart`][multipart] converts a `Faraday::Request#body` hash of key/value pairs into a multipart form request. * [`UrlEncoded`][url_encoded] converts a `Faraday::Request#body` hash of key/value pairs into a url-encoded request body. -* [`Json Request`][json-request] converts a `Faraday::Request#body` hash of key/value pairs into a json request body. +* [`Json Request`][json-request] converts a `Faraday::Request#body` hash of key/value pairs into a JSON request body. * [`Json Response`][json-response] parses response body into a hash of key/value pairs. * [`Retry`][retry] automatically retries requests that fail due to intermittent client or server errors (such as network hiccups). diff --git a/docs/middleware/request/instrumentation.md b/docs/middleware/request/instrumentation.md index 86912b62d..83adec6e6 100644 --- a/docs/middleware/request/instrumentation.md +++ b/docs/middleware/request/instrumentation.md @@ -5,7 +5,7 @@ permalink: /middleware/instrumentation hide: true prev_name: Retry Middleware prev_link: ./retry -next_name: Json Response Middleware +next_name: JSON Response Middleware next_link: ./json-response top_name: Back to Middleware top_link: ./list diff --git a/docs/middleware/request/json.md b/docs/middleware/request/json.md index a3b0dd250..943edb6f7 100644 --- a/docs/middleware/request/json.md +++ b/docs/middleware/request/json.md @@ -1,6 +1,6 @@ --- layout: documentation -title: "Json Request Middleware" +title: "JSON Request Middleware" permalink: /middleware/json-request hide: true prev_name: UrlEncoded Middleware @@ -11,9 +11,9 @@ top_name: Back to Middleware top_link: ./list --- -The `Json Request` middleware converts a `Faraday::Request#body` hash of key/value pairs into a json request body. +The `Json Request` middleware converts a `Faraday::Request#body` hash of key/value pairs into a JSON request body. The middleware also automatically sets the `Content-Type` header to `application/json`, -processes only requests with matching Content-type or those without a type and +processes only requests with matching Content-Type or those without a type and doesn't try to encode bodies that already are in string form. ### Example Usage diff --git a/docs/middleware/request/url_encoded.md b/docs/middleware/request/url_encoded.md index 0336b9c75..c913e3865 100644 --- a/docs/middleware/request/url_encoded.md +++ b/docs/middleware/request/url_encoded.md @@ -5,7 +5,7 @@ permalink: /middleware/url-encoded hide: true prev_name: Multipart Middleware prev_link: ./multipart -next_name: Json Request Middleware +next_name: JSON Request Middleware next_link: ./json-request top_name: Back to Middleware top_link: ./list diff --git a/docs/middleware/response/json.md b/docs/middleware/response/json.md index 11451e0e9..ca24ccd18 100644 --- a/docs/middleware/response/json.md +++ b/docs/middleware/response/json.md @@ -1,6 +1,6 @@ --- layout: documentation -title: "Json Response Middleware" +title: "JSON Response Middleware" permalink: /middleware/json-response hide: true prev_name: Instrumentation Middleware diff --git a/docs/middleware/response/logger.md b/docs/middleware/response/logger.md index 7aaa0a104..e728d51f1 100644 --- a/docs/middleware/response/logger.md +++ b/docs/middleware/response/logger.md @@ -3,7 +3,7 @@ layout: documentation title: "Logger Middleware" permalink: /middleware/logger hide: true -prev_name: Json Response Middleware +prev_name: JSON Response Middleware prev_link: ./json-response next_name: RaiseError Middleware next_link: ./raise-error