feat: support setting of request headers in feed config (#41)

* feat: support setting of request headers in feed config * feat: support setting of http headers in feedconfig Fixes #38
html2rss · Oct 5, 2019 · a7aca11 · a7aca11
1 parent f4ec8d1
commit a7aca11
Show file tree

Hide file tree

Showing 4 changed files with 121 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -48,7 +48,17 @@ Since 0.5.0 it is possible to scrape and process JSON.
 
 Adding `json: true` to the channel config will convert the JSON response to XML.
 
-Example:
+Feed config:
+
+```yaml
+channel:
+  url: https://example.com
+  title: "Example with JSON"
+  json: true
+# ...
+```
+
+Imagine this HTTP response:
 
 ```json
 {
@@ -73,6 +83,25 @@ Your items selector would be `data > datum`, the item's link selector would be `
 
 Under the hood it uses ActiveSupport's [`Hash.to_xml`](https://apidock.com/rails/Hash/to_xml) core extension for the JSON to XML conversion.
 
+## Set any HTTP header in the request
+
+You can add any HTTP headers to the request to the channel URL.
+You can use this to e.g. have Cookie or Authorization information being sent or to overwrite the User-Agent.
+
+```yaml
+channel:
+  url: https://example.com
+  title: "Example with http headers"
+  headers:
+    "User-Agent": "html2rss-request"
+    "X-Something": "Foobar"
+    "Authorization": "Token deadbea7"
+    "Cookie": "monster=MeWantCookie"
+# ...
+```
+
+The headers provided by the channel will be merged into the global headers.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

diff --git a/lib/html2rss/config.rb b/lib/html2rss/config.rb
@@ -40,7 +40,7 @@ def json?
     end
 
     def headers
-      global_config.fetch('headers', {})
+      global_config.fetch('headers', {}).merge(channel_config.fetch('headers', {}))
     end
 
     def attribute_options(name)

diff --git a/spec/fixtures/vcr_cassettes/httpbin-headers.yml b/spec/fixtures/vcr_cassettes/httpbin-headers.yml
diff --git a/spec/html2rss_spec.rb b/spec/html2rss_spec.rb
@@ -173,4 +173,40 @@
       end
     end
   end
+
+  context 'with config having channel headers and json: true' do
+    subject(:categories) do
+      VCR.use_cassette('httpbin-headers') {
+        described_class.feed(feed_config)
+      }.items.first.categories.map(&:content)
+    end
+
+    let(:feed_config) do
+      {
+        channel: {
+          url: 'https://httpbin.org/headers',
+          title: 'httpbin headers',
+          json: true,
+          headers: {
+            'User-Agent': 'html2rss-request',
+            'X-Something': 'Foobar',
+            'Authorization': 'Token deadbea7',
+            'Cookie': 'monster=MeWantCookie'
+          }
+        },
+        selectors: {
+          items: { selector: 'html' },
+          title: { selector: 'host' },
+          something: { selector: 'x-something' },
+          authorization: { selector: 'authorization' },
+          cookie: { selector: 'cookie' },
+          categories: %i[title something authorization cookie]
+        }
+      }
+    end
+
+    it 'has the headers' do
+      expect(categories).to include('httpbin.org', 'Foobar', 'Token deadbea7', 'monster=MeWantCookie')
+    end
+  end
 end