Requests Fail from getServerSideProps due to Proxy (400 Bad Request)

[adrienclay36]

Summary

At my company we work behind a proxy and develop applications to be used on a private network.

When I try to use either fetch or Axios to get data external to the application (as in, I'm not using NextJS Api routes, I'm getting data from other servers) I am getting an http proxy request error in both scenarios. It's always a 400 error for both public api's of any kind and my dotnet API server that's running on localhost. It appears it only happens in development, not when the app is deployed.

What's odd to me is that this doesn't happen when executing requests client side with either fetch or axios. Those work just fine. I really want to incorporate data availability on load with NextJS for this application, but I suppose I could drive forward with all client side data fetching only if I can't get this to work. it wouldn't be a huge deal but defeats at least 1/3 of the purpose of me using NextJS for this project. I'm mostly in it for the file routing, but ultimately the SSR was a huge draw for me.

I've tried multiple approaches so far (For context, this all would be taking place inside of getServerSideProps)

This happens for AXIOS and FETCH, it makes no difference.

1. Axios proxy settings

This method doesn't work, I'm not sure why, doesn't work with or without auth settings.

One interesting thing about this is that if I DONT include this, I can see in the axios error that it already has the proxy as "mycompany.proxy.com"

const response = await axios.get(URL, {  proxy: {
    host: 'proxy.company.com',
    port: 80,
    auth: {
      username: 'username',
      password: 'password'
    }
  }});

2. HTTPS Agent

Same error occurs here, no changes in output. Still a 400 bad request error.

import https from 'https';
const result = await axios.post(
    `https://${url}/dataendpoint`,
    body,
    {
      httpsAgent: new https.Agent({
        rejectUnauthorized: false
      })
    }
  )

3. Including Certifies manually:

With this, I get the error ECONNRESET, not sure what this means other than I read that it means the target url shut down the request prematurely for some reason.

const agent = new https.Agent({
    cert: fs.readFileSync('./path_to/cert.cer')
})
const response = await axios.get(URL, { httpsAgent: agent});

The only solution that works, and it only works for localhost, not public API's

if(process.env.NODE_ENV === 'development') {
    process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
}

const response = await fetch("localhost:3000/api/items");
console.log(await response.json()) // SUCCESS, got data

const response = await fetch("https://www.publicapi.com/items");
console.log(await response.json()) // throws invalid json response, unexpected token < in JSON at position 0, which I think means the request is still failing.

I am at a complete loss as to what to do here. Nothing is working and I can't find anything online that points to anything I feel like I haven't seen already. I feel like I've tried everything under the sun.

Additional information

No response

Example

No response

[icyJoseph]

So fetch("internal-url-to-your-companies-proxy") doesn't work?

Could you try with what they show in this article? https://www.scrapingbee.com/blog/proxy-node-fetch/

In Node 16, Next.js polyfills fetch with node-fetch, server side, so you'd just need to add the https-proxy-agent library.

If you make a script that just uses fetch server side, does it work? You'd need Node 18 at least, to get native fetch available, but I think it might be worth removing Next.js from the equation first, and see if you can hit your Proxy at all through server side fetching, aka with a Node.js script.

[adrienclay36]

Good call, I tried just that.. I'm on Node 18 by the way.

Here are three different configurations I've tried, with https agent, a tunnel agent, and the proxy agent mentioned in the article you've linked and now, while I am getting past 400, I am now receiving a self-signed certificate error:

var HttpsProxyAgent = require("https-proxy-agent");
var https = require("https");
var fs = require("fs");
const tunnel = require("tunnel");

const httpsAgent = new https.Agent({
    rejectUnauthorized: false,
    host: "proxy.company.com",
    port: 8080,
    cert: fs.readFileSync("root_ca.cer"),
})

const tunnelAgent = tunnel.httpsOverHttp({
    cert: fs.readFileSync("root_ca.cer"),
    proxy: {
        host: 'proxy.company.com',
        port: 8080,
        proxyAuth: 'username:password'
    }
});

var proxyAgent = new HttpsProxyAgent("proxy.company.com:8080");

fetch("https://internal-url.com/api/AvailableItems", { agent: httpsAgent }).then((response) => response.json().then((data) => console.log(data))).catch(error => console.log(error));


// RESULTING ERROR 

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11413:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error: self-signed certificate
      at TLSSocket.onConnectSecure (node:_tls_wrap:1540:34)
      at TLSSocket.emit (node:events:513:28)
      at TLSSocket._finishInit (node:_tls_wrap:959:8)
      at ssl.onhandshakedone (node:_tls_wrap:743:12) {
    code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
  }

I at least see now this is not a NextJS problem and a node problem in general.

[icyJoseph]

Oh wow, after googling a bit, I must say that this escapes my knowledge domain.

From what I gather, now you have to use NODE_TLS_REJECT_UNAUTHORIZED? Or passing a secure: false option, to the proxy object, but why would that be absolutely necessary I don't know...