Send requests with a self-signed certificate for auth

[taylordowns2000]

For an upcoming UNHCR project, we've been asked to make requests with a self-signed X509 certificate. Something like this: http://www.rajatswarup.com/blog/2007/03/10/using-certificates-with-curl/

Given a cert (stored as text, passed in as a value from the standard config object in `state.json) can we make all requests for that job expression use the self-signed cert.

@stuartc , this is one I'd love your thoughts on.

{
  "configuration": {
    "name": "primero-unhcr-3",
    "body": "...thebodyofthecerthere?"
  }
}

We'd then use the httpAgent option in axios:

{
  httpsAgent: new https.Agent({ ca: fs.readFileSync(certPath) });
}

https://stackoverflow.com/questions/51363855/how-to-configure-axios-to-use-ssl-certificate

or maybe ca is set to the x509 text from config?

[taylordowns2000]

Thoughts on how to implement this in a flexible way:

export a helper function from language-http/src/Adaptor.js

export function newAgent(options) {
  return state => {
    const opts = expandReferences(options)(state);
    return new https.Agent(opts);
  };
}

required state to use a self-signed cert

{
  "configuration": {
    "publicKey": "hithere",
    "privateKey":
      "213712903821908912jkldsadjklsajkdal7908391203===============dsa790421kl",
  },
  "data": {
    "a": 1,
  },
};

usage/signature

get('path', {
  // passing in options for axios
  body: state => state.data,
  https: newAgent({
    ca: state.configuration.privateKey,
    // strictSSL: false
  }),
});

[taylordowns2000]

Currently, all the tests are passing because we appear to send the correct httpsAgent options to axios—we even get validation that they work out OK because we now send back the entire axios response in state.response. See test/index.js lines 700, 676, and 653.

When we run a real expression with the following code, however:

post(
  `${state.configuration.url}/Users/login`,
  {
    headers: { 'content-type': 'application/json' },
    body: {
      email: state.configuration.email,
      password: state.configuration.password,
    },
    httpsAgent: newAgent({ rejectUnauthorized: false }),
  }
);

we get this (ubiquitous on github) error:

╭─────────────────────────────────────────────────────────────────────╮
│ ◰ ◱ ◲ ◳  OpenFn/core ~ @openfn/language-http#v3.0.2 (Node v12.19.1) │
╰─────────────────────────────────────────────────────────────────────╯
TypeError [ERR_INVALID_ARG_TYPE]: The "options.agent" property must be one of Agent-like Object, undefined, or false. Received an instance of Object
    at new ClientRequest (_http_client.js:133:11)
    at Object.request (https.js:316:10)
    at RedirectableRequest._performRequest (/home/taylor/devtools/adaptors/language-http/node_modules/follow-redirects/index.js:241:24)
    at new RedirectableRequest (/home/taylor/devtools/adaptors/language-http/node_modules/follow-redirects/index.js:60:8)
    at Object.request (/home/taylor/devtools/adaptors/language-http/node_modules/follow-redirects/index.js:437:14)
    at dispatchHttpRequest (/home/taylor/devtools/adaptors/language-http/node_modules/axios/lib/adapters/http.js:195:25)
    at new Promise (<anonymous>)
    at httpAdapter (/home/taylor/devtools/adaptors/language-http/node_modules/axios/lib/adapters/http.js:46:10)
    at dispatchRequest (/home/taylor/devtools/adaptors/language-http/node_modules/axios/lib/core/dispatchRequest.js:52:10) {
  code: 'ERR_INVALID_ARG_TYPE'
}

We're following the docs from axios, and I've read several bug threads where some people "just cant get it to work". (E.g., axios/axios#535 (comment))

Seems so simple: https://stackoverflow.com/questions/51363855/how-to-configure-axios-to-use-ssl-certificate

I'm super confused as to:

1. Why the tests pass
2. Why the job fails, when pointed at a real server with bad certs (one where we actually need to set rejectUnauthorized to false)
3. Why I can't get axios to behave

We've had two special requests come up related to this https.Agent(opts) usage... we need, at a bare minimum, to allow folks to accept bad SSL certs (with rejectUnauthorized: false) and to pass in their own certs, with { ca: 'somePrivateKey' }. @stuartc , any ideas on this?

Also, please note that we're using this fairly custom and messy expandRequestReferences function from common.http which carefully tries not to expand httpsAgent keys (other than options) and data keys if they're FormData instances.

I've ripped all that out and tried passing a new https.Agent({ rejectUnauthorized: false }) directly to axios itself (not more common.http) and even that doesn't seem to work.

[aleksa-krolls]

@taylordowns2000 Quick one to inform upcoming UNHCR design... does this change require we as consultants to spec anything differently? Or is this now a built-in featuee?

[taylordowns2000]

It will be available as planned, but is not yet being used on platform-app. Do you know if they have provided a test server for us already?

@aleksa-krolls , we've got access to this right now, but it seemingly only lets us test the api keys, rather than the cert. If you could ask them how/when we could start testing the self-signed cert, that would be wonderful!