[Feature] Enable CORS in gpt4all-chat/server.cpp

[zwilch]

Feature Request

To access the GPT4All API directly from a browser (such as Firefox), or through browser extensions (for Firefox and Chrome), as well as extensions in Thunderbird (similar to Firefox), the server.cpp file needs to support CORS (Cross-Origin Resource Sharing) and properly handle CORS Preflight OPTIONS requests from the browser.

1. Info about CORS
2. qhttpserver CORS
3. Edit gpt4all-chat/mysettings.cpp <= CORS Option enable
4. Edit gpt4all-chat/server.cpp implements some setHeaders like in 2. described

[cosmic-snow]

You've already commented in #1008, which enables CORS (or at least part of it). Said pull request includes a simple HTML file with a bit of JavaScript. I've just tested that again, and it still works.

So my questions are:

• Did you try that HTML example? Does it not work for you for some reason?
• If you need something else, can you describe it in detail and especially how it differs from what's already there?

[zwilch]

Hello cosmic-snow,

I have tried to make a simplified version for a firefox extension.

make a folder an put two file inside "manifest.json" and "background.js"

manifest.json

{
  "manifest_version": 3,
  "name": "GPT4ALL Localhost Fetch Extension",
  "version": "1.0",
  "description": "Using fetch to get data from gpt4all API on http://localhost:4891/v1",
  

  "background": {
     "scripts": ["background.js"]
  },
  "host_permissions":[
    "http://127.0.0.1:4891/*",
	"http://localhost:4891/*"
  ]
}

background.js

// background.js

// This event listener will be triggered when the extension is installed or updated
chrome.runtime.onInstalled.addListener(() => {
  console.log('Extension installed or updated.');
  main(); // start main()
});

// This event listener will be triggered when the extension is first installed
chrome.runtime.onStartup.addListener(() => {
  console.log('Extension started.');
});

async function gpt4allAPI(CORSmode){
console.log("call gpt4allAPI("+CORSmode+")");
const json_completion = JSON.stringify(
 {stream:false,
  temperature:0.6,
  max_tokens:100,
 messages:[{role:"user",content:"Hello."}],
  model: 'Nous Hermes 2 Mistral DPO'
 }
 );
 const completions = await fetch("http://127.0.0.1:4891/v1/chat/completions",{
	keepalive: true,
	method: "POST",
	mode: CORSmode, 
 headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
    'Access-Control-Allow-Origin': "*",
    'Access-Control-Allow-Headers': "*",
	"access-control-allow-methods":"GET, POST, OPTIONS",
    "access-control-expose-headers":"*"
	},
	body:json_completion
	});

	console.log("API.js completions",completions);
	var completionjson=null; //var declaration 
switch(CORSmode){
	case "no-cors":
		console.log("switch() no-cors");
		//completionjson = await completions.json();
		//console.log("API.js completionjson=",completionjson);
		break;
	case "cors":
		console.log("switch() cors");
		completionjson = await completions.json(); //trows a error on no-cors to access server response
		console.log("API.js completionjson=",completionjson);
	break;
}//switch
}//end gpt4allAPI()

async function  main(){
await gpt4allAPI("no-cors");
await gpt4allAPI("cors");
}//end main()

Load this Extension in Firefox

• open main menu and find "Add Ons and Themes"
• open AddOn Debugging tab (context menu behinde the wheel)
• in new Tab Debuging (or with about:debugging#/runtime/this-firefox)
• load a Temporary AddOn load (lgo to your folder and oad this manifest.json)
• with Inspector button you can inspect console and network from AddOn
• with reload it reloads and start again

What this AddOn do

• it connect to localhost:4891/v1/chat/completions
• and send as user "Hello" to API Endpoint of gpt4all (need to be started and enabled)
• first it use "no-corse"mode of fetch so it can connect without "CORS"
  • the server will respons to "Hello" with specified model (can be changed in script)
  • then the server finish the connection
  • with "no-cors" the JavaScript can not access the response (security reason)
• second fetch start with "cors" mode
  • fetch send a OPTION (CORS Preflight) to server, the server hast to respons to this
  • the server do not response so CORS is failed

Networking Console

First is "no-corse" it can access the gpt4all API, but Javascript can not access the content.
the response from server is correct you can watch in wireshark, but in javascript the type is "opaque" see here => https://stackoverflow.com/questions/54896998/how-to-process-fetch-response-from-an-opaque-type

With CORS (Cross-Origin Resource Sharing), the browser sends a preflight OPTION request before making a POST request to another domain. However, it seems that you're not receiving the correct response for this preflight request, leading to a failed POST API request due to a CORS error.

JavaScript Console

Conclusion

Without proper correspondence regarding CORS (Cross-Origin Resource Sharing) preflight OPTIONS, no one can implement a browser extension or Thunderbird extension for using gpt4all.

[cosmic-snow]

Alright, I'll look at that example of yours later. But before I invest any time in that:

• Did you try that HTML example? Does it not work for you for some reason?

It's in the demo section of PR #1008.

[zwilch]

insert in main() in background.js script

const models = await fetch("http://127.0.0.1:4891/v1/models");
const modelsData = await models.json();
console.log("modelsData",modelsData)

It is similary to #1008
it uses GET and not POST and the browser does not send OPTIONS CORS Prefligt to gpt4all server.cpp

GET Models is working.

GET Models Network Console

here is no OPTIONS send first from Browser and it get all models

before POST chat/completion is send, there is a OPTION send first from Browser, wich failes and does not get the right header back

[zwilch]

Here the trace in wireshark the Browser get status 404 for asking first with OPTION the server
May it would help the server answheres with "OK" Status Code 200 to the OPTION question from Browser.

Wireshark Browser to gpt4all server.cpp

OPTIONS /v1/chat/completions HTTP/1.1
Host: 127.0.0.1:4891
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0
Accept: /
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: POST
Access-Control-Request-Headers: access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,content-type
Origin: moz-extension://9bbd4855-8087-4878-913c-dbd5499e292a
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site

Wrieshark Answhere gpt4all server.cpp Error 404

HTTP/1.1 404 Not Found
Status Code: 404
Access-Control-Allow-Origin: *
Content-Type: application/x-empty
Content-Length: 0

[cosmic-snow]

Yes, the OPTIONS method of HTTP is not implemented in the chat server. That's something which needs to be done separately. The Qt framework's HTTP server (which is used by this application) doesn't have a full-fledged web API framework so that would have to be done explicitely (see the code from here).

I just wanted to make sure that there isn't some other problem on your end and the basic CORS example works.

I haven't looked at/tried your example yet, but will do so sometime today.

[cosmic-snow]

Alright, some initial notes on this:

• One thing I've found strange is that CORS should not require any preflight with the following: GET, POST, HEAD (a so-called "simple request", as explained in [1][2][4])
• You're using fetch() whereas my example uses a plain old XMLHttpRequest. I don't really know about fetch() but it should do the right thing with regards to CORS, according to the docs [3].
  • Although seeing as that doesn't work in your case, you could probably work the problem by falling back to the latter; as long as GPT4All doesn't implement OPTIONS, I mean.
• Seeing as this should, in fact, be a simple request, something else must be wrong.
  • => I think it's because Content-Type isn't any of the three "safe" ones, as defined by the new Fetch Standard spec (application/x-www-form-urlencoded, multipart/form-data, or text/plain), as opposed to the older CORS spec.
  • Therefore, fetch() wants to do a preflight regardless of the method being GET, POST, or HEAD.
• "mode": "no-cors" in your example should never work in a fetch(). That is as intended, because CORS is definitely required when using a local file, or I guess in your case, a browser extension.

So as mentioned, you could probably get around it through XmlHttpRequests, if there aren't any restrictions on that in an extension. But seeing as the server's output is application/json, it should support that OPTIONS request as per the spec. The proper response for that should probably look something like in [1].

I'll further edit this comment after investigating some more

[1]: https://developer.mozilla.org/en-US/docs/Glossary/Preflight_request
[2]: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests
[3]: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#supplying_request_options
[4]: https://developer.mozilla.org/en-US/docs/Web/API/fetch#exceptions

[zwilch]

Thank you
[2]: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests
helped me a lot to do a simple_request.

Seems you are right, with a "simple_request" its possible to access the json answhere.

Possible Solution with simple_request

manifest.json

{
  "manifest_version": 3,
  "name": "GPT4ALL Localhost Fetch Extension",
  "version": "1.0",
  "description": "Using fetch to get data from gpt4all API on http://localhost:4891/v1",
  

  "background": {
     "scripts": ["background.js"]
  },
  "host_permissions":[
    "http://127.0.0.1/",
    "http://localhost/"
  ]
}

background.js

// background.js

// This event listener will be triggered when the extension is installed or updated
chrome.runtime.onInstalled.addListener(() => {
  console.log('Extension installed or updated.');
  main(); // start main()
});
async function gpt4allAPIXMLHttpRequest(){
const json_completion = JSON.stringify(
 {stream:false,
  temperature:0.6,
  max_tokens:100,
  messages:[{role:"user",content:"Hello."}],
  model: 'Nous Hermes 2 Mistral DPO'
 }
 );

const http = new XMLHttpRequest()
        http.open('POST', 'http://127.0.0.1:4891/v1/chat/completions')
        http.setRequestHeader('Content-type', 'text/plain')
        http.send(json_completion) // Make sure to stringify
        http.onload = function() {
            // Do whatever with response
            console.log(http.responseText)
        }
	
}
async function gpt4allAPIfetch(CORSmode){
console.log("call gpt4allAPI("+CORSmode+")");

const json_completion = JSON.stringify(
 {stream:false,
  temperatur:0.6,
  max_tokens:100,
  messages:[{role:"user",content:"Hello."}],
  model: 'Nous Hermes 2 Mistral DPO'
 }
 );


var options={
	keepalive: true,
	method: "POST",
 headers: {
    Accept: 'application/json',
 	'Content-Type': 'text/plain',

	},
	body:json_completion
	};
if ( CORSmode) options.mode = CORSmode;

 const completions = await fetch("http://127.0.0.1:4891/v1/chat/completions",options);
 console.log("API.js completions",completions);
 var completionjson=null; //var declaration 
 try{
		 completionjson = await completions.json();
		 console.log("API.js completionjson=",completionjson);
 }catch(err){console.log("switch null err=",err);}
}//end gpt4allAPIfetch()

async function  main(){
const models = await fetch("http://127.0.0.1:4891/v1/models");
const modelsData = await models.json();
console.log("modelsData",modelsData);

console.log("main gpt4allAPIXMLHttpRequest()");
try{await gpt4allAPIXMLHttpRequest();}catch(e){console.log(e);}

console.log("main null");
try{await gpt4allAPIfetch(null);}catch(e){console.log(e);}
console.log("main cors");
try{await gpt4allAPIfetch("cors");}catch(e){console.log(e);}
}//end main()

CORS Simple Request Fetch/XMLHttpRequest

[cosmic-snow]

• I am looking into what it would take to respond with OPTIONS here, but it's a bit tricky. The link in your OP (cors nikhilm/qhttpserver#63) is not the same as what's in use here. You can look at the Qt implementation here, and I think it's just a mirror on GitHub, can't interact there.

• What I meant was that it needs to be one of the "safe" accepted Content-Type headers -- but the server responds with an "unsafe" application/json anyway. So implementing that OPTIONS is what should be done on a proper server.

• Looks like you got something working at least? I'll still take some time to see if something can be done with the server.

[zwilch]

with your Ressource [2} specifying a "CORS Simple Request" I had success with Fetch and with XMLHttpRequest.
I have editing it above within a working Firefox Extension Example.
You see no OPTION is send by Browser Firefox more.
You see the POST get "OK 200"
In Console you can see the fetch and XMLHttpRequest answhere from server.

With this Example to implementing CORS Simple Request we should be able to program Firefox/Thunderbird Extension to use GPT4ALL locally together with this Applications (FF/Thunderbird).

Thank you so much, may it implements in future also the CORS Preflight, so no developer does strugling with the unsuccessfully CORS Preflight.

[cosmic-snow]

I have not tested the following thoroughly, but it might be enough to make a browser's preflight request happy. I don't have the time to do more tests right now, though, so I'll leave this here for the moment:

    // server.cpp: Server::start()
    m_server->route("<arg>", QHttpServerRequest::Method::Options, // <arg> with QUrl is basically a wildcard match
        [](const QUrl &anyPath, const QHttpServerRequest &request, QHttpServerResponder &&responder) {
            auto headers = request.headers();
            // reuse and allow request's "Origin" and "Access-Control-Request-Headers" headers:
            QByteArray allowOrigin = "";
            QByteArray allowRequestHeaders = "";
            for (auto const& header: headers) {
                if (header.first == "Origin") {
                    allowOrigin = header.second;
                } else if (header.first == "Access-Control-Request-Headers") {
                    allowRequestHeaders = header.second;
                }
            }
            // send OPTIONS preflight response for CORS:
            responder.write(
                "", // empty body, only headers matter
                QHttpServerResponder::HeaderList {
                    std::make_pair("Connection", "Keep-Alive"),
                    std::make_pair("Access-Control-Allow-Origin", allowOrigin),
                    std::make_pair("Access-Control-Allow-Methods", "GET, POST, OPTIONS"),
                    std::make_pair("Access-Control-Allow-Headers", allowRequestHeaders),
                    std::make_pair("Access-Control-Max-Age", "86400"),
                },
                QHttpServerResponder::StatusCode::NoContent
            );
        }
    );

[iimez]

@zwilch Please try using

"host_permissions":[
  "http://127.0.0.1/"
]

like this and see if that allows you to avoid CORS entirely.
Port numbers are not allowed in host_permissions. I'm suspecting the extension is not picking up the setting correctly because the pattern is malformed.

see
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Match_patterns#sect2
and https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/host_permissions#format

[zwilch]

@iimez I have changed manifest.json to this:

@zwilch Please try using

"host_permissions":[
  "http://127.0.0.1/"
]

see
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Match_patterns#sect2
and
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/host_permissions#format

You are correct, this works perfectly! All POST requests from the browser now go through without the OPTION CORS preflight request before. This was a misconfiguration in "host_permissions" with no error related to manifest.json.

Thank you very much for this hint.