Remove request Migrate to superagent && Fix CI

[confused-Techie]

This PR intends to fully remove request from the codebase, as it's deprecated, and no longer receiving updates.

This starts by removing request from ./src/request.js which is the file all others use to preform any network requests, afterwards hopefully removing request from the downloading node scripts should prove much easier. But as tests can't run on Windows yet, I'll have to rely on CI to determine success in these changes, other than manual testing.

[confused-Techie]

We really need to figure out our testing situation. I was initially excited seeing green CI, but on closer inspection, the amount of passed tests shown as dots, does not match the amount we expect to test. Nor is there any kind of summary for how many tests have passed, like there should be. These tests are failing somewhere, and with our testing situation eating any output, there's no way to know where or how many. Most worrisome of all, this is reported as passed tests. But something is broken.

[confused-Techie]

So on this, I've done a lot of investigating what's going on with our tests. And I hate to say, I think it's a jasmine-focused issue, otherwise something we can't fix in this repo. I'm seeing lots of reports from jasmine-node (Which jasmine-focused uses under the hood) of it failing to capture any stacktrace or errors when requests using request fail, which I'm assuming transfer to superagent as well, since it's intended to be the direct replacement.

But at the very least, I've found a way locally to discover what's going on. Although a bit troublesome.

To get actually logs from these failed tests, you'll need to add the following snippet to the top of the file: ./node_modules/jasmine-node/lib/jasmine-node/cli.js With the snippet being:

process.on("uncaughtException", (err) => {
  fs.writeFileSync(`${__dirname}/log.txt`, JSON.stringify(err) + "\n", { encoding: "utf8", flag: "a+"});
  fs.writeFileSync(`${__dirname}/log.txt`, JSON.stringify(err.stack) + "\n", { encoding: "utf8", flag: "a+"});
});

Once you run your tests, and if they are failing without any output like seen here, a log.txt file will be created within ./node_modules/jasmine-node/lib/jasmine-node/log.txt where you can view the output of what's going on. It's a bit ugly, but quickly cleaning it up you can see what's failing.

Which for reference, in this current PR, here's the output:

This PR Output

TypeError [ERR_INVALID_ARG_TYPE] [ERR_INVALID_ARG_TYPE]: 
	The \"path\" argument must be of type string or an instance of Buffer or URL. 
	Received null\n    at readFile (node:fs:351:10)\n    at go$readFile 
	(D:\\Documents\\ppm\\node_modules\\npm\\node_modules\\graceful-fs\\graceful-fs.js:118:14)\n    
	at Proxy.readFile (D:\\Documents\\ppm\\node_modules\\npm\\node_modules\\graceful-fs\\graceful-fs.js:115:12)\n    
	at Object.readFile (D:\\Documents\\ppm\\node_modules\\season\\lib\\cson.js:195:17)\n    
	at D:\\Documents\\ppm\\src\\install.js:590:21\n    
	at iteratee (D:\\Documents\\ppm\\src\\install.js:621:53)\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:1959:13\n    
	at replenish (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:446:21)\n    
	at iterateeCallback (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:430:21)\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:327:20\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:1961:17\n    
	at D:\\Documents\\ppm\\src\\install.js:585:77\n    
	at wrapper (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:271:20)\n    
	at next (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:5795:24)\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:327:20\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:3681:19\n    
	at wrapper (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:271:20)\n    
	at replenish (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:441:29)\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:451:13\n    
	at eachOfLimit$1 (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:477:34)\n    
	at awaitable (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:211:32)\n    
	at eachOfSeries (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:813:16)\n    
	at awaitable (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:211:32)\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:3673:9\n    
	at awaitable (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:211:32)\n    
	at Object.series (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:4910:16)\n 
	at Install.installPackageDependencies (D:\\Documents\\ppm\\src\\install.js:387:20)\n    
	at D:\\Documents\\ppm\\src\\install.js:394:38\n    
	at nextTask (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:5789:27)\n    
	at next (D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:5797:13)\n    
	at D:\\Documents\\ppm\\node_modules\\async\\dist\\async.js:327:20\n    
	at Install.logCommandResults (D:\\Documents\\ppm\\src\\command.js:90:14)\n    
	at D:\\Documents\\ppm\\src\\install.js:181:23\n    
	at ChildProcess.onChildExit (D:\\Documents\\ppm\\src\\command.js:45:48)\n    
	at ChildProcess.emit (node:events:365:28)\n    at maybeClose (node:internal/child_process:1067:16)\n    
	at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)\n

[confused-Techie]

So on this one, as you can see CI was troublesome. I have found a way around the issue of the jasmine-focused straight up crashing reporting nothing as to what went wrong in the tests, and seems to be the case with many different exceptions that might occur. Since I saw them here and over on #88

So with that in mind I've added a fix to the CI here as well. While a bit verbose, it essentially preforms the duties that are reserved for the main file of jasmine-node. Meaning basically the whole main file has been copy pasted into our scripts. Which while normally would be very risky, I think is safe enough since jasmine-node hasn't seen in a commit in 4 years, so I'm not worried about that. But otherwise it does successfully give us output. So I'll leave those changes in here, since they were essentially to getting this PR up off the ground.

Once this is merged then we can go ahead and use these advances in CI to resolve the issues being seen by #88

[confused-Techie]

Also an edit, the single failing windows test is not the fault of this package. And is resolved in #86

[DeeDeeG]

Haven't gotten around to fully reviewing src/request.js.

Just had a few comments.

Hopefully makes the remaining review less to do.

[DeeDeeG]

Tests pass on my machine locally, and in CI, so I'm inclined to believe this doesn't break anything.

And i think it's really nice that src/request.js wraps the external dependency, so we can isolate the churn to there and have the rest of the code mostly unchanged.

I would love to get a chance to fully review all of the changes in src/requests.js, but if I never get around to that, it's at a "lite approve" 👍 from me, which is: I wish I could review it more thoroughly at the time of giving the "lite approve", but I feel somewhat responsible for the feature or change's future success, as a reviewer today, and would be very willing to assist in any troubleshooting, debugging, release management hoop jumping, etc. should there be any issue in the future.

And that I've done whatever due diligence I can do short of a full review of the code line by line to ensure this seems like it won't have show-stopper or large unintended fallouts from any issues.

I suppose I do want to mention there's nothing in the specs about proxies, at least not that I could find with a quick code search, so I hope that anyone who is familiar with using (HTTP, HTTPS) proxies can give that a manual kick in the tires, so to speak, give it a real-world test to make up for the lack of automated tests for it.

Yeah, I could live with this merging as-is. I am gearing up to try and review any more of it I can get to soon, but I don't want to delay it too much further on account of my review timeline.

P.S. this sort of assumes the review comments above are addressed or resolved. Sorry about those!

[confused-Techie]

@DeeDeeG Thanks for taking a look at this one. I know there's a lot of changes in here.

But as this change isn't really dependent on much, except maybe it's nice to have the CI changes available, overall this one can wait, so it'd be fine to wait until a proper review can occur. Otherwise if we want to merge as as, I'm also totally down for that as well.

But on your notes about proxies, worrying to know there's no testing for it. I will say, I'm not an expert in that field, so my changes interacting with proxies, is mostly straight out of docs for the services we use.

[DeeDeeG]

I never ended up properly reviewing this in ~2 weeks, but I would be comfortable merging as per my lite-approve/comment above.

That said, I'll point out Codacy has some actually interesting suggestions here and there, if you ignore all the whitespace-only suggestions and Codacy apparently preferring double-quote marks " for strings, rather than single-quote marks ', for who knows what reason (I prefer the single-quote marks myself???).

Anything else you'd want to see done with this? Or should we go for it and merge this?

(Also, sorry some recent PR's have made merge conflicts in the yarn.lock. I think yarn install can handle merge conflict markers and fix the conflict itself??)

[confused-Techie]

@DeeDeeG I appreciate you giving this PR some attention.

Although looking at Codacy, I'm personally not seeing anything super motivating unless I've just missed it.

Really seems we should probably get a codacy config file in this repo to start ignoring some of these pointless rules and warnings it's trying to enforce.

Beyond that, other than fixing the merge conflict I'm still confident in this PR, if we think it's good to go.

[DeeDeeG]

@confused-Techie the "var defined but never used" stuff seemed the most possibly interesting unless it was misinterpreting the point of those vars. (Ctrl + F "never used" on the Codacy, since there's so much whitespace and quote mark suggestions to ignore.)

And there's one about == vs ===, not that that's way important TBH.

But I honestly leave it to your discretion, since I should've brought these up sooner in review if they were important, this has been delayed quite a bit already, so it doesn't seem super fair to hold it up on tiny things like this.

[confused-Techie]

@DeeDeeG I've gone ahead and addressed really only one of the warnings.

The main reason being, is a few of them are coming from the bundled-node-version.js file, which is a direct copy of the jasmine test runner file we use.

And I'd really rather not mess with that in unknown ways tbh. So if we aren't too concerned about it, may be best to leave it alone.

But otherwise, I've gone ahead and resolved the conflicts, and updated the yarn.lock file.

But no worries about this one taking some time, it's not the most urgent PR. But hopefully we are good to go now, I really do appreciate you taking a look!

[DeeDeeG]

One of those "weird mysteries of CI". For only the NodeJS 18.x runs... The yarn install step says something weird about node-gyp...

info This package requires node-gyp, which is not currently installed. Yarn will attempt to automatically install it. If this fails, you can run "yarn global add node-gyp" to manually install it.

And it does install node-gyp (why does this even happen anyway?) and then... yarn doesn't run the repo's postinstall script???? For some reason?

So, there is no ./bin/node and the test step fails immediately after.

?????????????????????????

This doesn't happen for me locally. I have no idea what is going on with @actions/setup-node's stuff with latest NodeJS 18 right now. But this feels like very not related to this PR. I guess if I can reproduce the same CI failure on master branch, then it's definitely not this PR's fault and I'll be a re-approve again...

WTF GitHub Actions?

I don't even.

[confused-Techie]

@DeeDeeG Yeah this makes no sense. Unless it's some issue with the changes to node-gyp that's been added when I updated this branch from main.

But I would've expected we would see this on that original PR.

But tried rerunning, ensured we don't have any type of cache for our runners, but seems to consistently fail here.

But like you, can't get it to happen locally. Although, to be fair I'm not on Node 18 locally. So may need to install that and try again, but how weird.

[DeeDeeG]

Yeah, it happens on master branch if re-run, too. I suspect this will blow over without our doing anything in our repo.

FWIW, some Node+npm versions changed since 5 days ago when master branch CI last ran..., in terms of what @actions/setup-node is installing for Node 18.x...:

Before (working):

  node: v18.17.1
  npm: 9.6.7
  yarn: 1.22.19

After (buggy):

  node: v18.18.0
  npm: 9.8.1
  yarn: 1.22.19

I installed this version of node (v18.18.0) and npm (9.8.1) locally on macOS, and everything works fine. I seriously don't get it.

But the issue can reproduce on master branch, so it's not an issue with this PR. Just a weird CI thing. Hope it goes away on its own...

[DeeDeeG]

This is a bit of a "rubber stamp" Approve, since I didn't manage to review all of request.js and the jasmine changes in-detail, though I vaguely remember some time long ago I did confirm that the changes to jasmine are actually quite minimal... I think I did, anyway.

This PR has been open for quite a while and doing well in the repo's automated tests, so I'll give it an Approve and try to keep up with anything that might go wrong, for triage and issue resolution purposes.

CI failure is unrelated to this PR. It can be seen affecting a deliberate re-run of master branch as well. (https://github.com/pulsar-edit/ppm/actions/runs/6210843978) (I have no idea what caused this, other than it looks like @actions/setup-node's fault, maybe, and I hope it goes away on its own, since we didn't cause it. FWIW I have seen similar transient CI things like this before, I thin specifically with different @actions/setup-node version updates, and it just went away on its own those times. Weird stuff, but not our fault.)

Thanks for the efforts! Maybe this can even (incidentally) make debugging the test problems/errors on Windows a little less cryptic... Hopefully! As in, maybe we can actually see the output from the test suite on Windows now??? Worth checking... (Long-term goal: Get everything passing on Windows!)

Long way of saying "Approve".

[confused-Techie]

@DeeDeeG Thanks a ton for taking such a good look at this, and especially confirming this failure isn't the fault of this PR.

Suppose in the short term we could always disable testing on the newer node version, as it's meant really to warn us of things breaking, since we don't actually use anything other than 16 within Pulsar right now. Although, this does maybe indicate something breaking, but I'm very much hoping you are right, in that this will just go away on it's own.

But I'm very much on board with using these changes to see if we can finally enable, and fix Windows tests, as that would be quite awesome!

But with your approval, I'll happily merge this one, and hopefully soon take a look at running some Windows tests!