v4.0.0 #198

ghost · 2019-04-15T08:28:42Z

Supersedes #196

Add tests

Adds lib/browsers.json - contains UA and exact header information
Adds lib/headers.js - loads a random UA from lib/browsers.json
Replaces all occurrences of var with either const or let
Tiny regex modification to slightly improve document.getElementById
Adds optional support for Brotli and headers are adapted depending on whether or not it's available
Adds lib/brotli.js for the optional support (node's or user installed version)
Makes both request and brotli peer dependencies which must be installed by the user.
Moves sandbox(vm) related code to lib/sandbox.js and increases testability
Adds support for solving reCAPTCHA(v2)
Renames functions e.g. processsResponseBody -> onRequestComplete
Moves request options validation to it's own function: validateRequest

If a user attempts to run cloudscraper without first installing request, they will receive the following error message:

###
### The "request" library is not installed automatically anymore.
### But is a dependency of "request-promise".
### Please install it with:
### npm install request --save
###

{ Error: Cannot find module 'request'
<snipped>

The error message is automatically produced by request-promise.
Closes #147
Closes #62

codemanki · 2019-04-16T07:09:21Z

Just got back from a short vacation and so many things happened here O.o I'm gonna review all the changes asap

ghost · 2019-04-16T09:22:24Z

@codemanki You might want to hold off until I push these tests

ghost · 2019-04-16T11:03:11Z

The code coverage is sane again but not quite 100%
I'll cover the rest of it very soon. Feel free to review now.

index.js

ghost · 2019-04-18T07:55:03Z

Hey, I've changed my mind. I'm not going to add any extra additional tests for the time being but the PR does LGTM. I can address any issues if you see any.

lib/sandbox.js

ghost · 2019-04-18T08:22:00Z

Also to clarify, Brotli is a completely optional dependency but if anybody intends on having bulletproof headers, they should install it.

ghost · 2019-04-18T08:22:44Z

@VeNoMouS you might want to review that last comment I made.

ghost · 2019-04-18T08:27:16Z

One last thing to mention is that there is some percentage of the user base that is facing issues that this PR resolves by beating Cloudflare's updated BIC checks.

codemanki · 2019-04-21T09:54:34Z

There are no breaking changes, right?

ghost · 2019-04-21T09:56:51Z

No breaking changes other than what I noted above... request is now a peer dependency which the user must install themselves.

ghost · 2019-04-21T09:58:52Z

TBH, it's sitting here when it works better than the master branch. I consider the current release to be broken.

codemanki · 2019-04-21T09:59:33Z

I will release it today

…

On Sun, Apr 21, 2019, 11:58 Dwayne ***@***.***> wrote: TBH, it's sitting here when it works better than the master branch. I consider the current release to be broken. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#198 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACK6QEYBAREW6S4A5F4IL3PRQ3FZANCNFSM4HF5UTBA> .

ghost · 2019-04-21T10:03:52Z

@codemanki Before you do, check the resolved messages, they're not all resolved. I'm too busy to fix thoses but even without them being fixed...Still better than current release.

codemanki · 2019-04-21T13:22:09Z

@pro-src could you please change the target branch from master to https://github.com/codemanki/cloudscraper/tree/4.0.0 ?

ghost · 2019-04-21T13:24:30Z

Done.

index.js

codemanki · 2019-04-21T13:34:09Z

index.js

  }

  stringBody = body.toString('utf8');

  try {
    validate(options, response, stringBody);
  } catch (error) {
+    if (error instanceof CaptchaError && typeof options.onCaptcha === 'function') {


We need to document this

I would add a few examples as a part of that documentation. See the relevant issue for examples.

codemanki · 2019-04-21T13:43:57Z

@pro-src i will then merge this PR, will add my changes to the branch and will create a new PR. Otherwise I can't add my changes to your branch

codemanki · 2019-04-21T13:47:53Z

@pro-src Oh i see. It is indeed possible - https://help.github.com/en/articles/committing-changes-to-a-pull-request-branch-created-from-a-fork

ghost · 2019-04-21T13:56:00Z

AFAIK you should be able to change the target branch back to master should you need to and change the title, etc. etc. 😉

codemanki · 2019-04-21T14:03:10Z

@pro-src is it safe? :D Feels like disarming a bomb

ghost · 2019-04-21T14:04:58Z

Yes, it is safe. 💣

codemanki · 2019-04-22T15:05:50Z

I guess it is ready to be released. @pro-src do you want to take a quick look?

TravisBuddy · 2019-04-22T15:08:25Z

Travis tests have failed

Hey @pro-src,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

Node.js: node

View build log

npm test

> cloudscraper@4.0.0 test /home/travis/build/codemanki/cloudscraper
> npm run lint && nyc --reporter=html --reporter=text mocha


> cloudscraper@4.0.0 lint /home/travis/build/codemanki/cloudscraper
> eslint --ext .json --ext .js .


/home/travis/build/codemanki/cloudscraper/examples/solve-recaptcha.js
   3:24   error  Missing space before function parentheses      space-before-function-paren
   4:3    error  Expected space or tab after '//' in comment    spaced-comment
  18:1    error  Expected catch() or return                     promise/catch-or-return
  18:49   error  Strings must use singlequote                   quotes
  18:103  error  Newline required at end of file but not found  eol-last
  18:103  error  Missing semicolon                              semi

✖ 6 problems (6 errors, 0 warnings)
  5 errors and 0 warnings potentially fixable with the `--fix` option.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! cloudscraper@4.0.0 lint: `eslint --ext .json --ext .js .`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the cloudscraper@4.0.0 lint script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/travis/.npm/_logs/2019-04-22T15_16_02_639Z-debug.log
npm ERR! Test failed.  See above for more details.

Node.js: 6

View build log

npm test

> cloudscraper@4.0.0 test /home/travis/build/codemanki/cloudscraper
> npm run lint && nyc --reporter=html --reporter=text mocha


> cloudscraper@4.0.0 lint /home/travis/build/codemanki/cloudscraper
> eslint --ext .json --ext .js .


/home/travis/build/codemanki/cloudscraper/examples/solve-recaptcha.js
   3:24   error  Missing space before function parentheses      space-before-function-paren
   4:3    error  Expected space or tab after '//' in comment    spaced-comment
  18:1    error  Expected catch() or return                     promise/catch-or-return
  18:49   error  Strings must use singlequote                   quotes
  18:103  error  Newline required at end of file but not found  eol-last
  18:103  error  Missing semicolon                              semi

✖ 6 problems (6 errors, 0 warnings)
  5 errors and 0 warnings potentially fixable with the `--fix` option.


npm ERR! Linux 4.4.0-101-generic
npm ERR! argv "/home/travis/.nvm/versions/node/v6.17.1/bin/node" "/home/travis/.nvm/versions/node/v6.17.1/bin/npm" "run" "lint"
npm ERR! node v6.17.1
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! cloudscraper@4.0.0 lint: `eslint --ext .json --ext .js .`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the cloudscraper@4.0.0 lint script 'eslint --ext .json --ext .js .'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the cloudscraper package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     eslint --ext .json --ext .js .
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs cloudscraper
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls cloudscraper
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/travis/build/codemanki/cloudscraper/npm-debug.log
npm ERR! Test failed.  See above for more details.

TravisBuddy Request Identifier: b9d627a0-6511-11e9-a049-ff6f6c17598f

ghost · 2019-04-22T15:10:21Z

Probably should make a dummy solveCaptcha function so the linter doesn't complain or you could just modify that NPM script to exclude the examples directory.

I usually do the following before committing: npm run lint -- --fix

LGTM 👍

codemanki · 2019-04-22T15:21:01Z

Here we go... Thank you @pro-src for this PR :)

ghost mentioned this pull request Apr 15, 2019

(WIP) v4.0.0 #196

Closed

pro-src added 7 commits April 15, 2019 18:06

Fix UA randomization and header order

876ebe4

Let & Const

0462147

Adjust document.getElementById to allow empty nodes

3930c91

Add optional Brotli support

e32c285

Move request to peerDependencies

7e333be

Support reCAPTCHA solving

14eb381

Add missing use strict statements

12bec43

ghost mentioned this pull request Apr 15, 2019

CaptchaError: captcha on v3.9.1 #195

Closed

Add test/fixture for latest challenge

cc1313a

Add tests for v4

12809b3

ghost commented Apr 18, 2019

View reviewed changes

index.js Show resolved Hide resolved

ghost commented Apr 18, 2019

View reviewed changes

lib/sandbox.js Show resolved Hide resolved

This was referenced Apr 18, 2019

Remove request module dependency in v4 #147

Closed

Add support for recaptcha #62

Closed

Revadike approved these changes Apr 19, 2019

View reviewed changes

ghost changed the base branch from master to 4.0.0 April 21, 2019 13:23

codemanki reviewed Apr 21, 2019

View reviewed changes

index.js Show resolved Hide resolved

codemanki reviewed Apr 21, 2019

View reviewed changes

codemanki mentioned this pull request Apr 21, 2019

Update docs, add deprecation warning for 3.7.0 #187

Closed

Test if i can commit

f659287

codemanki changed the base branch from 4.0.0 to master April 21, 2019 14:12

codemanki added 3 commits April 22, 2019 11:29

Update docs and index.js timeout parsing

16c2659

Mention recaptcha

45ba695

Update package.json

d2a047c

codemanki added 2 commits April 22, 2019 17:14

Adding dummy solveReCAPTCHA method

2831e5c

Fix lint issues with recaptcha example

77e8929

codemanki merged commit 79760a7 into codemanki:master Apr 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v4.0.0 #198

v4.0.0 #198

ghost commented Apr 15, 2019 •

edited by ghost

Loading

codemanki commented Apr 16, 2019

ghost commented Apr 16, 2019

ghost commented Apr 16, 2019

ghost commented Apr 18, 2019 •

edited by ghost

Loading

ghost commented Apr 18, 2019

ghost commented Apr 18, 2019

ghost commented Apr 18, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019 •

edited by ghost

Loading

ghost commented Apr 21, 2019

codemanki commented Apr 21, 2019 via email

ghost commented Apr 21, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019

codemanki Apr 21, 2019

ghost Apr 21, 2019

codemanki commented Apr 21, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019

codemanki commented Apr 22, 2019

TravisBuddy commented Apr 22, 2019 •

edited

Loading

ghost commented Apr 22, 2019 •

edited by ghost

Loading

codemanki commented Apr 22, 2019

v4.0.0 #198

v4.0.0 #198

Conversation

ghost commented Apr 15, 2019 • edited by ghost Loading

codemanki commented Apr 16, 2019

ghost commented Apr 16, 2019

ghost commented Apr 16, 2019

ghost commented Apr 18, 2019 • edited by ghost Loading

ghost commented Apr 18, 2019

ghost commented Apr 18, 2019

ghost commented Apr 18, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019 • edited by ghost Loading

ghost commented Apr 21, 2019

codemanki commented Apr 21, 2019 via email

ghost commented Apr 21, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019

codemanki Apr 21, 2019

Choose a reason for hiding this comment

ghost Apr 21, 2019

Choose a reason for hiding this comment

codemanki commented Apr 21, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019

codemanki commented Apr 21, 2019

ghost commented Apr 21, 2019

codemanki commented Apr 22, 2019

TravisBuddy commented Apr 22, 2019 • edited Loading

Travis tests have failed

Node.js: node

Node.js: 6

TravisBuddy Request Identifier: b9d627a0-6511-11e9-a049-ff6f6c17598f

ghost commented Apr 22, 2019 • edited by ghost Loading

codemanki commented Apr 22, 2019

ghost commented Apr 15, 2019 •

edited by ghost

Loading

ghost commented Apr 18, 2019 •

edited by ghost

Loading

ghost commented Apr 21, 2019 •

edited by ghost

Loading

TravisBuddy commented Apr 22, 2019 •

edited

Loading

ghost commented Apr 22, 2019 •

edited by ghost

Loading