Skip to content

#510 Remove plugin exports and update docs #511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Sep 21, 2022
Merged

#510 Remove plugin exports and update docs #511

merged 3 commits into from
Sep 21, 2022

Conversation

ryanolee
Copy link
Contributor

What

Removing plugin exports as discussed in #510

Why

So that plugin code is not too tightly coupled to custom plugin implementations

How

Remove plugin exports and updated documentation

Notes

Note: This should not be merged until V6

@ryanolee
Copy link
Contributor Author

Will fix the tests relating to this later today / tomorrow (time permitting 😆)

s0ph1e
s0ph1e requested changes Sep 20, 2022
View reviewed changes
Copy link
Member

@s0ph1e s0ph1e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ryanolee

Thank you for creating the PR. Please fix the failing tests and update the readme according to the comments

README.md Outdated
* [saveResource](#saveresource)
* [generateFilename](#generatefilename)
* [getReference](#getreference)
- [website-scraper](#website-scraper)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are too many changes here, could you please leave only plugins-related items?

README.md Outdated
- [Log and debug](#log-and-debug)

##### Built-in plugins
Scraper has built-in plugins which are used by default if not overwritten with custom plugins. You can find them in [lib/plugins](https://github.com/website-scraper/node-website-scraper/tree/master/lib/plugins) directory or get them using
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the documentation also needs an update here

@ryanolee ryanolee force-pushed the feature/RemovePluginExport branch from 2c45544 to b2aee75 Compare September 20, 2022 22:04
@ryanolee ryanolee marked this pull request as ready for review September 20, 2022 22:09
@ryanolee
Copy link
Contributor Author

ryanolee commented Sep 20, 2022 •
edited
Loading

Hi @s0ph1e ,
Thanks for the reply! I think the extra changes the readme files came from one of my IDE extensions that annoyingly seems to run on pre commit. Removed the change for now and fixed up the tests / docs. (Last test seems to have failed due to rate limits) Not sure what we want to do with this PR as it probably does not need to go out until v6 of the scraper. 👀

@s0ph1e
Copy link
Member

s0ph1e commented Sep 21, 2022

Hey @ryanolee,

Thank you for the quick changes. I'm going to merge the PR to master (master stands for current WIP for the next version, v5 code will be in branch 5.x)

@s0ph1e s0ph1e merged commit de75674 into website-scraper:master Sep 21, 2022
yamanyadav8810 added a commit to yamanyadav8810/node-website-scraper that referenced this pull request Feb 14, 2023
@yamanyadav8810
Squashed commit of the following:
8aff1bc 
commit 65b8056
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Dec 6 01:50:13 2022 +0200

    Bump fs-extra from 10.1.0 to 11.1.0 (website-scraper#523)

    Bumps [fs-extra](https://github.com/jprichardson/node-fs-extra) from 10.1.0 to 11.1.0.
    - [Release notes](https://github.com/jprichardson/node-fs-extra/releases)
    - [Changelog](https://github.com/jprichardson/node-fs-extra/blob/master/CHANGELOG.md)
    - [Commits](jprichardson/node-fs-extra@10.1.0...11.1.0)

    ---
    updated-dependencies:
    - dependency-name: fs-extra
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 52a341a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Dec 5 20:36:07 2022 +0000

    Bump sinon from 14.0.2 to 15.0.0 (website-scraper#524)

commit faa8bab
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Thu Nov 17 00:11:58 2022 +0200

    Update stale.yml

commit 5886558
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Wed Nov 16 23:24:22 2022 +0200

    Update stale.yml

commit 2797213
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 21:09:28 2022 +0000

    Bump actions/setup-node from 2 to 3 (website-scraper#519)

commit d9d147a
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Wed Nov 16 23:06:02 2022 +0200

    Delete no-response.yml

commit 7b6cbb5
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 20:54:40 2022 +0000

    Bump paambaati/codeclimate-action from 3.0.0 to 3.2.0 (website-scraper#520)

commit 28d6b40
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Wed Nov 16 22:40:53 2022 +0200

    Delete codeql-analysis.yml

commit 145f31d
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Wed Nov 16 22:40:44 2022 +0200

    Create codeql.yml

commit 4237388
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 17:15:31 2022 +0000

    Bump normalize-url from 7.2.0 to 8.0.0 (website-scraper#517)

commit 9dc9ef6
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Wed Nov 16 19:04:39 2022 +0200

    Auto-update of Github Actions

commit b169e86
Author: Sofiia Antypenko <sophia@antipenko.pp.ua>
Date:   Sun Oct 9 09:56:22 2022 +0200

    Fix ENOENT when running from working directory without package.json (website-scraper#516)

commit de75674
Author: ryanolee <landgraabiv@gmail.com>
Date:   Wed Sep 21 21:28:24 2022 +0100

    Remove plugin exports and update docs (website-scraper#511), closes website-scraper#510

    * chore(website-scraper#510): Remove plugin exports and update docs

    * core(website-scraper#510) Update messaging on internal plugins

    Co-authored-by: ryanolee <rizza@rizza.net>

commit 6c5cfc5
Author: Sofiia Antypenko <sophia@antipenko.pp.ua>
Date:   Thu Sep 15 21:55:48 2022 +0200

    Revert "RFC: Refactor builtin plugins to expose class properties (website-scraper#506)" (website-scraper#509)

    This reverts commit 5dde8ec.

commit b845fb0
Author: Sofiia Antypenko <sophia@antipenko.pp.ua>
Date:   Mon Sep 12 10:01:34 2022 +0200

    Remove lodash from dependencies (website-scraper#508)

commit 5dde8ec
Author: ryanolee <landgraabiv@gmail.com>
Date:   Mon Sep 12 08:55:31 2022 +0100

    RFC: Refactor builtin plugins to expose class properties (website-scraper#506)

    * core: Refactor builtin plugins to expose class properties

    * chore: EsLint

    Co-authored-by: ryan lee <rol@bluetel.co.uk>

commit c6ff84e
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Thu Sep 1 11:55:54 2022 +0300

    Remove demo link from the README

commit 8d6b5aa
Author: s0ph1e <sophia@antipenko.pp.ua>
Date:   Tue Aug 30 12:23:56 2022 +0200

    5.3.0

commit 6bb4e20
Author: Sophia Antipenko <sophia@antipenko.pp.ua>
Date:   Mon Aug 29 21:56:22 2022 +0200

    Use encoding from resource text (website-scraper#504)

commit 07c4c02
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Sat Jul 2 00:36:57 2022 +0300

    Use codeclimate project id instead of secret

commit 525dc50
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jul 1 09:20:14 2022 +0200

    Bump cheerio from 1.0.0-rc.11 to 1.0.0-rc.12 (website-scraper#499)

    Bumps [cheerio](https://github.com/cheeriojs/cheerio) from 1.0.0-rc.11 to 1.0.0-rc.12.
    - [Release notes](https://github.com/cheeriojs/cheerio/releases)
    - [Changelog](https://github.com/cheeriojs/cheerio/blob/main/History.md)
    - [Commits](cheeriojs/cheerio@v1.0.0-rc.11...v1.0.0-rc.12)

    ---
    updated-dependencies:
    - dependency-name: cheerio
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 07e6858
Author: Sophia Antipenko <sophia@antipenko.pp.ua>
Date:   Thu Jun 23 08:57:14 2022 +0200

    Bump version 5.2.0

commit e8c6f9d
Author: Paul Hawxby <phawxby@gmail.com>
Date:   Thu Jun 23 07:24:45 2022 +0100

    fix: non-english char encoding, close website-scraper#454, close website-scraper#466 (website-scraper#496)

    * fix: non-english char encoding

    * fix: optional chaining not supported

    * fix: various tests

    not finished yet

    * fix: tests now pass

    * fix: testing and code complexity

    * fix: code climate

    * fix: change comments

    * fix: simplify code complexity

    * refactor: simplify logic and add more testing

    * fix: reduce cognitive complexity

    * fix: more cognitive complexity

    * chore: damn space

    * fix: broke a test

    * chore: update docs

    * test: re-enable utf8 test

    * chore: update request test

commit 3bd2555
Author: Sophia Antipenko <sophia@antipenko.pp.ua>
Date:   Sat Jun 11 14:43:32 2022 +0200

    Revert encoding fix website-scraper#482 (website-scraper#495)

commit c6f60b8
Author: Sophia Antipenko <sophia@antipenko.pp.ua>
Date:   Thu Jun 9 14:18:08 2022 +0200

    Remove link to gitter from CONTRIBUTING.md

commit 087d943
Author: Sophia Antipenko <sophia@antipenko.pp.ua>
Date:   Thu Jun 9 14:07:10 2022 +0200

    Bump version 5.1.0

commit 5a58f48
Author: Illia Antypenko <ilya@antipenko.pp.ua>
Date:   Thu Jun 9 14:04:51 2022 +0200

    Fix encoding issue for non-English websites, closes website-scraper#454 website-scraper#466 (website-scraper#482)

commit d80e9b0
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat May 28 08:57:53 2022 +0200

    Bump cheerio from 1.0.0-rc.10 to 1.0.0-rc.11 (website-scraper#491)

    Bumps [cheerio](https://github.com/cheeriojs/cheerio) from 1.0.0-rc.10 to 1.0.0-rc.11.
    - [Release notes](https://github.com/cheeriojs/cheerio/releases)
    - [Changelog](https://github.com/cheeriojs/cheerio/blob/main/History.md)
    - [Commits](cheeriojs/cheerio@v1.0.0-rc.10...v1.0.0-rc.11)

    ---
    updated-dependencies:
    - dependency-name: cheerio
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 9397b07
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue May 10 10:58:35 2022 +0200

    Bump sinon from 13.0.2 to 14.0.0 (website-scraper#488)

    Bumps [sinon](https://github.com/sinonjs/sinon) from 13.0.2 to 14.0.0.
    - [Release notes](https://github.com/sinonjs/sinon/releases)
    - [Changelog](https://github.com/sinonjs/sinon/blob/main/docs/changelog.md)
    - [Commits](sinonjs/sinon@v13.0.2...v14.0.0)

    ---
    updated-dependencies:
    - dependency-name: sinon
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 66b1e49
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue May 3 14:49:33 2022 +0200

    Bump mocha from 9.2.2 to 10.0.0 (website-scraper#486)

    Bumps [mocha](https://github.com/mochajs/mocha) from 9.2.2 to 10.0.0.
    - [Release notes](https://github.com/mochajs/mocha/releases)
    - [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
    - [Commits](mochajs/mocha@v9.2.2...v10.0.0)

    ---
    updated-dependencies:
    - dependency-name: mocha
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 848ca1a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Feb 6 19:26:26 2022 +0200

    Bump sinon from 12.0.1 to 13.0.0 (website-scraper#483)

    Bumps [sinon](https://github.com/sinonjs/sinon) from 12.0.1 to 13.0.0.
    - [Release notes](https://github.com/sinonjs/sinon/releases)
    - [Changelog](https://github.com/sinonjs/sinon/blob/master/docs/changelog.md)
    - [Commits](sinonjs/sinon@v12.0.1...v13.0.0)

    ---
    updated-dependencies:
    - dependency-name: sinon
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@s0ph1e s0ph1e s0ph1e approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ryanolee @s0ph1e