Skip to content

Commit

Permalink
馃帹 Initialize strategy system
Browse files Browse the repository at this point in the history
  • Loading branch information
KirianCaumes committed Nov 28, 2023
1 parent 4118d54 commit d7c70be
Show file tree
Hide file tree
Showing 7 changed files with 503 additions and 87 deletions.
5 changes: 4 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,8 @@
}
},
"typescript.preferences.importModuleSpecifier": "non-relative",
"javascript.preferences.importModuleSpecifier": "non-relative"
"javascript.preferences.importModuleSpecifier": "non-relative",
"markdownlint.config": {
"MD051": false
}
}
52 changes: 51 additions & 1 deletion README.MD
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,15 @@ Another (better ?) NodeJs library to fetch data from Discogs marketplace. 馃捒

## 馃搨 Installation (with npm)

This library works with two different `strategies` to scrape data: with a classic HTTP fetch request (Axios) or by creating a new instance of a browser (Playwright). Take a look at the "[About strategies](#馃-about-strategies)" chapter to learn more.

Run the following command in your project:

```sh
npm install discogs-marketplace-api-nodejs

# Or to avoid installing Playwright
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm install discogs-marketplace-api-nodejs
```

## 鈻讹笍 Quick Start
Expand Down Expand Up @@ -102,6 +107,31 @@ const result = await DiscogsMarketplace.search({
})
```

## 馃 About strategies

> Which `strategy` should I choose?
This project provides two ways to scrape data:

- `fetch` (default): Get the result with a classic HTTP fetch request (Axios).
- For the moment, this is the simplest and fastest solution to choose.
- `browser`: Get the result by creating a new instance of a browser (Playwright) to scrape the page.
- This can help bypass Cloudflare (or other) protection for some cases.
- Slightly slower than `fetch` (see [benchmarks](https://github.com/KirianCaumes/Discogs-Marketplace-API-NodeJS/releases/tag/v1.8.1))

**Both strategies will give you the exact same result.**

I choose to maintain this two solutions, because over time Discogs is changing: one day they activated a protection that prevented a `fetch` solution from working, so I implemented to `browser` solution. But a few days later, they disabled that protection and the original solution became the best again.

With both strategies, you are free to choose the one that suits you!

```js
const result = await DiscogsMarketplace.search({
// ...
strategy: 'fetch' // or 'browser'
})
```

## 馃搩 Data format

You can provide parameters to `DiscogsMarketplace.search` function according to this interface:
Expand Down Expand Up @@ -198,6 +228,12 @@ interface InputInterface {
* Default to `en`.
*/
lang?: LangType
/**
* Strategy to scrape data:
* - `fetch` (default): Get the result with a classic HTTP fetch request (Axios).
* - `browser`: Get the result by creating a new instance of a browser (Playwright) to scrape the page. This can help bypass Cloudflare (or other) protection for some cases.
*/
strategy?: 'fetch' | 'browser'
}
```

Expand Down Expand Up @@ -292,6 +328,20 @@ You can open a pull request with your new additions.

## 馃悰 Known issues/problems

### Linkedom

If you are using this library on a Typescript project, you might encounter issues with `linkedom`.

To fix it, use this in command line: `--skipLibCheck` or add that in you `tsconfig.json`:

```json
{
"compilerOptions": {
"skipLibCheck": true
}
}
```

### Playwright

Install system dependencies:
Expand All @@ -302,4 +352,4 @@ sudo npx playwright install-deps chromium

More information [here](https://playwright.dev/docs/browsers#install-system-dependencies).

If you found another problem, feel free to open an issue.
馃憠 If you find another problem, feel free to open an issue.
Loading

0 comments on commit d7c70be

Please sign in to comment.