This tool uses its own parsing of the HTML into JSON, then this package scrolls through it to find the desired information.
Read this for more information. Or visit this website for the API types
Copyright © 2024 DisQada
This tool is licensed under the Apache 2.0 License.
See the LICENSE file for more information.
The package breaks down the HTML into JSON nodes, following the concept that an HTML tag is represented as a JSON object/node.
Each node represents an HTML tag with it's tag name, attributes and children nodes. The children nodes may contain text as string values
Note that the parser completely ignores HTML comments
(<!--example-->)
First and most importantly, we need to have our HTML string ready in a variable.
If you're going to use the same HTML string for multiple uses, it's better to parse it yourself and then pass the JSON output to the functions (so the HTML string will be parsed only once). The following example shows how to do this:
import { parse } from '@disqada/scraper'
// Use another package to fetch the HTML from the web
const html = `
<html>
<head>
<title>Test Page</title>
</head>
<body>
<h1 id="title">Hello, world!</h1>
<p class="content">This is a test paragraph.</p>
</body>
</html>
`
const nodes = parse(html)
// Rest of the code ...import { findNode } from '@disqada/scraper'
const node = findNode(nodes, {
tag: 'h1',
attr: {
key: 'id',
value: 'title'
}
})
// node = {
// type: 'element',
// tagName: 'h1'
// attributes: [{key: 'id', value: 'title' }],
// children: []
// }import { grabText } from '@disqada/scraper'
const text = grabText(nodes, {
tag: 'p'
})
// text = 'This is a test paragraph.'The function grabText can be given a TextOptions object that specifies some configurations for the search process. Note that it's optional.
Click on the blue highlighted
TextOptionsto read more.
import { grabAttr } from '@disqada/scraper'
const attr = grabAttr(nodes, { tag: 'p' }, 'class')
// attr = 'content'You can download an HTML file and its parsed JSON file under the scrap folder in the root path of your project outside runtime by calling the download command CLI.
Note that either
--urlor--pathmust be given
| Arg name | required | Description |
|---|---|---|
--file |
true | Name of the downloaded HTML and parsed JSON file |
--url |
false | Link of the web page |
--path |
false | Path of a local HTML file (the HTML file will be copied to the scrap folder) |
scraper --url='https://example.com/sample' --file='sample'scraper --path='./samples/v1/index.html' --file='sample1'
