feat: HTTP request tool

[michael-radency]

Summary

Tool to visit a website

Related tickets and issues

https://linear.app/n8n/issue/AI-162/tool-to-visit-a-website

[netroy]

We could use @mozilla/readability and jsdom here to cleanly extract the content that's likely relevant to an end-user.

Something like this perhaps:

import { JSDOM } from 'jsdom'
import { Readability } from '@mozilla/readability'

const dom = await JSDOM.fromURL(url)
const article = new Readability(dom.window.document, {
    keepClasses: true,
}).parse()

and then use article.content.

we could also consider using turndown to convert the html into markdown, which LLM tend to handle better than html IMO.

import Turndown from 'turndown'
const markdown = turndown.turndown(article.content)

[michael-radency]

@netroy
what would be advantages over html-to-text + Cheerio? since we already using such setup for Html node

[netroy]

cheerio is great to either use css selectors to extract text, but leaves the burden of determining the semantics in the markup to the end user.

First download an article via curl https://www.bbc.com/news/articles/cldd6x6gglxo > news.html.

Then try this with cheerio:

const fs = require('fs');
const cheerio = require('cheerio');
const html = fs.readFileSync('news.html', 'utf8');
const $ = cheerio.load(html);
console.log($('body').text());

I got this

With just readability:

(async () => {
	const { JSDOM } = require('jsdom');
	const { Readability } = require('@mozilla/readability');
	const Turndown = require('turndown');

	const dom = await JSDOM.fromFile('news.html');
	const article = new Readability(dom.window.document, {
		keepClasses: true,
	}).parse();
	console.log(article.textContent);
})();

I got this

With readability + turndown:

(async () => {
	const { JSDOM } = require('jsdom');
	const { Readability } = require('@mozilla/readability');
	const Turndown = require('turndown');

	const dom = await JSDOM.fromFile('news.html');
	const article = new Readability(dom.window.document, {
		keepClasses: true,
	}).parse();
	const turndown = new Turndown({
		headingStyle: 'atx',
		hr: '---',
		bulletListMarker: '-',
		codeBlockStyle: 'fenced',
	});
	const markdown = turndown.turndown(article.content);
	console.log(markdown);
})();

I got this

Perhaps we should add a "Extract as Markdown" option in the node to determine if we want to use markup to reduce semantic noise in the extracted text?