Web fetch

Asset Crawler for common Web pages

Usage

As a Node.js package

import { savePage } from 'web-fetch';

savePage({
    source: 'http://URL.to/one/of/your/old/posts/'
});

Docker example

https://github.com/kaiyuanshe/KYS-service

As a Command Line tool

web-fetch http://URL.to/one/of/your/old/posts/

In GitHub actions

name: Fetch Web pages
on:
    issues:
        types:
            - opened
jobs:
    Fetch-and-Save:
        runs-on: ubuntu-latest
        permissions:
            contents: write
            issues: write
        steps:
            - uses: actions/checkout@v3

            - uses: pnpm/action-setup@v2
              with:
                  version: 9
            - uses: actions/setup-node@v3
              with:
                  node-version: 18
                  cache: pnpm
            - uses: browser-actions/setup-chrome@v1

            - name: Setup Web-fetch
              run: pnpm i web-fetch -g

            - name: Fetch first URL in Issue Body
              run: web-fetch $(echo "${{ github.event.issue.Body }}" | grep -Eo "https?://\S+")

            - uses: stefanzweifel/git-auto-commit-action@v5
              with:
                  commit_message: '${{ github.event.issue.title }}'

Supported Structure

Web-fetch/source/config.ts

Lines 10 to 29 in d4c574a

	export const body_tag = [
	'article',
	'main',
	'.article',
	'.content',
	'.post',
	'.blog',
	'.main',
	'.container',
	'body'
	];
	export const meta_tag = {
	title: ['h1', 'h2', '.title', 'title'].map(likeOf),
	authors: ['.author', '.publisher', '.creator', '.editor'].map(likeOf),
	date: ['.date', '.time', '.publish', '.create'].map(likeOf),
	updated: ['.update', '.edit', '.modif'].map(likeOf),
	categories: ['.breadcrumb', '.categor'].map(likeOf),
	tags: ['.tag', '.label'].map(likeOf)
	};

Renderer

1. Puppeteer (default)

2. JSDOM

Wrapper

1. Hexo plugin