Skip to content

chenyanming/shrface

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
img
 
 
 
 
 
 
 
 
 
 
 
 

SHRFACE

https://melpa.org/packages/shrface-badge.svg

Table of Contents

Description

This package extends shr / eww with org features and analysis capability. It can be used in dash-docs, eww, nov.el, mu/mu4e, anki.el, etc. It is also able to export HTML buffer to Org buffer/file, or download/archive web pages.

img/eww.png

  • Configurable org-like heading faces, headline bullets, item bullets,paragraph indentation, fill-column, item bullet, versatile hyper links(http/https/file/mailto/etc) face and so on.
  • Browse the internet or local html file with eww just like org mode.
  • Read dash docsets with dash-docs and the beauty of org faces.
  • Read epub files with nov.el , just like org mode.
  • Read html email with mu/mu4e , the same reading experience just like org mode without formatting html to org file.
  • Switch/jump the headlines just like org-mode in eww and nov.el with imenu
  • Toggle/cycle the headlines just like org-mode in eww and nov.el with outline minor mode , org-cycle/org-shifttab, or shrface-outline-cycle / shrface-outline-cycle-buffer.
  • Enable indentation just like org-mode in eww and nov.el with org-indent-mode
  • Analysis capability:
    • Headline analysis: List all headlines with clickable texts, or jump around with ivy.
    • URL analysis: List all classified URL with clickable texts in a buffer, or jump around with ivy.
  • Export HTML buffer to org buffer/file using shr engine (no Pandoc is needed).

The project target is to apply org features and analysis capability to shr, and all libraries that render HTML with shr (Simple HTML Renderer).

As EmacsWiki says:

shr.el is an HTML renderer in Emacs as of version 24.4 (based on libxml2, it was originally a part of Gnus). It’s the basis of the web browser eww.

Installation

It’s available on Melpa :

M-x package-install shrface

Configuration

Quick Start

use-package

I recommend configure with use-package, it is more elegant:

(use-package shrface
  :defer t
  :config
  (shrface-basic)
  (shrface-trial)
  (shrface-default-keybindings) ; setup default keybindings
  (setq shrface-href-versatile t))

(use-package eww
  :defer t
  :init
  (add-hook 'eww-after-render-hook #'shrface-mode)
  :config
  (require 'shrface))

(use-package nov
  :defer t
  :init
  (add-hook 'nov-mode-hook #'shrface-mode)
  :config
  (require 'shrface)
  (setq nov-shr-rendering-functions '((img . nov-render-img) (title . nov-render-title)))
  (setq nov-shr-rendering-functions (append nov-shr-rendering-functions shr-external-rendering-functions)))

(use-package anki
  :defer t
  :load-path "~/.emacs.d/lisp/anki/"
  :init
  (add-hook 'anki-mode-hook #'shrface-mode)
  (autoload 'anki "anki")
  (autoload 'anki-browser "anki")
  (autoload 'anki-list-decks "anki")
  :config
  (require 'shrface)
  (setq anki-shr-rendering-functions (append anki-shr-rendering-functions shr-external-rendering-functions))
  (setq sql-sqlite-program "/usr/bin/sqlite3")
  (setq anki-collection-dir "/Users/chandamon/Library/Application Support/Anki2/User 1"))

Font setup

The default font is variable-pitch.

(set-face-attribute 'variable-pitch nil :font (format "%s:pixelsize=%d" "iA Writer Quattro S" 15))

Keybinding Settings

You can to call shrface-default-keybindings to enable the recommended keybindings.

(defun shrface-default-keybindings ()
  (interactive)
  "Sets up the default keybindings for `shrface-mode'."
  (define-key shrface-mode-map (kbd "TAB") 'shrface-outline-cycle)
  (define-key shrface-mode-map (kbd "<backtab>") 'shrface-outline-cycle-buffer)
  (define-key shrface-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key shrface-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key shrface-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key shrface-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key shrface-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

Since the keybindings may conflict with other modes, such as nov-mode-map, eww-mode-map, mu4e-mode-map. If you want to enable shrface’s keybindings on these modes, you have to bind the functions to those maps as well.

Here is the keybinding example:

(with-eval-after-load 'nov
  (define-key nov-mode-map (kbd "<tab>") 'shrface-outline-cycle)
  (define-key nov-mode-map (kbd "S-<tab>") 'shrface-outline-cycle-buffer)
  (define-key nov-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key nov-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key nov-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key nov-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key nov-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

(with-eval-after-load 'eww
  (define-key eww-mode-map (kbd "<tab>") 'shrface-outline-cycle)
  (define-key eww-mode-map (kbd "S-<tab>") 'shrface-outline-cycle-buffer)
  (define-key eww-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key eww-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key eww-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key eww-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key eww-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

(with-eval-after-load 'mu4e
  (define-key mu4e-view-mode-map (kbd "<tab>") 'shrface-outline-cycle)
  (define-key mu4e-view-mode-map (kbd "S-<tab>") 'shrface-outline-cycle-buffer)
  (define-key mu4e-view-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key mu4e-view-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key mu4e-view-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key mu4e-view-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key mu4e-view-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

In additional to the keys provided by shrface-mode, the following features should also work, you can test and find which fits your requirement:

org-mode

  1. org-cycle,
  2. org-shifttab
  3. org-content
  4. org-overview

etc.

outline-minor-mode

imenu-mode

Features Anatomy

shrface-mode

shrface-mode is a minor mode that help you enable/disable the following additional features:

  1. imenu
  2. outline-minior-mode
  3. org-indent-mode
  4. Setup shr faces

Please be careful, every time you run shrface-mode, faces will be enable/disable globally for all shr rendered buffers. Since I have not found a good way to enable/disable the faces per buffer yet.

If you want to enable shrface just when you need to, you should avoid to use add-hook like the recommended settings above, and just enable/disable shrface-mode via M-x.

shrface-occur

List all headlines, and you can easily jump between them by mouse click.

shrface-links

List all URLs and classify them. You can easily go to the occurrence (Left Click/Enter), copy the URL (Middle Click) or browse the URL (Right Click). Besides, if all-the-icons is available, web icon for each link will be shown.

shrface-links-counsel

List all URLs with counsel.

  • The URLs are listed in order based the position on the buffer.
  • The first candidate to select is the next url counting from the current cursor position. In this case, you can use C-M-m (ivy-call), C-M-n (ivy-next-line-and-call), and C-M-p (ivy-previous-line-and-call) to jump around all URLs without losing your position.
  • C-o to fire the action menu on the selected candidate.
  • C-c C-o to fire the ivy-occur

shrface-headline-counsel

List all headlines with counsel.

  • It is a better solution than imenu and shrface-occur.
  • It can work without the headline bullets. You can disable the bullets via (setq shrface-toggle-bullets t) and still be able to jump around the headlines.
  • The first candidate to select is the current context headline of the current cursor position. In this case, you can use C-M-m (ivy-call), C-M-n (ivy-next-line-and-call), and C-M-p (ivy-previous-line-and-call) to jump around all headlines without losing your position.
  • C-o to fire the action menu on the selected candidate.
  • C-c C-o to fire the ivy-occur

shrface-links-helm

List all URLs with helm.

  • TAB to preview the link
  • RET to goto the link

shrface-headline-helm

List all headlines with helm.

  • TAB to preview the headline
  • RET to goto the headline

shrface-links-consult

List all links with consult.

shrface-headline-consult

List all headlines with consult.

shrface-next-headline, shrface-previous-headline

These two headline functions are designed to replace outline-next-headline and outline-previous-headline. They scan the headline number text properties and jump to the headlines which means it can work under no bullets circumstance if (setq shrface-toggle-bullets t).

Headline bullets (h1 to h6)

Customize the headline bullets

You can configure your favorite bullets up to 6 levels of headings (cycled through if less than 6 bullets in setting).

You can set it with:

(setq shrface-bullets-bullet-list ("" "" "" ""))

PS: The bullets setting can be derived from org-bullets-bullet-list or org-superstar-headline-bullets-list, if org-bullets or org-superstar is available.

Toggle headline bullets locally/temporary

The quick way to toggle(disable/enable) headline bullets locally/temporary:

M-x shrface-toggle-bullets

Please notice: In mu4e-view-mode, using shrface-toggle-bullets will toggle bullets globally.

Disable headline bullets globally

If you do not like headline bullets, disable them globally by:

(setq shrface-toggle-bullets t)

Please notice, the following features are also disabled:

  1. function shrface-occur
  2. variable shrface-mode

However, the following features are still be able to use:

  1. function shrface-links
  2. function shrface-links-counsel
  3. function shrface-headline-counsel
  4. function shrface-previous-headline
  5. function shrface-next-headline
  6. function shrface-links-helm
  7. function shrface-headline-helm
  8. function shrface-links-consult
  9. function shrface-headline-consult

Item bullet

You can configure your favorite item bullet for shrface

You can set it with:

(setq shrface-item-bullet "-")

PS: Only one type of item bullet is supported, prettified by shrface-item-bullet-face

Paragraph indentation and fill column

Both shrface-paragraph-indentation and shrface-paragraph-fill-column are obsolete.

Pleas use org-indent-mode to control the indentation, and shr-width to adjust the text width (fill-column).

Versatile URL

You can enable versatile URL faces support simply by:

(setq shrface-href-versatile t)

The following types of URL can be customized.

  • http
  • https
  • ftp
  • file
  • mailto
  • other

Supported faces

Here are the faces supported:

;;; Faces for `shrface-basic

(defcustom shrface-bullets-bullet-list
  (or (bound-and-true-p org-bullets-bullet-list)
      (bound-and-true-p org-superstar-headline-bullets-list)
      '(""
        ""
        ""
        ""))
  "Bullets for headings"
  :group 'shrface
  :type '(repeat (string :tag "Bullet character")))

(defface shrface-href-face '((t :inherit org-link))
  "Default <href> face if `shrface-href-versatile' is nil"
  :group 'shrface-faces)

(defface shrface-href-other-face '((t :inherit org-link :foreground "#87cefa"))
  "Face used for <href> other than http:// https:// ftp://
file:// mailto:// if `shrface-href-versatile' is NON-nil. For
example, it can be used for fontifying charter links with epub
files when using nov.el."
  :group 'shrface-faces)

(defface shrface-href-http-face '((t :inherit org-link :foreground "#39CCCC"))
  "Face used for <href>, http:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-https-face '((t :inherit org-link :foreground "#7FDBFF"))
  "Face used for <href>, https:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-ftp-face '((t :inherit org-link :foreground "#3D9970"))
  "Face used for <href>, ftp:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-file-face '((t :inherit org-link :foreground "#2ECC40"))
  "Face used for <href>, file:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-mailto-face '((t :inherit org-link :foreground "#FF851B"))
  "Face used for <href>, mailto:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-h1-face '((t :inherit org-level-1))
  "Face used for <h1> headlines."
  :group 'shrface-faces)

(defface shrface-h2-face '((t :inherit org-level-2))
  "Face used for <h2> headlines."
  :group 'shrface-faces)

(defface shrface-h3-face '((t :inherit org-level-3))
  "Face used for <h3> headlines."
  :group 'shrface-faces)

(defface shrface-h4-face  '((t :inherit org-level-4))
  "Face used for <h4> headlines."
  :group 'shrface-faces)

(defface shrface-h5-face  '((t :inherit org-level-5))
  "Face used for <h5> headlines."
  :group 'shrface-faces)

(defface shrface-h6-face '((t :inherit org-level-6))
  "Face used for <h6> headlines."
  :group 'shrface-faces)

(defface shrface-verbatim '((t :inherit org-verbatim))
  "Face used for verbatim/emphasis - <em>."
  :group 'shrface-faces)

(defface shrface-item-bullet-face '((t :inherit org-list-dt))
  "Face used for unordered list bullet"
  :group 'shrface-faces)

(defface shrface-item-number-face '((t :inherit org-list-dt))
  "Face used for ordered list numbers"
  :group 'shrface-faces)

(defface shrface-description-list-term-face '((t :inherit org-list-dt))
  "Face used for description list terms <dt>"
  :group 'shrface-faces)

(defface shrface-figure '((t :inherit org-table))
  "Face used for figure <figure>, e.g. figure captions."
  :group 'shrface-faces)

;;; Faces for `shrface-analysis' realted buffers

(defface shrface-links-title-face '((t :inherit default))
  "Face used for *shrface-links* title"
  :group 'shrface-analysis-faces)

(defface shrface-links-url-face '((t :inherit font-lock-comment-face))
  "Face used for *shrface-links* url"
  :group 'shrface-analysis-faces)

(defface shrface-links-mouse-face '((t :inherit mode-line-highlight))
  "Face used for *shrface-links* mouse face"
  :group 'shrface-analysis-faces)

Experimental face(s)

;;; Faces for `shrface-trail' realted buffers

(defface shrface-code '((t :inherit org-code))
  "TODO Face used for inline code"
  :group 'shrface-faces)

Export Html to Org

Exporting HTML to Org doesn’t need to enable shrface-mode, or run shrface-basic / shrface-trail, you just make sure you (require 'shrface) before using the exporting features:

  • shrface-html-export-as-org: Export current html buffer to an org buffer.
  • shrface-html-export-to-org: Export current html buffer to an org file.
  • shrface-org-title: Dynamically bound and overwrite the org title.
  • shrface-request-url: Dynamically bound the request url, so that shrface can fix the relative/absolute urls based on the request url.

If you use org-web-tools to download the http page into org, you can also override its pandoc engine with shr engine as following:

(advice-add 'org-web-tools--html-to-org-with-pandoc :override 'shrface-html-convert-as-org-string)

Here are two simple examples to use shrface-html-export-as-org, and shrface-html-export-to-org, using Request.

Download an URL and output an Org buffer/file:

(defun request-url-as-org (url)
  (interactive "sRequest url: ")
  (require 'shrface)
  (require 'request)
  (request url
    :parser 'buffer-string
    :headers '(("User-Agent" . "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"))
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
                (let ((shrface-request-url url))
                  (shrface-html-export-as-org data))))))

(defun request-url-to-org (url)
  (interactive "sRequest url: ")
  (require 'shrface)
  (require 'request)
  (request url
    :parser 'buffer-string
    :headers '(("User-Agent" . "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"))
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
                (let ((shrface-request-url url))
                  (shrface-html-export-to-org data "request.org"))))))

If you use org-web-tools, it is very easy to capture the web as a readable format:

(defun request-url-readable (url)
  (interactive "sRequest url: ")
  (require 'shrface)
  (require 'request)
  (require 'org-web-tools)
  (request url
    :parser 'buffer-string
    :headers '(("User-Agent" . "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"))
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
                (let* ((web (org-web-tools--eww-readable data))
                       (title (car web))
                       (html (cdr web))
                       (shrface-org-title title)
                       (shrface-request-url url))
                  (shrface-html-export-as-org html))))))

(Optional) Enable source codes highlight

You can install shr-tag-pre-highlight.el to enable source codes highlight and background color.

(use-package shr-tag-pre-highlight
  :ensure t
  :after shr
  :config
  (add-to-list 'shr-external-rendering-functions
               '(pre . shr-tag-pre-highlight))
  (when (version< emacs-version "26")
    (with-eval-after-load 'eww
      (advice-add 'eww-display-html :around
                  'eww-display-html--override-shr-external-rendering-functions))))

Hacking the shr-tag-pre-highlight.el

If you want to add indentation, background color, #+BEGIN_SRC lang, and #+END_SRC for, you can overwrite the function as following:

(require 'shr-tag-pre-highlight)
(add-to-list 'shr-external-rendering-functions '(pre . shrface-shr-tag-pre-highlight))
(defun shrface-shr-tag-pre-highlight (pre)
    "Highlighting code in PRE."
    (let* ((shr-folding-mode 'none)
           (shr-current-font 'default)
           (code (with-temp-buffer
                   (shr-generic pre)
                   ;; (indent-rigidly (point-min) (point-max) 2)
                   (buffer-string)))
           (lang (or (shr-tag-pre-highlight-guess-language-attr pre)
                     (let ((sym (language-detection-string code)))
                       (and sym (symbol-name sym)))))
           (mode (and lang
                      (shr-tag-pre-highlight--get-lang-mode lang))))
      (shr-ensure-newline)
      (shr-ensure-newline)
      (setq start (point))
      (insert
       (propertize (concat "#+BEGIN_SRC " lang "\n") 'face 'org-block-begin-line)
       (or (and (fboundp mode)
                (with-demoted-errors "Error while fontifying: %S"
                  (shr-tag-pre-highlight-fontify code mode)))
           code)
       (propertize "#+END_SRC" 'face 'org-block-end-line ))
      (shr-ensure-newline)
      (setq end (point))
      (if light
          (add-face-text-property start end '(:background "#D8DEE9" :extend t))
        (add-face-text-property start end '(:background "#292b2e" :extend t)))
      (shr-ensure-newline)
      (insert "\n")))

Sometimes a wrong language is detected, but it is still great for highlight, even just for eye pleasing.

Screenshots when both enable shrface and the code highlights

img/code-highlight.png

News/Logs

TBD

Version ???

  • Add shrface-links-consult
  • Add shrface-headline-consult

2021-01-23

Version 2.6.3:

  • Add shrface-html-export-as-org
  • Add shrface-html-export-to-org
  • Add shrface-tag-u
  • Add shrface-tag-strong

2021-01-20

Version 2.6.2:

  • Add keys for shrface-mode.
  • Move settings to shrface-mode. So that we can use shrface-mode on other shr rendered buffers via M-x.

2020-11-08

Version 2.6.1:

  • Add: shrface-outline-cycle
  • Add: shrface-outline-cycle-buffer

2020-05-07

Version 2.6:

  • New Function: shrface-headline-helm and shrface-links-helm

2020-05-02

Version 2.5:

  • New Function: shrface-headline-counsel: better solution than imenu to list all headlines
  • New Function: shrface-next-headline: better solution than outline-next-headline, it can work without headline bullets
  • New Function: shrface-previous-headline: better solution than outline-previous-headline, it can work without headline bullets

2020-05-01

Version 2.4:

  • Improved: Improve the user experience for shrface-links-counsel
  • New face: shrface-figure

2020-04-29

Version 2.3:

  • New customization: shrface-toggle-bullets Quick way to toggle the headline bullets.

2020-04-26

Version 2.2:

  • New customization: shrface-imenu-depth The maximum level for Imenu access to shrface headlines.
  • New Feature: shrface-links and shrface-links-counsel The first shrface-analysis feature, to list all possible URL in a new buffer.

2020-04-23

Version 2.1:

  • New Feature: shrface-occur
  • New faces:
    • shrface-href-http-face
    • shrface-href-https-face
    • shrface-href-ftp-face
    • shrface-href-file-face
    • shrface-href-mailto-face
    • shrface-href-other-face

2020-04-20

Version 2.0:

  • New face: shrface-description-list-term-face

2020-04-19

Version 1.9:

  • New Functions: shrface-basic and shrface-trail

2020-04-18

Version 1.8:

  • New face: shrface-item-number-face
  • New Minor Mode: shrface-mode

2020-04-17

Version 1.7:

  • New feature: shrface-item-bullet
  • New face: shrface-item-bullet-face

Version 1.6:

  • New feature: org-indent-mode support (Enabled by default)

2020-04-16

Version 1.5:

  • New feature: outline minior mode support (Enabled by default, but not the keybindings)

2020-04-15

Version 1.4:

  • New feature: imenu support

2020-04-13

Version 1.3:

  • New face: shrface-code (Experimental face, disabled by default)

2020-04-12

Version 1.2:

  • New face: shrface-verbatim

2020-04-11

Version 1.1:

  • Fixed bug: Wrong indentation handling make some items in paragraph disappear (such as images)

2020-04-10

Version 1.0:

  • New face: shrface-bullets-bullet-list
  • New face: shrface-h1-face
  • New face: shrface-h2-face
  • New face: shrface-h3-face
  • New face: shrface-h4-face
  • New face: shrface-h5-face
  • New face: shrface-h6-face
  • New face: shrface-h6-face
  • New face: shrface-href-face
  • New customizable variable: shrface-paragraph-indentation
  • New customizable variable: shrface-paragraph-fill-column

About

Extend eww/nov with org-mode features, archive web pages to org files with shr.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published