diff --git a/archivebox-0.5.3/.pc/.quilt_patches b/archivebox-0.5.3/.pc/.quilt_patches new file mode 100644 index 0000000..6857a8d --- /dev/null +++ b/archivebox-0.5.3/.pc/.quilt_patches @@ -0,0 +1 @@ +debian/patches diff --git a/archivebox-0.5.3/.pc/.quilt_series b/archivebox-0.5.3/.pc/.quilt_series new file mode 100644 index 0000000..c206706 --- /dev/null +++ b/archivebox-0.5.3/.pc/.quilt_series @@ -0,0 +1 @@ +series diff --git a/archivebox-0.5.3/.pc/.version b/archivebox-0.5.3/.pc/.version new file mode 100644 index 0000000..0cfbf08 --- /dev/null +++ b/archivebox-0.5.3/.pc/.version @@ -0,0 +1 @@ +2 diff --git a/archivebox-0.5.3/.pc/applied-patches b/archivebox-0.5.3/.pc/applied-patches new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/MANIFEST.in b/archivebox-0.5.3/MANIFEST.in new file mode 100644 index 0000000..c9ae153 --- /dev/null +++ b/archivebox-0.5.3/MANIFEST.in @@ -0,0 +1,4 @@ +graft archivebox +global-exclude .DS_Store +global-exclude __pycache__ +global-exclude *.pyc diff --git a/archivebox-0.5.3/PKG-INFO b/archivebox-0.5.3/PKG-INFO new file mode 100644 index 0000000..b6534de --- /dev/null +++ b/archivebox-0.5.3/PKG-INFO @@ -0,0 +1,591 @@ +Metadata-Version: 2.1 +Name: archivebox +Version: 0.5.3 +Summary: The self-hosted internet archive. +Home-page: https://github.com/ArchiveBox/ArchiveBox +Author: Nick Sweeting +Author-email: git@nicksweeting.com +License: MIT +Project-URL: Source, https://github.com/ArchiveBox/ArchiveBox +Project-URL: Documentation, https://github.com/ArchiveBox/ArchiveBox/wiki +Project-URL: Bug Tracker, https://github.com/ArchiveBox/ArchiveBox/issues +Project-URL: Changelog, https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog +Project-URL: Roadmap, https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap +Project-URL: Community, https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community +Project-URL: Donate, https://github.com/ArchiveBox/ArchiveBox/wiki/Donations +Description:
+ +

ArchiveBox
The open-source self-hosted web archive.

+ + ▶️ Quickstart | + Demo | + Github | + Documentation | + Info & Motivation | + Community | + Roadmap + + 
+        "Your own personal internet archive" (网站存档 / 爬虫)
+
+ + + + + + + + + + +
+
+ + ArchiveBox is a powerful self-hosted internet archiving solution written in Python 3. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on the configuration and the content it detects. + + Your archive can be managed through the command line with commands like `archivebox add`, through the built-in Web UI `archivebox server`, or via the Python library API (beta). It can ingest bookmarks from a browser or service like Pocket/Pinboard, your entire browsing history, RSS feeds, or URLs one at a time. You can also schedule regular/realtime imports with `archivebox schedule`. + + The main index is a self-contained `index.sqlite3` file, and each snapshot is stored as a folder `data/archive//`, with an easy-to-read `index.html` and `index.json` within. For each page, ArchiveBox auto-extracts many types of assets/media and saves them in standard formats, with out-of-the-box support for: several types of HTML snapshots (wget, Chrome headless, singlefile), PDF snapshotting, screenshotting, WARC archiving, git repositories, images, audio, video, subtitles, article text, and more. The snapshots are browseable and managable offline through the filesystem, the built-in webserver, or the Python library API. + + ### Quickstart + + It works on Linux/BSD (Intel and ARM CPUs with `docker`/`apt`/`pip3`), macOS (with `docker`/`brew`/`pip3`), and Windows (beta with `docker`/`pip3`). + + ```bash + pip3 install archivebox + archivebox --version + # install extras as-needed, or use one of full setup methods below to get everything out-of-the-box + + mkdir ~/archivebox && cd ~/archivebox # this can be anywhere + archivebox init + + archivebox add 'https://example.com' + archivebox add --depth=1 'https://example.com' + archivebox schedule --every=day https://getpocket.com/users/USERNAME/feed/all + archivebox oneshot --extract=title,favicon,media https://www.youtube.com/watch?v=dQw4w9WgXcQ + archivebox help # to see more options + ``` + + *(click to expand the sections below for full setup instructions)* + +
+ Get ArchiveBox with docker-compose on any platform (recommended, everything included out-of-the-box) + + First make sure you have Docker installed: https://docs.docker.com/get-docker/ +

+ This is the recommended way to run ArchiveBox because it includes *all* the extractors like chrome, wget, youtube-dl, git, etc., as well as full-text search with sonic, and many other great features. + + ```bash + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml + docker-compose run archivebox init + docker-compose run archivebox --version + + # start the webserver and open the UI (optional) + docker-compose run archivebox manage createsuperuser + docker-compose up -d + open http://127.0.0.1:8000 + + # you can also add links and manage your archive via the CLI: + docker-compose run archivebox add 'https://example.com' + docker-compose run archivebox status + docker-compose run archivebox help # to see more options + ``` + +
+ +
+ Get ArchiveBox with docker on any platform + + First make sure you have Docker installed: https://docs.docker.com/get-docker/
+ ```bash + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + docker run -v $PWD:/data -it archivebox/archivebox init + docker run -v $PWD:/data -it archivebox/archivebox --version + + # start the webserver and open the UI (optional) + docker run -v $PWD:/data -it archivebox/archivebox manage createsuperuser + docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add links and manage your archive via the CLI: + docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com' + docker run -v $PWD:/data -it archivebox/archivebox status + docker run -v $PWD:/data -it archivebox/archivebox help # to see more options + ``` + +
+ +
+ Get ArchiveBox with apt on Ubuntu >=20.04 + + ```bash + sudo add-apt-repository -u ppa:archivebox/archivebox + sudo apt install archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + + For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`: + ```bash + deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main + deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main + ``` + (you may need to install some other dependencies manually however) + +
+ +
+ Get ArchiveBox with brew on macOS >=10.13 + + ```bash + brew install archivebox/archivebox/archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + +
+ +
+ Get ArchiveBox with pip on any platform + + ```bash + pip3 install archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + # Install any missing extras like wget/git/chrome/etc. manually as needed + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + +
+ + --- + +
+ +
+ + DEMO: archivebox.zervice.io/ + For more information, see the full Quickstart guide, Usage, and Configuration docs. +
+ + --- + + + # Overview + + ArchiveBox is a command line tool, self-hostable web-archiving server, and Python library all-in-one. It can be installed on Docker, macOS, and Linux/BSD, and Windows. You can download and install it as a Debian/Ubuntu package, Homebrew package, Python3 package, or a Docker image. No matter which install method you choose, they all provide the same CLI, Web UI, and on-disk data format. + + To use ArchiveBox you start by creating a folder for your data to live in (it can be anywhere on your system), and running `archivebox init` inside of it. That will create a sqlite3 index and an `ArchiveBox.conf` file. After that, you can continue to add/export/manage/etc using the CLI `archivebox help`, or you can run the Web UI (recommended). If you only want to archive a single site, you can run `archivebox oneshot` to avoid having to create a whole collection. + + The CLI is considered "stable", the ArchiveBox Python API and REST APIs are "beta", and the [desktop app](https://github.com/ArchiveBox/desktop) is "alpha". + + At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer). You can also self-host your archivebox server on a public domain to provide archive.org-style public access to your site snapshots. + +
+ CLI Screenshot + Desktop index screenshot + Desktop details page Screenshot + Desktop details page Screenshot
+ Demo | Usage | Screenshots +
+ . . . . . . . . . . . . . . . . . . . . . . . . . . . . +

+ + + ## Key Features + + - [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally + - [**Few dependencies**](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) + - [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - Easy to set up **[scheduled importing](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources** + - Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC + - ~~**Suitable for paywalled / [authenticated content](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes) + - **Doesn't require a constantly-running daemon**, proxy, or native app + - Provides a CLI, Python API, self-hosted web UI, and REST API (WIP) + - Architected to be able to run [**many varieties of scripts during archiving**](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. to extract media, summarize articles, [scroll pages](https://github.com/ArchiveBox/ArchiveBox/issues/80), [close modals](https://github.com/ArchiveBox/ArchiveBox/issues/175), expand comment threads, etc. + - Can also [**mirror content to 3rd-party archiving services**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy + + ## Input formats + + ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more! + + ```bash + echo 'http://example.com' | archivebox add + archivebox add 'https://example.com/some/page' + archivebox add < ~/Downloads/firefox_bookmarks_export.html + archivebox add < any_text_with_urls_in_it.txt + archivebox add --depth=1 'https://example.com/some/downloads.html' + archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12' + ``` + + - Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more) + - RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format + - Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more + + See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples. + + It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand. + + ## Output formats + + All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`. + + The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.) + + ```bash + ls ./archive// + ``` + + - **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details + - **Title:** `title` title of the site + - **Favicon:** `favicon.ico` favicon of the site + - **Headers:** `headers.json` Any HTTP headers the site returns are saved in a json file + - **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile + - **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present + - **WARC:** `warc/.gz` gzipped WARC of all the resources fetched while archiving + - **PDF:** `output.pdf` Printed PDF of site using headless chrome + - **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome + - **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome + - **Readability:** `article.html/json` Article text extraction using Readability + - **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org + - **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl + - **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links + - _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._ + + It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file. + + ## Dependencies + + You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled. + + If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) or the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install). + + ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more. + + ## Caveats + + If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process. + ```bash + # don't do this: + archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere' + archivebox add 'https://example.com/any/url/you/want/to/keep/secret/' + + # without first disabling share the URL with 3rd party APIs: + archivebox config --set SAVE_ARCHIVE_DOT_ORG=False # disable saving all URLs in Archive.org + archivebox config --set SAVE_FAVICON=False # optional: only the domain is leaked, not full URL + archivebox config --set CHROME_BINARY=chromium # optional: switch to chromium to avoid Chrome phoning home to Google + ``` + + Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details. + ```bash + # visiting an archived page with malicious JS: + https://127.0.0.1:8000/archive/1602401954/example.com/index.html + + # example.com/index.js can now make a request to read everything: + https://127.0.0.1:8000/index.html + https://127.0.0.1:8000/archive/* + # then example.com/index.js can send it off to some evil server + ``` + + Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash: + ```bash + archivebox add 'https://example.com#2020-10-24' + ... + archivebox add 'https://example.com#2020-10-25' + ``` + + --- + +
+ +
+ + --- + + # Background & Motivation + + Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity. + + Whether it's to resist censorship by saving articles before they get taken down or edited, or + just to save a collection of early 2010's flash games you love to play, having the tools to + archive internet content enables to you save the stuff you care most about before it disappears. + +
+
+ Image from WTF is Link Rot?...
 +
+ + The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. + I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about. + + Because modern websites are complicated and often rely on dynamic content, + ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats. + + All the archived links are stored by date bookmarked in `./archive/`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM. + + ## Comparison to Other Projects + + ▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.** + + comparison The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations. + + #### User Interface & Intended Purpose + + ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend. + + #### Private Local Archives vs Centralized Public Archives + + Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle. + + #### Storage Requirements + + Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files. + + ## Learn more + + Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web! + + + + - [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists) + _Community-maintained indexes of archiving tools and institutions._ + - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) + _Open source tools and projects in the internet archiving space._ + - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List) + _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._ + - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities) + _A collection of the most active internet archiving communities and initiatives._ + - Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) + - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post. + - Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter. + + --- + + # Documentation + + + + We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation. + + You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder. + + ## Getting Started + + - [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) + - [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) + - [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + + ## Reference + + - [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) + - [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) + - [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) + - [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site) + - [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) + - [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) + - [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) + - [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) + - [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) + - [Python API](https://docs.archivebox.io/en/latest/modules.html) + - REST API (coming soon...) + + ## More Info + + - [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues) + - [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) + - [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) + - [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) + - [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation) + - [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + + --- + + # ArchiveBox Development + + All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap. + + ### Setup the dev environment + + First, install the system dependencies from the "Bare Metal" section above. + Then you can clone the ArchiveBox repo and install + ```python3 + git clone https://github.com/ArchiveBox/ArchiveBox && cd ArchiveBox + git checkout master # or the branch you want to test + git submodule update --init --recursive + git pull --recurse-submodules + + # Install ArchiveBox + python dependencies + python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev] + # or with pipenv: pipenv install --dev && pipenv shell + + # Install node dependencies + npm install + + # Optional: install extractor dependencies manually or with helper script + ./bin/setup.sh + + # Optional: develop via docker by mounting the code dir into the container + # if you edit e.g. ./archivebox/core/models.py on the docker host, runserver + # inside the container will reload and pick up your changes + docker build . -t archivebox + docker run -it -p 8000:8000 \ + -v $PWD/data:/data \ + -v $PWD/archivebox:/app/archivebox \ + archivebox server 0.0.0.0:8000 --debug --reload + ``` + + ### Common development tasks + + See the `./bin/` folder and read the source of the bash scripts within. + You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`. + + #### Run the linters + + ```bash + ./bin/lint.sh + ``` + (uses `flake8` and `mypy`) + + #### Run the integration tests + + ```bash + ./bin/test.sh + ``` + (uses `pytest -s`) + + #### Make migrations or enter a django shell + + ```bash + cd archivebox/ + ./manage.py makemigrations + + cd data/ + archivebox shell + ``` + (uses `pytest -s`) + + #### Build the docs, pip package, and docker image + + ```bash + ./bin/build.sh + + # or individually: + ./bin/build_docs.sh + ./bin/build_pip.sh + ./bin/build_deb.sh + ./bin/build_brew.sh + ./bin/build_docker.sh + ``` + + #### Roll a release + + ```bash + ./bin/release.sh + ``` + (bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages) + + + --- + +
+

+ +
+ This project is maintained mostly in my spare time with the help from generous contributors and Monadical.com. +

+ +
+ Sponsor us on Github +
+
+ +
+ + + + +

+ +
+ +Platform: UNKNOWN +Classifier: License :: OSI Approved :: MIT License +Classifier: Natural Language :: English +Classifier: Operating System :: OS Independent +Classifier: Development Status :: 4 - Beta +Classifier: Topic :: Utilities +Classifier: Topic :: System :: Archiving +Classifier: Topic :: System :: Archiving :: Backup +Classifier: Topic :: System :: Recovery Tools +Classifier: Topic :: Sociology :: History +Classifier: Topic :: Internet :: WWW/HTTP +Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search +Classifier: Topic :: Internet :: WWW/HTTP :: WSGI :: Application +Classifier: Topic :: Software Development :: Libraries :: Python Modules +Classifier: Intended Audience :: Developers +Classifier: Intended Audience :: Education +Classifier: Intended Audience :: End Users/Desktop +Classifier: Intended Audience :: Information Technology +Classifier: Intended Audience :: Legal Industry +Classifier: Intended Audience :: System Administrators +Classifier: Environment :: Console +Classifier: Environment :: Web Environment +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Framework :: Django +Classifier: Typing :: Typed +Requires-Python: >=3.7 +Description-Content-Type: text/markdown +Provides-Extra: dev diff --git a/archivebox-0.5.3/README.md b/archivebox-0.5.3/README.md new file mode 100644 index 0000000..2e35783 --- /dev/null +++ b/archivebox-0.5.3/README.md @@ -0,0 +1,545 @@ +
+ +

ArchiveBox
The open-source self-hosted web archive.

+ +▶️ Quickstart | +Demo | +Github | +Documentation | +Info & Motivation | +Community | +Roadmap + +
+"Your own personal internet archive" (网站存档 / 爬虫)
+
+ + + + + + + + + + +
+
+ +ArchiveBox is a powerful self-hosted internet archiving solution written in Python 3. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on the configuration and the content it detects. + +Your archive can be managed through the command line with commands like `archivebox add`, through the built-in Web UI `archivebox server`, or via the Python library API (beta). It can ingest bookmarks from a browser or service like Pocket/Pinboard, your entire browsing history, RSS feeds, or URLs one at a time. You can also schedule regular/realtime imports with `archivebox schedule`. + +The main index is a self-contained `index.sqlite3` file, and each snapshot is stored as a folder `data/archive//`, with an easy-to-read `index.html` and `index.json` within. For each page, ArchiveBox auto-extracts many types of assets/media and saves them in standard formats, with out-of-the-box support for: several types of HTML snapshots (wget, Chrome headless, singlefile), PDF snapshotting, screenshotting, WARC archiving, git repositories, images, audio, video, subtitles, article text, and more. The snapshots are browseable and managable offline through the filesystem, the built-in webserver, or the Python library API. + +### Quickstart + +It works on Linux/BSD (Intel and ARM CPUs with `docker`/`apt`/`pip3`), macOS (with `docker`/`brew`/`pip3`), and Windows (beta with `docker`/`pip3`). + +```bash +pip3 install archivebox +archivebox --version +# install extras as-needed, or use one of full setup methods below to get everything out-of-the-box + +mkdir ~/archivebox && cd ~/archivebox # this can be anywhere +archivebox init + +archivebox add 'https://example.com' +archivebox add --depth=1 'https://example.com' +archivebox schedule --every=day https://getpocket.com/users/USERNAME/feed/all +archivebox oneshot --extract=title,favicon,media https://www.youtube.com/watch?v=dQw4w9WgXcQ +archivebox help # to see more options +``` + +*(click to expand the sections below for full setup instructions)* + +
+Get ArchiveBox with docker-compose on any platform (recommended, everything included out-of-the-box) + +First make sure you have Docker installed: https://docs.docker.com/get-docker/ +

+This is the recommended way to run ArchiveBox because it includes *all* the extractors like chrome, wget, youtube-dl, git, etc., as well as full-text search with sonic, and many other great features. + +```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml +docker-compose run archivebox init +docker-compose run archivebox --version + +# start the webserver and open the UI (optional) +docker-compose run archivebox manage createsuperuser +docker-compose up -d +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker-compose run archivebox add 'https://example.com' +docker-compose run archivebox status +docker-compose run archivebox help # to see more options +``` + +
+ +
+Get ArchiveBox with docker on any platform + +First make sure you have Docker installed: https://docs.docker.com/get-docker/
+```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +docker run -v $PWD:/data -it archivebox/archivebox init +docker run -v $PWD:/data -it archivebox/archivebox --version + +# start the webserver and open the UI (optional) +docker run -v $PWD:/data -it archivebox/archivebox manage createsuperuser +docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com' +docker run -v $PWD:/data -it archivebox/archivebox status +docker run -v $PWD:/data -it archivebox/archivebox help # to see more options +``` + +
+ +
+Get ArchiveBox with apt on Ubuntu >=20.04 + +```bash +sudo add-apt-repository -u ppa:archivebox/archivebox +sudo apt install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`: +```bash +deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +``` +(you may need to install some other dependencies manually however) + +
+ +
+Get ArchiveBox with brew on macOS >=10.13 + +```bash +brew install archivebox/archivebox/archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
+ +
+Get ArchiveBox with pip on any platform + +```bash +pip3 install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version +# Install any missing extras like wget/git/chrome/etc. manually as needed + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
+ +--- + +
+ +
+ +DEMO: archivebox.zervice.io/ +For more information, see the full Quickstart guide, Usage, and Configuration docs. +
+ +--- + + +# Overview + +ArchiveBox is a command line tool, self-hostable web-archiving server, and Python library all-in-one. It can be installed on Docker, macOS, and Linux/BSD, and Windows. You can download and install it as a Debian/Ubuntu package, Homebrew package, Python3 package, or a Docker image. No matter which install method you choose, they all provide the same CLI, Web UI, and on-disk data format. + +To use ArchiveBox you start by creating a folder for your data to live in (it can be anywhere on your system), and running `archivebox init` inside of it. That will create a sqlite3 index and an `ArchiveBox.conf` file. After that, you can continue to add/export/manage/etc using the CLI `archivebox help`, or you can run the Web UI (recommended). If you only want to archive a single site, you can run `archivebox oneshot` to avoid having to create a whole collection. + +The CLI is considered "stable", the ArchiveBox Python API and REST APIs are "beta", and the [desktop app](https://github.com/ArchiveBox/desktop) is "alpha". + +At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer). You can also self-host your archivebox server on a public domain to provide archive.org-style public access to your site snapshots. + +
+CLI Screenshot +Desktop index screenshot +Desktop details page Screenshot +Desktop details page Screenshot
+Demo | Usage | Screenshots +
+. . . . . . . . . . . . . . . . . . . . . . . . . . . . +

+ + +## Key Features + +- [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally +- [**Few dependencies**](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) +- [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) +- Easy to set up **[scheduled importing](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources** +- Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC +- ~~**Suitable for paywalled / [authenticated content](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes) +- **Doesn't require a constantly-running daemon**, proxy, or native app +- Provides a CLI, Python API, self-hosted web UI, and REST API (WIP) +- Architected to be able to run [**many varieties of scripts during archiving**](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. to extract media, summarize articles, [scroll pages](https://github.com/ArchiveBox/ArchiveBox/issues/80), [close modals](https://github.com/ArchiveBox/ArchiveBox/issues/175), expand comment threads, etc. +- Can also [**mirror content to 3rd-party archiving services**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy + +## Input formats + +ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more! + +```bash +echo 'http://example.com' | archivebox add +archivebox add 'https://example.com/some/page' +archivebox add < ~/Downloads/firefox_bookmarks_export.html +archivebox add < any_text_with_urls_in_it.txt +archivebox add --depth=1 'https://example.com/some/downloads.html' +archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12' +``` + +- Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more) +- RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format +- Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more + +See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples. + +It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand. + +## Output formats + +All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`. + +The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.) + +```bash + ls ./archive// +``` + +- **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details +- **Title:** `title` title of the site +- **Favicon:** `favicon.ico` favicon of the site +- **Headers:** `headers.json` Any HTTP headers the site returns are saved in a json file +- **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile +- **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present +- **WARC:** `warc/.gz` gzipped WARC of all the resources fetched while archiving +- **PDF:** `output.pdf` Printed PDF of site using headless chrome +- **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome +- **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome +- **Readability:** `article.html/json` Article text extraction using Readability +- **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org +- **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl +- **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links +- _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._ + +It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file. + +## Dependencies + +You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled. + +If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) or the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install). + +ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more. + +## Caveats + +If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process. +```bash +# don't do this: +archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere' +archivebox add 'https://example.com/any/url/you/want/to/keep/secret/' + +# without first disabling share the URL with 3rd party APIs: +archivebox config --set SAVE_ARCHIVE_DOT_ORG=False # disable saving all URLs in Archive.org +archivebox config --set SAVE_FAVICON=False # optional: only the domain is leaked, not full URL +archivebox config --set CHROME_BINARY=chromium # optional: switch to chromium to avoid Chrome phoning home to Google +``` + +Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details. +```bash +# visiting an archived page with malicious JS: +https://127.0.0.1:8000/archive/1602401954/example.com/index.html + +# example.com/index.js can now make a request to read everything: +https://127.0.0.1:8000/index.html +https://127.0.0.1:8000/archive/* +# then example.com/index.js can send it off to some evil server +``` + +Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash: +```bash +archivebox add 'https://example.com#2020-10-24' +... +archivebox add 'https://example.com#2020-10-25' +``` + +--- + +
+ +
+ +--- + +# Background & Motivation + +Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity. + +Whether it's to resist censorship by saving articles before they get taken down or edited, or +just to save a collection of early 2010's flash games you love to play, having the tools to +archive internet content enables to you save the stuff you care most about before it disappears. + +
+
+ Image from WTF is Link Rot?...
 +
+ +The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. +I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about. + +Because modern websites are complicated and often rely on dynamic content, +ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats. + +All the archived links are stored by date bookmarked in `./archive/`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM. + +## Comparison to Other Projects + +▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.** + +comparison The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations. + +#### User Interface & Intended Purpose + +ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend. + +#### Private Local Archives vs Centralized Public Archives + +Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle. + +#### Storage Requirements + +Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files. + +## Learn more + +Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web! + + + +- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists) + _Community-maintained indexes of archiving tools and institutions._ + - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) + _Open source tools and projects in the internet archiving space._ + - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List) + _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._ + - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities) + _A collection of the most active internet archiving communities and initiatives._ +- Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post. +- Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter. + +--- + +# Documentation + + + +We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation. + +You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder. + +## Getting Started + +- [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) +- [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) +- [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + +## Reference + +- [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) +- [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) +- [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) +- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site) +- [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) +- [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) +- [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) +- [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) +- [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) +- [Python API](https://docs.archivebox.io/en/latest/modules.html) +- REST API (coming soon...) + +## More Info + +- [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues) +- [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) +- [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) +- [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation) +- [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + +--- + +# ArchiveBox Development + +All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap. + +### Setup the dev environment + +First, install the system dependencies from the "Bare Metal" section above. +Then you can clone the ArchiveBox repo and install +```python3 +git clone https://github.com/ArchiveBox/ArchiveBox && cd ArchiveBox +git checkout master # or the branch you want to test +git submodule update --init --recursive +git pull --recurse-submodules + +# Install ArchiveBox + python dependencies +python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev] +# or with pipenv: pipenv install --dev && pipenv shell + +# Install node dependencies +npm install + +# Optional: install extractor dependencies manually or with helper script +./bin/setup.sh + +# Optional: develop via docker by mounting the code dir into the container +# if you edit e.g. ./archivebox/core/models.py on the docker host, runserver +# inside the container will reload and pick up your changes +docker build . -t archivebox +docker run -it -p 8000:8000 \ + -v $PWD/data:/data \ + -v $PWD/archivebox:/app/archivebox \ + archivebox server 0.0.0.0:8000 --debug --reload +``` + +### Common development tasks + +See the `./bin/` folder and read the source of the bash scripts within. +You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`. + +#### Run the linters + +```bash +./bin/lint.sh +``` +(uses `flake8` and `mypy`) + +#### Run the integration tests + +```bash +./bin/test.sh +``` +(uses `pytest -s`) + +#### Make migrations or enter a django shell + +```bash +cd archivebox/ +./manage.py makemigrations + +cd data/ +archivebox shell +``` +(uses `pytest -s`) + +#### Build the docs, pip package, and docker image + +```bash +./bin/build.sh + +# or individually: +./bin/build_docs.sh +./bin/build_pip.sh +./bin/build_deb.sh +./bin/build_brew.sh +./bin/build_docker.sh +``` + +#### Roll a release + +```bash +./bin/release.sh +``` +(bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages) + + +--- + +
+

+ +
+This project is maintained mostly in my spare time with the help from generous contributors and Monadical.com. +

+ +
+Sponsor us on Github +
+
+ +
+ + + + +

+ +
diff --git a/archivebox-0.5.3/archivebox.egg-info/PKG-INFO b/archivebox-0.5.3/archivebox.egg-info/PKG-INFO new file mode 100644 index 0000000..b6534de --- /dev/null +++ b/archivebox-0.5.3/archivebox.egg-info/PKG-INFO @@ -0,0 +1,591 @@ +Metadata-Version: 2.1 +Name: archivebox +Version: 0.5.3 +Summary: The self-hosted internet archive. +Home-page: https://github.com/ArchiveBox/ArchiveBox +Author: Nick Sweeting +Author-email: git@nicksweeting.com +License: MIT +Project-URL: Source, https://github.com/ArchiveBox/ArchiveBox +Project-URL: Documentation, https://github.com/ArchiveBox/ArchiveBox/wiki +Project-URL: Bug Tracker, https://github.com/ArchiveBox/ArchiveBox/issues +Project-URL: Changelog, https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog +Project-URL: Roadmap, https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap +Project-URL: Community, https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community +Project-URL: Donate, https://github.com/ArchiveBox/ArchiveBox/wiki/Donations +Description:
+ +

ArchiveBox
The open-source self-hosted web archive.

+ + ▶️ Quickstart | + Demo | + Github | + Documentation | + Info & Motivation | + Community | + Roadmap + + 
+        "Your own personal internet archive" (网站存档 / 爬虫)
+
+ + + + + + + + + + +
+
+ + ArchiveBox is a powerful self-hosted internet archiving solution written in Python 3. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on the configuration and the content it detects. + + Your archive can be managed through the command line with commands like `archivebox add`, through the built-in Web UI `archivebox server`, or via the Python library API (beta). It can ingest bookmarks from a browser or service like Pocket/Pinboard, your entire browsing history, RSS feeds, or URLs one at a time. You can also schedule regular/realtime imports with `archivebox schedule`. + + The main index is a self-contained `index.sqlite3` file, and each snapshot is stored as a folder `data/archive//`, with an easy-to-read `index.html` and `index.json` within. For each page, ArchiveBox auto-extracts many types of assets/media and saves them in standard formats, with out-of-the-box support for: several types of HTML snapshots (wget, Chrome headless, singlefile), PDF snapshotting, screenshotting, WARC archiving, git repositories, images, audio, video, subtitles, article text, and more. The snapshots are browseable and managable offline through the filesystem, the built-in webserver, or the Python library API. + + ### Quickstart + + It works on Linux/BSD (Intel and ARM CPUs with `docker`/`apt`/`pip3`), macOS (with `docker`/`brew`/`pip3`), and Windows (beta with `docker`/`pip3`). + + ```bash + pip3 install archivebox + archivebox --version + # install extras as-needed, or use one of full setup methods below to get everything out-of-the-box + + mkdir ~/archivebox && cd ~/archivebox # this can be anywhere + archivebox init + + archivebox add 'https://example.com' + archivebox add --depth=1 'https://example.com' + archivebox schedule --every=day https://getpocket.com/users/USERNAME/feed/all + archivebox oneshot --extract=title,favicon,media https://www.youtube.com/watch?v=dQw4w9WgXcQ + archivebox help # to see more options + ``` + + *(click to expand the sections below for full setup instructions)* + +
+ Get ArchiveBox with docker-compose on any platform (recommended, everything included out-of-the-box) + + First make sure you have Docker installed: https://docs.docker.com/get-docker/ +

+ This is the recommended way to run ArchiveBox because it includes *all* the extractors like chrome, wget, youtube-dl, git, etc., as well as full-text search with sonic, and many other great features. + + ```bash + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml + docker-compose run archivebox init + docker-compose run archivebox --version + + # start the webserver and open the UI (optional) + docker-compose run archivebox manage createsuperuser + docker-compose up -d + open http://127.0.0.1:8000 + + # you can also add links and manage your archive via the CLI: + docker-compose run archivebox add 'https://example.com' + docker-compose run archivebox status + docker-compose run archivebox help # to see more options + ``` + +
+ +
+ Get ArchiveBox with docker on any platform + + First make sure you have Docker installed: https://docs.docker.com/get-docker/
+ ```bash + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + docker run -v $PWD:/data -it archivebox/archivebox init + docker run -v $PWD:/data -it archivebox/archivebox --version + + # start the webserver and open the UI (optional) + docker run -v $PWD:/data -it archivebox/archivebox manage createsuperuser + docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add links and manage your archive via the CLI: + docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com' + docker run -v $PWD:/data -it archivebox/archivebox status + docker run -v $PWD:/data -it archivebox/archivebox help # to see more options + ``` + +
+ +
+ Get ArchiveBox with apt on Ubuntu >=20.04 + + ```bash + sudo add-apt-repository -u ppa:archivebox/archivebox + sudo apt install archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + + For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`: + ```bash + deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main + deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main + ``` + (you may need to install some other dependencies manually however) + +
+ +
+ Get ArchiveBox with brew on macOS >=10.13 + + ```bash + brew install archivebox/archivebox/archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + +
+ +
+ Get ArchiveBox with pip on any platform + + ```bash + pip3 install archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + # Install any missing extras like wget/git/chrome/etc. manually as needed + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + +
+ + --- + +
+ +
+ + DEMO: archivebox.zervice.io/ + For more information, see the full Quickstart guide, Usage, and Configuration docs. +
+ + --- + + + # Overview + + ArchiveBox is a command line tool, self-hostable web-archiving server, and Python library all-in-one. It can be installed on Docker, macOS, and Linux/BSD, and Windows. You can download and install it as a Debian/Ubuntu package, Homebrew package, Python3 package, or a Docker image. No matter which install method you choose, they all provide the same CLI, Web UI, and on-disk data format. + + To use ArchiveBox you start by creating a folder for your data to live in (it can be anywhere on your system), and running `archivebox init` inside of it. That will create a sqlite3 index and an `ArchiveBox.conf` file. After that, you can continue to add/export/manage/etc using the CLI `archivebox help`, or you can run the Web UI (recommended). If you only want to archive a single site, you can run `archivebox oneshot` to avoid having to create a whole collection. + + The CLI is considered "stable", the ArchiveBox Python API and REST APIs are "beta", and the [desktop app](https://github.com/ArchiveBox/desktop) is "alpha". + + At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer). You can also self-host your archivebox server on a public domain to provide archive.org-style public access to your site snapshots. + +
+ CLI Screenshot + Desktop index screenshot + Desktop details page Screenshot + Desktop details page Screenshot
+ Demo | Usage | Screenshots +
+ . . . . . . . . . . . . . . . . . . . . . . . . . . . . +

+ + + ## Key Features + + - [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally + - [**Few dependencies**](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) + - [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - Easy to set up **[scheduled importing](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources** + - Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC + - ~~**Suitable for paywalled / [authenticated content](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes) + - **Doesn't require a constantly-running daemon**, proxy, or native app + - Provides a CLI, Python API, self-hosted web UI, and REST API (WIP) + - Architected to be able to run [**many varieties of scripts during archiving**](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. to extract media, summarize articles, [scroll pages](https://github.com/ArchiveBox/ArchiveBox/issues/80), [close modals](https://github.com/ArchiveBox/ArchiveBox/issues/175), expand comment threads, etc. + - Can also [**mirror content to 3rd-party archiving services**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy + + ## Input formats + + ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more! + + ```bash + echo 'http://example.com' | archivebox add + archivebox add 'https://example.com/some/page' + archivebox add < ~/Downloads/firefox_bookmarks_export.html + archivebox add < any_text_with_urls_in_it.txt + archivebox add --depth=1 'https://example.com/some/downloads.html' + archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12' + ``` + + - Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more) + - RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format + - Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more + + See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples. + + It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand. + + ## Output formats + + All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`. + + The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.) + + ```bash + ls ./archive// + ``` + + - **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details + - **Title:** `title` title of the site + - **Favicon:** `favicon.ico` favicon of the site + - **Headers:** `headers.json` Any HTTP headers the site returns are saved in a json file + - **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile + - **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present + - **WARC:** `warc/.gz` gzipped WARC of all the resources fetched while archiving + - **PDF:** `output.pdf` Printed PDF of site using headless chrome + - **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome + - **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome + - **Readability:** `article.html/json` Article text extraction using Readability + - **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org + - **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl + - **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links + - _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._ + + It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file. + + ## Dependencies + + You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled. + + If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) or the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install). + + ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more. + + ## Caveats + + If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process. + ```bash + # don't do this: + archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere' + archivebox add 'https://example.com/any/url/you/want/to/keep/secret/' + + # without first disabling share the URL with 3rd party APIs: + archivebox config --set SAVE_ARCHIVE_DOT_ORG=False # disable saving all URLs in Archive.org + archivebox config --set SAVE_FAVICON=False # optional: only the domain is leaked, not full URL + archivebox config --set CHROME_BINARY=chromium # optional: switch to chromium to avoid Chrome phoning home to Google + ``` + + Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details. + ```bash + # visiting an archived page with malicious JS: + https://127.0.0.1:8000/archive/1602401954/example.com/index.html + + # example.com/index.js can now make a request to read everything: + https://127.0.0.1:8000/index.html + https://127.0.0.1:8000/archive/* + # then example.com/index.js can send it off to some evil server + ``` + + Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash: + ```bash + archivebox add 'https://example.com#2020-10-24' + ... + archivebox add 'https://example.com#2020-10-25' + ``` + + --- + +
+ +
+ + --- + + # Background & Motivation + + Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity. + + Whether it's to resist censorship by saving articles before they get taken down or edited, or + just to save a collection of early 2010's flash games you love to play, having the tools to + archive internet content enables to you save the stuff you care most about before it disappears. + +
+
+ Image from WTF is Link Rot?...
 +
+ + The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. + I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about. + + Because modern websites are complicated and often rely on dynamic content, + ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats. + + All the archived links are stored by date bookmarked in `./archive/`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM. + + ## Comparison to Other Projects + + ▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.** + + comparison The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations. + + #### User Interface & Intended Purpose + + ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend. + + #### Private Local Archives vs Centralized Public Archives + + Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle. + + #### Storage Requirements + + Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files. + + ## Learn more + + Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web! + + + + - [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists) + _Community-maintained indexes of archiving tools and institutions._ + - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) + _Open source tools and projects in the internet archiving space._ + - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List) + _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._ + - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities) + _A collection of the most active internet archiving communities and initiatives._ + - Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) + - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post. + - Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter. + + --- + + # Documentation + + + + We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation. + + You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder. + + ## Getting Started + + - [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) + - [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) + - [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + + ## Reference + + - [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) + - [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) + - [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) + - [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site) + - [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) + - [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) + - [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) + - [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) + - [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) + - [Python API](https://docs.archivebox.io/en/latest/modules.html) + - REST API (coming soon...) + + ## More Info + + - [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues) + - [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) + - [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) + - [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) + - [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation) + - [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + + --- + + # ArchiveBox Development + + All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap. + + ### Setup the dev environment + + First, install the system dependencies from the "Bare Metal" section above. + Then you can clone the ArchiveBox repo and install + ```python3 + git clone https://github.com/ArchiveBox/ArchiveBox && cd ArchiveBox + git checkout master # or the branch you want to test + git submodule update --init --recursive + git pull --recurse-submodules + + # Install ArchiveBox + python dependencies + python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev] + # or with pipenv: pipenv install --dev && pipenv shell + + # Install node dependencies + npm install + + # Optional: install extractor dependencies manually or with helper script + ./bin/setup.sh + + # Optional: develop via docker by mounting the code dir into the container + # if you edit e.g. ./archivebox/core/models.py on the docker host, runserver + # inside the container will reload and pick up your changes + docker build . -t archivebox + docker run -it -p 8000:8000 \ + -v $PWD/data:/data \ + -v $PWD/archivebox:/app/archivebox \ + archivebox server 0.0.0.0:8000 --debug --reload + ``` + + ### Common development tasks + + See the `./bin/` folder and read the source of the bash scripts within. + You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`. + + #### Run the linters + + ```bash + ./bin/lint.sh + ``` + (uses `flake8` and `mypy`) + + #### Run the integration tests + + ```bash + ./bin/test.sh + ``` + (uses `pytest -s`) + + #### Make migrations or enter a django shell + + ```bash + cd archivebox/ + ./manage.py makemigrations + + cd data/ + archivebox shell + ``` + (uses `pytest -s`) + + #### Build the docs, pip package, and docker image + + ```bash + ./bin/build.sh + + # or individually: + ./bin/build_docs.sh + ./bin/build_pip.sh + ./bin/build_deb.sh + ./bin/build_brew.sh + ./bin/build_docker.sh + ``` + + #### Roll a release + + ```bash + ./bin/release.sh + ``` + (bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages) + + + --- + +
+

+ +
+ This project is maintained mostly in my spare time with the help from generous contributors and Monadical.com. +

+ +
+ Sponsor us on Github +
+
+ +
+ + + + +

+ +
+ +Platform: UNKNOWN +Classifier: License :: OSI Approved :: MIT License +Classifier: Natural Language :: English +Classifier: Operating System :: OS Independent +Classifier: Development Status :: 4 - Beta +Classifier: Topic :: Utilities +Classifier: Topic :: System :: Archiving +Classifier: Topic :: System :: Archiving :: Backup +Classifier: Topic :: System :: Recovery Tools +Classifier: Topic :: Sociology :: History +Classifier: Topic :: Internet :: WWW/HTTP +Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search +Classifier: Topic :: Internet :: WWW/HTTP :: WSGI :: Application +Classifier: Topic :: Software Development :: Libraries :: Python Modules +Classifier: Intended Audience :: Developers +Classifier: Intended Audience :: Education +Classifier: Intended Audience :: End Users/Desktop +Classifier: Intended Audience :: Information Technology +Classifier: Intended Audience :: Legal Industry +Classifier: Intended Audience :: System Administrators +Classifier: Environment :: Console +Classifier: Environment :: Web Environment +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Framework :: Django +Classifier: Typing :: Typed +Requires-Python: >=3.7 +Description-Content-Type: text/markdown +Provides-Extra: dev diff --git a/archivebox-0.5.3/archivebox.egg-info/SOURCES.txt b/archivebox-0.5.3/archivebox.egg-info/SOURCES.txt new file mode 100644 index 0000000..6d08f24 --- /dev/null +++ b/archivebox-0.5.3/archivebox.egg-info/SOURCES.txt @@ -0,0 +1,129 @@ +MANIFEST.in +README.md +setup.cfg +setup.py +archivebox/.flake8 +archivebox/LICENSE +archivebox/README.md +archivebox/__init__.py +archivebox/__main__.py +archivebox/config.py +archivebox/config_stubs.py +archivebox/logging_util.py +archivebox/main.py +archivebox/manage.py +archivebox/mypy.ini +archivebox/package.json +archivebox/system.py +archivebox/util.py +archivebox.egg-info/PKG-INFO +archivebox.egg-info/SOURCES.txt +archivebox.egg-info/dependency_links.txt +archivebox.egg-info/entry_points.txt +archivebox.egg-info/requires.txt +archivebox.egg-info/top_level.txt +archivebox/cli/__init__.py +archivebox/cli/archivebox_add.py +archivebox/cli/archivebox_config.py +archivebox/cli/archivebox_help.py +archivebox/cli/archivebox_init.py +archivebox/cli/archivebox_list.py +archivebox/cli/archivebox_manage.py +archivebox/cli/archivebox_oneshot.py +archivebox/cli/archivebox_remove.py +archivebox/cli/archivebox_schedule.py +archivebox/cli/archivebox_server.py +archivebox/cli/archivebox_shell.py +archivebox/cli/archivebox_status.py +archivebox/cli/archivebox_update.py +archivebox/cli/archivebox_version.py +archivebox/cli/tests.py +archivebox/core/__init__.py +archivebox/core/admin.py +archivebox/core/apps.py +archivebox/core/forms.py +archivebox/core/mixins.py +archivebox/core/models.py +archivebox/core/settings.py +archivebox/core/tests.py +archivebox/core/urls.py +archivebox/core/views.py +archivebox/core/welcome_message.py +archivebox/core/wsgi.py +archivebox/core/management/commands/archivebox.py +archivebox/core/migrations/0001_initial.py +archivebox/core/migrations/0002_auto_20200625_1521.py +archivebox/core/migrations/0003_auto_20200630_1034.py +archivebox/core/migrations/0004_auto_20200713_1552.py +archivebox/core/migrations/0005_auto_20200728_0326.py +archivebox/core/migrations/0006_auto_20201012_1520.py +archivebox/core/migrations/0007_archiveresult.py +archivebox/core/migrations/0008_auto_20210105_1421.py +archivebox/core/migrations/__init__.py +archivebox/core/templatetags/__init__.py +archivebox/core/templatetags/core_tags.py +archivebox/extractors/__init__.py +archivebox/extractors/archive_org.py +archivebox/extractors/dom.py +archivebox/extractors/favicon.py +archivebox/extractors/git.py +archivebox/extractors/headers.py +archivebox/extractors/media.py +archivebox/extractors/mercury.py +archivebox/extractors/pdf.py +archivebox/extractors/readability.py +archivebox/extractors/screenshot.py +archivebox/extractors/singlefile.py +archivebox/extractors/title.py +archivebox/extractors/wget.py +archivebox/index/__init__.py +archivebox/index/csv.py +archivebox/index/html.py +archivebox/index/json.py +archivebox/index/schema.py +archivebox/index/sql.py +archivebox/parsers/__init__.py +archivebox/parsers/generic_html.py +archivebox/parsers/generic_json.py +archivebox/parsers/generic_rss.py +archivebox/parsers/generic_txt.py +archivebox/parsers/medium_rss.py +archivebox/parsers/netscape_html.py +archivebox/parsers/pinboard_rss.py +archivebox/parsers/pocket_api.py +archivebox/parsers/pocket_html.py +archivebox/parsers/shaarli_rss.py +archivebox/parsers/wallabag_atom.py +archivebox/search/__init__.py +archivebox/search/utils.py +archivebox/search/backends/__init__.py +archivebox/search/backends/ripgrep.py +archivebox/search/backends/sonic.py +archivebox/themes/admin/actions_as_select.html +archivebox/themes/admin/app_index.html +archivebox/themes/admin/base.html +archivebox/themes/admin/grid_change_list.html +archivebox/themes/admin/login.html +archivebox/themes/admin/snapshots_grid.html +archivebox/themes/default/add_links.html +archivebox/themes/default/base.html +archivebox/themes/default/link_details.html +archivebox/themes/default/main_index.html +archivebox/themes/default/main_index_minimal.html +archivebox/themes/default/main_index_row.html +archivebox/themes/default/core/snapshot_list.html +archivebox/themes/default/static/add.css +archivebox/themes/default/static/admin.css +archivebox/themes/default/static/archive.png +archivebox/themes/default/static/bootstrap.min.css +archivebox/themes/default/static/external.png +archivebox/themes/default/static/jquery.dataTables.min.css +archivebox/themes/default/static/jquery.dataTables.min.js +archivebox/themes/default/static/jquery.min.js +archivebox/themes/default/static/sort_asc.png +archivebox/themes/default/static/sort_both.png +archivebox/themes/default/static/sort_desc.png +archivebox/themes/default/static/spinner.gif +archivebox/themes/legacy/main_index.html +archivebox/themes/legacy/main_index_row.html +archivebox/vendor/__init__.py \ No newline at end of file diff --git a/archivebox-0.5.3/archivebox.egg-info/dependency_links.txt b/archivebox-0.5.3/archivebox.egg-info/dependency_links.txt new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/archivebox-0.5.3/archivebox.egg-info/dependency_links.txt @@ -0,0 +1 @@ + diff --git a/archivebox-0.5.3/archivebox.egg-info/entry_points.txt b/archivebox-0.5.3/archivebox.egg-info/entry_points.txt new file mode 100644 index 0000000..14fdb7e --- /dev/null +++ b/archivebox-0.5.3/archivebox.egg-info/entry_points.txt @@ -0,0 +1,3 @@ +[console_scripts] +archivebox = archivebox.cli:main + diff --git a/archivebox-0.5.3/archivebox.egg-info/requires.txt b/archivebox-0.5.3/archivebox.egg-info/requires.txt new file mode 100644 index 0000000..64d30de --- /dev/null +++ b/archivebox-0.5.3/archivebox.egg-info/requires.txt @@ -0,0 +1,26 @@ +atomicwrites==1.4.0 +croniter==0.3.34 +dateparser +django-extensions==3.0.3 +django==3.1.3 +ipython +mypy-extensions==0.4.3 +python-crontab==2.5.1 +requests==2.24.0 +w3lib==1.22.0 +youtube-dl + +[dev] +bottle +django-stubs +flake8 +ipdb +mypy +pytest +recommonmark +setuptools +sphinx +sphinx-rtd-theme +stdeb +twine +wheel diff --git a/archivebox-0.5.3/archivebox.egg-info/top_level.txt b/archivebox-0.5.3/archivebox.egg-info/top_level.txt new file mode 100644 index 0000000..74056b6 --- /dev/null +++ b/archivebox-0.5.3/archivebox.egg-info/top_level.txt @@ -0,0 +1 @@ +archivebox diff --git a/archivebox-0.5.3/archivebox/.flake8 b/archivebox-0.5.3/archivebox/.flake8 new file mode 100644 index 0000000..dd6ba8e --- /dev/null +++ b/archivebox-0.5.3/archivebox/.flake8 @@ -0,0 +1,6 @@ +[flake8] +ignore = D100,D101,D102,D103,D104,D105,D202,D203,D205,D400,E131,E241,E252,E266,E272,E701,E731,W293,W503,W291,W391 +select = F,E9,W +max-line-length = 130 +max-complexity = 10 +exclude = migrations,tests,node_modules,vendor,static,venv,.venv,.venv2,.docker-venv diff --git a/archivebox-0.5.3/archivebox/LICENSE b/archivebox-0.5.3/archivebox/LICENSE new file mode 100644 index 0000000..ea201f9 --- /dev/null +++ b/archivebox-0.5.3/archivebox/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2020 Nick Sweeting + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/archivebox-0.5.3/archivebox/README.md b/archivebox-0.5.3/archivebox/README.md new file mode 100644 index 0000000..2e35783 --- /dev/null +++ b/archivebox-0.5.3/archivebox/README.md @@ -0,0 +1,545 @@ +
+ +

ArchiveBox
The open-source self-hosted web archive.

+ +▶️ Quickstart | +Demo | +Github | +Documentation | +Info & Motivation | +Community | +Roadmap + +
+"Your own personal internet archive" (网站存档 / 爬虫)
+
+ + + + + + + + + + +
+
+ +ArchiveBox is a powerful self-hosted internet archiving solution written in Python 3. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on the configuration and the content it detects. + +Your archive can be managed through the command line with commands like `archivebox add`, through the built-in Web UI `archivebox server`, or via the Python library API (beta). It can ingest bookmarks from a browser or service like Pocket/Pinboard, your entire browsing history, RSS feeds, or URLs one at a time. You can also schedule regular/realtime imports with `archivebox schedule`. + +The main index is a self-contained `index.sqlite3` file, and each snapshot is stored as a folder `data/archive//`, with an easy-to-read `index.html` and `index.json` within. For each page, ArchiveBox auto-extracts many types of assets/media and saves them in standard formats, with out-of-the-box support for: several types of HTML snapshots (wget, Chrome headless, singlefile), PDF snapshotting, screenshotting, WARC archiving, git repositories, images, audio, video, subtitles, article text, and more. The snapshots are browseable and managable offline through the filesystem, the built-in webserver, or the Python library API. + +### Quickstart + +It works on Linux/BSD (Intel and ARM CPUs with `docker`/`apt`/`pip3`), macOS (with `docker`/`brew`/`pip3`), and Windows (beta with `docker`/`pip3`). + +```bash +pip3 install archivebox +archivebox --version +# install extras as-needed, or use one of full setup methods below to get everything out-of-the-box + +mkdir ~/archivebox && cd ~/archivebox # this can be anywhere +archivebox init + +archivebox add 'https://example.com' +archivebox add --depth=1 'https://example.com' +archivebox schedule --every=day https://getpocket.com/users/USERNAME/feed/all +archivebox oneshot --extract=title,favicon,media https://www.youtube.com/watch?v=dQw4w9WgXcQ +archivebox help # to see more options +``` + +*(click to expand the sections below for full setup instructions)* + +
+Get ArchiveBox with docker-compose on any platform (recommended, everything included out-of-the-box) + +First make sure you have Docker installed: https://docs.docker.com/get-docker/ +

+This is the recommended way to run ArchiveBox because it includes *all* the extractors like chrome, wget, youtube-dl, git, etc., as well as full-text search with sonic, and many other great features. + +```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml +docker-compose run archivebox init +docker-compose run archivebox --version + +# start the webserver and open the UI (optional) +docker-compose run archivebox manage createsuperuser +docker-compose up -d +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker-compose run archivebox add 'https://example.com' +docker-compose run archivebox status +docker-compose run archivebox help # to see more options +``` + +
+ +
+Get ArchiveBox with docker on any platform + +First make sure you have Docker installed: https://docs.docker.com/get-docker/
+```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +docker run -v $PWD:/data -it archivebox/archivebox init +docker run -v $PWD:/data -it archivebox/archivebox --version + +# start the webserver and open the UI (optional) +docker run -v $PWD:/data -it archivebox/archivebox manage createsuperuser +docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com' +docker run -v $PWD:/data -it archivebox/archivebox status +docker run -v $PWD:/data -it archivebox/archivebox help # to see more options +``` + +
+ +
+Get ArchiveBox with apt on Ubuntu >=20.04 + +```bash +sudo add-apt-repository -u ppa:archivebox/archivebox +sudo apt install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`: +```bash +deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +``` +(you may need to install some other dependencies manually however) + +
+ +
+Get ArchiveBox with brew on macOS >=10.13 + +```bash +brew install archivebox/archivebox/archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
+ +
+Get ArchiveBox with pip on any platform + +```bash +pip3 install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version +# Install any missing extras like wget/git/chrome/etc. manually as needed + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
+ +--- + +
+ +
+ +DEMO: archivebox.zervice.io/ +For more information, see the full Quickstart guide, Usage, and Configuration docs. +
+ +--- + + +# Overview + +ArchiveBox is a command line tool, self-hostable web-archiving server, and Python library all-in-one. It can be installed on Docker, macOS, and Linux/BSD, and Windows. You can download and install it as a Debian/Ubuntu package, Homebrew package, Python3 package, or a Docker image. No matter which install method you choose, they all provide the same CLI, Web UI, and on-disk data format. + +To use ArchiveBox you start by creating a folder for your data to live in (it can be anywhere on your system), and running `archivebox init` inside of it. That will create a sqlite3 index and an `ArchiveBox.conf` file. After that, you can continue to add/export/manage/etc using the CLI `archivebox help`, or you can run the Web UI (recommended). If you only want to archive a single site, you can run `archivebox oneshot` to avoid having to create a whole collection. + +The CLI is considered "stable", the ArchiveBox Python API and REST APIs are "beta", and the [desktop app](https://github.com/ArchiveBox/desktop) is "alpha". + +At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer). You can also self-host your archivebox server on a public domain to provide archive.org-style public access to your site snapshots. + +
+CLI Screenshot +Desktop index screenshot +Desktop details page Screenshot +Desktop details page Screenshot
+Demo | Usage | Screenshots +
+. . . . . . . . . . . . . . . . . . . . . . . . . . . . +

+ + +## Key Features + +- [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally +- [**Few dependencies**](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) +- [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) +- Easy to set up **[scheduled importing](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources** +- Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC +- ~~**Suitable for paywalled / [authenticated content](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes) +- **Doesn't require a constantly-running daemon**, proxy, or native app +- Provides a CLI, Python API, self-hosted web UI, and REST API (WIP) +- Architected to be able to run [**many varieties of scripts during archiving**](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. to extract media, summarize articles, [scroll pages](https://github.com/ArchiveBox/ArchiveBox/issues/80), [close modals](https://github.com/ArchiveBox/ArchiveBox/issues/175), expand comment threads, etc. +- Can also [**mirror content to 3rd-party archiving services**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy + +## Input formats + +ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more! + +```bash +echo 'http://example.com' | archivebox add +archivebox add 'https://example.com/some/page' +archivebox add < ~/Downloads/firefox_bookmarks_export.html +archivebox add < any_text_with_urls_in_it.txt +archivebox add --depth=1 'https://example.com/some/downloads.html' +archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12' +``` + +- Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more) +- RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format +- Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more + +See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples. + +It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand. + +## Output formats + +All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`. + +The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.) + +```bash + ls ./archive// +``` + +- **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details +- **Title:** `title` title of the site +- **Favicon:** `favicon.ico` favicon of the site +- **Headers:** `headers.json` Any HTTP headers the site returns are saved in a json file +- **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile +- **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present +- **WARC:** `warc/.gz` gzipped WARC of all the resources fetched while archiving +- **PDF:** `output.pdf` Printed PDF of site using headless chrome +- **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome +- **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome +- **Readability:** `article.html/json` Article text extraction using Readability +- **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org +- **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl +- **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links +- _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._ + +It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file. + +## Dependencies + +You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled. + +If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) or the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install). + +ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more. + +## Caveats + +If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process. +```bash +# don't do this: +archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere' +archivebox add 'https://example.com/any/url/you/want/to/keep/secret/' + +# without first disabling share the URL with 3rd party APIs: +archivebox config --set SAVE_ARCHIVE_DOT_ORG=False # disable saving all URLs in Archive.org +archivebox config --set SAVE_FAVICON=False # optional: only the domain is leaked, not full URL +archivebox config --set CHROME_BINARY=chromium # optional: switch to chromium to avoid Chrome phoning home to Google +``` + +Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details. +```bash +# visiting an archived page with malicious JS: +https://127.0.0.1:8000/archive/1602401954/example.com/index.html + +# example.com/index.js can now make a request to read everything: +https://127.0.0.1:8000/index.html +https://127.0.0.1:8000/archive/* +# then example.com/index.js can send it off to some evil server +``` + +Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash: +```bash +archivebox add 'https://example.com#2020-10-24' +... +archivebox add 'https://example.com#2020-10-25' +``` + +--- + +
+ +
+ +--- + +# Background & Motivation + +Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity. + +Whether it's to resist censorship by saving articles before they get taken down or edited, or +just to save a collection of early 2010's flash games you love to play, having the tools to +archive internet content enables to you save the stuff you care most about before it disappears. + +
+
+ Image from WTF is Link Rot?...
 +
+ +The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. +I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about. + +Because modern websites are complicated and often rely on dynamic content, +ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats. + +All the archived links are stored by date bookmarked in `./archive/`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM. + +## Comparison to Other Projects + +▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.** + +comparison The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations. + +#### User Interface & Intended Purpose + +ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend. + +#### Private Local Archives vs Centralized Public Archives + +Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle. + +#### Storage Requirements + +Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files. + +## Learn more + +Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web! + + + +- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists) + _Community-maintained indexes of archiving tools and institutions._ + - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) + _Open source tools and projects in the internet archiving space._ + - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List) + _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._ + - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities) + _A collection of the most active internet archiving communities and initiatives._ +- Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post. +- Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter. + +--- + +# Documentation + + + +We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation. + +You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder. + +## Getting Started + +- [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) +- [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) +- [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + +## Reference + +- [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) +- [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) +- [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) +- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site) +- [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) +- [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) +- [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) +- [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) +- [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) +- [Python API](https://docs.archivebox.io/en/latest/modules.html) +- REST API (coming soon...) + +## More Info + +- [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues) +- [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) +- [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) +- [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation) +- [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + +--- + +# ArchiveBox Development + +All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap. + +### Setup the dev environment + +First, install the system dependencies from the "Bare Metal" section above. +Then you can clone the ArchiveBox repo and install +```python3 +git clone https://github.com/ArchiveBox/ArchiveBox && cd ArchiveBox +git checkout master # or the branch you want to test +git submodule update --init --recursive +git pull --recurse-submodules + +# Install ArchiveBox + python dependencies +python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev] +# or with pipenv: pipenv install --dev && pipenv shell + +# Install node dependencies +npm install + +# Optional: install extractor dependencies manually or with helper script +./bin/setup.sh + +# Optional: develop via docker by mounting the code dir into the container +# if you edit e.g. ./archivebox/core/models.py on the docker host, runserver +# inside the container will reload and pick up your changes +docker build . -t archivebox +docker run -it -p 8000:8000 \ + -v $PWD/data:/data \ + -v $PWD/archivebox:/app/archivebox \ + archivebox server 0.0.0.0:8000 --debug --reload +``` + +### Common development tasks + +See the `./bin/` folder and read the source of the bash scripts within. +You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`. + +#### Run the linters + +```bash +./bin/lint.sh +``` +(uses `flake8` and `mypy`) + +#### Run the integration tests + +```bash +./bin/test.sh +``` +(uses `pytest -s`) + +#### Make migrations or enter a django shell + +```bash +cd archivebox/ +./manage.py makemigrations + +cd data/ +archivebox shell +``` +(uses `pytest -s`) + +#### Build the docs, pip package, and docker image + +```bash +./bin/build.sh + +# or individually: +./bin/build_docs.sh +./bin/build_pip.sh +./bin/build_deb.sh +./bin/build_brew.sh +./bin/build_docker.sh +``` + +#### Roll a release + +```bash +./bin/release.sh +``` +(bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages) + + +--- + +
+

+ +
+This project is maintained mostly in my spare time with the help from generous contributors and Monadical.com. +

+ +
+Sponsor us on Github +
+
+ +
+ + + + +

+ +
diff --git a/archivebox-0.5.3/archivebox/__init__.py b/archivebox-0.5.3/archivebox/__init__.py new file mode 100644 index 0000000..b0c00b6 --- /dev/null +++ b/archivebox-0.5.3/archivebox/__init__.py @@ -0,0 +1 @@ +__package__ = 'archivebox' diff --git a/archivebox-0.5.3/archivebox/__main__.py b/archivebox-0.5.3/archivebox/__main__.py new file mode 100755 index 0000000..8afaa27 --- /dev/null +++ b/archivebox-0.5.3/archivebox/__main__.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox' + +import sys + +from .cli import main + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/__init__.py b/archivebox-0.5.3/archivebox/cli/__init__.py new file mode 100644 index 0000000..f9a55ef --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/__init__.py @@ -0,0 +1,144 @@ +__package__ = 'archivebox.cli' +__command__ = 'archivebox' + +import os +import sys +import argparse + +from typing import Optional, Dict, List, IO, Union +from pathlib import Path + +from ..config import OUTPUT_DIR + +from importlib import import_module + +CLI_DIR = Path(__file__).resolve().parent + +# these common commands will appear sorted before any others for ease-of-use +meta_cmds = ('help', 'version') +main_cmds = ('init', 'info', 'config') +archive_cmds = ('add', 'remove', 'update', 'list', 'status') + +fake_db = ("oneshot",) + +display_first = (*meta_cmds, *main_cmds, *archive_cmds) + +# every imported command module must have these properties in order to be valid +required_attrs = ('__package__', '__command__', 'main') + +# basic checks to make sure imported files are valid subcommands +is_cli_module = lambda fname: fname.startswith('archivebox_') and fname.endswith('.py') +is_valid_cli_module = lambda module, subcommand: ( + all(hasattr(module, attr) for attr in required_attrs) + and module.__command__.split(' ')[-1] == subcommand +) + + +def list_subcommands() -> Dict[str, str]: + """find and import all valid archivebox_.py files in CLI_DIR""" + + COMMANDS = [] + for filename in os.listdir(CLI_DIR): + if is_cli_module(filename): + subcommand = filename.replace('archivebox_', '').replace('.py', '') + module = import_module('.archivebox_{}'.format(subcommand), __package__) + assert is_valid_cli_module(module, subcommand) + COMMANDS.append((subcommand, module.main.__doc__)) + globals()[subcommand] = module.main + + display_order = lambda cmd: ( + display_first.index(cmd[0]) + if cmd[0] in display_first else + 100 + len(cmd[0]) + ) + + return dict(sorted(COMMANDS, key=display_order)) + + +def run_subcommand(subcommand: str, + subcommand_args: List[str]=None, + stdin: Optional[IO]=None, + pwd: Union[Path, str, None]=None) -> None: + """Run a given ArchiveBox subcommand with the given list of args""" + + if subcommand not in meta_cmds: + from ..config import setup_django + setup_django(in_memory_db=subcommand in fake_db, check_db=subcommand in archive_cmds) + + module = import_module('.archivebox_{}'.format(subcommand), __package__) + module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore + + +SUBCOMMANDS = list_subcommands() + +class NotProvided: + pass + + +def main(args: Optional[List[str]]=NotProvided, stdin: Optional[IO]=NotProvided, pwd: Optional[str]=None) -> None: + args = sys.argv[1:] if args is NotProvided else args + stdin = sys.stdin if stdin is NotProvided else stdin + + subcommands = list_subcommands() + parser = argparse.ArgumentParser( + prog=__command__, + description='ArchiveBox: The self-hosted internet archive', + add_help=False, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--help', '-h', + action='store_true', + help=subcommands['help'], + ) + group.add_argument( + '--version', + action='store_true', + help=subcommands['version'], + ) + group.add_argument( + "subcommand", + type=str, + help= "The name of the subcommand to run", + nargs='?', + choices=subcommands.keys(), + default=None, + ) + parser.add_argument( + "subcommand_args", + help="Arguments for the subcommand", + nargs=argparse.REMAINDER, + ) + command = parser.parse_args(args or ()) + + if command.version: + command.subcommand = 'version' + elif command.help or command.subcommand is None: + command.subcommand = 'help' + + if command.subcommand not in ('help', 'version', 'status'): + from ..logging_util import log_cli_command + + log_cli_command( + subcommand=command.subcommand, + subcommand_args=command.subcommand_args, + stdin=stdin, + pwd=pwd or OUTPUT_DIR + ) + + run_subcommand( + subcommand=command.subcommand, + subcommand_args=command.subcommand_args, + stdin=stdin, + pwd=pwd or OUTPUT_DIR, + ) + + +__all__ = ( + 'SUBCOMMANDS', + 'list_subcommands', + 'run_subcommand', + *SUBCOMMANDS.keys(), +) + + diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_add.py b/archivebox-0.5.3/archivebox/cli/archivebox_add.py new file mode 100644 index 0000000..41c7554 --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_add.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox add' + +import sys +import argparse + +from typing import List, Optional, IO + +from ..main import add +from ..util import docstring +from ..config import OUTPUT_DIR, ONLY_NEW +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(add.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=add.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--update-all', #'-n', + action='store_true', + default=not ONLY_NEW, # when ONLY_NEW=True we skip updating old links + help="Also retry previously skipped/failed links when adding new links", + ) + parser.add_argument( + '--index-only', #'-o', + action='store_true', + help="Add the links to the main index without archiving them", + ) + parser.add_argument( + 'urls', + nargs='*', + type=str, + default=None, + help=( + 'URLs or paths to archive e.g.:\n' + ' https://getpocket.com/users/USERNAME/feed/all\n' + ' https://example.com/some/rss/feed.xml\n' + ' https://example.com\n' + ' ~/Downloads/firefox_bookmarks_export.html\n' + ' ~/Desktop/sites_list.csv\n' + ) + ) + parser.add_argument( + "--depth", + action="store", + default=0, + choices=[0, 1], + type=int, + help="Recursively archive all linked pages up to this many hops away" + ) + parser.add_argument( + "--overwrite", + default=False, + action="store_true", + help="Re-archive URLs from scratch, overwriting any existing files" + ) + parser.add_argument( + "--init", #'-i', + action='store_true', + help="Init/upgrade the curent data directory before adding", + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + command = parser.parse_args(args or ()) + urls = command.urls + stdin_urls = accept_stdin(stdin) + if (stdin_urls and urls) or (not stdin and not urls): + stderr( + '[X] You must pass URLs/paths to add via stdin or CLI arguments.\n', + color='red', + ) + raise SystemExit(2) + add( + urls=stdin_urls or urls, + depth=command.depth, + update_all=command.update_all, + index_only=command.index_only, + overwrite=command.overwrite, + init=command.init, + extractors=command.extract, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) + + +# TODO: Implement these +# +# parser.add_argument( +# '--mirror', #'-m', +# action='store_true', +# help='Archive an entire site (finding all linked pages below it on the same domain)', +# ) +# parser.add_argument( +# '--crawler', #'-r', +# choices=('depth_first', 'breadth_first'), +# help='Controls which crawler to use in order to find outlinks in a given page', +# default=None, +# ) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_config.py b/archivebox-0.5.3/archivebox/cli/archivebox_config.py new file mode 100644 index 0000000..f81286c --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_config.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox config' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import config +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(config.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=config.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--get', #'-g', + action='store_true', + help="Get the value for the given config KEYs", + ) + group.add_argument( + '--set', #'-s', + action='store_true', + help="Set the given KEY=VALUE config values", + ) + group.add_argument( + '--reset', #'-s', + action='store_true', + help="Reset the given KEY config values to their defaults", + ) + parser.add_argument( + 'config_options', + nargs='*', + type=str, + help='KEY or KEY=VALUE formatted config values to get or set', + ) + command = parser.parse_args(args or ()) + config_options_str = accept_stdin(stdin) + + config( + config_options_str=config_options_str, + config_options=command.config_options, + get=command.get, + set=command.set, + reset=command.reset, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_help.py b/archivebox-0.5.3/archivebox/cli/archivebox_help.py new file mode 100755 index 0000000..46f17cb --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_help.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox help' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import help +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(help.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=help.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + help(out_dir=pwd or OUTPUT_DIR) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_init.py b/archivebox-0.5.3/archivebox/cli/archivebox_init.py new file mode 100755 index 0000000..6255ef2 --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_init.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox init' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import init +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(init.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=init.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--force', # '-f', + action='store_true', + help='Ignore unrecognized files in current directory and initialize anyway', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + init( + force=command.force, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_list.py b/archivebox-0.5.3/archivebox/cli/archivebox_list.py new file mode 100644 index 0000000..3838cf6 --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_list.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox list' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import list_all +from ..util import docstring +from ..config import OUTPUT_DIR +from ..index import ( + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, +) +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(list_all.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=list_all.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--csv', #'-c', + type=str, + help="Print the output in CSV format with the given columns, e.g.: timestamp,url,extension", + default=None, + ) + group.add_argument( + '--json', #'-j', + action='store_true', + help="Print the output in JSON format with all columns included.", + ) + group.add_argument( + '--html', + action='store_true', + help="Print the output in HTML format" + ) + parser.add_argument( + '--with-headers', + action='store_true', + help='Include the headers in the output document' + ) + parser.add_argument( + '--sort', #'-s', + type=str, + help="List the links sorted using the given key, e.g. timestamp or updated.", + default=None, + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="List only links bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="List only links bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--status', + type=str, + choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'), + default='indexed', + help=( + 'List only links or data directories that have the given status\n' + f' indexed {get_indexed_folders.__doc__} (the default)\n' + f' archived {get_archived_folders.__doc__}\n' + f' unarchived {get_unarchived_folders.__doc__}\n' + '\n' + f' present {get_present_folders.__doc__}\n' + f' valid {get_valid_folders.__doc__}\n' + f' invalid {get_invalid_folders.__doc__}\n' + '\n' + f' duplicate {get_duplicate_folders.__doc__}\n' + f' orphaned {get_orphaned_folders.__doc__}\n' + f' corrupted {get_corrupted_folders.__doc__}\n' + f' unrecognized {get_unrecognized_folders.__doc__}\n' + ) + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + default=None, + help='List only URLs matching these filter patterns.' + ) + command = parser.parse_args(args or ()) + filter_patterns_str = accept_stdin(stdin) + + if command.with_headers and not (command.json or command.html or command.csv): + stderr( + '[X] --with-headers can only be used with --json, --html or --csv options.\n', + color='red', + ) + raise SystemExit(2) + + matching_folders = list_all( + filter_patterns_str=filter_patterns_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + status=command.status, + after=command.after, + before=command.before, + sort=command.sort, + csv=command.csv, + json=command.json, + html=command.html, + with_headers=command.with_headers, + out_dir=pwd or OUTPUT_DIR, + ) + raise SystemExit(not matching_folders) + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_manage.py b/archivebox-0.5.3/archivebox/cli/archivebox_manage.py new file mode 100644 index 0000000..f05604e --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_manage.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox manage' + +import sys + +from typing import Optional, List, IO + +from ..main import manage +from ..util import docstring +from ..config import OUTPUT_DIR + + +@docstring(manage.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + manage( + args=args, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_oneshot.py b/archivebox-0.5.3/archivebox/cli/archivebox_oneshot.py new file mode 100644 index 0000000..af68bac --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_oneshot.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox oneshot' + +import sys +import argparse + +from pathlib import Path +from typing import List, Optional, IO + +from ..main import oneshot +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(oneshot.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=oneshot.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + 'url', + type=str, + default=None, + help=( + 'URLs or paths to archive e.g.:\n' + ' https://getpocket.com/users/USERNAME/feed/all\n' + ' https://example.com/some/rss/feed.xml\n' + ' https://example.com\n' + ' ~/Downloads/firefox_bookmarks_export.html\n' + ' ~/Desktop/sites_list.csv\n' + ) + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + parser.add_argument( + '--out-dir', + type=str, + default=OUTPUT_DIR, + help= "Path to save the single archive folder to, e.g. ./example.com_archive" + ) + command = parser.parse_args(args or ()) + url = command.url + stdin_url = accept_stdin(stdin) + if (stdin_url and url) or (not stdin and not url): + stderr( + '[X] You must pass a URL/path to add via stdin or CLI arguments.\n', + color='red', + ) + raise SystemExit(2) + + oneshot( + url=stdin_url or url, + out_dir=Path(command.out_dir).resolve(), + extractors=command.extract, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_remove.py b/archivebox-0.5.3/archivebox/cli/archivebox_remove.py new file mode 100644 index 0000000..cb073e9 --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_remove.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox remove' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import remove +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(remove.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=remove.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--yes', # '-y', + action='store_true', + help='Remove links instantly without prompting to confirm.', + ) + parser.add_argument( + '--delete', # '-r', + action='store_true', + help=( + "In addition to removing the link from the index, " + "also delete its archived content and metadata folder." + ), + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="List only URLs bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="List only URLs bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex','tag'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + help='URLs matching this filter pattern will be removed from the index.' + ) + command = parser.parse_args(args or ()) + filter_str = accept_stdin(stdin) + + remove( + filter_str=filter_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + before=command.before, + after=command.after, + yes=command.yes, + delete=command.delete, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_schedule.py b/archivebox-0.5.3/archivebox/cli/archivebox_schedule.py new file mode 100644 index 0000000..ec5e914 --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_schedule.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox schedule' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import schedule +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(schedule.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=schedule.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--quiet', '-q', + action='store_true', + help=("Don't warn about storage space."), + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--add', # '-a', + action='store_true', + help='Add a new scheduled ArchiveBox update job to cron', + ) + parser.add_argument( + '--every', # '-e', + type=str, + default=None, + help='Run ArchiveBox once every [timeperiod] (hour/day/month/year or cron format e.g. "0 0 * * *")', + ) + parser.add_argument( + '--depth', # '-d', + type=int, + default=0, + help='Depth to archive to [0] or 1, see "add" command help for more info.', + ) + group.add_argument( + '--clear', # '-c' + action='store_true', + help=("Stop all ArchiveBox scheduled runs (remove cron jobs)"), + ) + group.add_argument( + '--show', # '-s' + action='store_true', + help=("Print a list of currently active ArchiveBox cron jobs"), + ) + group.add_argument( + '--foreground', '-f', + action='store_true', + help=("Launch ArchiveBox scheduler as a long-running foreground task " + "instead of using cron."), + ) + group.add_argument( + '--run-all', # '-a', + action='store_true', + help=("Run all the scheduled jobs once immediately, independent of " + "their configured schedules, can be used together with --foreground"), + ) + parser.add_argument( + 'import_path', + nargs='?', + type=str, + default=None, + help=("Check this path and import any new links on every run " + "(can be either local file or remote URL)"), + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + schedule( + add=command.add, + show=command.show, + clear=command.clear, + foreground=command.foreground, + run_all=command.run_all, + quiet=command.quiet, + every=command.every, + depth=command.depth, + import_path=command.import_path, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_server.py b/archivebox-0.5.3/archivebox/cli/archivebox_server.py new file mode 100644 index 0000000..dbacf7e --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_server.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox server' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import server +from ..util import docstring +from ..config import OUTPUT_DIR, BIND_ADDR +from ..logging_util import SmartFormatter, reject_stdin + +@docstring(server.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=server.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + 'runserver_args', + nargs='*', + type=str, + default=[BIND_ADDR], + help='Arguments to pass to Django runserver' + ) + parser.add_argument( + '--reload', + action='store_true', + help='Enable auto-reloading when code or templates change', + ) + parser.add_argument( + '--debug', + action='store_true', + help='Enable DEBUG=True mode with more verbose errors', + ) + parser.add_argument( + '--init', + action='store_true', + help='Run archivebox init before starting the server', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + server( + runserver_args=command.runserver_args, + reload=command.reload, + debug=command.debug, + init=command.init, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_shell.py b/archivebox-0.5.3/archivebox/cli/archivebox_shell.py new file mode 100644 index 0000000..bcd5fdd --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_shell.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox shell' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import shell +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(shell.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=shell.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + shell( + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_status.py b/archivebox-0.5.3/archivebox/cli/archivebox_status.py new file mode 100644 index 0000000..2bef19c --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_status.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox status' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import status +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(status.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=status.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + status(out_dir=pwd or OUTPUT_DIR) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_update.py b/archivebox-0.5.3/archivebox/cli/archivebox_update.py new file mode 100644 index 0000000..6748096 --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_update.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox update' + +import sys +import argparse + +from typing import List, Optional, IO + +from ..main import update +from ..util import docstring +from ..config import OUTPUT_DIR +from ..index import ( + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, +) +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(update.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=update.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--only-new', #'-n', + action='store_true', + help="Don't attempt to retry previously skipped/failed links when updating", + ) + parser.add_argument( + '--index-only', #'-o', + action='store_true', + help="Update the main index without archiving any content", + ) + parser.add_argument( + '--resume', #'-r', + type=float, + help='Resume the update process from a given timestamp', + default=None, + ) + parser.add_argument( + '--overwrite', #'-x', + action='store_true', + help='Ignore existing archived content and overwrite with new versions (DANGEROUS)', + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="Update only links bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="Update only links bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--status', + type=str, + choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'), + default='indexed', + help=( + 'Update only links or data directories that have the given status\n' + f' indexed {get_indexed_folders.__doc__} (the default)\n' + f' archived {get_archived_folders.__doc__}\n' + f' unarchived {get_unarchived_folders.__doc__}\n' + '\n' + f' present {get_present_folders.__doc__}\n' + f' valid {get_valid_folders.__doc__}\n' + f' invalid {get_invalid_folders.__doc__}\n' + '\n' + f' duplicate {get_duplicate_folders.__doc__}\n' + f' orphaned {get_orphaned_folders.__doc__}\n' + f' corrupted {get_corrupted_folders.__doc__}\n' + f' unrecognized {get_unrecognized_folders.__doc__}\n' + ) + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + default=None, + help='Update only URLs matching these filter patterns.' + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + command = parser.parse_args(args or ()) + filter_patterns_str = accept_stdin(stdin) + + update( + resume=command.resume, + only_new=command.only_new, + index_only=command.index_only, + overwrite=command.overwrite, + filter_patterns_str=filter_patterns_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + status=command.status, + after=command.after, + before=command.before, + out_dir=pwd or OUTPUT_DIR, + extractors=command.extract, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/archivebox_version.py b/archivebox-0.5.3/archivebox/cli/archivebox_version.py new file mode 100755 index 0000000..e7922f3 --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/archivebox_version.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox version' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import version +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(version.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=version.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--quiet', '-q', + action='store_true', + help='Only print ArchiveBox version number and nothing else.', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + version( + quiet=command.quiet, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/archivebox/cli/tests.py b/archivebox-0.5.3/archivebox/cli/tests.py new file mode 100755 index 0000000..4d7016a --- /dev/null +++ b/archivebox-0.5.3/archivebox/cli/tests.py @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' + + +import os +import sys +import shutil +import unittest +from pathlib import Path + +from contextlib import contextmanager + +TEST_CONFIG = { + 'USE_COLOR': 'False', + 'SHOW_PROGRESS': 'False', + + 'OUTPUT_DIR': 'data.tests', + + 'SAVE_ARCHIVE_DOT_ORG': 'False', + 'SAVE_TITLE': 'False', + + 'USE_CURL': 'False', + 'USE_WGET': 'False', + 'USE_GIT': 'False', + 'USE_CHROME': 'False', + 'USE_YOUTUBEDL': 'False', +} + +OUTPUT_DIR = 'data.tests' +os.environ.update(TEST_CONFIG) + +from ..main import init +from ..index import load_main_index +from ..config import ( + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, +) + +from . import ( + archivebox_init, + archivebox_add, + archivebox_remove, +) + +HIDE_CLI_OUTPUT = True + +test_urls = ''' +https://example1.com/what/is/happening.html?what=1#how-about-this=1 +https://example2.com/what/is/happening/?what=1#how-about-this=1 +HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f +https://example4.com/what/is/happening.html +https://example5.com/ +https://example6.com + +http://example7.com +[https://example8.com/what/is/this.php?what=1] +[and http://example9.com?what=1&other=3#and-thing=2] +https://example10.com#and-thing=2 " +abcdef +sdflkf[what](https://subb.example12.com/who/what.php?whoami=1#whatami=2)?am=hi +example13.bada +and example14.badb +htt://example15.badc +''' + +stdout = sys.stdout +stderr = sys.stderr + + +@contextmanager +def output_hidden(show_failing=True): + if not HIDE_CLI_OUTPUT: + yield + return + + sys.stdout = open('stdout.txt', 'w+') + sys.stderr = open('stderr.txt', 'w+') + try: + yield + sys.stdout.close() + sys.stderr.close() + sys.stdout = stdout + sys.stderr = stderr + except: + sys.stdout.close() + sys.stderr.close() + sys.stdout = stdout + sys.stderr = stderr + if show_failing: + with open('stdout.txt', 'r') as f: + print(f.read()) + with open('stderr.txt', 'r') as f: + print(f.read()) + raise + finally: + os.remove('stdout.txt') + os.remove('stderr.txt') + + +class TestInit(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + + def tearDown(self): + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + def test_basic_init(self): + with output_hidden(): + archivebox_init.main([]) + + assert (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists() + assert (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists() + assert (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists() + assert len(load_main_index(out_dir=OUTPUT_DIR)) == 0 + + def test_conflicting_init(self): + with open(Path(OUTPUT_DIR) / 'test_conflict.txt', 'w+') as f: + f.write('test') + + try: + with output_hidden(show_failing=False): + archivebox_init.main([]) + assert False, 'Init should have exited with an exception' + except SystemExit: + pass + + assert not (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists() + assert not (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists() + assert not (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists() + try: + load_main_index(out_dir=OUTPUT_DIR) + assert False, 'load_main_index should raise an exception when no index is present' + except: + pass + + def test_no_dirty_state(self): + with output_hidden(): + init() + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + with output_hidden(): + init() + + +class TestAdd(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + with output_hidden(): + init() + + def tearDown(self): + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + def test_add_arg_url(self): + with output_hidden(): + archivebox_add.main(['https://getpocket.com/users/nikisweeting/feed/all']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 30 + + def test_add_arg_file(self): + test_file = Path(OUTPUT_DIR) / 'test.txt' + with open(test_file, 'w+') as f: + f.write(test_urls) + + with output_hidden(): + archivebox_add.main([test_file]) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 12 + os.remove(test_file) + + def test_add_stdin_url(self): + with output_hidden(): + archivebox_add.main([], stdin=test_urls) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 12 + + +class TestRemove(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + with output_hidden(): + init() + archivebox_add.main([], stdin=test_urls) + + # def tearDown(self): + # shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + + def test_remove_exact(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', 'https://example5.com/']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 11 + + def test_remove_regex(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', '--filter-type=regex', r'http(s)?:\/\/(.+\.)?(example\d\.com)']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 4 + + def test_remove_domain(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', '--filter-type=domain', 'example5.com', 'example6.com']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 10 + + def test_remove_none(self): + try: + with output_hidden(show_failing=False): + archivebox_remove.main(['--yes', '--delete', 'https://doesntexist.com']) + assert False, 'Should raise if no URLs match' + except: + pass + + +if __name__ == '__main__': + if '--verbose' in sys.argv or '-v' in sys.argv: + HIDE_CLI_OUTPUT = False + + unittest.main() diff --git a/archivebox-0.5.3/archivebox/config.py b/archivebox-0.5.3/archivebox/config.py new file mode 100644 index 0000000..9a3f9a7 --- /dev/null +++ b/archivebox-0.5.3/archivebox/config.py @@ -0,0 +1,1081 @@ +""" +ArchiveBox config definitons (including defaults and dynamic config options). + +Config Usage Example: + + archivebox config --set MEDIA_TIMEOUT=600 + env MEDIA_TIMEOUT=600 USE_COLOR=False ... archivebox [subcommand] ... + +Config Precedence Order: + + 1. cli args (--update-all / --index-only / etc.) + 2. shell environment vars (env USE_COLOR=False archivebox add '...') + 3. config file (echo "SAVE_FAVICON=False" >> ArchiveBox.conf) + 4. defaults (defined below in Python) + +Documentation: + + https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration + +""" + +__package__ = 'archivebox' + +import os +import io +import re +import sys +import json +import getpass +import shutil +import django + +from hashlib import md5 +from pathlib import Path +from typing import Optional, Type, Tuple, Dict, Union, List +from subprocess import run, PIPE, DEVNULL +from configparser import ConfigParser +from collections import defaultdict + +from .config_stubs import ( + SimpleConfigValueDict, + ConfigValue, + ConfigDict, + ConfigDefaultValue, + ConfigDefaultDict, +) + +############################### Config Schema ################################## + +CONFIG_SCHEMA: Dict[str, ConfigDefaultDict] = { + 'SHELL_CONFIG': { + 'IS_TTY': {'type': bool, 'default': lambda _: sys.stdout.isatty()}, + 'USE_COLOR': {'type': bool, 'default': lambda c: c['IS_TTY']}, + 'SHOW_PROGRESS': {'type': bool, 'default': lambda c: c['IS_TTY']}, + 'IN_DOCKER': {'type': bool, 'default': False}, + # TODO: 'SHOW_HINTS': {'type: bool, 'default': True}, + }, + + 'GENERAL_CONFIG': { + 'OUTPUT_DIR': {'type': str, 'default': None}, + 'CONFIG_FILE': {'type': str, 'default': None}, + 'ONLY_NEW': {'type': bool, 'default': True}, + 'TIMEOUT': {'type': int, 'default': 60}, + 'MEDIA_TIMEOUT': {'type': int, 'default': 3600}, + 'OUTPUT_PERMISSIONS': {'type': str, 'default': '755'}, + 'RESTRICT_FILE_NAMES': {'type': str, 'default': 'windows'}, + 'URL_BLACKLIST': {'type': str, 'default': r'\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$'}, # to avoid downloading code assets as their own pages + }, + + 'SERVER_CONFIG': { + 'SECRET_KEY': {'type': str, 'default': None}, + 'BIND_ADDR': {'type': str, 'default': lambda c: ['127.0.0.1:8000', '0.0.0.0:8000'][c['IN_DOCKER']]}, + 'ALLOWED_HOSTS': {'type': str, 'default': '*'}, + 'DEBUG': {'type': bool, 'default': False}, + 'PUBLIC_INDEX': {'type': bool, 'default': True}, + 'PUBLIC_SNAPSHOTS': {'type': bool, 'default': True}, + 'PUBLIC_ADD_VIEW': {'type': bool, 'default': False}, + 'FOOTER_INFO': {'type': str, 'default': 'Content is hosted for personal archiving purposes only. Contact server owner for any takedown requests.'}, + 'ACTIVE_THEME': {'type': str, 'default': 'default'}, + }, + + 'ARCHIVE_METHOD_TOGGLES': { + 'SAVE_TITLE': {'type': bool, 'default': True, 'aliases': ('FETCH_TITLE',)}, + 'SAVE_FAVICON': {'type': bool, 'default': True, 'aliases': ('FETCH_FAVICON',)}, + 'SAVE_WGET': {'type': bool, 'default': True, 'aliases': ('FETCH_WGET',)}, + 'SAVE_WGET_REQUISITES': {'type': bool, 'default': True, 'aliases': ('FETCH_WGET_REQUISITES',)}, + 'SAVE_SINGLEFILE': {'type': bool, 'default': True, 'aliases': ('FETCH_SINGLEFILE',)}, + 'SAVE_READABILITY': {'type': bool, 'default': True, 'aliases': ('FETCH_READABILITY',)}, + 'SAVE_MERCURY': {'type': bool, 'default': True, 'aliases': ('FETCH_MERCURY',)}, + 'SAVE_PDF': {'type': bool, 'default': True, 'aliases': ('FETCH_PDF',)}, + 'SAVE_SCREENSHOT': {'type': bool, 'default': True, 'aliases': ('FETCH_SCREENSHOT',)}, + 'SAVE_DOM': {'type': bool, 'default': True, 'aliases': ('FETCH_DOM',)}, + 'SAVE_HEADERS': {'type': bool, 'default': True, 'aliases': ('FETCH_HEADERS',)}, + 'SAVE_WARC': {'type': bool, 'default': True, 'aliases': ('FETCH_WARC',)}, + 'SAVE_GIT': {'type': bool, 'default': True, 'aliases': ('FETCH_GIT',)}, + 'SAVE_MEDIA': {'type': bool, 'default': True, 'aliases': ('FETCH_MEDIA',)}, + 'SAVE_ARCHIVE_DOT_ORG': {'type': bool, 'default': True, 'aliases': ('SUBMIT_ARCHIVE_DOT_ORG',)}, + }, + + 'ARCHIVE_METHOD_OPTIONS': { + 'RESOLUTION': {'type': str, 'default': '1440,2000', 'aliases': ('SCREENSHOT_RESOLUTION',)}, + 'GIT_DOMAINS': {'type': str, 'default': 'github.com,bitbucket.org,gitlab.com'}, + 'CHECK_SSL_VALIDITY': {'type': bool, 'default': True}, + + 'CURL_USER_AGENT': {'type': str, 'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) curl/{CURL_VERSION}'}, + 'WGET_USER_AGENT': {'type': str, 'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) wget/{WGET_VERSION}'}, + 'CHROME_USER_AGENT': {'type': str, 'default': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36'}, + + 'COOKIES_FILE': {'type': str, 'default': None}, + 'CHROME_USER_DATA_DIR': {'type': str, 'default': None}, + + 'CHROME_HEADLESS': {'type': bool, 'default': True}, + 'CHROME_SANDBOX': {'type': bool, 'default': lambda c: not c['IN_DOCKER']}, + 'YOUTUBEDL_ARGS': {'type': list, 'default': ['--write-description', + '--write-info-json', + '--write-annotations', + '--write-thumbnail', + '--no-call-home', + '--user-agent', + '--all-subs', + '--extract-audio', + '--keep-video', + '--ignore-errors', + '--geo-bypass', + '--audio-format', 'mp3', + '--audio-quality', '320K', + '--embed-thumbnail', + '--add-metadata']}, + + 'WGET_ARGS': {'type': list, 'default': ['--no-verbose', + '--adjust-extension', + '--convert-links', + '--force-directories', + '--backup-converted', + '--span-hosts', + '--no-parent', + '-e', 'robots=off', + ]}, + 'CURL_ARGS': {'type': list, 'default': ['--silent', + '--location', + '--compressed' + ]}, + 'GIT_ARGS': {'type': list, 'default': ['--recursive']}, + }, + + 'SEARCH_BACKEND_CONFIG' : { + 'USE_INDEXING_BACKEND': {'type': bool, 'default': True}, + 'USE_SEARCHING_BACKEND': {'type': bool, 'default': True}, + 'SEARCH_BACKEND_ENGINE': {'type': str, 'default': 'ripgrep'}, + 'SEARCH_BACKEND_HOST_NAME': {'type': str, 'default': 'localhost'}, + 'SEARCH_BACKEND_PORT': {'type': int, 'default': 1491}, + 'SEARCH_BACKEND_PASSWORD': {'type': str, 'default': 'SecretPassword'}, + # SONIC + 'SONIC_COLLECTION': {'type': str, 'default': 'archivebox'}, + 'SONIC_BUCKET': {'type': str, 'default': 'snapshots'}, + }, + + 'DEPENDENCY_CONFIG': { + 'USE_CURL': {'type': bool, 'default': True}, + 'USE_WGET': {'type': bool, 'default': True}, + 'USE_SINGLEFILE': {'type': bool, 'default': True}, + 'USE_READABILITY': {'type': bool, 'default': True}, + 'USE_MERCURY': {'type': bool, 'default': True}, + 'USE_GIT': {'type': bool, 'default': True}, + 'USE_CHROME': {'type': bool, 'default': True}, + 'USE_NODE': {'type': bool, 'default': True}, + 'USE_YOUTUBEDL': {'type': bool, 'default': True}, + 'USE_RIPGREP': {'type': bool, 'default': True}, + + 'CURL_BINARY': {'type': str, 'default': 'curl'}, + 'GIT_BINARY': {'type': str, 'default': 'git'}, + 'WGET_BINARY': {'type': str, 'default': 'wget'}, + 'SINGLEFILE_BINARY': {'type': str, 'default': 'single-file'}, + 'READABILITY_BINARY': {'type': str, 'default': 'readability-extractor'}, + 'MERCURY_BINARY': {'type': str, 'default': 'mercury-parser'}, + 'YOUTUBEDL_BINARY': {'type': str, 'default': 'youtube-dl'}, + 'NODE_BINARY': {'type': str, 'default': 'node'}, + 'RIPGREP_BINARY': {'type': str, 'default': 'rg'}, + 'CHROME_BINARY': {'type': str, 'default': None}, + + 'POCKET_CONSUMER_KEY': {'type': str, 'default': None}, + 'POCKET_ACCESS_TOKENS': {'type': dict, 'default': {}}, + }, +} + + +########################## Backwards-Compatibility ############################# + + +# for backwards compatibility with old config files, check old/deprecated names for each key +CONFIG_ALIASES = { + alias: key + for section in CONFIG_SCHEMA.values() + for key, default in section.items() + for alias in default.get('aliases', ()) +} +USER_CONFIG = {key for section in CONFIG_SCHEMA.values() for key in section.keys()} + +def get_real_name(key: str) -> str: + """get the current canonical name for a given deprecated config key""" + return CONFIG_ALIASES.get(key.upper().strip(), key.upper().strip()) + + + +################################ Constants ##################################### + +PACKAGE_DIR_NAME = 'archivebox' +TEMPLATES_DIR_NAME = 'themes' + +ARCHIVE_DIR_NAME = 'archive' +SOURCES_DIR_NAME = 'sources' +LOGS_DIR_NAME = 'logs' +STATIC_DIR_NAME = 'static' +SQL_INDEX_FILENAME = 'index.sqlite3' +JSON_INDEX_FILENAME = 'index.json' +HTML_INDEX_FILENAME = 'index.html' +ROBOTS_TXT_FILENAME = 'robots.txt' +FAVICON_FILENAME = 'favicon.ico' +CONFIG_FILENAME = 'ArchiveBox.conf' + +DEFAULT_CLI_COLORS = { + 'reset': '\033[00;00m', + 'lightblue': '\033[01;30m', + 'lightyellow': '\033[01;33m', + 'lightred': '\033[01;35m', + 'red': '\033[01;31m', + 'green': '\033[01;32m', + 'blue': '\033[01;34m', + 'white': '\033[01;37m', + 'black': '\033[01;30m', +} +ANSI = {k: '' for k in DEFAULT_CLI_COLORS.keys()} + +COLOR_DICT = defaultdict(lambda: [(0, 0, 0), (0, 0, 0)], { + '00': [(0, 0, 0), (0, 0, 0)], + '30': [(0, 0, 0), (0, 0, 0)], + '31': [(255, 0, 0), (128, 0, 0)], + '32': [(0, 200, 0), (0, 128, 0)], + '33': [(255, 255, 0), (128, 128, 0)], + '34': [(0, 0, 255), (0, 0, 128)], + '35': [(255, 0, 255), (128, 0, 128)], + '36': [(0, 255, 255), (0, 128, 128)], + '37': [(255, 255, 255), (255, 255, 255)], +}) + +STATICFILE_EXTENSIONS = { + # 99.999% of the time, URLs ending in these extensions are static files + # that can be downloaded as-is, not html pages that need to be rendered + 'gif', 'jpeg', 'jpg', 'png', 'tif', 'tiff', 'wbmp', 'ico', 'jng', 'bmp', + 'svg', 'svgz', 'webp', 'ps', 'eps', 'ai', + 'mp3', 'mp4', 'm4a', 'mpeg', 'mpg', 'mkv', 'mov', 'webm', 'm4v', + 'flv', 'wmv', 'avi', 'ogg', 'ts', 'm3u8', + 'pdf', 'txt', 'rtf', 'rtfd', 'doc', 'docx', 'ppt', 'pptx', 'xls', 'xlsx', + 'atom', 'rss', 'css', 'js', 'json', + 'dmg', 'iso', 'img', + 'rar', 'war', 'hqx', 'zip', 'gz', 'bz2', '7z', + + # Less common extensions to consider adding later + # jar, swf, bin, com, exe, dll, deb + # ear, hqx, eot, wmlc, kml, kmz, cco, jardiff, jnlp, run, msi, msp, msm, + # pl pm, prc pdb, rar, rpm, sea, sit, tcl tk, der, pem, crt, xpi, xspf, + # ra, mng, asx, asf, 3gpp, 3gp, mid, midi, kar, jad, wml, htc, mml + + # These are always treated as pages, not as static files, never add them: + # html, htm, shtml, xhtml, xml, aspx, php, cgi +} + + + +############################## Derived Config ################################## + + +DYNAMIC_CONFIG_SCHEMA: ConfigDefaultDict = { + 'TERM_WIDTH': {'default': lambda c: lambda: shutil.get_terminal_size((100, 10)).columns}, + 'USER': {'default': lambda c: getpass.getuser() or os.getlogin()}, + 'ANSI': {'default': lambda c: DEFAULT_CLI_COLORS if c['USE_COLOR'] else {k: '' for k in DEFAULT_CLI_COLORS.keys()}}, + + 'PACKAGE_DIR': {'default': lambda c: Path(__file__).resolve().parent}, + 'TEMPLATES_DIR': {'default': lambda c: c['PACKAGE_DIR'] / TEMPLATES_DIR_NAME}, + + 'OUTPUT_DIR': {'default': lambda c: Path(c['OUTPUT_DIR']).resolve() if c['OUTPUT_DIR'] else Path(os.curdir).resolve()}, + 'ARCHIVE_DIR': {'default': lambda c: c['OUTPUT_DIR'] / ARCHIVE_DIR_NAME}, + 'SOURCES_DIR': {'default': lambda c: c['OUTPUT_DIR'] / SOURCES_DIR_NAME}, + 'LOGS_DIR': {'default': lambda c: c['OUTPUT_DIR'] / LOGS_DIR_NAME}, + 'CONFIG_FILE': {'default': lambda c: Path(c['CONFIG_FILE']).resolve() if c['CONFIG_FILE'] else c['OUTPUT_DIR'] / CONFIG_FILENAME}, + 'COOKIES_FILE': {'default': lambda c: c['COOKIES_FILE'] and Path(c['COOKIES_FILE']).resolve()}, + 'CHROME_USER_DATA_DIR': {'default': lambda c: find_chrome_data_dir() if c['CHROME_USER_DATA_DIR'] is None else (Path(c['CHROME_USER_DATA_DIR']).resolve() if c['CHROME_USER_DATA_DIR'] else None)}, # None means unset, so we autodetect it with find_chrome_Data_dir(), but emptystring '' means user manually set it to '', and we should store it as None + 'URL_BLACKLIST_PTN': {'default': lambda c: c['URL_BLACKLIST'] and re.compile(c['URL_BLACKLIST'] or '', re.IGNORECASE | re.UNICODE | re.MULTILINE)}, + + 'ARCHIVEBOX_BINARY': {'default': lambda c: sys.argv[0]}, + 'VERSION': {'default': lambda c: json.loads((Path(c['PACKAGE_DIR']) / 'package.json').read_text().strip())['version']}, + 'GIT_SHA': {'default': lambda c: c['VERSION'].split('+')[-1] or 'unknown'}, + + 'PYTHON_BINARY': {'default': lambda c: sys.executable}, + 'PYTHON_ENCODING': {'default': lambda c: sys.stdout.encoding.upper()}, + 'PYTHON_VERSION': {'default': lambda c: '{}.{}.{}'.format(*sys.version_info[:3])}, + + 'DJANGO_BINARY': {'default': lambda c: django.__file__.replace('__init__.py', 'bin/django-admin.py')}, + 'DJANGO_VERSION': {'default': lambda c: '{}.{}.{} {} ({})'.format(*django.VERSION)}, + + 'USE_CURL': {'default': lambda c: c['USE_CURL'] and (c['SAVE_FAVICON'] or c['SAVE_TITLE'] or c['SAVE_ARCHIVE_DOT_ORG'])}, + 'CURL_VERSION': {'default': lambda c: bin_version(c['CURL_BINARY']) if c['USE_CURL'] else None}, + 'CURL_USER_AGENT': {'default': lambda c: c['CURL_USER_AGENT'].format(**c)}, + 'CURL_ARGS': {'default': lambda c: c['CURL_ARGS'] or []}, + 'SAVE_FAVICON': {'default': lambda c: c['USE_CURL'] and c['SAVE_FAVICON']}, + 'SAVE_ARCHIVE_DOT_ORG': {'default': lambda c: c['USE_CURL'] and c['SAVE_ARCHIVE_DOT_ORG']}, + + 'USE_WGET': {'default': lambda c: c['USE_WGET'] and (c['SAVE_WGET'] or c['SAVE_WARC'])}, + 'WGET_VERSION': {'default': lambda c: bin_version(c['WGET_BINARY']) if c['USE_WGET'] else None}, + 'WGET_AUTO_COMPRESSION': {'default': lambda c: wget_supports_compression(c) if c['USE_WGET'] else False}, + 'WGET_USER_AGENT': {'default': lambda c: c['WGET_USER_AGENT'].format(**c)}, + 'SAVE_WGET': {'default': lambda c: c['USE_WGET'] and c['SAVE_WGET']}, + 'SAVE_WARC': {'default': lambda c: c['USE_WGET'] and c['SAVE_WARC']}, + 'WGET_ARGS': {'default': lambda c: c['WGET_ARGS'] or []}, + + 'RIPGREP_VERSION': {'default': lambda c: bin_version(c['RIPGREP_BINARY']) if c['USE_RIPGREP'] else None}, + + 'USE_SINGLEFILE': {'default': lambda c: c['USE_SINGLEFILE'] and c['SAVE_SINGLEFILE']}, + 'SINGLEFILE_VERSION': {'default': lambda c: bin_version(c['SINGLEFILE_BINARY']) if c['USE_SINGLEFILE'] else None}, + + 'USE_READABILITY': {'default': lambda c: c['USE_READABILITY'] and c['SAVE_READABILITY']}, + 'READABILITY_VERSION': {'default': lambda c: bin_version(c['READABILITY_BINARY']) if c['USE_READABILITY'] else None}, + + 'USE_MERCURY': {'default': lambda c: c['USE_MERCURY'] and c['SAVE_MERCURY']}, + 'MERCURY_VERSION': {'default': lambda c: '1.0.0' if shutil.which(str(bin_path(c['MERCURY_BINARY']))) else None}, # mercury is unversioned + + 'USE_GIT': {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']}, + 'GIT_VERSION': {'default': lambda c: bin_version(c['GIT_BINARY']) if c['USE_GIT'] else None}, + 'SAVE_GIT': {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']}, + + 'USE_YOUTUBEDL': {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']}, + 'YOUTUBEDL_VERSION': {'default': lambda c: bin_version(c['YOUTUBEDL_BINARY']) if c['USE_YOUTUBEDL'] else None}, + 'SAVE_MEDIA': {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']}, + 'YOUTUBEDL_ARGS': {'default': lambda c: c['YOUTUBEDL_ARGS'] or []}, + + 'USE_CHROME': {'default': lambda c: c['USE_CHROME'] and (c['SAVE_PDF'] or c['SAVE_SCREENSHOT'] or c['SAVE_DOM'] or c['SAVE_SINGLEFILE'])}, + 'CHROME_BINARY': {'default': lambda c: c['CHROME_BINARY'] if c['CHROME_BINARY'] else find_chrome_binary()}, + 'CHROME_VERSION': {'default': lambda c: bin_version(c['CHROME_BINARY']) if c['USE_CHROME'] else None}, + + 'SAVE_PDF': {'default': lambda c: c['USE_CHROME'] and c['SAVE_PDF']}, + 'SAVE_SCREENSHOT': {'default': lambda c: c['USE_CHROME'] and c['SAVE_SCREENSHOT']}, + 'SAVE_DOM': {'default': lambda c: c['USE_CHROME'] and c['SAVE_DOM']}, + 'SAVE_SINGLEFILE': {'default': lambda c: c['USE_CHROME'] and c['SAVE_SINGLEFILE'] and c['USE_NODE']}, + 'SAVE_READABILITY': {'default': lambda c: c['USE_READABILITY'] and c['USE_NODE']}, + 'SAVE_MERCURY': {'default': lambda c: c['USE_MERCURY'] and c['USE_NODE']}, + + 'USE_NODE': {'default': lambda c: c['USE_NODE'] and (c['SAVE_READABILITY'] or c['SAVE_SINGLEFILE'] or c['SAVE_MERCURY'])}, + 'NODE_VERSION': {'default': lambda c: bin_version(c['NODE_BINARY']) if c['USE_NODE'] else None}, + + 'DEPENDENCIES': {'default': lambda c: get_dependency_info(c)}, + 'CODE_LOCATIONS': {'default': lambda c: get_code_locations(c)}, + 'EXTERNAL_LOCATIONS': {'default': lambda c: get_external_locations(c)}, + 'DATA_LOCATIONS': {'default': lambda c: get_data_locations(c)}, + 'CHROME_OPTIONS': {'default': lambda c: get_chrome_info(c)}, +} + + + +################################### Helpers #################################### + + +def load_config_val(key: str, + default: ConfigDefaultValue=None, + type: Optional[Type]=None, + aliases: Optional[Tuple[str, ...]]=None, + config: Optional[ConfigDict]=None, + env_vars: Optional[os._Environ]=None, + config_file_vars: Optional[Dict[str, str]]=None) -> ConfigValue: + """parse bool, int, and str key=value pairs from env""" + + + config_keys_to_check = (key, *(aliases or ())) + for key in config_keys_to_check: + if env_vars: + val = env_vars.get(key) + if val: + break + if config_file_vars: + val = config_file_vars.get(key) + if val: + break + + if type is None or val is None: + if callable(default): + assert isinstance(config, dict) + return default(config) + + return default + + elif type is bool: + if val.lower() in ('true', 'yes', '1'): + return True + elif val.lower() in ('false', 'no', '0'): + return False + else: + raise ValueError(f'Invalid configuration option {key}={val} (expected a boolean: True/False)') + + elif type is str: + if val.lower() in ('true', 'false', 'yes', 'no', '1', '0'): + raise ValueError(f'Invalid configuration option {key}={val} (expected a string)') + return val.strip() + + elif type is int: + if not val.isdigit(): + raise ValueError(f'Invalid configuration option {key}={val} (expected an integer)') + return int(val) + + elif type is list or type is dict: + return json.loads(val) + + raise Exception('Config values can only be str, bool, int or json') + + +def load_config_file(out_dir: str=None) -> Optional[Dict[str, str]]: + """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf""" + + out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve() + config_path = Path(out_dir) / CONFIG_FILENAME + if config_path.exists(): + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(config_path) + # flatten into one namespace + config_file_vars = { + key.upper(): val + for section, options in config_file.items() + for key, val in options.items() + } + # print('[i] Loaded config file', os.path.abspath(config_path)) + # print(config_file_vars) + return config_file_vars + return None + + +def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict: + """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf""" + + from .system import atomic_write + + CONFIG_HEADER = ( + """# This is the config file for your ArchiveBox collection. + # + # You can add options here manually in INI format, or automatically by running: + # archivebox config --set KEY=VALUE + # + # If you modify this file manually, make sure to update your archive after by running: + # archivebox init + # + # A list of all possible config with documentation and examples can be found here: + # https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration + + """) + + out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve() + config_path = Path(out_dir) / CONFIG_FILENAME + + if not config_path.exists(): + atomic_write(config_path, CONFIG_HEADER) + + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(config_path) + + with open(config_path, 'r') as old: + atomic_write(f'{config_path}.bak', old.read()) + + find_section = lambda key: [name for name, opts in CONFIG_SCHEMA.items() if key in opts][0] + + # Set up sections in empty config file + for key, val in config.items(): + section = find_section(key) + if section in config_file: + existing_config = dict(config_file[section]) + else: + existing_config = {} + config_file[section] = {**existing_config, key: val} + + # always make sure there's a SECRET_KEY defined for Django + existing_secret_key = None + if 'SERVER_CONFIG' in config_file and 'SECRET_KEY' in config_file['SERVER_CONFIG']: + existing_secret_key = config_file['SERVER_CONFIG']['SECRET_KEY'] + + if (not existing_secret_key) or ('not a valid secret' in existing_secret_key): + from django.utils.crypto import get_random_string + chars = 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.' + random_secret_key = get_random_string(50, chars) + if 'SERVER_CONFIG' in config_file: + config_file['SERVER_CONFIG']['SECRET_KEY'] = random_secret_key + else: + config_file['SERVER_CONFIG'] = {'SECRET_KEY': random_secret_key} + + with open(config_path, 'w+') as new: + config_file.write(new) + + try: + # validate the config by attempting to re-parse it + CONFIG = load_all_config() + return { + key.upper(): CONFIG.get(key.upper()) + for key in config.keys() + } + except: + # something went horribly wrong, rever to the previous version + with open(f'{config_path}.bak', 'r') as old: + atomic_write(config_path, old.read()) + + if Path(f'{config_path}.bak').exists(): + os.remove(f'{config_path}.bak') + + return {} + + + +def load_config(defaults: ConfigDefaultDict, + config: Optional[ConfigDict]=None, + out_dir: Optional[str]=None, + env_vars: Optional[os._Environ]=None, + config_file_vars: Optional[Dict[str, str]]=None) -> ConfigDict: + + env_vars = env_vars or os.environ + config_file_vars = config_file_vars or load_config_file(out_dir=out_dir) + + extended_config: ConfigDict = config.copy() if config else {} + for key, default in defaults.items(): + try: + extended_config[key] = load_config_val( + key, + default=default['default'], + type=default.get('type'), + aliases=default.get('aliases'), + config=extended_config, + env_vars=env_vars, + config_file_vars=config_file_vars, + ) + except KeyboardInterrupt: + raise SystemExit(0) + except Exception as e: + stderr() + stderr(f'[X] Error while loading configuration value: {key}', color='red', config=extended_config) + stderr(' {}: {}'.format(e.__class__.__name__, e)) + stderr() + stderr(' Check your config for mistakes and try again (your archive data is unaffected).') + stderr() + stderr(' For config documentation and examples see:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration') + stderr() + raise + raise SystemExit(2) + + return extended_config + +# def write_config(config: ConfigDict): + +# with open(os.path.join(config['OUTPUT_DIR'], CONFIG_FILENAME), 'w+') as f: + + +# Logging Helpers +def stdout(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if color: + strs = [ansi[color], ' '.join(str(a) for a in args), ansi['reset'], '\n'] + else: + strs = [' '.join(str(a) for a in args), '\n'] + + sys.stdout.write(prefix + ''.join(strs)) + +def stderr(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if color: + strs = [ansi[color], ' '.join(str(a) for a in args), ansi['reset'], '\n'] + else: + strs = [' '.join(str(a) for a in args), '\n'] + + sys.stderr.write(prefix + ''.join(strs)) + +def hint(text: Union[Tuple[str, ...], List[str], str], prefix=' ', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if isinstance(text, str): + stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text, **ansi)) + else: + stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text[0], **ansi)) + for line in text[1:]: + stderr('{} {}'.format(prefix, line)) + + +# Dependency Metadata Helpers +def bin_version(binary: Optional[str]) -> Optional[str]: + """check the presence and return valid version line of a specified binary""" + + abspath = bin_path(binary) + if not binary or not abspath: + return None + + try: + version_str = run([abspath, "--version"], stdout=PIPE).stdout.strip().decode() + # take first 3 columns of first line of version info + return ' '.join(version_str.split('\n')[0].strip().split()[:3]) + except OSError: + pass + # stderr(f'[X] Unable to find working version of dependency: {binary}', color='red') + # stderr(' Make sure it\'s installed, then confirm it\'s working by running:') + # stderr(f' {binary} --version') + # stderr() + # stderr(' If you don\'t want to install it, you can disable it via config. See here for more info:') + # stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Install') + return None + +def bin_path(binary: Optional[str]) -> Optional[str]: + if binary is None: + return None + + node_modules_bin = Path('.') / 'node_modules' / '.bin' / binary + if node_modules_bin.exists(): + return str(node_modules_bin.resolve()) + + return shutil.which(str(Path(binary).expanduser())) or shutil.which(str(binary)) or binary + +def bin_hash(binary: Optional[str]) -> Optional[str]: + if binary is None: + return None + abs_path = bin_path(binary) + if abs_path is None or not Path(abs_path).exists(): + return None + + file_hash = md5() + with io.open(abs_path, mode='rb') as f: + for chunk in iter(lambda: f.read(io.DEFAULT_BUFFER_SIZE), b''): + file_hash.update(chunk) + + return f'md5:{file_hash.hexdigest()}' + +def find_chrome_binary() -> Optional[str]: + """find any installed chrome binaries in the default locations""" + # Precedence: Chromium, Chrome, Beta, Canary, Unstable, Dev + # make sure data dir finding precedence order always matches binary finding order + default_executable_paths = ( + 'chromium-browser', + 'chromium', + '/Applications/Chromium.app/Contents/MacOS/Chromium', + 'chrome', + 'google-chrome', + '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', + 'google-chrome-stable', + 'google-chrome-beta', + 'google-chrome-canary', + '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary', + 'google-chrome-unstable', + 'google-chrome-dev', + ) + for name in default_executable_paths: + full_path_exists = shutil.which(name) + if full_path_exists: + return name + + return None + +def find_chrome_data_dir() -> Optional[str]: + """find any installed chrome user data directories in the default locations""" + # Precedence: Chromium, Chrome, Beta, Canary, Unstable, Dev + # make sure data dir finding precedence order always matches binary finding order + default_profile_paths = ( + '~/.config/chromium', + '~/Library/Application Support/Chromium', + '~/AppData/Local/Chromium/User Data', + '~/.config/chrome', + '~/.config/google-chrome', + '~/Library/Application Support/Google/Chrome', + '~/AppData/Local/Google/Chrome/User Data', + '~/.config/google-chrome-stable', + '~/.config/google-chrome-beta', + '~/Library/Application Support/Google/Chrome Canary', + '~/AppData/Local/Google/Chrome SxS/User Data', + '~/.config/google-chrome-unstable', + '~/.config/google-chrome-dev', + ) + for path in default_profile_paths: + full_path = Path(path).resolve() + if full_path.exists(): + return full_path + return None + +def wget_supports_compression(config): + try: + cmd = [ + config['WGET_BINARY'], + "--compression=auto", + "--help", + ] + return not run(cmd, stdout=DEVNULL, stderr=DEVNULL).returncode + except (FileNotFoundError, OSError): + return False + +def get_code_locations(config: ConfigDict) -> SimpleConfigValueDict: + return { + 'PACKAGE_DIR': { + 'path': (config['PACKAGE_DIR']).resolve(), + 'enabled': True, + 'is_valid': (config['PACKAGE_DIR'] / '__main__.py').exists(), + }, + 'TEMPLATES_DIR': { + 'path': (config['TEMPLATES_DIR']).resolve(), + 'enabled': True, + 'is_valid': (config['TEMPLATES_DIR'] / config['ACTIVE_THEME'] / 'static').exists(), + }, + # 'NODE_MODULES_DIR': { + # 'path': , + # 'enabled': , + # 'is_valid': (...).exists(), + # }, + } + +def get_external_locations(config: ConfigDict) -> ConfigValue: + abspath = lambda path: None if path is None else Path(path).resolve() + return { + 'CHROME_USER_DATA_DIR': { + 'path': abspath(config['CHROME_USER_DATA_DIR']), + 'enabled': config['USE_CHROME'] and config['CHROME_USER_DATA_DIR'], + 'is_valid': False if config['CHROME_USER_DATA_DIR'] is None else (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(), + }, + 'COOKIES_FILE': { + 'path': abspath(config['COOKIES_FILE']), + 'enabled': config['USE_WGET'] and config['COOKIES_FILE'], + 'is_valid': False if config['COOKIES_FILE'] is None else Path(config['COOKIES_FILE']).exists(), + }, + } + +def get_data_locations(config: ConfigDict) -> ConfigValue: + return { + 'OUTPUT_DIR': { + 'path': config['OUTPUT_DIR'].resolve(), + 'enabled': True, + 'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(), + }, + 'SOURCES_DIR': { + 'path': config['SOURCES_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['SOURCES_DIR'].exists(), + }, + 'LOGS_DIR': { + 'path': config['LOGS_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['LOGS_DIR'].exists(), + }, + 'ARCHIVE_DIR': { + 'path': config['ARCHIVE_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['ARCHIVE_DIR'].exists(), + }, + 'CONFIG_FILE': { + 'path': config['CONFIG_FILE'].resolve(), + 'enabled': True, + 'is_valid': config['CONFIG_FILE'].exists(), + }, + 'SQL_INDEX': { + 'path': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).resolve(), + 'enabled': True, + 'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(), + }, + } + +def get_dependency_info(config: ConfigDict) -> ConfigValue: + return { + 'ARCHIVEBOX_BINARY': { + 'path': bin_path(config['ARCHIVEBOX_BINARY']), + 'version': config['VERSION'], + 'hash': bin_hash(config['ARCHIVEBOX_BINARY']), + 'enabled': True, + 'is_valid': True, + }, + 'PYTHON_BINARY': { + 'path': bin_path(config['PYTHON_BINARY']), + 'version': config['PYTHON_VERSION'], + 'hash': bin_hash(config['PYTHON_BINARY']), + 'enabled': True, + 'is_valid': bool(config['DJANGO_VERSION']), + }, + 'DJANGO_BINARY': { + 'path': bin_path(config['DJANGO_BINARY']), + 'version': config['DJANGO_VERSION'], + 'hash': bin_hash(config['DJANGO_BINARY']), + 'enabled': True, + 'is_valid': bool(config['DJANGO_VERSION']), + }, + 'CURL_BINARY': { + 'path': bin_path(config['CURL_BINARY']), + 'version': config['CURL_VERSION'], + 'hash': bin_hash(config['PYTHON_BINARY']), + 'enabled': config['USE_CURL'], + 'is_valid': bool(config['CURL_VERSION']), + }, + 'WGET_BINARY': { + 'path': bin_path(config['WGET_BINARY']), + 'version': config['WGET_VERSION'], + 'hash': bin_hash(config['WGET_BINARY']), + 'enabled': config['USE_WGET'], + 'is_valid': bool(config['WGET_VERSION']), + }, + 'NODE_BINARY': { + 'path': bin_path(config['NODE_BINARY']), + 'version': config['NODE_VERSION'], + 'hash': bin_hash(config['NODE_BINARY']), + 'enabled': config['USE_NODE'], + 'is_valid': bool(config['SINGLEFILE_VERSION']), + }, + 'SINGLEFILE_BINARY': { + 'path': bin_path(config['SINGLEFILE_BINARY']), + 'version': config['SINGLEFILE_VERSION'], + 'hash': bin_hash(config['SINGLEFILE_BINARY']), + 'enabled': config['USE_SINGLEFILE'], + 'is_valid': bool(config['SINGLEFILE_VERSION']), + }, + 'READABILITY_BINARY': { + 'path': bin_path(config['READABILITY_BINARY']), + 'version': config['READABILITY_VERSION'], + 'hash': bin_hash(config['READABILITY_BINARY']), + 'enabled': config['USE_READABILITY'], + 'is_valid': bool(config['READABILITY_VERSION']), + }, + 'MERCURY_BINARY': { + 'path': bin_path(config['MERCURY_BINARY']), + 'version': config['MERCURY_VERSION'], + 'hash': bin_hash(config['MERCURY_BINARY']), + 'enabled': config['USE_MERCURY'], + 'is_valid': bool(config['MERCURY_VERSION']), + }, + 'GIT_BINARY': { + 'path': bin_path(config['GIT_BINARY']), + 'version': config['GIT_VERSION'], + 'hash': bin_hash(config['GIT_BINARY']), + 'enabled': config['USE_GIT'], + 'is_valid': bool(config['GIT_VERSION']), + }, + 'YOUTUBEDL_BINARY': { + 'path': bin_path(config['YOUTUBEDL_BINARY']), + 'version': config['YOUTUBEDL_VERSION'], + 'hash': bin_hash(config['YOUTUBEDL_BINARY']), + 'enabled': config['USE_YOUTUBEDL'], + 'is_valid': bool(config['YOUTUBEDL_VERSION']), + }, + 'CHROME_BINARY': { + 'path': bin_path(config['CHROME_BINARY']), + 'version': config['CHROME_VERSION'], + 'hash': bin_hash(config['CHROME_BINARY']), + 'enabled': config['USE_CHROME'], + 'is_valid': bool(config['CHROME_VERSION']), + }, + 'RIPGREP_BINARY': { + 'path': bin_path(config['RIPGREP_BINARY']), + 'version': config['RIPGREP_VERSION'], + 'hash': bin_hash(config['RIPGREP_BINARY']), + 'enabled': config['USE_RIPGREP'], + 'is_valid': bool(config['RIPGREP_VERSION']), + }, + # TODO: add an entry for the sonic search backend? + # 'SONIC_BINARY': { + # 'path': bin_path(config['SONIC_BINARY']), + # 'version': config['SONIC_VERSION'], + # 'hash': bin_hash(config['SONIC_BINARY']), + # 'enabled': config['USE_SONIC'], + # 'is_valid': bool(config['SONIC_VERSION']), + # }, + } + +def get_chrome_info(config: ConfigDict) -> ConfigValue: + return { + 'TIMEOUT': config['TIMEOUT'], + 'RESOLUTION': config['RESOLUTION'], + 'CHECK_SSL_VALIDITY': config['CHECK_SSL_VALIDITY'], + 'CHROME_BINARY': config['CHROME_BINARY'], + 'CHROME_HEADLESS': config['CHROME_HEADLESS'], + 'CHROME_SANDBOX': config['CHROME_SANDBOX'], + 'CHROME_USER_AGENT': config['CHROME_USER_AGENT'], + 'CHROME_USER_DATA_DIR': config['CHROME_USER_DATA_DIR'], + } + + +# ****************************************************************************** +# ****************************************************************************** +# ******************************** Load Config ********************************* +# ******* (compile the defaults, configs, and metadata all into CONFIG) ******** +# ****************************************************************************** +# ****************************************************************************** + + +def load_all_config(): + CONFIG: ConfigDict = {} + for section_name, section_config in CONFIG_SCHEMA.items(): + CONFIG = load_config(section_config, CONFIG) + + return load_config(DYNAMIC_CONFIG_SCHEMA, CONFIG) + +# add all final config values in CONFIG to globals in this file +CONFIG = load_all_config() +globals().update(CONFIG) +# this lets us do: from .config import DEBUG, MEDIA_TIMEOUT, ... + + +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** + + + +########################### System Environment Setup ########################### + + +# Set timezone to UTC and umask to OUTPUT_PERMISSIONS +os.environ["TZ"] = 'UTC' +os.umask(0o777 - int(OUTPUT_PERMISSIONS, base=8)) # noqa: F821 + +# add ./node_modules/.bin to $PATH so we can use node scripts in extractors +NODE_BIN_PATH = str((Path(CONFIG["OUTPUT_DIR"]).absolute() / 'node_modules' / '.bin')) +sys.path.append(NODE_BIN_PATH) + + + + +########################### Config Validity Checkers ########################### + + +def check_system_config(config: ConfigDict=CONFIG) -> None: + ### Check system environment + if config['USER'] == 'root': + stderr('[!] ArchiveBox should never be run as root!', color='red') + stderr(' For more information, see the security overview documentation:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root') + raise SystemExit(2) + + ### Check Python environment + if sys.version_info[:3] < (3, 6, 0): + stderr(f'[X] Python version is not new enough: {config["PYTHON_VERSION"]} (>3.6 is required)', color='red') + stderr(' See https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#python for help upgrading your Python installation.') + raise SystemExit(2) + + if config['PYTHON_ENCODING'] not in ('UTF-8', 'UTF8'): + stderr(f'[X] Your system is running python3 scripts with a bad locale setting: {config["PYTHON_ENCODING"]} (it should be UTF-8).', color='red') + stderr(' To fix it, add the line "export PYTHONIOENCODING=UTF-8" to your ~/.bashrc file (without quotes)') + stderr(' Or if you\'re using ubuntu/debian, run "dpkg-reconfigure locales"') + stderr('') + stderr(' Confirm that it\'s fixed by opening a new shell and running:') + stderr(' python3 -c "import sys; print(sys.stdout.encoding)" # should output UTF-8') + raise SystemExit(2) + + # stderr('[i] Using Chrome binary: {}'.format(shutil.which(CHROME_BINARY) or CHROME_BINARY)) + # stderr('[i] Using Chrome data dir: {}'.format(os.path.abspath(CHROME_USER_DATA_DIR))) + if config['CHROME_USER_DATA_DIR'] is not None: + if not (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(): + stderr('[X] Could not find profile "Default" in CHROME_USER_DATA_DIR.', color='red') + stderr(f' {config["CHROME_USER_DATA_DIR"]}') + stderr(' Make sure you set it to a Chrome user data directory containing a Default profile folder.') + stderr(' For more info see:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#CHROME_USER_DATA_DIR') + if '/Default' in str(config['CHROME_USER_DATA_DIR']): + stderr() + stderr(' Try removing /Default from the end e.g.:') + stderr(' CHROME_USER_DATA_DIR="{}"'.format(config['CHROME_USER_DATA_DIR'].split('/Default')[0])) + raise SystemExit(2) + + +def check_dependencies(config: ConfigDict=CONFIG, show_help: bool=True) -> None: + invalid_dependencies = [ + (name, info) for name, info in config['DEPENDENCIES'].items() + if info['enabled'] and not info['is_valid'] + ] + if invalid_dependencies and show_help: + stderr(f'[!] Warning: Missing {len(invalid_dependencies)} recommended dependencies', color='lightyellow') + for dependency, info in invalid_dependencies: + stderr( + ' ! {}: {} ({})'.format( + dependency, + info['path'] or 'unable to find binary', + info['version'] or 'unable to detect version', + ) + ) + if dependency in ('SINGLEFILE_BINARY', 'READABILITY_BINARY', 'MERCURY_BINARY'): + hint(('npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"', + f'or archivebox config --set SAVE_{dependency.rsplit("_", 1)[0]}=False to silence this warning', + ''), prefix=' ') + stderr('') + + if config['TIMEOUT'] < 5: + stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red') + stderr(' You must allow *at least* 5 seconds for indexing and archive methods to run succesfully.') + stderr(' (Setting it to somewhere between 30 and 3000 seconds is recommended)') + stderr() + stderr(' If you want to make ArchiveBox run faster, disable specific archive methods instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles') + stderr() + + elif config['USE_CHROME'] and config['TIMEOUT'] < 15: + stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red') + stderr(' Chrome will fail to archive all sites if set to less than ~15 seconds.') + stderr(' (Setting it to somewhere between 30 and 300 seconds is recommended)') + stderr() + stderr(' If you want to make ArchiveBox run faster, disable specific archive methods instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles') + stderr() + + if config['USE_YOUTUBEDL'] and config['MEDIA_TIMEOUT'] < 20: + stderr(f'[!] Warning: MEDIA_TIMEOUT is set too low! (currently set to MEDIA_TIMEOUT={config["MEDIA_TIMEOUT"]} seconds)', color='red') + stderr(' Youtube-dl will fail to archive all media if set to less than ~20 seconds.') + stderr(' (Setting it somewhere over 60 seconds is recommended)') + stderr() + stderr(' If you want to disable media archiving entirely, set SAVE_MEDIA=False instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_media') + stderr() + +def check_data_folder(out_dir: Union[str, Path, None]=None, config: ConfigDict=CONFIG) -> None: + output_dir = out_dir or config['OUTPUT_DIR'] + assert isinstance(output_dir, (str, Path)) + + sql_index_exists = (Path(output_dir) / SQL_INDEX_FILENAME).exists() + if not sql_index_exists: + stderr('[X] No archivebox index found in the current directory.', color='red') + stderr(f' {output_dir}', color='lightyellow') + stderr() + stderr(' {lightred}Hint{reset}: Are you running archivebox in the right folder?'.format(**config['ANSI'])) + stderr(' cd path/to/your/archive/folder') + stderr(' archivebox [command]') + stderr() + stderr(' {lightred}Hint{reset}: To create a new archive collection or import existing data in this folder, run:'.format(**config['ANSI'])) + stderr(' archivebox init') + raise SystemExit(2) + + from .index.sql import list_migrations + + pending_migrations = [name for status, name in list_migrations() if not status] + + if (not sql_index_exists) or pending_migrations: + if sql_index_exists: + pending_operation = f'apply the {len(pending_migrations)} pending migrations' + else: + pending_operation = 'generate the new SQL main index' + + stderr('[X] This collection was created with an older version of ArchiveBox and must be upgraded first.', color='lightyellow') + stderr(f' {output_dir}') + stderr() + stderr(f' To upgrade it to the latest version and {pending_operation} run:') + stderr(' archivebox init') + raise SystemExit(3) + + sources_dir = Path(output_dir) / SOURCES_DIR_NAME + if not sources_dir.exists(): + sources_dir.mkdir() + + + +def setup_django(out_dir: Path=None, check_db=False, config: ConfigDict=CONFIG, in_memory_db=False) -> None: + check_system_config() + + output_dir = out_dir or Path(config['OUTPUT_DIR']) + + assert isinstance(output_dir, Path) and isinstance(config['PACKAGE_DIR'], Path) + + try: + import django + sys.path.append(str(config['PACKAGE_DIR'])) + os.environ.setdefault('OUTPUT_DIR', str(output_dir)) + assert (config['PACKAGE_DIR'] / 'core' / 'settings.py').exists(), 'settings.py was not found at archivebox/core/settings.py' + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings') + + if in_memory_db: + # Put the db in memory and run migrations in case any command requires it + from django.core.management import call_command + os.environ.setdefault("ARCHIVEBOX_DATABASE_NAME", ":memory:") + django.setup() + call_command("migrate", interactive=False, verbosity=0) + else: + django.setup() + + if check_db: + sql_index_path = Path(output_dir) / SQL_INDEX_FILENAME + assert sql_index_path.exists(), ( + f'No database file {SQL_INDEX_FILENAME} found in OUTPUT_DIR: {config["OUTPUT_DIR"]}') + except KeyboardInterrupt: + raise SystemExit(2) diff --git a/archivebox-0.5.3/archivebox/config_stubs.py b/archivebox-0.5.3/archivebox/config_stubs.py new file mode 100644 index 0000000..988f58a --- /dev/null +++ b/archivebox-0.5.3/archivebox/config_stubs.py @@ -0,0 +1,113 @@ +from pathlib import Path +from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any, List +from mypy_extensions import TypedDict + + + +SimpleConfigValue = Union[str, bool, int, None, Pattern, Dict[str, Any]] +SimpleConfigValueDict = Dict[str, SimpleConfigValue] +SimpleConfigValueGetter = Callable[[], SimpleConfigValue] +ConfigValue = Union[SimpleConfigValue, SimpleConfigValueDict, SimpleConfigValueGetter] + + +class BaseConfig(TypedDict): + pass + +class ConfigDict(BaseConfig, total=False): + """ + # Regenerate by pasting this quine into `archivebox shell` 🥚 + from archivebox.config import ConfigDict, CONFIG_DEFAULTS + print('class ConfigDict(BaseConfig, total=False):') + print(' ' + '"'*3 + ConfigDict.__doc__ + '"'*3) + for section, configs in CONFIG_DEFAULTS.items(): + for key, attrs in configs.items(): + Type, default = attrs['type'], attrs['default'] + if default is None: + print(f' {key}: Optional[{Type.__name__}]') + else: + print(f' {key}: {Type.__name__}') + print() + """ + IS_TTY: bool + USE_COLOR: bool + SHOW_PROGRESS: bool + IN_DOCKER: bool + + PACKAGE_DIR: Path + OUTPUT_DIR: Path + CONFIG_FILE: Path + ONLY_NEW: bool + TIMEOUT: int + MEDIA_TIMEOUT: int + OUTPUT_PERMISSIONS: str + RESTRICT_FILE_NAMES: str + URL_BLACKLIST: str + + SECRET_KEY: Optional[str] + BIND_ADDR: str + ALLOWED_HOSTS: str + DEBUG: bool + PUBLIC_INDEX: bool + PUBLIC_SNAPSHOTS: bool + FOOTER_INFO: str + ACTIVE_THEME: str + + SAVE_TITLE: bool + SAVE_FAVICON: bool + SAVE_WGET: bool + SAVE_WGET_REQUISITES: bool + SAVE_SINGLEFILE: bool + SAVE_READABILITY: bool + SAVE_MERCURY: bool + SAVE_PDF: bool + SAVE_SCREENSHOT: bool + SAVE_DOM: bool + SAVE_WARC: bool + SAVE_GIT: bool + SAVE_MEDIA: bool + SAVE_ARCHIVE_DOT_ORG: bool + + RESOLUTION: str + GIT_DOMAINS: str + CHECK_SSL_VALIDITY: bool + CURL_USER_AGENT: str + WGET_USER_AGENT: str + CHROME_USER_AGENT: str + COOKIES_FILE: Union[str, Path, None] + CHROME_USER_DATA_DIR: Union[str, Path, None] + CHROME_HEADLESS: bool + CHROME_SANDBOX: bool + + USE_CURL: bool + USE_WGET: bool + USE_SINGLEFILE: bool + USE_READABILITY: bool + USE_MERCURY: bool + USE_GIT: bool + USE_CHROME: bool + USE_YOUTUBEDL: bool + CURL_BINARY: str + GIT_BINARY: str + WGET_BINARY: str + SINGLEFILE_BINARY: str + READABILITY_BINARY: str + MERCURY_BINARY: str + YOUTUBEDL_BINARY: str + CHROME_BINARY: Optional[str] + + YOUTUBEDL_ARGS: List[str] + WGET_ARGS: List[str] + CURL_ARGS: List[str] + GIT_ARGS: List[str] + + +ConfigDefaultValueGetter = Callable[[ConfigDict], ConfigValue] +ConfigDefaultValue = Union[ConfigValue, ConfigDefaultValueGetter] + +ConfigDefault = TypedDict('ConfigDefault', { + 'default': ConfigDefaultValue, + 'type': Optional[Type], + 'aliases': Optional[Tuple[str, ...]], +}, total=False) + +ConfigDefaultDict = Dict[str, ConfigDefault] diff --git a/archivebox-0.5.3/archivebox/core/__init__.py b/archivebox-0.5.3/archivebox/core/__init__.py new file mode 100644 index 0000000..3e1d607 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/__init__.py @@ -0,0 +1 @@ +__package__ = 'archivebox.core' diff --git a/archivebox-0.5.3/archivebox/core/admin.py b/archivebox-0.5.3/archivebox/core/admin.py new file mode 100644 index 0000000..832bea3 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/admin.py @@ -0,0 +1,257 @@ +__package__ = 'archivebox.core' + +from io import StringIO +from contextlib import redirect_stdout + +from django.contrib import admin +from django.urls import path +from django.utils.html import format_html +from django.utils.safestring import mark_safe +from django.shortcuts import render, redirect +from django.contrib.auth import get_user_model +from django import forms + +from core.models import Snapshot, Tag +from core.forms import AddLinkForm, TagField + +from core.mixins import SearchResultsAdminMixin + +from index.html import snapshot_icons +from util import htmldecode, urldecode, ansi_to_html +from logging_util import printable_filesize +from main import add, remove +from config import OUTPUT_DIR +from extractors import archive_links + +# TODO: https://stackoverflow.com/questions/40760880/add-custom-button-to-django-admin-panel + +def update_snapshots(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], out_dir=OUTPUT_DIR) +update_snapshots.short_description = "Archive" + +def update_titles(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], overwrite=True, methods=('title','favicon'), out_dir=OUTPUT_DIR) +update_titles.short_description = "Pull title" + +def overwrite_snapshots(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], overwrite=True, out_dir=OUTPUT_DIR) +overwrite_snapshots.short_description = "Re-archive (overwrite)" + +def verify_snapshots(modeladmin, request, queryset): + for snapshot in queryset: + print(snapshot.timestamp, snapshot.url, snapshot.is_archived, snapshot.archive_size, len(snapshot.history)) + +verify_snapshots.short_description = "Check" + +def delete_snapshots(modeladmin, request, queryset): + remove(snapshots=queryset, yes=True, delete=True, out_dir=OUTPUT_DIR) + +delete_snapshots.short_description = "Delete" + + +class SnapshotAdminForm(forms.ModelForm): + tags = TagField(required=False) + + class Meta: + model = Snapshot + fields = "__all__" + + def save(self, commit=True): + # Based on: https://stackoverflow.com/a/49933068/3509554 + + # Get the unsave instance + instance = forms.ModelForm.save(self, False) + tags = self.cleaned_data.pop("tags") + + #update save_m2m + def new_save_m2m(): + instance.save_tags(tags) + + # Do we need to save all changes now? + self.save_m2m = new_save_m2m + if commit: + instance.save() + + return instance + + +class SnapshotAdmin(SearchResultsAdminMixin, admin.ModelAdmin): + list_display = ('added', 'title_str', 'url_str', 'files', 'size') + sort_fields = ('title_str', 'url_str', 'added') + readonly_fields = ('id', 'url', 'timestamp', 'num_outputs', 'is_archived', 'url_hash', 'added', 'updated') + search_fields = ['url', 'timestamp', 'title', 'tags__name'] + fields = (*readonly_fields, 'title', 'tags') + list_filter = ('added', 'updated', 'tags') + ordering = ['-added'] + actions = [delete_snapshots, overwrite_snapshots, update_snapshots, update_titles, verify_snapshots] + actions_template = 'admin/actions_as_select.html' + form = SnapshotAdminForm + + def get_urls(self): + urls = super().get_urls() + custom_urls = [ + path('grid/', self.admin_site.admin_view(self.grid_view),name='grid') + ] + return custom_urls + urls + + def get_queryset(self, request): + return super().get_queryset(request).prefetch_related('tags') + + def tag_list(self, obj): + return ', '.join(obj.tags.values_list('name', flat=True)) + + def id_str(self, obj): + return format_html( + '{}', + obj.url_hash[:8], + ) + + def title_str(self, obj): + canon = obj.as_link().canonical_outputs() + tags = ''.join( + format_html('{} ', tag.id, tag) + for tag in obj.tags.all() + if str(tag).strip() + ) + return format_html( + '' + '' + '' + '' + '{}' + '', + obj.archive_path, + obj.archive_path, canon['favicon_path'], + obj.archive_path, + 'fetched' if obj.latest_title or obj.title else 'pending', + urldecode(htmldecode(obj.latest_title or obj.title or ''))[:128] or 'Pending...' + ) + mark_safe(f' {tags}') + + def files(self, obj): + return snapshot_icons(obj) + + def size(self, obj): + archive_size = obj.archive_size + if archive_size: + size_txt = printable_filesize(archive_size) + if archive_size > 52428800: + size_txt = mark_safe(f'{size_txt}') + else: + size_txt = mark_safe('...') + return format_html( + '{}', + obj.archive_path, + size_txt, + ) + + def url_str(self, obj): + return format_html( + '{}', + obj.url, + obj.url.split('://www.', 1)[-1].split('://', 1)[-1][:64], + ) + + def grid_view(self, request): + + # cl = self.get_changelist_instance(request) + + # Save before monkey patching to restore for changelist list view + saved_change_list_template = self.change_list_template + saved_list_per_page = self.list_per_page + saved_list_max_show_all = self.list_max_show_all + + # Monkey patch here plus core_tags.py + self.change_list_template = 'admin/grid_change_list.html' + self.list_per_page = 20 + self.list_max_show_all = self.list_per_page + + # Call monkey patched view + rendered_response = self.changelist_view(request) + + # Restore values + self.change_list_template = saved_change_list_template + self.list_per_page = saved_list_per_page + self.list_max_show_all = saved_list_max_show_all + + return rendered_response + + + id_str.short_description = 'ID' + title_str.short_description = 'Title' + url_str.short_description = 'Original URL' + + id_str.admin_order_field = 'id' + title_str.admin_order_field = 'title' + url_str.admin_order_field = 'url' + +class TagAdmin(admin.ModelAdmin): + list_display = ('slug', 'name', 'id') + sort_fields = ('id', 'name', 'slug') + readonly_fields = ('id',) + search_fields = ('id', 'name', 'slug') + fields = (*readonly_fields, 'name', 'slug') + + +class ArchiveBoxAdmin(admin.AdminSite): + site_header = 'ArchiveBox' + index_title = 'Links' + site_title = 'Index' + + def get_urls(self): + return [ + path('core/snapshot/add/', self.add_view, name='Add'), + ] + super().get_urls() + + def add_view(self, request): + if not request.user.is_authenticated: + return redirect(f'/admin/login/?next={request.path}') + + request.current_app = self.name + context = { + **self.each_context(request), + 'title': 'Add URLs', + } + + if request.method == 'GET': + context['form'] = AddLinkForm() + + elif request.method == 'POST': + form = AddLinkForm(request.POST) + if form.is_valid(): + url = form.cleaned_data["url"] + print(f'[+] Adding URL: {url}') + depth = 0 if form.cleaned_data["depth"] == "0" else 1 + input_kwargs = { + "urls": url, + "depth": depth, + "update_all": False, + "out_dir": OUTPUT_DIR, + } + add_stdout = StringIO() + with redirect_stdout(add_stdout): + add(**input_kwargs) + print(add_stdout.getvalue()) + + context.update({ + "stdout": ansi_to_html(add_stdout.getvalue().strip()), + "form": AddLinkForm() + }) + else: + context["form"] = form + + return render(template_name='add_links.html', request=request, context=context) + +admin.site = ArchiveBoxAdmin() +admin.site.register(get_user_model()) +admin.site.register(Snapshot, SnapshotAdmin) +admin.site.register(Tag, TagAdmin) +admin.site.disable_action('delete_selected') diff --git a/archivebox-0.5.3/archivebox/core/apps.py b/archivebox-0.5.3/archivebox/core/apps.py new file mode 100644 index 0000000..26f78a8 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class CoreConfig(AppConfig): + name = 'core' diff --git a/archivebox-0.5.3/archivebox/core/forms.py b/archivebox-0.5.3/archivebox/core/forms.py new file mode 100644 index 0000000..86b29bb --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/forms.py @@ -0,0 +1,67 @@ +__package__ = 'archivebox.core' + +from django import forms + +from ..util import URL_REGEX +from ..vendor.taggit_utils import edit_string_for_tags, parse_tags + +CHOICES = ( + ('0', 'depth = 0 (archive just these URLs)'), + ('1', 'depth = 1 (archive these URLs and all URLs one hop away)'), +) + +from ..extractors import get_default_archive_methods + +ARCHIVE_METHODS = [ + (name, name) + for name, _, _ in get_default_archive_methods() +] + + +class AddLinkForm(forms.Form): + url = forms.RegexField(label="URLs (one per line)", regex=URL_REGEX, min_length='6', strip=True, widget=forms.Textarea, required=True) + depth = forms.ChoiceField(label="Archive depth", choices=CHOICES, widget=forms.RadioSelect, initial='0') + archive_methods = forms.MultipleChoiceField( + required=False, + widget=forms.SelectMultiple, + choices=ARCHIVE_METHODS, + ) +class TagWidgetMixin: + def format_value(self, value): + if value is not None and not isinstance(value, str): + value = edit_string_for_tags(value) + return super().format_value(value) + +class TagWidget(TagWidgetMixin, forms.TextInput): + pass + +class TagField(forms.CharField): + widget = TagWidget + + def clean(self, value): + value = super().clean(value) + try: + return parse_tags(value) + except ValueError: + raise forms.ValidationError( + "Please provide a comma-separated list of tags." + ) + + def has_changed(self, initial_value, data_value): + # Always return False if the field is disabled since self.bound_data + # always uses the initial value in this case. + if self.disabled: + return False + + try: + data_value = self.clean(data_value) + except forms.ValidationError: + pass + + if initial_value is None: + initial_value = [] + + initial_value = [tag.name for tag in initial_value] + initial_value.sort() + + return initial_value != data_value diff --git a/archivebox-0.5.3/archivebox/core/management/commands/archivebox.py b/archivebox-0.5.3/archivebox/core/management/commands/archivebox.py new file mode 100644 index 0000000..a68b5d9 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/management/commands/archivebox.py @@ -0,0 +1,18 @@ +__package__ = 'archivebox' + +from django.core.management.base import BaseCommand + + +from .cli import run_subcommand + + +class Command(BaseCommand): + help = 'Run an ArchiveBox CLI subcommand (e.g. add, remove, list, etc)' + + def add_arguments(self, parser): + parser.add_argument('subcommand', type=str, help='The subcommand you want to run') + parser.add_argument('command_args', nargs='*', help='Arguments to pass to the subcommand') + + + def handle(self, *args, **kwargs): + run_subcommand(kwargs['subcommand'], args=kwargs['command_args']) diff --git a/archivebox-0.5.3/archivebox/core/migrations/0001_initial.py b/archivebox-0.5.3/archivebox/core/migrations/0001_initial.py new file mode 100644 index 0000000..73ac78e --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0001_initial.py @@ -0,0 +1,27 @@ +# Generated by Django 2.2 on 2019-05-01 03:27 + +from django.db import migrations, models +import uuid + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ] + + operations = [ + migrations.CreateModel( + name='Snapshot', + fields=[ + ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)), + ('url', models.URLField(unique=True)), + ('timestamp', models.CharField(default=None, max_length=32, null=True, unique=True)), + ('title', models.CharField(default=None, max_length=128, null=True)), + ('tags', models.CharField(default=None, max_length=256, null=True)), + ('added', models.DateTimeField(auto_now_add=True)), + ('updated', models.DateTimeField(default=None, null=True)), + ], + ), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/0002_auto_20200625_1521.py b/archivebox-0.5.3/archivebox/core/migrations/0002_auto_20200625_1521.py new file mode 100644 index 0000000..4811282 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0002_auto_20200625_1521.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.7 on 2020-06-25 15:21 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0001_initial'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(default=None, max_length=32, null=True), + ), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/0003_auto_20200630_1034.py b/archivebox-0.5.3/archivebox/core/migrations/0003_auto_20200630_1034.py new file mode 100644 index 0000000..61fd472 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0003_auto_20200630_1034.py @@ -0,0 +1,38 @@ +# Generated by Django 3.0.7 on 2020-06-30 10:34 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0002_auto_20200625_1521'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='added', + field=models.DateTimeField(auto_now_add=True, db_index=True), + ), + migrations.AlterField( + model_name='snapshot', + name='tags', + field=models.CharField(db_index=True, default=None, max_length=256, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(db_index=True, default=None, max_length=32, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='title', + field=models.CharField(db_index=True, default=None, max_length=128, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='updated', + field=models.DateTimeField(db_index=True, default=None, null=True), + ), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/0004_auto_20200713_1552.py b/archivebox-0.5.3/archivebox/core/migrations/0004_auto_20200713_1552.py new file mode 100644 index 0000000..6983662 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0004_auto_20200713_1552.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.7 on 2020-07-13 15:52 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0003_auto_20200630_1034'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(db_index=True, default=None, max_length=32, unique=True), + preserve_default=False, + ), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/0005_auto_20200728_0326.py b/archivebox-0.5.3/archivebox/core/migrations/0005_auto_20200728_0326.py new file mode 100644 index 0000000..f367aeb --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0005_auto_20200728_0326.py @@ -0,0 +1,28 @@ +# Generated by Django 3.0.7 on 2020-07-28 03:26 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0004_auto_20200713_1552'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='tags', + field=models.CharField(blank=True, db_index=True, max_length=256, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='title', + field=models.CharField(blank=True, db_index=True, max_length=128, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='updated', + field=models.DateTimeField(blank=True, db_index=True, null=True), + ), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/0006_auto_20201012_1520.py b/archivebox-0.5.3/archivebox/core/migrations/0006_auto_20201012_1520.py new file mode 100644 index 0000000..694c990 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0006_auto_20201012_1520.py @@ -0,0 +1,70 @@ +# Generated by Django 3.0.8 on 2020-10-12 15:20 + +from django.db import migrations, models +from django.utils.text import slugify + +def forwards_func(apps, schema_editor): + SnapshotModel = apps.get_model("core", "Snapshot") + TagModel = apps.get_model("core", "Tag") + + db_alias = schema_editor.connection.alias + snapshots = SnapshotModel.objects.all() + for snapshot in snapshots: + tags = snapshot.tags + tag_set = ( + set(tag.strip() for tag in (snapshot.tags_old or '').split(',')) + ) + tag_set.discard("") + + for tag in tag_set: + to_add, _ = TagModel.objects.get_or_create(name=tag, slug=slugify(tag)) + snapshot.tags.add(to_add) + + +def reverse_func(apps, schema_editor): + SnapshotModel = apps.get_model("core", "Snapshot") + TagModel = apps.get_model("core", "Tag") + + db_alias = schema_editor.connection.alias + snapshots = SnapshotModel.objects.all() + for snapshot in snapshots: + tags = snapshot.tags.values_list("name", flat=True) + snapshot.tags_old = ",".join([tag for tag in tags]) + snapshot.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0005_auto_20200728_0326'), + ] + + operations = [ + migrations.RenameField( + model_name='snapshot', + old_name='tags', + new_name='tags_old', + ), + migrations.CreateModel( + name='Tag', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('name', models.CharField(max_length=100, unique=True, verbose_name='name')), + ('slug', models.SlugField(max_length=100, unique=True, verbose_name='slug')), + ], + options={ + 'verbose_name': 'Tag', + 'verbose_name_plural': 'Tags', + }, + ), + migrations.AddField( + model_name='snapshot', + name='tags', + field=models.ManyToManyField(to='core.Tag'), + ), + migrations.RunPython(forwards_func, reverse_func), + migrations.RemoveField( + model_name='snapshot', + name='tags_old', + ), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/0007_archiveresult.py b/archivebox-0.5.3/archivebox/core/migrations/0007_archiveresult.py new file mode 100644 index 0000000..a780376 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0007_archiveresult.py @@ -0,0 +1,97 @@ +# Generated by Django 3.0.8 on 2020-11-04 12:25 + +import json +from pathlib import Path + +from django.db import migrations, models +import django.db.models.deletion + +from config import CONFIG +from index.json import to_json + +try: + JSONField = models.JSONField +except AttributeError: + import jsonfield + JSONField = jsonfield.JSONField + + +def forwards_func(apps, schema_editor): + from core.models import EXTRACTORS + + Snapshot = apps.get_model("core", "Snapshot") + ArchiveResult = apps.get_model("core", "ArchiveResult") + + snapshots = Snapshot.objects.all() + for snapshot in snapshots: + out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp + + try: + with open(out_dir / "index.json", "r") as f: + fs_index = json.load(f) + except Exception as e: + continue + + history = fs_index["history"] + + for extractor in history: + for result in history[extractor]: + ArchiveResult.objects.create(extractor=extractor, snapshot=snapshot, cmd=result["cmd"], cmd_version=result["cmd_version"], + start_ts=result["start_ts"], end_ts=result["end_ts"], status=result["status"], pwd=result["pwd"], output=result["output"]) + + +def verify_json_index_integrity(snapshot): + results = snapshot.archiveresult_set.all() + out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp + with open(out_dir / "index.json", "r") as f: + index = json.load(f) + + history = index["history"] + index_results = [result for extractor in history for result in history[extractor]] + flattened_results = [result["start_ts"] for result in index_results] + + missing_results = [result for result in results if result.start_ts.isoformat() not in flattened_results] + + for missing in missing_results: + index["history"][missing.extractor].append({"cmd": missing.cmd, "cmd_version": missing.cmd_version, "end_ts": missing.end_ts.isoformat(), + "start_ts": missing.start_ts.isoformat(), "pwd": missing.pwd, "output": missing.output, + "schema": "ArchiveResult", "status": missing.status}) + + json_index = to_json(index) + with open(out_dir / "index.json", "w") as f: + f.write(json_index) + + +def reverse_func(apps, schema_editor): + Snapshot = apps.get_model("core", "Snapshot") + ArchiveResult = apps.get_model("core", "ArchiveResult") + for snapshot in Snapshot.objects.all(): + verify_json_index_integrity(snapshot) + + ArchiveResult.objects.all().delete() + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0006_auto_20201012_1520'), + ] + + operations = [ + migrations.CreateModel( + name='ArchiveResult', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('cmd', JSONField()), + ('pwd', models.CharField(max_length=256)), + ('cmd_version', models.CharField(max_length=32)), + ('status', models.CharField(choices=[('succeeded', 'succeeded'), ('failed', 'failed'), ('skipped', 'skipped')], max_length=16)), + ('output', models.CharField(max_length=512)), + ('start_ts', models.DateTimeField()), + ('end_ts', models.DateTimeField()), + ('extractor', models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('wget', 'wget'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('headers', 'headers'), ('archive_org', 'archive_org')], max_length=32)), + ('snapshot', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Snapshot')), + ], + ), + migrations.RunPython(forwards_func, reverse_func), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/0008_auto_20210105_1421.py b/archivebox-0.5.3/archivebox/core/migrations/0008_auto_20210105_1421.py new file mode 100644 index 0000000..e5b3387 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/migrations/0008_auto_20210105_1421.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.3 on 2021-01-05 14:21 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0007_archiveresult'), + ] + + operations = [ + migrations.AlterField( + model_name='archiveresult', + name='cmd_version', + field=models.CharField(blank=True, default=None, max_length=32, null=True), + ), + ] diff --git a/archivebox-0.5.3/archivebox/core/migrations/__init__.py b/archivebox-0.5.3/archivebox/core/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/archivebox/core/mixins.py b/archivebox-0.5.3/archivebox/core/mixins.py new file mode 100644 index 0000000..538ca1e --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/mixins.py @@ -0,0 +1,23 @@ +from django.contrib import messages + +from archivebox.search import query_search_index + +class SearchResultsAdminMixin(object): + def get_search_results(self, request, queryset, search_term): + ''' Enhances the search queryset with results from the search backend. + ''' + qs, use_distinct = \ + super(SearchResultsAdminMixin, self).get_search_results( + request, queryset, search_term) + + search_term = search_term.strip() + if not search_term: + return qs, use_distinct + try: + qsearch = query_search_index(search_term) + except Exception as err: + messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}') + else: + qs = queryset & qsearch + finally: + return qs, use_distinct diff --git a/archivebox-0.5.3/archivebox/core/models.py b/archivebox-0.5.3/archivebox/core/models.py new file mode 100644 index 0000000..13d75b6 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/models.py @@ -0,0 +1,194 @@ +__package__ = 'archivebox.core' + +import uuid + +from django.db import models, transaction +from django.utils.functional import cached_property +from django.utils.text import slugify +from django.db.models import Case, When, Value, IntegerField + +from ..util import parse_date +from ..index.schema import Link +from ..extractors import get_default_archive_methods, ARCHIVE_METHODS_INDEXING_PRECEDENCE + +EXTRACTORS = [(extractor[0], extractor[0]) for extractor in get_default_archive_methods()] +STATUS_CHOICES = [ + ("succeeded", "succeeded"), + ("failed", "failed"), + ("skipped", "skipped") +] + +try: + JSONField = models.JSONField +except AttributeError: + import jsonfield + JSONField = jsonfield.JSONField + + +class Tag(models.Model): + """ + Based on django-taggit model + """ + name = models.CharField(verbose_name="name", unique=True, blank=False, max_length=100) + slug = models.SlugField(verbose_name="slug", unique=True, max_length=100) + + class Meta: + verbose_name = "Tag" + verbose_name_plural = "Tags" + + def __str__(self): + return self.name + + def slugify(self, tag, i=None): + slug = slugify(tag) + if i is not None: + slug += "_%d" % i + return slug + + def save(self, *args, **kwargs): + if self._state.adding and not self.slug: + self.slug = self.slugify(self.name) + + with transaction.atomic(): + slugs = set( + type(self) + ._default_manager.filter(slug__startswith=self.slug) + .values_list("slug", flat=True) + ) + + i = None + while True: + slug = self.slugify(self.name, i) + if slug not in slugs: + self.slug = slug + return super().save(*args, **kwargs) + i = 1 if i is None else i+1 + else: + return super().save(*args, **kwargs) + + +class Snapshot(models.Model): + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) + + url = models.URLField(unique=True) + timestamp = models.CharField(max_length=32, unique=True, db_index=True) + + title = models.CharField(max_length=128, null=True, blank=True, db_index=True) + + added = models.DateTimeField(auto_now_add=True, db_index=True) + updated = models.DateTimeField(null=True, blank=True, db_index=True) + tags = models.ManyToManyField(Tag) + + keys = ('url', 'timestamp', 'title', 'tags', 'updated') + + def __repr__(self) -> str: + title = self.title or '-' + return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})' + + def __str__(self) -> str: + title = self.title or '-' + return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})' + + @classmethod + def from_json(cls, info: dict): + info = {k: v for k, v in info.items() if k in cls.keys} + return cls(**info) + + def as_json(self, *args) -> dict: + args = args or self.keys + return { + key: getattr(self, key) + if key != 'tags' else self.tags_str() + for key in args + } + + def as_link(self) -> Link: + return Link.from_json(self.as_json()) + + def as_link_with_details(self) -> Link: + from ..index import load_link_details + return load_link_details(self.as_link()) + + def tags_str(self) -> str: + return ','.join(self.tags.order_by('name').values_list('name', flat=True)) + + @cached_property + def bookmarked(self): + return parse_date(self.timestamp) + + @cached_property + def is_archived(self): + return self.as_link().is_archived + + @cached_property + def num_outputs(self): + return self.archiveresult_set.filter(status='succeeded').count() + + @cached_property + def url_hash(self): + return self.as_link().url_hash + + @cached_property + def base_url(self): + return self.as_link().base_url + + @cached_property + def link_dir(self): + return self.as_link().link_dir + + @cached_property + def archive_path(self): + return self.as_link().archive_path + + @cached_property + def archive_size(self): + return self.as_link().archive_size + + @cached_property + def history(self): + # TODO: use ArchiveResult for this instead of json + return self.as_link_with_details().history + + @cached_property + def latest_title(self): + if ('title' in self.history + and self.history['title'] + and (self.history['title'][-1].status == 'succeeded') + and self.history['title'][-1].output.strip()): + return self.history['title'][-1].output.strip() + return None + + def save_tags(self, tags=()): + tags_id = [] + for tag in tags: + tags_id.append(Tag.objects.get_or_create(name=tag)[0].id) + self.tags.clear() + self.tags.add(*tags_id) + + +class ArchiveResultManager(models.Manager): + def indexable(self, sorted: bool = True): + INDEXABLE_METHODS = [ r[0] for r in ARCHIVE_METHODS_INDEXING_PRECEDENCE ] + qs = self.get_queryset().filter(extractor__in=INDEXABLE_METHODS,status='succeeded') + + if sorted: + precedence = [ When(extractor=method, then=Value(precedence)) for method, precedence in ARCHIVE_METHODS_INDEXING_PRECEDENCE ] + qs = qs.annotate(indexing_precedence=Case(*precedence, default=Value(1000),output_field=IntegerField())).order_by('indexing_precedence') + return qs + + +class ArchiveResult(models.Model): + snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE) + cmd = JSONField() + pwd = models.CharField(max_length=256) + cmd_version = models.CharField(max_length=32, default=None, null=True, blank=True) + output = models.CharField(max_length=512) + start_ts = models.DateTimeField() + end_ts = models.DateTimeField() + status = models.CharField(max_length=16, choices=STATUS_CHOICES) + extractor = models.CharField(choices=EXTRACTORS, max_length=32) + + objects = ArchiveResultManager() + + def __str__(self): + return self.extractor diff --git a/archivebox-0.5.3/archivebox/core/settings.py b/archivebox-0.5.3/archivebox/core/settings.py new file mode 100644 index 0000000..e8ed6b1 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/settings.py @@ -0,0 +1,165 @@ +__package__ = 'archivebox.core' + +import os +import sys + +from pathlib import Path +from django.utils.crypto import get_random_string + +from ..config import ( # noqa: F401 + DEBUG, + SECRET_KEY, + ALLOWED_HOSTS, + PACKAGE_DIR, + ACTIVE_THEME, + TEMPLATES_DIR_NAME, + SQL_INDEX_FILENAME, + OUTPUT_DIR, +) + + +IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3] +IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ +IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3] + +################################################################################ +### Django Core Settings +################################################################################ + +WSGI_APPLICATION = 'core.wsgi.application' +ROOT_URLCONF = 'core.urls' + +LOGIN_URL = '/accounts/login/' +LOGOUT_REDIRECT_URL = '/' +PASSWORD_RESET_URL = '/accounts/password_reset/' +APPEND_SLASH = True + +INSTALLED_APPS = [ + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', + 'django.contrib.admin', + + 'core', + + 'django_extensions', +] + + +MIDDLEWARE = [ + 'django.middleware.security.SecurityMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.common.CommonMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', +] + +AUTHENTICATION_BACKENDS = [ + 'django.contrib.auth.backends.ModelBackend', +] + + +################################################################################ +### Staticfile and Template Settings +################################################################################ + +STATIC_URL = '/static/' + +STATICFILES_DIRS = [ + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / ACTIVE_THEME / 'static'), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'default' / 'static'), +] + +TEMPLATE_DIRS = [ + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / ACTIVE_THEME), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'default'), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME), +] + +TEMPLATES = [ + { + 'BACKEND': 'django.template.backends.django.DjangoTemplates', + 'DIRS': TEMPLATE_DIRS, + 'APP_DIRS': True, + 'OPTIONS': { + 'context_processors': [ + 'django.template.context_processors.debug', + 'django.template.context_processors.request', + 'django.contrib.auth.context_processors.auth', + 'django.contrib.messages.context_processors.messages', + ], + }, + }, +] + + +################################################################################ +### External Service Settings +################################################################################ + +DATABASE_FILE = Path(OUTPUT_DIR) / SQL_INDEX_FILENAME +DATABASE_NAME = os.environ.get("ARCHIVEBOX_DATABASE_NAME", DATABASE_FILE) + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': DATABASE_NAME, + } +} + +EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' + + +################################################################################ +### Security Settings +################################################################################ + +SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.') + +ALLOWED_HOSTS = ALLOWED_HOSTS.split(',') + +SECURE_BROWSER_XSS_FILTER = True +SECURE_CONTENT_TYPE_NOSNIFF = True + +CSRF_COOKIE_SECURE = False +SESSION_COOKIE_SECURE = False +SESSION_COOKIE_DOMAIN = None +SESSION_COOKIE_AGE = 1209600 # 2 weeks +SESSION_EXPIRE_AT_BROWSER_CLOSE = False +SESSION_SAVE_EVERY_REQUEST = True + +AUTH_PASSWORD_VALIDATORS = [ + {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'}, + {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'}, + {'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator'}, + {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'}, +] + + +################################################################################ +### Shell Settings +################################################################################ + +SHELL_PLUS = 'ipython' +SHELL_PLUS_PRINT_SQL = False +IPYTHON_ARGUMENTS = ['--no-confirm-exit', '--no-banner'] +IPYTHON_KERNEL_DISPLAY_NAME = 'ArchiveBox Django Shell' +if IS_SHELL: + os.environ['PYTHONSTARTUP'] = str(Path(PACKAGE_DIR) / 'core' / 'welcome_message.py') + + +################################################################################ +### Internationalization & Localization Settings +################################################################################ + +LANGUAGE_CODE = 'en-us' +TIME_ZONE = 'UTC' +USE_I18N = False +USE_L10N = False +USE_TZ = False + +DATETIME_FORMAT = 'Y-m-d g:iA' +SHORT_DATETIME_FORMAT = 'Y-m-d h:iA' diff --git a/archivebox-0.5.3/archivebox/core/templatetags/__init__.py b/archivebox-0.5.3/archivebox/core/templatetags/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/archivebox/core/templatetags/core_tags.py b/archivebox-0.5.3/archivebox/core/templatetags/core_tags.py new file mode 100644 index 0000000..25f0685 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/templatetags/core_tags.py @@ -0,0 +1,47 @@ +from django import template +from django.urls import reverse +from django.contrib.admin.templatetags.base import InclusionAdminNode +from django.templatetags.static import static + + +from typing import Union + +from core.models import ArchiveResult + +register = template.Library() + +@register.simple_tag +def snapshot_image(snapshot): + result = ArchiveResult.objects.filter(snapshot=snapshot, extractor='screenshot', status='succeeded').first() + if result: + return reverse('LinkAssets', args=[f'{str(snapshot.timestamp)}/{result.output}']) + + return static('archive.png') + +@register.filter +def file_size(num_bytes: Union[int, float]) -> str: + for count in ['Bytes','KB','MB','GB']: + if num_bytes > -1024.0 and num_bytes < 1024.0: + return '%3.1f %s' % (num_bytes, count) + num_bytes /= 1024.0 + return '%3.1f %s' % (num_bytes, 'TB') + +def result_list(cl): + """ + Monkey patched result + """ + num_sorted_fields = 0 + return { + 'cl': cl, + 'num_sorted_fields': num_sorted_fields, + 'results': cl.result_list, + } + +@register.tag(name='snapshots_grid') +def result_list_tag(parser, token): + return InclusionAdminNode( + parser, token, + func=result_list, + template_name='snapshots_grid.html', + takes_context=False, + ) diff --git a/archivebox-0.5.3/archivebox/core/tests.py b/archivebox-0.5.3/archivebox/core/tests.py new file mode 100644 index 0000000..4d66077 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/tests.py @@ -0,0 +1,3 @@ +#from django.test import TestCase + +# Create your tests here. diff --git a/archivebox-0.5.3/archivebox/core/urls.py b/archivebox-0.5.3/archivebox/core/urls.py new file mode 100644 index 0000000..b8e4baf --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/urls.py @@ -0,0 +1,36 @@ +from django.contrib import admin + +from django.urls import path, include +from django.views import static +from django.conf import settings +from django.views.generic.base import RedirectView + +from core.views import MainIndex, LinkDetails, PublicArchiveView, AddView + + +# print('DEBUG', settings.DEBUG) + +urlpatterns = [ + path('robots.txt', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'robots.txt'}), + path('favicon.ico', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'favicon.ico'}), + + path('docs/', RedirectView.as_view(url='https://github.com/ArchiveBox/ArchiveBox/wiki'), name='Docs'), + + path('archive/', RedirectView.as_view(url='/')), + path('archive/', LinkDetails.as_view(), name='LinkAssets'), + + path('admin/core/snapshot/add/', RedirectView.as_view(url='/add/')), + path('add/', AddView.as_view()), + + path('accounts/login/', RedirectView.as_view(url='/admin/login/')), + path('accounts/logout/', RedirectView.as_view(url='/admin/logout/')), + + + path('accounts/', include('django.contrib.auth.urls')), + path('admin/', admin.site.urls), + + path('index.html', RedirectView.as_view(url='/')), + path('index.json', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'index.json'}), + path('', MainIndex.as_view(), name='Home'), + path('public/', PublicArchiveView.as_view(), name='public-index'), +] diff --git a/archivebox-0.5.3/archivebox/core/views.py b/archivebox-0.5.3/archivebox/core/views.py new file mode 100644 index 0000000..b46e364 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/views.py @@ -0,0 +1,173 @@ +__package__ = 'archivebox.core' + +from io import StringIO +from contextlib import redirect_stdout + +from django.shortcuts import render, redirect + +from django.http import HttpResponse +from django.views import View, static +from django.views.generic.list import ListView +from django.views.generic import FormView +from django.contrib.auth.mixins import UserPassesTestMixin + +from core.models import Snapshot +from core.forms import AddLinkForm + +from ..config import ( + OUTPUT_DIR, + PUBLIC_INDEX, + PUBLIC_SNAPSHOTS, + PUBLIC_ADD_VIEW, + VERSION, + FOOTER_INFO, +) +from main import add +from ..util import base_url, ansi_to_html +from ..index.html import snapshot_icons + + +class MainIndex(View): + template = 'main_index.html' + + def get(self, request): + if request.user.is_authenticated: + return redirect('/admin/core/snapshot/') + + if PUBLIC_INDEX: + return redirect('public-index') + + return redirect(f'/admin/login/?next={request.path}') + + +class LinkDetails(View): + def get(self, request, path): + # missing trailing slash -> redirect to index + if '/' not in path: + return redirect(f'{path}/index.html') + + if not request.user.is_authenticated and not PUBLIC_SNAPSHOTS: + return redirect(f'/admin/login/?next={request.path}') + + try: + slug, archivefile = path.split('/', 1) + except (IndexError, ValueError): + slug, archivefile = path.split('/', 1)[0], 'index.html' + + all_pages = list(Snapshot.objects.all()) + + # slug is a timestamp + by_ts = {page.timestamp: page for page in all_pages} + try: + # print('SERVING STATICFILE', by_ts[slug].link_dir, request.path, path) + response = static.serve(request, archivefile, document_root=by_ts[slug].link_dir, show_indexes=True) + response["Link"] = f'<{by_ts[slug].url}>; rel="canonical"' + return response + except KeyError: + pass + + # slug is a hash + by_hash = {page.url_hash: page for page in all_pages} + try: + timestamp = by_hash[slug].timestamp + return redirect(f'/archive/{timestamp}/{archivefile}') + except KeyError: + pass + + # slug is a URL + by_url = {page.base_url: page for page in all_pages} + try: + # TODO: add multiple snapshot support by showing index of all snapshots + # for given url instead of redirecting to timestamp index + timestamp = by_url[base_url(path)].timestamp + return redirect(f'/archive/{timestamp}/index.html') + except KeyError: + pass + + return HttpResponse( + 'No archived link matches the given timestamp or hash.', + content_type="text/plain", + status=404, + ) + +class PublicArchiveView(ListView): + template = 'snapshot_list.html' + model = Snapshot + paginate_by = 100 + ordering = ['title'] + + def get_context_data(self, **kwargs): + return { + **super().get_context_data(**kwargs), + 'VERSION': VERSION, + 'FOOTER_INFO': FOOTER_INFO, + } + + def get_queryset(self, **kwargs): + qs = super().get_queryset(**kwargs) + query = self.request.GET.get('q') + if query: + qs = qs.filter(title__icontains=query) + for snapshot in qs: + snapshot.icons = snapshot_icons(snapshot) + return qs + + def get(self, *args, **kwargs): + if PUBLIC_INDEX or self.request.user.is_authenticated: + response = super().get(*args, **kwargs) + return response + else: + return redirect(f'/admin/login/?next={self.request.path}') + + +class AddView(UserPassesTestMixin, FormView): + template_name = "add_links.html" + form_class = AddLinkForm + + def get_initial(self): + """Prefill the AddLinkForm with the 'url' GET parameter""" + if self.request.method == 'GET': + url = self.request.GET.get('url', None) + if url: + return {'url': url} + else: + return super().get_initial() + + def test_func(self): + return PUBLIC_ADD_VIEW or self.request.user.is_authenticated + + def get_context_data(self, **kwargs): + return { + **super().get_context_data(**kwargs), + 'title': "Add URLs", + # We can't just call request.build_absolute_uri in the template, because it would include query parameters + 'absolute_add_path': self.request.build_absolute_uri(self.request.path), + 'VERSION': VERSION, + 'FOOTER_INFO': FOOTER_INFO, + } + + def form_valid(self, form): + url = form.cleaned_data["url"] + print(f'[+] Adding URL: {url}') + depth = 0 if form.cleaned_data["depth"] == "0" else 1 + extractors = ','.join(form.cleaned_data["archive_methods"]) + input_kwargs = { + "urls": url, + "depth": depth, + "update_all": False, + "out_dir": OUTPUT_DIR, + } + if extractors: + input_kwargs.update({"extractors": extractors}) + add_stdout = StringIO() + with redirect_stdout(add_stdout): + add(**input_kwargs) + print(add_stdout.getvalue()) + + context = self.get_context_data() + + context.update({ + "stdout": ansi_to_html(add_stdout.getvalue().strip()), + "form": AddLinkForm() + }) + return render(template_name=self.template_name, request=self.request, context=context) diff --git a/archivebox-0.5.3/archivebox/core/welcome_message.py b/archivebox-0.5.3/archivebox/core/welcome_message.py new file mode 100644 index 0000000..ed5d2d7 --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/welcome_message.py @@ -0,0 +1,5 @@ +from archivebox.logging_util import log_shell_welcome_msg + + +if __name__ == '__main__': + log_shell_welcome_msg() diff --git a/archivebox-0.5.3/archivebox/core/wsgi.py b/archivebox-0.5.3/archivebox/core/wsgi.py new file mode 100644 index 0000000..f933afa --- /dev/null +++ b/archivebox-0.5.3/archivebox/core/wsgi.py @@ -0,0 +1,16 @@ +""" +WSGI config for archivebox project. + +It exposes the WSGI callable as a module-level variable named ``application``. + +For more information on this file, see +https://docs.djangoproject.com/en/2.1/howto/deployment/wsgi/ +""" + +import os + +from django.core.wsgi import get_wsgi_application + +os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'archivebox.settings') + +application = get_wsgi_application() diff --git a/archivebox-0.5.3/archivebox/extractors/__init__.py b/archivebox-0.5.3/archivebox/extractors/__init__.py new file mode 100644 index 0000000..a4acef0 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/__init__.py @@ -0,0 +1,182 @@ +__package__ = 'archivebox.extractors' + +import os +from pathlib import Path + +from typing import Optional, List, Iterable, Union +from datetime import datetime +from django.db.models import QuerySet + +from ..index.schema import Link +from ..index.sql import write_link_to_sql_index +from ..index import ( + load_link_details, + write_link_details, +) +from ..util import enforce_types +from ..logging_util import ( + log_archiving_started, + log_archiving_paused, + log_archiving_finished, + log_link_archiving_started, + log_link_archiving_finished, + log_archive_method_started, + log_archive_method_finished, +) +from ..search import write_search_index + +from .title import should_save_title, save_title +from .favicon import should_save_favicon, save_favicon +from .wget import should_save_wget, save_wget +from .singlefile import should_save_singlefile, save_singlefile +from .readability import should_save_readability, save_readability +from .mercury import should_save_mercury, save_mercury +from .pdf import should_save_pdf, save_pdf +from .screenshot import should_save_screenshot, save_screenshot +from .dom import should_save_dom, save_dom +from .git import should_save_git, save_git +from .media import should_save_media, save_media +from .archive_org import should_save_archive_dot_org, save_archive_dot_org +from .headers import should_save_headers, save_headers + + +def get_default_archive_methods(): + return [ + ('title', should_save_title, save_title), + ('favicon', should_save_favicon, save_favicon), + ('wget', should_save_wget, save_wget), + ('singlefile', should_save_singlefile, save_singlefile), + ('pdf', should_save_pdf, save_pdf), + ('screenshot', should_save_screenshot, save_screenshot), + ('dom', should_save_dom, save_dom), + ('readability', should_save_readability, save_readability), #keep readability below wget and singlefile, as it depends on them + ('mercury', should_save_mercury, save_mercury), + ('git', should_save_git, save_git), + ('media', should_save_media, save_media), + ('headers', should_save_headers, save_headers), + ('archive_org', should_save_archive_dot_org, save_archive_dot_org), + ] + +ARCHIVE_METHODS_INDEXING_PRECEDENCE = [('readability', 1), ('singlefile', 2), ('dom', 3), ('wget', 4)] + +@enforce_types +def ignore_methods(to_ignore: List[str]): + ARCHIVE_METHODS = get_default_archive_methods() + methods = filter(lambda x: x[0] not in to_ignore, ARCHIVE_METHODS) + methods = map(lambda x: x[0], methods) + return list(methods) + +@enforce_types +def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> Link: + """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp""" + + # TODO: Remove when the input is changed to be a snapshot. Suboptimal approach. + from core.models import Snapshot, ArchiveResult + try: + snapshot = Snapshot.objects.get(url=link.url) # TODO: This will be unnecessary once everything is a snapshot + except Snapshot.DoesNotExist: + snapshot = write_link_to_sql_index(link) + + ARCHIVE_METHODS = get_default_archive_methods() + + if methods: + ARCHIVE_METHODS = [ + method for method in ARCHIVE_METHODS + if method[0] in methods + ] + + out_dir = out_dir or Path(link.link_dir) + try: + is_new = not Path(out_dir).exists() + if is_new: + os.makedirs(out_dir) + + link = load_link_details(link, out_dir=out_dir) + write_link_details(link, out_dir=out_dir, skip_sql_index=False) + log_link_archiving_started(link, out_dir, is_new) + link = link.overwrite(updated=datetime.now()) + stats = {'skipped': 0, 'succeeded': 0, 'failed': 0} + + for method_name, should_run, method_function in ARCHIVE_METHODS: + try: + if method_name not in link.history: + link.history[method_name] = [] + + if should_run(link, out_dir) or overwrite: + log_archive_method_started(method_name) + + result = method_function(link=link, out_dir=out_dir) + + link.history[method_name].append(result) + + stats[result.status] += 1 + log_archive_method_finished(result) + write_search_index(link=link, texts=result.index_texts) + ArchiveResult.objects.create(snapshot=snapshot, extractor=method_name, cmd=result.cmd, cmd_version=result.cmd_version, + output=result.output, pwd=result.pwd, start_ts=result.start_ts, end_ts=result.end_ts, status=result.status) + + else: + # print('{black} X {}{reset}'.format(method_name, **ANSI)) + stats['skipped'] += 1 + except Exception as e: + raise Exception('Exception in archive_methods.save_{}(Link(url={}))'.format( + method_name, + link.url, + )) from e + + # print(' ', stats) + + try: + latest_title = link.history['title'][-1].output.strip() + if latest_title and len(latest_title) >= len(link.title or ''): + link = link.overwrite(title=latest_title) + except Exception: + pass + + write_link_details(link, out_dir=out_dir, skip_sql_index=False) + + log_link_archiving_finished(link, link.link_dir, is_new, stats) + + except KeyboardInterrupt: + try: + write_link_details(link, out_dir=link.link_dir) + except: + pass + raise + + except Exception as err: + print(' ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err)) + raise + + return link + +@enforce_types +def archive_links(all_links: Union[Iterable[Link], QuerySet], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> List[Link]: + + if type(all_links) is QuerySet: + num_links: int = all_links.count() + get_link = lambda x: x.as_link() + all_links = all_links.iterator() + else: + num_links: int = len(all_links) + get_link = lambda x: x + + if num_links == 0: + return [] + + log_archiving_started(num_links) + idx: int = 0 + try: + for link in all_links: + idx += 1 + to_archive = get_link(link) + archive_link(to_archive, overwrite=overwrite, methods=methods, out_dir=Path(link.link_dir)) + except KeyboardInterrupt: + log_archiving_paused(num_links, idx, link.timestamp) + raise SystemExit(0) + except BaseException: + print() + raise + + log_archiving_finished(num_links) + return all_links diff --git a/archivebox-0.5.3/archivebox/extractors/archive_org.py b/archivebox-0.5.3/archivebox/extractors/archive_org.py new file mode 100644 index 0000000..f5598d6 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/archive_org.py @@ -0,0 +1,112 @@ +__package__ = 'archivebox.extractors' + + +from pathlib import Path +from typing import Optional, List, Dict, Tuple +from collections import defaultdict + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, +) +from ..config import ( + TIMEOUT, + CURL_ARGS, + CHECK_SSL_VALIDITY, + SAVE_ARCHIVE_DOT_ORG, + CURL_BINARY, + CURL_VERSION, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_archive_dot_org(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "archive.org.txt").exists(): + # if open(path, 'r').read().strip() != 'None': + return False + + return SAVE_ARCHIVE_DOT_ORG + +@enforce_types +def save_archive_dot_org(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """submit site to archive.org for archiving via their service, save returned archive url""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'archive.org.txt' + archive_org_url = None + submit_url = 'https://web.archive.org/save/{}'.format(link.url) + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--head', + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + submit_url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + content_location, errors = parse_archive_dot_org_response(result.stdout) + if content_location: + archive_org_url = content_location[0] + elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]: + archive_org_url = None + # raise ArchiveError('Archive.org denied by {}/robots.txt'.format(domain(link.url))) + elif errors: + raise ArchiveError(', '.join(errors)) + else: + raise ArchiveError('Failed to find "content-location" URL header in Archive.org response.') + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + if output and not isinstance(output, Exception): + # instead of writing None when archive.org rejects the url write the + # url to resubmit it to archive.org. This is so when the user visits + # the URL in person, it will attempt to re-archive it, and it'll show the + # nicer error message explaining why the url was rejected if it fails. + archive_org_url = archive_org_url or submit_url + with open(str(out_dir / output), 'w', encoding='utf-8') as f: + f.write(archive_org_url) + chmod_file('archive.org.txt', cwd=str(out_dir)) + output = archive_org_url + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) + +@enforce_types +def parse_archive_dot_org_response(response: bytes) -> Tuple[List[str], List[str]]: + # Parse archive.org response headers + headers: Dict[str, List[str]] = defaultdict(list) + + # lowercase all the header names and store in dict + for header in response.splitlines(): + if b':' not in header or not header.strip(): + continue + name, val = header.decode().split(':', 1) + headers[name.lower().strip()].append(val.strip()) + + # Get successful archive url in "content-location" header or any errors + content_location = headers.get('content-location', headers['location']) + errors = headers['x-archive-wayback-runtime-error'] + return content_location, errors + diff --git a/archivebox-0.5.3/archivebox/extractors/dom.py b/archivebox-0.5.3/archivebox/extractors/dom.py new file mode 100644 index 0000000..babbe71 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/dom.py @@ -0,0 +1,69 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file, atomic_write +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_DOM, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_dom(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / 'output.html').exists(): + return False + + return SAVE_DOM + +@enforce_types +def save_dom(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """print HTML of site to file using chrome --dump-html""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'output.html' + output_path = out_dir / output + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--dump-dom', + link.url + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + atomic_write(output_path, result.stdout) + + if result.returncode: + hints = result.stderr.decode() + raise ArchiveError('Failed to save DOM', hints) + + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/favicon.py b/archivebox-0.5.3/archivebox/extractors/favicon.py new file mode 100644 index 0000000..5e7c1fb --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/favicon.py @@ -0,0 +1,64 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput +from ..system import chmod_file, run +from ..util import enforce_types, domain +from ..config import ( + TIMEOUT, + SAVE_FAVICON, + CURL_BINARY, + CURL_ARGS, + CURL_VERSION, + CHECK_SSL_VALIDITY, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_favicon(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if (Path(out_dir) / 'favicon.ico').exists(): + return False + + return SAVE_FAVICON + +@enforce_types +def save_favicon(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download site favicon from google's favicon api""" + + out_dir = out_dir or link.link_dir + output: ArchiveOutput = 'favicon.ico' + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--max-time', str(timeout), + '--output', str(output), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + 'https://www.google.com/s2/favicons?domain={}'.format(domain(link.url)), + ] + status = 'pending' + timer = TimedProgress(timeout, prefix=' ') + try: + run(cmd, cwd=str(out_dir), timeout=timeout) + chmod_file(output, cwd=str(out_dir)) + status = 'succeeded' + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/git.py b/archivebox-0.5.3/archivebox/extractors/git.py new file mode 100644 index 0000000..fd20d4b --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/git.py @@ -0,0 +1,90 @@ +__package__ = 'archivebox.extractors' + + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + domain, + extension, + without_query, + without_fragment, +) +from ..config import ( + TIMEOUT, + SAVE_GIT, + GIT_BINARY, + GIT_ARGS, + GIT_VERSION, + GIT_DOMAINS, + CHECK_SSL_VALIDITY +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_git(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + if (out_dir / "git").exists(): + return False + + is_clonable_url = ( + (domain(link.url) in GIT_DOMAINS) + or (extension(link.url) == 'git') + ) + if not is_clonable_url: + return False + + return SAVE_GIT + + +@enforce_types +def save_git(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using git""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'git' + output_path = out_dir / output + output_path.mkdir(exist_ok=True) + cmd = [ + GIT_BINARY, + 'clone', + *GIT_ARGS, + *([] if CHECK_SSL_VALIDITY else ['-c', 'http.sslVerify=false']), + without_query(without_fragment(link.url)), + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(output_path), timeout=timeout + 1) + if result.returncode == 128: + # ignore failed re-download when the folder already exists + pass + elif result.returncode > 0: + hints = 'Got git response code: {}.'.format(result.returncode) + raise ArchiveError('Failed to save git clone', hints) + + chmod_file(output, cwd=str(out_dir)) + + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=GIT_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/headers.py b/archivebox-0.5.3/archivebox/extractors/headers.py new file mode 100644 index 0000000..4e69dec --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/headers.py @@ -0,0 +1,69 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput +from ..system import atomic_write +from ..util import ( + enforce_types, + get_headers, +) +from ..config import ( + TIMEOUT, + CURL_BINARY, + CURL_ARGS, + CURL_USER_AGENT, + CURL_VERSION, + CHECK_SSL_VALIDITY, + SAVE_HEADERS +) +from ..logging_util import TimedProgress + +@enforce_types +def should_save_headers(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + + output = Path(out_dir or link.link_dir) / 'headers.json' + return not output.exists() and SAVE_HEADERS + + +@enforce_types +def save_headers(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """Download site headers""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() + output: ArchiveOutput = 'headers.json' + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--head', + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + link.url, + ] + try: + json_headers = get_headers(link.url, timeout=timeout) + output_folder.mkdir(exist_ok=True) + atomic_write(str(output_folder / "headers.json"), json_headers) + except (Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/media.py b/archivebox-0.5.3/archivebox/extractors/media.py new file mode 100644 index 0000000..3792fd2 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/media.py @@ -0,0 +1,81 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, +) +from ..config import ( + MEDIA_TIMEOUT, + SAVE_MEDIA, + YOUTUBEDL_ARGS, + YOUTUBEDL_BINARY, + YOUTUBEDL_VERSION, + CHECK_SSL_VALIDITY +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_media(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or link.link_dir + + if is_static_file(link.url): + return False + + if (out_dir / "media").exists(): + return False + + return SAVE_MEDIA + +@enforce_types +def save_media(link: Link, out_dir: Optional[Path]=None, timeout: int=MEDIA_TIMEOUT) -> ArchiveResult: + """Download playlists or individual video, audio, and subtitles using youtube-dl""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'media' + output_path = out_dir / output + output_path.mkdir(exist_ok=True) + cmd = [ + YOUTUBEDL_BINARY, + *YOUTUBEDL_ARGS, + *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate']), + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(output_path), timeout=timeout + 1) + chmod_file(output, cwd=str(out_dir)) + if result.returncode: + if (b'ERROR: Unsupported URL' in result.stderr + or b'HTTP Error 404' in result.stderr + or b'HTTP Error 403' in result.stderr + or b'URL could be a direct video link' in result.stderr + or b'Unable to extract container ID' in result.stderr): + # These happen too frequently on non-media pages to warrant printing to console + pass + else: + hints = ( + 'Got youtube-dl response code: {}.'.format(result.returncode), + *result.stderr.decode().split('\n'), + ) + raise ArchiveError('Failed to save media', hints) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=YOUTUBEDL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/mercury.py b/archivebox-0.5.3/archivebox/extractors/mercury.py new file mode 100644 index 0000000..741c329 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/mercury.py @@ -0,0 +1,104 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from subprocess import CompletedProcess +from typing import Optional, List +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, atomic_write +from ..util import ( + enforce_types, + is_static_file, + +) +from ..config import ( + TIMEOUT, + SAVE_MERCURY, + DEPENDENCIES, + MERCURY_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def ShellError(cmd: List[str], result: CompletedProcess, lines: int=20) -> ArchiveError: + # parse out last line of stderr + return ArchiveError( + f'Got {cmd[0]} response code: {result.returncode}).', + *( + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', lines)[-lines:] + if line.strip() + ), + ) + + +@enforce_types +def should_save_mercury(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + output = Path(out_dir or link.link_dir) / 'mercury' + return SAVE_MERCURY and MERCURY_VERSION and (not output.exists()) + + +@enforce_types +def save_mercury(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download reader friendly version using @postlight/mercury-parser""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() / "mercury" + output = str(output_folder) + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + # Get plain text version of article + cmd = [ + DEPENDENCIES['MERCURY_BINARY']['path'], + link.url, + "--format=text" + ] + result = run(cmd, cwd=out_dir, timeout=timeout) + try: + article_text = json.loads(result.stdout) + except json.JSONDecodeError: + raise ShellError(cmd, result) + + # Get HTML version of article + cmd = [ + DEPENDENCIES['MERCURY_BINARY']['path'], + link.url + ] + result = run(cmd, cwd=out_dir, timeout=timeout) + try: + article_json = json.loads(result.stdout) + except json.JSONDecodeError: + raise ShellError(cmd, result) + + output_folder.mkdir(exist_ok=True) + atomic_write(str(output_folder / "content.html"), article_json.pop("content")) + atomic_write(str(output_folder / "content.txt"), article_text["content"]) + atomic_write(str(output_folder / "article.json"), article_json) + + # Check for common failure cases + if (result.returncode > 0): + raise ShellError(cmd, result) + except (ArchiveError, Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=MERCURY_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/pdf.py b/archivebox-0.5.3/archivebox/extractors/pdf.py new file mode 100644 index 0000000..1b0201e --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/pdf.py @@ -0,0 +1,68 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_PDF, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_pdf(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "output.pdf").exists(): + return False + + return SAVE_PDF + + +@enforce_types +def save_pdf(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """print PDF of site to file using chrome --headless""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'output.pdf' + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--print-to-pdf', + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + if result.returncode: + hints = (result.stderr or result.stdout).decode() + raise ArchiveError('Failed to save PDF', hints) + + chmod_file('output.pdf', cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/readability.py b/archivebox-0.5.3/archivebox/extractors/readability.py new file mode 100644 index 0000000..9da620b --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/readability.py @@ -0,0 +1,124 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from tempfile import NamedTemporaryFile + +from typing import Optional +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, atomic_write +from ..util import ( + enforce_types, + download_url, + is_static_file, + +) +from ..config import ( + TIMEOUT, + CURL_BINARY, + SAVE_READABILITY, + DEPENDENCIES, + READABILITY_VERSION, +) +from ..logging_util import TimedProgress + +@enforce_types +def get_html(link: Link, path: Path) -> str: + """ + Try to find wget, singlefile and then dom files. + If none is found, download the url again. + """ + canonical = link.canonical_outputs() + abs_path = path.absolute() + sources = [canonical["singlefile_path"], canonical["wget_path"], canonical["dom_path"]] + document = None + for source in sources: + try: + with open(abs_path / source, "r") as f: + document = f.read() + break + except (FileNotFoundError, TypeError): + continue + if document is None: + return download_url(link.url) + else: + return document + +@enforce_types +def should_save_readability(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + output = Path(out_dir or link.link_dir) / 'readability' + return SAVE_READABILITY and READABILITY_VERSION and (not output.exists()) + + +@enforce_types +def save_readability(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download reader friendly version using @mozilla/readability""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() / "readability" + output = str(output_folder) + + # Readability Docs: https://github.com/mozilla/readability + + status = 'succeeded' + # fake command to show the user so they have something to try debugging if get_html fails + cmd = [ + CURL_BINARY, + link.url + ] + readability_content = None + timer = TimedProgress(timeout, prefix=' ') + try: + document = get_html(link, out_dir) + temp_doc = NamedTemporaryFile(delete=False) + temp_doc.write(document.encode("utf-8")) + temp_doc.close() + + cmd = [ + DEPENDENCIES['READABILITY_BINARY']['path'], + temp_doc.name + ] + + result = run(cmd, cwd=out_dir, timeout=timeout) + result_json = json.loads(result.stdout) + output_folder.mkdir(exist_ok=True) + readability_content = result_json.pop("textContent") + atomic_write(str(output_folder / "content.html"), result_json.pop("content")) + atomic_write(str(output_folder / "content.txt"), readability_content) + atomic_write(str(output_folder / "article.json"), result_json) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + hints = ( + 'Got readability response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0): + raise ArchiveError('Readability was not able to archive the page', hints) + except (Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=READABILITY_VERSION, + output=output, + status=status, + index_texts= [readability_content] if readability_content else [], + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/screenshot.py b/archivebox-0.5.3/archivebox/extractors/screenshot.py new file mode 100644 index 0000000..325584e --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/screenshot.py @@ -0,0 +1,67 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_SCREENSHOT, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_screenshot(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "screenshot.png").exists(): + return False + + return SAVE_SCREENSHOT + +@enforce_types +def save_screenshot(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """take screenshot of site using chrome --headless""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'screenshot.png' + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--screenshot', + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + if result.returncode: + hints = (result.stderr or result.stdout).decode() + raise ArchiveError('Failed to save screenshot', hints) + + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/singlefile.py b/archivebox-0.5.3/archivebox/extractors/singlefile.py new file mode 100644 index 0000000..2e5c389 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/singlefile.py @@ -0,0 +1,90 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_SINGLEFILE, + DEPENDENCIES, + SINGLEFILE_VERSION, + CHROME_BINARY, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_singlefile(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + output = out_dir / 'singlefile.html' + return SAVE_SINGLEFILE and SINGLEFILE_VERSION and (not output.exists()) + + +@enforce_types +def save_singlefile(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using single-file""" + + out_dir = out_dir or Path(link.link_dir) + output = str(out_dir.absolute() / "singlefile.html") + + browser_args = chrome_args(TIMEOUT=0) + + # SingleFile CLI Docs: https://github.com/gildas-lormeau/SingleFile/tree/master/cli + browser_args = '--browser-args={}'.format(json.dumps(browser_args[1:])) + cmd = [ + DEPENDENCIES['SINGLEFILE_BINARY']['path'], + '--browser-executable-path={}'.format(CHROME_BINARY), + browser_args, + link.url, + output + ] + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + hints = ( + 'Got single-file response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0): + raise ArchiveError('SingleFile was not able to archive the page', hints) + chmod_file(output) + except (Exception, OSError) as err: + status = 'failed' + # TODO: Make this prettier. This is necessary to run the command (escape JSON internal quotes). + cmd[2] = browser_args.replace('"', "\\\"") + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=SINGLEFILE_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/title.py b/archivebox-0.5.3/archivebox/extractors/title.py new file mode 100644 index 0000000..28cb128 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/title.py @@ -0,0 +1,130 @@ +__package__ = 'archivebox.extractors' + +import re +from html.parser import HTMLParser +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..util import ( + enforce_types, + is_static_file, + download_url, + htmldecode, +) +from ..config import ( + TIMEOUT, + CHECK_SSL_VALIDITY, + SAVE_TITLE, + CURL_BINARY, + CURL_ARGS, + CURL_VERSION, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + + +HTML_TITLE_REGEX = re.compile( + r'' # start matching text after tag + r'(.[^<>]+)', # get everything up to these symbols + re.IGNORECASE | re.MULTILINE | re.DOTALL | re.UNICODE, +) + + +class TitleParser(HTMLParser): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.title_tag = "" + self.title_og = "" + self.inside_title_tag = False + + @property + def title(self): + return self.title_tag or self.title_og or None + + def handle_starttag(self, tag, attrs): + if tag.lower() == "title" and not self.title_tag: + self.inside_title_tag = True + elif tag.lower() == "meta" and not self.title_og: + attrs = dict(attrs) + if attrs.get("property") == "og:title" and attrs.get("content"): + self.title_og = attrs.get("content") + + def handle_data(self, data): + if self.inside_title_tag and data: + self.title_tag += data.strip() + + def handle_endtag(self, tag): + if tag.lower() == "title": + self.inside_title_tag = False + + +@enforce_types +def should_save_title(link: Link, out_dir: Optional[str]=None) -> bool: + # if link already has valid title, skip it + if link.title and not link.title.lower().startswith('http'): + return False + + if is_static_file(link.url): + return False + + return SAVE_TITLE + +def extract_title_with_regex(html): + match = re.search(HTML_TITLE_REGEX, html) + output = htmldecode(match.group(1).strip()) if match else None + return output + +@enforce_types +def save_title(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """try to guess the page's title from its content""" + + from core.models import Snapshot + + output: ArchiveOutput = None + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + html = download_url(link.url, timeout=timeout) + try: + # try using relatively strict html parser first + parser = TitleParser() + parser.feed(html) + output = parser.title + if output is None: + raise + except Exception: + # fallback to regex that can handle broken/malformed html + output = extract_title_with_regex(html) + + # if title is better than the one in the db, update db with new title + if isinstance(output, str) and output: + if not link.title or len(output) >= len(link.title): + Snapshot.objects.filter(url=link.url, + timestamp=link.timestamp)\ + .update(title=output) + else: + raise ArchiveError('Unable to detect page title') + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/archivebox/extractors/wget.py b/archivebox-0.5.3/archivebox/extractors/wget.py new file mode 100644 index 0000000..331f636 --- /dev/null +++ b/archivebox-0.5.3/archivebox/extractors/wget.py @@ -0,0 +1,184 @@ +__package__ = 'archivebox.extractors' + +import re +from pathlib import Path + +from typing import Optional +from datetime import datetime + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + without_scheme, + without_fragment, + without_query, + path, + domain, + urldecode, +) +from ..config import ( + WGET_ARGS, + TIMEOUT, + SAVE_WGET, + SAVE_WARC, + WGET_BINARY, + WGET_VERSION, + RESTRICT_FILE_NAMES, + CHECK_SSL_VALIDITY, + SAVE_WGET_REQUISITES, + WGET_AUTO_COMPRESSION, + WGET_USER_AGENT, + COOKIES_FILE, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_wget(link: Link, out_dir: Optional[Path]=None) -> bool: + output_path = wget_output_path(link) + out_dir = out_dir or Path(link.link_dir) + if output_path and (out_dir / output_path).exists(): + return False + + return SAVE_WGET + + +@enforce_types +def save_wget(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using wget""" + + out_dir = out_dir or link.link_dir + if SAVE_WARC: + warc_dir = out_dir / "warc" + warc_dir.mkdir(exist_ok=True) + warc_path = warc_dir / str(int(datetime.now().timestamp())) + + # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html + output: ArchiveOutput = None + cmd = [ + WGET_BINARY, + # '--server-response', # print headers for better error parsing + *WGET_ARGS, + '--timeout={}'.format(timeout), + *(['--restrict-file-names={}'.format(RESTRICT_FILE_NAMES)] if RESTRICT_FILE_NAMES else []), + *(['--warc-file={}'.format(str(warc_path))] if SAVE_WARC else []), + *(['--page-requisites'] if SAVE_WGET_REQUISITES else []), + *(['--user-agent={}'.format(WGET_USER_AGENT)] if WGET_USER_AGENT else []), + *(['--load-cookies', COOKIES_FILE] if COOKIES_FILE else []), + *(['--compression=auto'] if WGET_AUTO_COMPRESSION else []), + *([] if SAVE_WARC else ['--timestamping']), + *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate', '--no-hsts']), + link.url, + ] + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + output = wget_output_path(link) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + files_downloaded = ( + int(output_tail[-1].strip().split(' ', 2)[1] or 0) + if 'Downloaded:' in output_tail[-1] + else 0 + ) + hints = ( + 'Got wget response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0 and files_downloaded < 1) or output is None: + if b'403: Forbidden' in result.stderr: + raise ArchiveError('403 Forbidden (try changing WGET_USER_AGENT)', hints) + if b'404: Not Found' in result.stderr: + raise ArchiveError('404 Not Found', hints) + if b'ERROR 500: Internal Server Error' in result.stderr: + raise ArchiveError('500 Internal Server Error', hints) + raise ArchiveError('Wget failed or got an error from the server', hints) + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=WGET_VERSION, + output=output, + status=status, + **timer.stats, + ) + + +@enforce_types +def wget_output_path(link: Link) -> Optional[str]: + """calculate the path to the wgetted .html file, since wget may + adjust some paths to be different than the base_url path. + + See docs on wget --adjust-extension (-E) + """ + if is_static_file(link.url): + return without_scheme(without_fragment(link.url)) + + # Wget downloads can save in a number of different ways depending on the url: + # https://example.com + # > example.com/index.html + # https://example.com?v=zzVa_tX1OiI + # > example.com/index.html?v=zzVa_tX1OiI.html + # https://www.example.com/?v=zzVa_tX1OiI + # > example.com/index.html?v=zzVa_tX1OiI.html + + # https://example.com/abc + # > example.com/abc.html + # https://example.com/abc/ + # > example.com/abc/index.html + # https://example.com/abc?v=zzVa_tX1OiI.html + # > example.com/abc?v=zzVa_tX1OiI.html + # https://example.com/abc/?v=zzVa_tX1OiI.html + # > example.com/abc/index.html?v=zzVa_tX1OiI.html + + # https://example.com/abc/test.html + # > example.com/abc/test.html + # https://example.com/abc/test?v=zzVa_tX1OiI + # > example.com/abc/test?v=zzVa_tX1OiI.html + # https://example.com/abc/test/?v=zzVa_tX1OiI + # > example.com/abc/test/index.html?v=zzVa_tX1OiI.html + + # There's also lots of complexity around how the urlencoding and renaming + # is done for pages with query and hash fragments or extensions like shtml / htm / php / etc + + # Since the wget algorithm for -E (appending .html) is incredibly complex + # and there's no way to get the computed output path from wget + # in order to avoid having to reverse-engineer how they calculate it, + # we just look in the output folder read the filename wget used from the filesystem + full_path = without_fragment(without_query(path(link.url))).strip('/') + search_dir = Path(link.link_dir) / domain(link.url).replace(":", "+") / urldecode(full_path) + for _ in range(4): + if search_dir.exists(): + if search_dir.is_dir(): + html_files = [ + f for f in search_dir.iterdir() + if re.search(".+\\.[Ss]?[Hh][Tt][Mm][Ll]?$", str(f), re.I | re.M) + ] + if html_files: + return str(html_files[0].relative_to(link.link_dir)) + + # Move up one directory level + search_dir = search_dir.parent + + if str(search_dir) == link.link_dir: + break + + return None diff --git a/archivebox-0.5.3/archivebox/index/__init__.py b/archivebox-0.5.3/archivebox/index/__init__.py new file mode 100644 index 0000000..8eab1d3 --- /dev/null +++ b/archivebox-0.5.3/archivebox/index/__init__.py @@ -0,0 +1,617 @@ +__package__ = 'archivebox.index' + +import os +import shutil +import json as pyjson +from pathlib import Path + +from itertools import chain +from typing import List, Tuple, Dict, Optional, Iterable +from collections import OrderedDict +from contextlib import contextmanager +from urllib.parse import urlparse +from django.db.models import QuerySet, Q + +from ..util import ( + scheme, + enforce_types, + ExtendedEncoder, +) +from ..config import ( + ARCHIVE_DIR_NAME, + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + OUTPUT_DIR, + TIMEOUT, + URL_BLACKLIST_PTN, + stderr, + OUTPUT_PERMISSIONS +) +from ..logging_util import ( + TimedProgress, + log_indexing_process_started, + log_indexing_process_finished, + log_indexing_started, + log_indexing_finished, + log_parsing_finished, + log_deduping_finished, +) + +from .schema import Link, ArchiveResult +from .html import ( + write_html_link_details, +) +from .json import ( + parse_json_link_details, + write_json_link_details, +) +from .sql import ( + write_sql_main_index, + write_sql_link_details, +) + +from ..search import search_backend_enabled, query_search_index + +### Link filtering and checking + +@enforce_types +def merge_links(a: Link, b: Link) -> Link: + """deterministially merge two links, favoring longer field values over shorter, + and "cleaner" values over worse ones. + """ + assert a.base_url == b.base_url, f'Cannot merge two links with different URLs ({a.base_url} != {b.base_url})' + + # longest url wins (because a fuzzy url will always be shorter) + url = a.url if len(a.url) > len(b.url) else b.url + + # best title based on length and quality + possible_titles = [ + title + for title in (a.title, b.title) + if title and title.strip() and '://' not in title + ] + title = None + if len(possible_titles) == 2: + title = max(possible_titles, key=lambda t: len(t)) + elif len(possible_titles) == 1: + title = possible_titles[0] + + # earliest valid timestamp + timestamp = ( + a.timestamp + if float(a.timestamp or 0) < float(b.timestamp or 0) else + b.timestamp + ) + + # all unique, truthy tags + tags_set = ( + set(tag.strip() for tag in (a.tags or '').split(',')) + | set(tag.strip() for tag in (b.tags or '').split(',')) + ) + tags = ','.join(tags_set) or None + + # all unique source entries + sources = list(set(a.sources + b.sources)) + + # all unique history entries for the combined archive methods + all_methods = set(list(a.history.keys()) + list(a.history.keys())) + history = { + method: (a.history.get(method) or []) + (b.history.get(method) or []) + for method in all_methods + } + for method in all_methods: + deduped_jsons = { + pyjson.dumps(result, sort_keys=True, cls=ExtendedEncoder) + for result in history[method] + } + history[method] = list(reversed(sorted( + (ArchiveResult.from_json(pyjson.loads(result)) for result in deduped_jsons), + key=lambda result: result.start_ts, + ))) + + return Link( + url=url, + timestamp=timestamp, + title=title, + tags=tags, + sources=sources, + history=history, + ) + + +@enforce_types +def validate_links(links: Iterable[Link]) -> List[Link]: + timer = TimedProgress(TIMEOUT * 4) + try: + links = archivable_links(links) # remove chrome://, about:, mailto: etc. + links = sorted_links(links) # deterministically sort the links based on timestamp, url + links = fix_duplicate_links(links) # merge/dedupe duplicate timestamps & urls + finally: + timer.end() + + return list(links) + +@enforce_types +def archivable_links(links: Iterable[Link]) -> Iterable[Link]: + """remove chrome://, about:// or other schemed links that cant be archived""" + for link in links: + try: + urlparse(link.url) + except ValueError: + continue + if scheme(link.url) not in ('http', 'https', 'ftp'): + continue + if URL_BLACKLIST_PTN and URL_BLACKLIST_PTN.search(link.url): + continue + + yield link + + +@enforce_types +def fix_duplicate_links(sorted_links: Iterable[Link]) -> Iterable[Link]: + """ + ensures that all non-duplicate links have monotonically increasing timestamps + """ + # from core.models import Snapshot + + unique_urls: OrderedDict[str, Link] = OrderedDict() + + for link in sorted_links: + if link.url in unique_urls: + # merge with any other links that share the same url + link = merge_links(unique_urls[link.url], link) + unique_urls[link.url] = link + + return unique_urls.values() + + +@enforce_types +def sorted_links(links: Iterable[Link]) -> Iterable[Link]: + sort_func = lambda link: (link.timestamp.split('.', 1)[0], link.url) + return sorted(links, key=sort_func, reverse=True) + + +@enforce_types +def links_after_timestamp(links: Iterable[Link], resume: Optional[float]=None) -> Iterable[Link]: + if not resume: + yield from links + return + + for link in links: + try: + if float(link.timestamp) <= resume: + yield link + except (ValueError, TypeError): + print('Resume value and all timestamp values must be valid numbers.') + + +@enforce_types +def lowest_uniq_timestamp(used_timestamps: OrderedDict, timestamp: str) -> str: + """resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2""" + + timestamp = timestamp.split('.')[0] + nonce = 0 + + # first try 152323423 before 152323423.0 + if timestamp not in used_timestamps: + return timestamp + + new_timestamp = '{}.{}'.format(timestamp, nonce) + while new_timestamp in used_timestamps: + nonce += 1 + new_timestamp = '{}.{}'.format(timestamp, nonce) + + return new_timestamp + + + +### Main Links Index + +@contextmanager +@enforce_types +def timed_index_update(out_path: Path): + log_indexing_started(out_path) + timer = TimedProgress(TIMEOUT * 2, prefix=' ') + try: + yield + finally: + timer.end() + + assert out_path.exists(), f'Failed to write index file: {out_path}' + log_indexing_finished(out_path) + + +@enforce_types +def write_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None: + """Writes links to sqlite3 file for a given list of links""" + + log_indexing_process_started(len(links)) + + try: + with timed_index_update(out_dir / SQL_INDEX_FILENAME): + write_sql_main_index(links, out_dir=out_dir) + os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes + + except (KeyboardInterrupt, SystemExit): + stderr('[!] Warning: Still writing index to disk...', color='lightyellow') + stderr(' Run archivebox init to fix any inconsistencies from an ungraceful exit.') + with timed_index_update(out_dir / SQL_INDEX_FILENAME): + write_sql_main_index(links, out_dir=out_dir) + os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes + raise SystemExit(0) + + log_indexing_process_finished() + +@enforce_types +def load_main_index(out_dir: Path=OUTPUT_DIR, warn: bool=True) -> List[Link]: + """parse and load existing index with any new links from import_path merged in""" + from core.models import Snapshot + try: + return Snapshot.objects.all() + + except (KeyboardInterrupt, SystemExit): + raise SystemExit(0) + +@enforce_types +def load_main_index_meta(out_dir: Path=OUTPUT_DIR) -> Optional[dict]: + index_path = out_dir / JSON_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + meta_dict = pyjson.load(f) + meta_dict.pop('links') + return meta_dict + + return None + + +@enforce_types +def parse_links_from_source(source_path: str, root_url: Optional[str]=None) -> Tuple[List[Link], List[Link]]: + + from ..parsers import parse_links + + new_links: List[Link] = [] + + # parse and validate the import file + raw_links, parser_name = parse_links(source_path, root_url=root_url) + new_links = validate_links(raw_links) + + if parser_name: + num_parsed = len(raw_links) + log_parsing_finished(num_parsed, parser_name) + + return new_links + +@enforce_types +def fix_duplicate_links_in_index(snapshots: QuerySet, links: Iterable[Link]) -> Iterable[Link]: + """ + Given a list of in-memory Links, dedupe and merge them with any conflicting Snapshots in the DB. + """ + unique_urls: OrderedDict[str, Link] = OrderedDict() + + for link in links: + index_link = snapshots.filter(url=link.url) + if index_link: + link = merge_links(index_link[0].as_link(), link) + + unique_urls[link.url] = link + + return unique_urls.values() + +@enforce_types +def dedupe_links(snapshots: QuerySet, + new_links: List[Link]) -> List[Link]: + """ + The validation of links happened at a different stage. This method will + focus on actual deduplication and timestamp fixing. + """ + + # merge existing links in out_dir and new links + dedup_links = fix_duplicate_links_in_index(snapshots, new_links) + + new_links = [ + link for link in new_links + if not snapshots.filter(url=link.url).exists() + ] + + dedup_links_dict = {link.url: link for link in dedup_links} + + # Replace links in new_links with the dedup version + for i in range(len(new_links)): + if new_links[i].url in dedup_links_dict.keys(): + new_links[i] = dedup_links_dict[new_links[i].url] + log_deduping_finished(len(new_links)) + + return new_links + +### Link Details Index + +@enforce_types +def write_link_details(link: Link, out_dir: Optional[str]=None, skip_sql_index: bool=False) -> None: + out_dir = out_dir or link.link_dir + + write_json_link_details(link, out_dir=out_dir) + write_html_link_details(link, out_dir=out_dir) + if not skip_sql_index: + write_sql_link_details(link) + + +@enforce_types +def load_link_details(link: Link, out_dir: Optional[str]=None) -> Link: + """check for an existing link archive in the given directory, + and load+merge it into the given link dict + """ + out_dir = out_dir or link.link_dir + + existing_link = parse_json_link_details(out_dir) + if existing_link: + return merge_links(existing_link, link) + + return link + + + +LINK_FILTERS = { + 'exact': lambda pattern: Q(url=pattern), + 'substring': lambda pattern: Q(url__icontains=pattern), + 'regex': lambda pattern: Q(url__iregex=pattern), + 'domain': lambda pattern: Q(url__istartswith=f"http://{pattern}") | Q(url__istartswith=f"https://{pattern}") | Q(url__istartswith=f"ftp://{pattern}"), + 'tag': lambda pattern: Q(tags__name=pattern), +} + +@enforce_types +def q_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet: + q_filter = Q() + for pattern in filter_patterns: + try: + q_filter = q_filter | LINK_FILTERS[filter_type](pattern) + except KeyError: + stderr() + stderr( + f'[X] Got invalid pattern for --filter-type={filter_type}:', + color='red', + ) + stderr(f' {pattern}') + raise SystemExit(2) + return snapshots.filter(q_filter) + +def search_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='search') -> QuerySet: + if not search_backend_enabled(): + stderr() + stderr( + '[X] The search backend is not enabled, set config.USE_SEARCHING_BACKEND = True', + color='red', + ) + raise SystemExit(2) + from core.models import Snapshot + + qsearch = Snapshot.objects.none() + for pattern in filter_patterns: + try: + qsearch |= query_search_index(pattern) + except: + raise SystemExit(2) + + return snapshots & qsearch + +@enforce_types +def snapshot_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet: + if filter_type != 'search': + return q_filter(snapshots, filter_patterns, filter_type) + else: + return search_filter(snapshots, filter_patterns, filter_type) + + +def get_indexed_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links without checking archive status or data directory validity""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in links + } + +def get_archived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links that are archived with a valid data directory""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_archived, links) + } + +def get_unarchived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links that are unarchived with no data directory or an empty data directory""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_unarchived, links) + } + +def get_present_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that actually exist in the archive/ folder""" + + all_folders = {} + + for entry in (out_dir / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + link = None + try: + link = parse_json_link_details(entry.path) + except Exception: + pass + + all_folders[entry.name] = link + + return all_folders + +def get_valid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs with a valid index matched to the main index and archived content""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_valid, links) + } + +def get_invalid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized""" + duplicate = get_duplicate_folders(snapshots, out_dir=OUTPUT_DIR) + orphaned = get_orphaned_folders(snapshots, out_dir=OUTPUT_DIR) + corrupted = get_corrupted_folders(snapshots, out_dir=OUTPUT_DIR) + unrecognized = get_unrecognized_folders(snapshots, out_dir=OUTPUT_DIR) + return {**duplicate, **orphaned, **corrupted, **unrecognized} + + +def get_duplicate_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that conflict with other directories that have the same link URL or timestamp""" + by_url = {} + by_timestamp = {} + duplicate_folders = {} + + data_folders = ( + str(entry) + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir() + if entry.is_dir() and not snapshots.filter(timestamp=entry.name).exists() + ) + + for path in chain(snapshots.iterator(), data_folders): + link = None + if type(path) is not str: + path = path.as_link().link_dir + + try: + link = parse_json_link_details(path) + except Exception: + pass + + if link: + # link folder has same timestamp as different link folder + by_timestamp[link.timestamp] = by_timestamp.get(link.timestamp, 0) + 1 + if by_timestamp[link.timestamp] > 1: + duplicate_folders[path] = link + + # link folder has same url as different link folder + by_url[link.url] = by_url.get(link.url, 0) + 1 + if by_url[link.url] > 1: + duplicate_folders[path] = link + return duplicate_folders + +def get_orphaned_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that contain a valid index but aren't listed in the main index""" + orphaned_folders = {} + + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + link = None + try: + link = parse_json_link_details(str(entry)) + except Exception: + pass + + if link and not snapshots.filter(timestamp=entry.name).exists(): + # folder is a valid link data dir with index details, but it's not in the main index + orphaned_folders[str(entry)] = link + + return orphaned_folders + +def get_corrupted_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that don't contain a valid index and aren't listed in the main index""" + corrupted = {} + for snapshot in snapshots.iterator(): + link = snapshot.as_link() + if is_corrupt(link): + corrupted[link.link_dir] = link + return corrupted + +def get_unrecognized_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that don't contain recognizable archive data and aren't listed in the main index""" + unrecognized_folders: Dict[str, Optional[Link]] = {} + + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + index_exists = (entry / "index.json").exists() + link = None + try: + link = parse_json_link_details(str(entry)) + except KeyError: + # Try to fix index + if index_exists: + try: + # Last attempt to repair the detail index + link_guessed = parse_json_link_details(str(entry), guess=True) + write_json_link_details(link_guessed, out_dir=str(entry)) + link = parse_json_link_details(str(entry)) + except Exception: + pass + + if index_exists and link is None: + # index exists but it's corrupted or unparseable + unrecognized_folders[str(entry)] = link + + elif not index_exists: + # link details index doesn't exist and the folder isn't in the main index + timestamp = entry.name + if not snapshots.filter(timestamp=timestamp).exists(): + unrecognized_folders[str(entry)] = link + + return unrecognized_folders + + +def is_valid(link: Link) -> bool: + dir_exists = Path(link.link_dir).exists() + index_exists = (Path(link.link_dir) / "index.json").exists() + if not dir_exists: + # unarchived links are not included in the valid list + return False + if dir_exists and not index_exists: + return False + if dir_exists and index_exists: + try: + parsed_link = parse_json_link_details(link.link_dir, guess=True) + return link.url == parsed_link.url + except Exception: + pass + return False + +def is_corrupt(link: Link) -> bool: + if not Path(link.link_dir).exists(): + # unarchived links are not considered corrupt + return False + + if is_valid(link): + return False + + return True + +def is_archived(link: Link) -> bool: + return is_valid(link) and link.is_archived + +def is_unarchived(link: Link) -> bool: + if not Path(link.link_dir).exists(): + return True + return not link.is_archived + + +def fix_invalid_folder_locations(out_dir: Path=OUTPUT_DIR) -> Tuple[List[str], List[str]]: + fixed = [] + cant_fix = [] + for entry in os.scandir(out_dir / ARCHIVE_DIR_NAME): + if entry.is_dir(follow_symlinks=True): + if (Path(entry.path) / 'index.json').exists(): + try: + link = parse_json_link_details(entry.path) + except KeyError: + link = None + if not link: + continue + + if not entry.path.endswith(f'/{link.timestamp}'): + dest = out_dir / ARCHIVE_DIR_NAME / link.timestamp + if dest.exists(): + cant_fix.append(entry.path) + else: + shutil.move(entry.path, dest) + fixed.append(dest) + timestamp = entry.path.rsplit('/', 1)[-1] + assert link.link_dir == entry.path + assert link.timestamp == timestamp + write_json_link_details(link, out_dir=entry.path) + + return fixed, cant_fix diff --git a/archivebox-0.5.3/archivebox/index/csv.py b/archivebox-0.5.3/archivebox/index/csv.py new file mode 100644 index 0000000..804e646 --- /dev/null +++ b/archivebox-0.5.3/archivebox/index/csv.py @@ -0,0 +1,37 @@ +__package__ = 'archivebox.index' + +from typing import List, Optional, Any + +from ..util import enforce_types +from .schema import Link + + +@enforce_types +def links_to_csv(links: List[Link], + cols: Optional[List[str]]=None, + header: bool=True, + separator: str=',', + ljust: int=0) -> str: + + cols = cols or ['timestamp', 'is_archived', 'url'] + + header_str = '' + if header: + header_str = separator.join(col.ljust(ljust) for col in cols) + + row_strs = ( + link.to_csv(cols=cols, ljust=ljust, separator=separator) + for link in links + ) + + return '\n'.join((header_str, *row_strs)) + + +@enforce_types +def to_csv(obj: Any, cols: List[str], separator: str=',', ljust: int=0) -> str: + from .json import to_json + + return separator.join( + to_json(getattr(obj, col), indent=None).ljust(ljust) + for col in cols + ) diff --git a/archivebox-0.5.3/archivebox/index/html.py b/archivebox-0.5.3/archivebox/index/html.py new file mode 100644 index 0000000..a62e2c7 --- /dev/null +++ b/archivebox-0.5.3/archivebox/index/html.py @@ -0,0 +1,164 @@ +__package__ = 'archivebox.index' + +from datetime import datetime +from typing import List, Optional, Iterator, Mapping +from pathlib import Path + +from django.utils.html import format_html +from collections import defaultdict + +from .schema import Link +from ..system import atomic_write +from ..logging_util import printable_filesize +from ..util import ( + enforce_types, + ts_to_date, + urlencode, + htmlencode, + urldecode, +) +from ..config import ( + OUTPUT_DIR, + VERSION, + GIT_SHA, + FOOTER_INFO, + HTML_INDEX_FILENAME, +) + +MAIN_INDEX_TEMPLATE = 'main_index.html' +MINIMAL_INDEX_TEMPLATE = 'main_index_minimal.html' +LINK_DETAILS_TEMPLATE = 'link_details.html' +TITLE_LOADING_MSG = 'Not yet archived...' + + +### Main Links Index + +@enforce_types +def parse_html_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[str]: + """parse an archive index html file and return the list of urls""" + + index_path = Path(out_dir) / HTML_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + for line in f: + if 'class="link-url"' in line: + yield line.split('"')[1] + return () + +@enforce_types +def generate_index_from_links(links: List[Link], with_headers: bool): + if with_headers: + output = main_index_template(links) + else: + output = main_index_template(links, template=MINIMAL_INDEX_TEMPLATE) + return output + +@enforce_types +def main_index_template(links: List[Link], template: str=MAIN_INDEX_TEMPLATE) -> str: + """render the template for the entire main index""" + + return render_django_template(template, { + 'version': VERSION, + 'git_sha': GIT_SHA, + 'num_links': str(len(links)), + 'date_updated': datetime.now().strftime('%Y-%m-%d'), + 'time_updated': datetime.now().strftime('%Y-%m-%d %H:%M'), + 'links': [link._asdict(extended=True) for link in links], + 'FOOTER_INFO': FOOTER_INFO, + }) + + +### Link Details Index + +@enforce_types +def write_html_link_details(link: Link, out_dir: Optional[str]=None) -> None: + out_dir = out_dir or link.link_dir + + rendered_html = link_details_template(link) + atomic_write(str(Path(out_dir) / HTML_INDEX_FILENAME), rendered_html) + + +@enforce_types +def link_details_template(link: Link) -> str: + + from ..extractors.wget import wget_output_path + + link_info = link._asdict(extended=True) + + return render_django_template(LINK_DETAILS_TEMPLATE, { + **link_info, + **link_info['canonical'], + 'title': htmlencode( + link.title + or (link.base_url if link.is_archived else TITLE_LOADING_MSG) + ), + 'url_str': htmlencode(urldecode(link.base_url)), + 'archive_url': urlencode( + wget_output_path(link) + or (link.domain if link.is_archived else '') + ) or 'about:blank', + 'extension': link.extension or 'html', + 'tags': link.tags or 'untagged', + 'size': printable_filesize(link.archive_size) if link.archive_size else 'pending', + 'status': 'archived' if link.is_archived else 'not yet archived', + 'status_color': 'success' if link.is_archived else 'danger', + 'oldest_archive_date': ts_to_date(link.oldest_archive_date), + }) + +@enforce_types +def render_django_template(template: str, context: Mapping[str, str]) -> str: + """render a given html template string with the given template content""" + from django.template.loader import render_to_string + + return render_to_string(template, context) + + +def snapshot_icons(snapshot) -> str: + from core.models import EXTRACTORS + + archive_results = snapshot.archiveresult_set.filter(status="succeeded") + link = snapshot.as_link() + path = link.archive_path + canon = link.canonical_outputs() + output = "" + output_template = '<a href="/{}/{}" class="exists-{}" title="{}">{} </a>' + icons = { + "singlefile": "❶", + "wget": "🆆", + "dom": "🅷", + "pdf": "📄", + "screenshot": "💻", + "media": "📼", + "git": "🅶", + "archive_org": "🏛", + "readability": "🆁", + "mercury": "🅼", + "warc": "📦" + } + exclude = ["favicon", "title", "headers", "archive_org"] + # Missing specific entry for WARC + + extractor_items = defaultdict(lambda: None) + for extractor, _ in EXTRACTORS: + for result in archive_results: + if result.extractor == extractor: + extractor_items[extractor] = result + + for extractor, _ in EXTRACTORS: + if extractor not in exclude: + exists = extractor_items[extractor] is not None + output += output_template.format(path, canon[f"{extractor}_path"], str(exists), + extractor, icons.get(extractor, "?")) + if extractor == "wget": + # warc isn't technically it's own extractor, so we have to add it after wget + exists = list((Path(path) / canon["warc_path"]).glob("*.warc.gz")) + output += output_template.format(exists[0] if exists else '#', canon["warc_path"], str(bool(exists)), "warc", icons.get("warc", "?")) + + if extractor == "archive_org": + # The check for archive_org is different, so it has to be handled separately + target_path = Path(path) / "archive.org.txt" + exists = target_path.exists() + output += '<a href="{}" class="exists-{}" title="{}">{}</a> '.format(canon["archive_org_path"], str(exists), + "archive_org", icons.get("archive_org", "?")) + + return format_html(f'<span class="files-icons" style="font-size: 1.1em; opacity: 0.8">{output}<span>') diff --git a/archivebox-0.5.3/archivebox/index/json.py b/archivebox-0.5.3/archivebox/index/json.py new file mode 100644 index 0000000..f24b969 --- /dev/null +++ b/archivebox-0.5.3/archivebox/index/json.py @@ -0,0 +1,154 @@ +__package__ = 'archivebox.index' + +import os +import sys +import json as pyjson +from pathlib import Path + +from datetime import datetime +from typing import List, Optional, Iterator, Any, Union + +from .schema import Link +from ..system import atomic_write +from ..util import enforce_types +from ..config import ( + VERSION, + OUTPUT_DIR, + FOOTER_INFO, + GIT_SHA, + DEPENDENCIES, + JSON_INDEX_FILENAME, + ARCHIVE_DIR_NAME, + ANSI +) + + +MAIN_INDEX_HEADER = { + 'info': 'This is an index of site data archived by ArchiveBox: The self-hosted web archive.', + 'schema': 'archivebox.index.json', + 'copyright_info': FOOTER_INFO, + 'meta': { + 'project': 'ArchiveBox', + 'version': VERSION, + 'git_sha': GIT_SHA, + 'website': 'https://ArchiveBox.io', + 'docs': 'https://github.com/ArchiveBox/ArchiveBox/wiki', + 'source': 'https://github.com/ArchiveBox/ArchiveBox', + 'issues': 'https://github.com/ArchiveBox/ArchiveBox/issues', + 'dependencies': DEPENDENCIES, + }, +} + +@enforce_types +def generate_json_index_from_links(links: List[Link], with_headers: bool): + if with_headers: + output = { + **MAIN_INDEX_HEADER, + 'num_links': len(links), + 'updated': datetime.now(), + 'last_run_cmd': sys.argv, + 'links': links, + } + else: + output = links + return to_json(output, indent=4, sort_keys=True) + + +@enforce_types +def parse_json_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]: + """parse an archive index json file and return the list of links""" + + index_path = Path(out_dir) / JSON_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + links = pyjson.load(f)['links'] + for link_json in links: + try: + yield Link.from_json(link_json) + except KeyError: + try: + detail_index_path = Path(OUTPUT_DIR) / ARCHIVE_DIR_NAME / link_json['timestamp'] + yield parse_json_link_details(str(detail_index_path)) + except KeyError: + # as a last effort, try to guess the missing values out of existing ones + try: + yield Link.from_json(link_json, guess=True) + except KeyError: + print(" {lightyellow}! Failed to load the index.json from {}".format(detail_index_path, **ANSI)) + continue + return () + +### Link Details Index + +@enforce_types +def write_json_link_details(link: Link, out_dir: Optional[str]=None) -> None: + """write a json file with some info about the link""" + + out_dir = out_dir or link.link_dir + path = Path(out_dir) / JSON_INDEX_FILENAME + atomic_write(str(path), link._asdict(extended=True)) + + +@enforce_types +def parse_json_link_details(out_dir: Union[Path, str], guess: Optional[bool]=False) -> Optional[Link]: + """load the json link index from a given directory""" + existing_index = Path(out_dir) / JSON_INDEX_FILENAME + if existing_index.exists(): + with open(existing_index, 'r', encoding='utf-8') as f: + try: + link_json = pyjson.load(f) + return Link.from_json(link_json, guess) + except pyjson.JSONDecodeError: + pass + return None + + +@enforce_types +def parse_json_links_details(out_dir: Union[Path, str]) -> Iterator[Link]: + """read through all the archive data folders and return the parsed links""" + + for entry in os.scandir(Path(out_dir) / ARCHIVE_DIR_NAME): + if entry.is_dir(follow_symlinks=True): + if (Path(entry.path) / 'index.json').exists(): + try: + link = parse_json_link_details(entry.path) + except KeyError: + link = None + if link: + yield link + + + +### Helpers + +class ExtendedEncoder(pyjson.JSONEncoder): + """ + Extended json serializer that supports serializing several model + fields and objects + """ + + def default(self, obj): + cls_name = obj.__class__.__name__ + + if hasattr(obj, '_asdict'): + return obj._asdict() + + elif isinstance(obj, bytes): + return obj.decode() + + elif isinstance(obj, datetime): + return obj.isoformat() + + elif isinstance(obj, Exception): + return '{}: {}'.format(obj.__class__.__name__, obj) + + elif cls_name in ('dict_items', 'dict_keys', 'dict_values'): + return tuple(obj) + + return pyjson.JSONEncoder.default(self, obj) + + +@enforce_types +def to_json(obj: Any, indent: Optional[int]=4, sort_keys: bool=True, cls=ExtendedEncoder) -> str: + return pyjson.dumps(obj, indent=indent, sort_keys=sort_keys, cls=ExtendedEncoder) + diff --git a/archivebox-0.5.3/archivebox/index/schema.py b/archivebox-0.5.3/archivebox/index/schema.py new file mode 100644 index 0000000..bc3a25d --- /dev/null +++ b/archivebox-0.5.3/archivebox/index/schema.py @@ -0,0 +1,448 @@ +""" + +WARNING: THIS FILE IS ALL LEGACY CODE TO BE REMOVED. + +DO NOT ADD ANY NEW FEATURES TO THIS FILE, NEW CODE GOES HERE: core/models.py + +""" + +__package__ = 'archivebox.index' + +from pathlib import Path + +from datetime import datetime, timedelta + +from typing import List, Dict, Any, Optional, Union + +from dataclasses import dataclass, asdict, field, fields + + +from ..system import get_dir_size + +from ..config import OUTPUT_DIR, ARCHIVE_DIR_NAME + +class ArchiveError(Exception): + def __init__(self, message, hints=None): + super().__init__(message) + self.hints = hints + +LinkDict = Dict[str, Any] + +ArchiveOutput = Union[str, Exception, None] + +@dataclass(frozen=True) +class ArchiveResult: + cmd: List[str] + pwd: Optional[str] + cmd_version: Optional[str] + output: ArchiveOutput + status: str + start_ts: datetime + end_ts: datetime + index_texts: Union[List[str], None] = None + schema: str = 'ArchiveResult' + + def __post_init__(self): + self.typecheck() + + def _asdict(self): + return asdict(self) + + def typecheck(self) -> None: + assert self.schema == self.__class__.__name__ + assert isinstance(self.status, str) and self.status + assert isinstance(self.start_ts, datetime) + assert isinstance(self.end_ts, datetime) + assert isinstance(self.cmd, list) + assert all(isinstance(arg, str) and arg for arg in self.cmd) + assert self.pwd is None or isinstance(self.pwd, str) and self.pwd + assert self.cmd_version is None or isinstance(self.cmd_version, str) and self.cmd_version + assert self.output is None or isinstance(self.output, (str, Exception)) + if isinstance(self.output, str): + assert self.output + + @classmethod + def guess_ts(_cls, dict_info): + from ..util import parse_date + parsed_timestamp = parse_date(dict_info["timestamp"]) + start_ts = parsed_timestamp + end_ts = parsed_timestamp + timedelta(seconds=int(dict_info["duration"])) + return start_ts, end_ts + + @classmethod + def from_json(cls, json_info, guess=False): + from ..util import parse_date + + info = { + key: val + for key, val in json_info.items() + if key in cls.field_names() + } + if guess: + keys = info.keys() + if "start_ts" not in keys: + info["start_ts"], info["end_ts"] = cls.guess_ts(json_info) + else: + info['start_ts'] = parse_date(info['start_ts']) + info['end_ts'] = parse_date(info['end_ts']) + if "pwd" not in keys: + info["pwd"] = str(Path(OUTPUT_DIR) / ARCHIVE_DIR_NAME / json_info["timestamp"]) + if "cmd_version" not in keys: + info["cmd_version"] = "Undefined" + if "cmd" not in keys: + info["cmd"] = [] + else: + info['start_ts'] = parse_date(info['start_ts']) + info['end_ts'] = parse_date(info['end_ts']) + info['cmd_version'] = info.get('cmd_version') + if type(info["cmd"]) is str: + info["cmd"] = [info["cmd"]] + return cls(**info) + + def to_dict(self, *keys) -> dict: + if keys: + return {k: v for k, v in asdict(self).items() if k in keys} + return asdict(self) + + def to_json(self, indent=4, sort_keys=True) -> str: + from .json import to_json + + return to_json(self, indent=indent, sort_keys=sort_keys) + + def to_csv(self, cols: Optional[List[str]]=None, separator: str=',', ljust: int=0) -> str: + from .csv import to_csv + + return to_csv(self, csv_col=cols or self.field_names(), separator=separator, ljust=ljust) + + @classmethod + def field_names(cls): + return [f.name for f in fields(cls)] + + @property + def duration(self) -> int: + return (self.end_ts - self.start_ts).seconds + +@dataclass(frozen=True) +class Link: + timestamp: str + url: str + title: Optional[str] + tags: Optional[str] + sources: List[str] + history: Dict[str, List[ArchiveResult]] = field(default_factory=lambda: {}) + updated: Optional[datetime] = None + schema: str = 'Link' + + + def __str__(self) -> str: + return f'[{self.timestamp}] {self.url} "{self.title}"' + + def __post_init__(self): + self.typecheck() + + def overwrite(self, **kwargs): + """pure functional version of dict.update that returns a new instance""" + return Link(**{**self._asdict(), **kwargs}) + + def __eq__(self, other): + if not isinstance(other, Link): + return NotImplemented + return self.url == other.url + + def __gt__(self, other): + if not isinstance(other, Link): + return NotImplemented + if not self.timestamp or not other.timestamp: + return + return float(self.timestamp) > float(other.timestamp) + + def typecheck(self) -> None: + from ..config import stderr, ANSI + try: + assert self.schema == self.__class__.__name__ + assert isinstance(self.timestamp, str) and self.timestamp + assert self.timestamp.replace('.', '').isdigit() + assert isinstance(self.url, str) and '://' in self.url + assert self.updated is None or isinstance(self.updated, datetime) + assert self.title is None or (isinstance(self.title, str) and self.title) + assert self.tags is None or isinstance(self.tags, str) + assert isinstance(self.sources, list) + assert all(isinstance(source, str) and source for source in self.sources) + assert isinstance(self.history, dict) + for method, results in self.history.items(): + assert isinstance(method, str) and method + assert isinstance(results, list) + assert all(isinstance(result, ArchiveResult) for result in results) + except Exception: + stderr('{red}[X] Error while loading link! [{}] {} "{}"{reset}'.format(self.timestamp, self.url, self.title, **ANSI)) + raise + + def _asdict(self, extended=False): + info = { + 'schema': 'Link', + 'url': self.url, + 'title': self.title or None, + 'timestamp': self.timestamp, + 'updated': self.updated or None, + 'tags': self.tags or None, + 'sources': self.sources or [], + 'history': self.history or {}, + } + if extended: + info.update({ + 'link_dir': self.link_dir, + 'archive_path': self.archive_path, + + 'hash': self.url_hash, + 'base_url': self.base_url, + 'scheme': self.scheme, + 'domain': self.domain, + 'path': self.path, + 'basename': self.basename, + 'extension': self.extension, + 'is_static': self.is_static, + + 'bookmarked_date': self.bookmarked_date, + 'updated_date': self.updated_date, + 'oldest_archive_date': self.oldest_archive_date, + 'newest_archive_date': self.newest_archive_date, + + 'is_archived': self.is_archived, + 'num_outputs': self.num_outputs, + 'num_failures': self.num_failures, + + 'latest': self.latest_outputs(), + 'canonical': self.canonical_outputs(), + }) + return info + + def as_snapshot(self): + from core.models import Snapshot + return Snapshot.objects.get(url=self.url) + + @classmethod + def from_json(cls, json_info, guess=False): + from ..util import parse_date + + info = { + key: val + for key, val in json_info.items() + if key in cls.field_names() + } + info['updated'] = parse_date(info.get('updated')) + info['sources'] = info.get('sources') or [] + + json_history = info.get('history') or {} + cast_history = {} + + for method, method_history in json_history.items(): + cast_history[method] = [] + for json_result in method_history: + assert isinstance(json_result, dict), 'Items in Link["history"][method] must be dicts' + cast_result = ArchiveResult.from_json(json_result, guess) + cast_history[method].append(cast_result) + + info['history'] = cast_history + return cls(**info) + + def to_json(self, indent=4, sort_keys=True) -> str: + from .json import to_json + + return to_json(self, indent=indent, sort_keys=sort_keys) + + def to_csv(self, cols: Optional[List[str]]=None, separator: str=',', ljust: int=0) -> str: + from .csv import to_csv + + return to_csv(self, cols=cols or self.field_names(), separator=separator, ljust=ljust) + + @classmethod + def field_names(cls): + return [f.name for f in fields(cls)] + + @property + def link_dir(self) -> str: + from ..config import CONFIG + return str(Path(CONFIG['ARCHIVE_DIR']) / self.timestamp) + + @property + def archive_path(self) -> str: + from ..config import ARCHIVE_DIR_NAME + return '{}/{}'.format(ARCHIVE_DIR_NAME, self.timestamp) + + @property + def archive_size(self) -> float: + try: + return get_dir_size(self.archive_path)[0] + except Exception: + return 0 + + ### URL Helpers + @property + def url_hash(self): + from ..util import hashurl + + return hashurl(self.url) + + @property + def scheme(self) -> str: + from ..util import scheme + return scheme(self.url) + + @property + def extension(self) -> str: + from ..util import extension + return extension(self.url) + + @property + def domain(self) -> str: + from ..util import domain + return domain(self.url) + + @property + def path(self) -> str: + from ..util import path + return path(self.url) + + @property + def basename(self) -> str: + from ..util import basename + return basename(self.url) + + @property + def base_url(self) -> str: + from ..util import base_url + return base_url(self.url) + + ### Pretty Printing Helpers + @property + def bookmarked_date(self) -> Optional[str]: + from ..util import ts_to_date + + max_ts = (datetime.now() + timedelta(days=30)).timestamp() + + if self.timestamp and self.timestamp.replace('.', '').isdigit(): + if 0 < float(self.timestamp) < max_ts: + return ts_to_date(datetime.fromtimestamp(float(self.timestamp))) + else: + return str(self.timestamp) + return None + + + @property + def updated_date(self) -> Optional[str]: + from ..util import ts_to_date + return ts_to_date(self.updated) if self.updated else None + + @property + def archive_dates(self) -> List[datetime]: + return [ + result.start_ts + for method in self.history.keys() + for result in self.history[method] + ] + + @property + def oldest_archive_date(self) -> Optional[datetime]: + return min(self.archive_dates, default=None) + + @property + def newest_archive_date(self) -> Optional[datetime]: + return max(self.archive_dates, default=None) + + ### Archive Status Helpers + @property + def num_outputs(self) -> int: + return self.as_snapshot().num_outputs + + @property + def num_failures(self) -> int: + return sum(1 + for method in self.history.keys() + for result in self.history[method] + if result.status == 'failed') + + @property + def is_static(self) -> bool: + from ..util import is_static_file + return is_static_file(self.url) + + @property + def is_archived(self) -> bool: + from ..config import ARCHIVE_DIR + from ..util import domain + + output_paths = ( + domain(self.url), + 'output.pdf', + 'screenshot.png', + 'output.html', + 'media', + 'singlefile.html' + ) + + return any( + (Path(ARCHIVE_DIR) / self.timestamp / path).exists() + for path in output_paths + ) + + def latest_outputs(self, status: str=None) -> Dict[str, ArchiveOutput]: + """get the latest output that each archive method produced for link""" + + ARCHIVE_METHODS = ( + 'title', 'favicon', 'wget', 'warc', 'singlefile', 'pdf', + 'screenshot', 'dom', 'git', 'media', 'archive_org', + ) + latest: Dict[str, ArchiveOutput] = {} + for archive_method in ARCHIVE_METHODS: + # get most recent succesful result in history for each archive method + history = self.history.get(archive_method) or [] + history = list(filter(lambda result: result.output, reversed(history))) + if status is not None: + history = list(filter(lambda result: result.status == status, history)) + + history = list(history) + if history: + latest[archive_method] = history[0].output + else: + latest[archive_method] = None + return latest + + + def canonical_outputs(self) -> Dict[str, Optional[str]]: + """predict the expected output paths that should be present after archiving""" + + from ..extractors.wget import wget_output_path + canonical = { + 'index_path': 'index.html', + 'favicon_path': 'favicon.ico', + 'google_favicon_path': 'https://www.google.com/s2/favicons?domain={}'.format(self.domain), + 'wget_path': wget_output_path(self), + 'warc_path': 'warc', + 'singlefile_path': 'singlefile.html', + 'readability_path': 'readability/content.html', + 'mercury_path': 'mercury/content.html', + 'pdf_path': 'output.pdf', + 'screenshot_path': 'screenshot.png', + 'dom_path': 'output.html', + 'archive_org_path': 'https://web.archive.org/web/{}'.format(self.base_url), + 'git_path': 'git', + 'media_path': 'media', + } + if self.is_static: + # static binary files like PDF and images are handled slightly differently. + # they're just downloaded once and aren't archived separately multiple times, + # so the wget, screenshot, & pdf urls should all point to the same file + + static_path = wget_output_path(self) + canonical.update({ + 'title': self.basename, + 'wget_path': static_path, + 'pdf_path': static_path, + 'screenshot_path': static_path, + 'dom_path': static_path, + 'singlefile_path': static_path, + 'readability_path': static_path, + 'mercury_path': static_path, + }) + return canonical + diff --git a/archivebox-0.5.3/archivebox/index/sql.py b/archivebox-0.5.3/archivebox/index/sql.py new file mode 100644 index 0000000..1e99f67 --- /dev/null +++ b/archivebox-0.5.3/archivebox/index/sql.py @@ -0,0 +1,106 @@ +__package__ = 'archivebox.index' + +from io import StringIO +from pathlib import Path +from typing import List, Tuple, Iterator +from django.db.models import QuerySet +from django.db import transaction + +from .schema import Link +from ..util import enforce_types +from ..config import OUTPUT_DIR + + +### Main Links Index + +@enforce_types +def parse_sql_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]: + from core.models import Snapshot + + return ( + Link.from_json(page.as_json(*Snapshot.keys)) + for page in Snapshot.objects.all() + ) + +@enforce_types +def remove_from_sql_main_index(snapshots: QuerySet, out_dir: Path=OUTPUT_DIR) -> None: + with transaction.atomic(): + snapshots.delete() + +@enforce_types +def write_link_to_sql_index(link: Link): + from core.models import Snapshot + info = {k: v for k, v in link._asdict().items() if k in Snapshot.keys} + tags = info.pop("tags") + if tags is None: + tags = [] + + try: + info["timestamp"] = Snapshot.objects.get(url=link.url).timestamp + except Snapshot.DoesNotExist: + while Snapshot.objects.filter(timestamp=info["timestamp"]).exists(): + info["timestamp"] = str(float(info["timestamp"]) + 1.0) + + snapshot, _ = Snapshot.objects.update_or_create(url=link.url, defaults=info) + snapshot.save_tags(tags) + return snapshot + + +@enforce_types +def write_sql_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None: + with transaction.atomic(): + for link in links: + write_link_to_sql_index(link) + + +@enforce_types +def write_sql_link_details(link: Link, out_dir: Path=OUTPUT_DIR) -> None: + from core.models import Snapshot + + with transaction.atomic(): + try: + snap = Snapshot.objects.get(url=link.url) + except Snapshot.DoesNotExist: + snap = write_link_to_sql_index(link) + snap.title = link.title + + tag_set = ( + set(tag.strip() for tag in (link.tags or '').split(',')) + ) + tag_list = list(tag_set) or [] + + snap.save() + snap.save_tags(tag_list) + + + +@enforce_types +def list_migrations(out_dir: Path=OUTPUT_DIR) -> List[Tuple[bool, str]]: + from django.core.management import call_command + out = StringIO() + call_command("showmigrations", list=True, stdout=out) + out.seek(0) + migrations = [] + for line in out.readlines(): + if line.strip() and ']' in line: + status_str, name_str = line.strip().split(']', 1) + is_applied = 'X' in status_str + migration_name = name_str.strip() + migrations.append((is_applied, migration_name)) + + return migrations + +@enforce_types +def apply_migrations(out_dir: Path=OUTPUT_DIR) -> List[str]: + from django.core.management import call_command + null, out = StringIO(), StringIO() + call_command("makemigrations", interactive=False, stdout=null) + call_command("migrate", interactive=False, stdout=out) + out.seek(0) + + return [line.strip() for line in out.readlines() if line.strip()] + +@enforce_types +def get_admins(out_dir: Path=OUTPUT_DIR) -> List[str]: + from django.contrib.auth.models import User + return User.objects.filter(is_superuser=True) diff --git a/archivebox-0.5.3/archivebox/logging_util.py b/archivebox-0.5.3/archivebox/logging_util.py new file mode 100644 index 0000000..f2b8673 --- /dev/null +++ b/archivebox-0.5.3/archivebox/logging_util.py @@ -0,0 +1,569 @@ +__package__ = 'archivebox' + +import re +import os +import sys +import time +import argparse +from math import log +from multiprocessing import Process +from pathlib import Path + +from datetime import datetime +from dataclasses import dataclass +from typing import Optional, List, Dict, Union, IO, TYPE_CHECKING + +if TYPE_CHECKING: + from .index.schema import Link, ArchiveResult + +from .util import enforce_types +from .config import ( + ConfigDict, + OUTPUT_DIR, + PYTHON_ENCODING, + ANSI, + IS_TTY, + TERM_WIDTH, + SHOW_PROGRESS, + SOURCES_DIR_NAME, + stderr, +) + +@dataclass +class RuntimeStats: + """mutable stats counter for logging archiving timing info to CLI output""" + + skipped: int = 0 + succeeded: int = 0 + failed: int = 0 + + parse_start_ts: Optional[datetime] = None + parse_end_ts: Optional[datetime] = None + + index_start_ts: Optional[datetime] = None + index_end_ts: Optional[datetime] = None + + archiving_start_ts: Optional[datetime] = None + archiving_end_ts: Optional[datetime] = None + +# globals are bad, mmkay +_LAST_RUN_STATS = RuntimeStats() + + + +class SmartFormatter(argparse.HelpFormatter): + """Patched formatter that prints newlines in argparse help strings""" + def _split_lines(self, text, width): + if '\n' in text: + return text.splitlines() + return argparse.HelpFormatter._split_lines(self, text, width) + + +def reject_stdin(caller: str, stdin: Optional[IO]=sys.stdin) -> None: + """Tell the user they passed stdin to a command that doesn't accept it""" + + if stdin and not stdin.isatty(): + stdin_raw_text = stdin.read().strip() + if stdin_raw_text: + stderr(f'[X] The "{caller}" command does not accept stdin.', color='red') + stderr(f' Run archivebox "{caller} --help" to see usage and examples.') + stderr() + raise SystemExit(1) + + +def accept_stdin(stdin: Optional[IO]=sys.stdin) -> Optional[str]: + """accept any standard input and return it as a string or None""" + if not stdin: + return None + elif stdin and not stdin.isatty(): + stdin_str = stdin.read().strip() + return stdin_str or None + return None + + +class TimedProgress: + """Show a progress bar and measure elapsed time until .end() is called""" + + def __init__(self, seconds, prefix=''): + self.SHOW_PROGRESS = SHOW_PROGRESS + if self.SHOW_PROGRESS: + self.p = Process(target=progress_bar, args=(seconds, prefix)) + self.p.start() + + self.stats = {'start_ts': datetime.now(), 'end_ts': None} + + def end(self): + """immediately end progress, clear the progressbar line, and save end_ts""" + + end_ts = datetime.now() + self.stats['end_ts'] = end_ts + + if self.SHOW_PROGRESS: + # terminate if we havent already terminated + try: + # kill the progress bar subprocess + try: + self.p.close() # must be closed *before* its terminnated + except: + pass + self.p.terminate() + self.p.join() + + + # clear whole terminal line + try: + sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset'])) + except (IOError, BrokenPipeError): + # ignore when the parent proc has stopped listening to our stdout + pass + except ValueError: + pass + + +@enforce_types +def progress_bar(seconds: int, prefix: str='') -> None: + """show timer in the form of progress bar, with percentage and seconds remaining""" + chunk = '█' if PYTHON_ENCODING == 'UTF-8' else '#' + last_width = TERM_WIDTH() + chunks = last_width - len(prefix) - 20 # number of progress chunks to show (aka max bar width) + try: + for s in range(seconds * chunks): + max_width = TERM_WIDTH() + if max_width < last_width: + # when the terminal size is shrunk, we have to write a newline + # otherwise the progress bar will keep wrapping incorrectly + sys.stdout.write('\r\n') + sys.stdout.flush() + chunks = max_width - len(prefix) - 20 + pct_complete = s / chunks / seconds * 100 + log_pct = (log(pct_complete or 1, 10) / 2) * 100 # everyone likes faster progress bars ;) + bar_width = round(log_pct/(100/chunks)) + last_width = max_width + + # ████████████████████ 0.9% (1/60sec) + sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format( + prefix, + ANSI['green' if pct_complete < 80 else 'lightyellow'], + (chunk * bar_width).ljust(chunks), + ANSI['reset'], + round(pct_complete, 1), + round(s/chunks), + seconds, + )) + sys.stdout.flush() + time.sleep(1 / chunks) + + # ██████████████████████████████████ 100.0% (60/60sec) + sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format( + prefix, + ANSI['red'], + chunk * chunks, + ANSI['reset'], + 100.0, + seconds, + seconds, + )) + sys.stdout.flush() + # uncomment to have it disappear when it hits 100% instead of staying full red: + # time.sleep(0.5) + # sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset'])) + # sys.stdout.flush() + except (KeyboardInterrupt, BrokenPipeError): + print() + pass + + +def log_cli_command(subcommand: str, subcommand_args: List[str], stdin: Optional[str], pwd: str): + from .config import VERSION, ANSI + cmd = ' '.join(('archivebox', subcommand, *subcommand_args)) + stderr('{black}[i] [{now}] ArchiveBox v{VERSION}: {cmd}{reset}'.format( + now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'), + VERSION=VERSION, + cmd=cmd, + **ANSI, + )) + stderr('{black} > {pwd}{reset}'.format(pwd=pwd, **ANSI)) + stderr() + +### Parsing Stage + + +def log_importing_started(urls: Union[str, List[str]], depth: int, index_only: bool): + _LAST_RUN_STATS.parse_start_ts = datetime.now() + print('{green}[+] [{}] Adding {} links to index (crawl depth={}){}...{reset}'.format( + _LAST_RUN_STATS.parse_start_ts.strftime('%Y-%m-%d %H:%M:%S'), + len(urls) if isinstance(urls, list) else len(urls.split('\n')), + depth, + ' (index only)' if index_only else '', + **ANSI, + )) + +def log_source_saved(source_file: str): + print(' > Saved verbatim input to {}/{}'.format(SOURCES_DIR_NAME, source_file.rsplit('/', 1)[-1])) + +def log_parsing_finished(num_parsed: int, parser_name: str): + _LAST_RUN_STATS.parse_end_ts = datetime.now() + print(' > Parsed {} URLs from input ({})'.format(num_parsed, parser_name)) + +def log_deduping_finished(num_new_links: int): + print(' > Found {} new URLs not already in index'.format(num_new_links)) + + +def log_crawl_started(new_links): + print() + print('{green}[*] Starting crawl of {} sites 1 hop out from starting point{reset}'.format(len(new_links), **ANSI)) + +### Indexing Stage + +def log_indexing_process_started(num_links: int): + start_ts = datetime.now() + _LAST_RUN_STATS.index_start_ts = start_ts + print() + print('{black}[*] [{}] Writing {} links to main index...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + **ANSI, + )) + + +def log_indexing_process_finished(): + end_ts = datetime.now() + _LAST_RUN_STATS.index_end_ts = end_ts + + +def log_indexing_started(out_path: str): + if IS_TTY: + sys.stdout.write(f' > {out_path}') + + +def log_indexing_finished(out_path: str): + print(f'\r √ {out_path}') + + +### Archiving Stage + +def log_archiving_started(num_links: int, resume: Optional[float]=None): + start_ts = datetime.now() + _LAST_RUN_STATS.archiving_start_ts = start_ts + print() + if resume: + print('{green}[▶] [{}] Resuming archive updating for {} pages starting from {}...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + resume, + **ANSI, + )) + else: + print('{green}[▶] [{}] Starting archiving of {} snapshots in index...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + **ANSI, + )) + +def log_archiving_paused(num_links: int, idx: int, timestamp: str): + end_ts = datetime.now() + _LAST_RUN_STATS.archiving_end_ts = end_ts + print() + print('\n{lightyellow}[X] [{now}] Downloading paused on link {timestamp} ({idx}/{total}){reset}'.format( + **ANSI, + now=end_ts.strftime('%Y-%m-%d %H:%M:%S'), + idx=idx+1, + timestamp=timestamp, + total=num_links, + )) + print() + print(' {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI)) + print(' archivebox server # then visit http://127.0.0.1:8000') + print(' Continue archiving where you left off by running:') + print(' archivebox update --resume={}'.format(timestamp)) + +def log_archiving_finished(num_links: int): + end_ts = datetime.now() + _LAST_RUN_STATS.archiving_end_ts = end_ts + assert _LAST_RUN_STATS.archiving_start_ts is not None + seconds = end_ts.timestamp() - _LAST_RUN_STATS.archiving_start_ts.timestamp() + if seconds > 60: + duration = '{0:.2f} min'.format(seconds / 60) + else: + duration = '{0:.2f} sec'.format(seconds) + + print() + print('{}[√] [{}] Update of {} pages complete ({}){}'.format( + ANSI['green'], + end_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + duration, + ANSI['reset'], + )) + print(' - {} links skipped'.format(_LAST_RUN_STATS.skipped)) + print(' - {} links updated'.format(_LAST_RUN_STATS.succeeded + _LAST_RUN_STATS.failed)) + print(' - {} links had errors'.format(_LAST_RUN_STATS.failed)) + print() + print(' {lightred}Hint:{reset} To manage your archive in a Web UI, run:'.format(**ANSI)) + print(' archivebox server 0.0.0.0:8000') + + +def log_link_archiving_started(link: "Link", link_dir: str, is_new: bool): + # [*] [2019-03-22 13:46:45] "Log Structured Merge Trees - ben stopford" + # http://www.benstopford.com/2015/02/14/log-structured-merge-trees/ + # > output/archive/1478739709 + + print('\n[{symbol_color}{symbol}{reset}] [{symbol_color}{now}{reset}] "{title}"'.format( + symbol_color=ANSI['green' if is_new else 'black'], + symbol='+' if is_new else '√', + now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'), + title=link.title or link.base_url, + **ANSI, + )) + print(' {blue}{url}{reset}'.format(url=link.url, **ANSI)) + print(' {} {}'.format( + '>' if is_new else '√', + pretty_path(link_dir), + )) + +def log_link_archiving_finished(link: "Link", link_dir: str, is_new: bool, stats: dict): + total = sum(stats.values()) + + if stats['failed'] > 0 : + _LAST_RUN_STATS.failed += 1 + elif stats['skipped'] == total: + _LAST_RUN_STATS.skipped += 1 + else: + _LAST_RUN_STATS.succeeded += 1 + + +def log_archive_method_started(method: str): + print(' > {}'.format(method)) + + +def log_archive_method_finished(result: "ArchiveResult"): + """quote the argument with whitespace in a command so the user can + copy-paste the outputted string directly to run the cmd + """ + # Prettify CMD string and make it safe to copy-paste by quoting arguments + quoted_cmd = ' '.join( + '"{}"'.format(arg) if ' ' in arg else arg + for arg in result.cmd + ) + + if result.status == 'failed': + if result.output.__class__.__name__ == 'TimeoutExpired': + duration = (result.end_ts - result.start_ts).seconds + hint_header = [ + '{lightyellow}Extractor timed out after {}s.{reset}'.format(duration, **ANSI), + ] + else: + hint_header = [ + '{lightyellow}Extractor failed:{reset}'.format(**ANSI), + ' {reset}{} {red}{}{reset}'.format( + result.output.__class__.__name__.replace('ArchiveError', ''), + result.output, + **ANSI, + ), + ] + + # Prettify error output hints string and limit to five lines + hints = getattr(result.output, 'hints', None) or () + if hints: + hints = hints if isinstance(hints, (list, tuple)) else hints.split('\n') + hints = ( + ' {}{}{}'.format(ANSI['lightyellow'], line.strip(), ANSI['reset']) + for line in hints[:5] if line.strip() + ) + + + # Collect and prefix output lines with indentation + output_lines = [ + *hint_header, + *hints, + '{}Run to see full output:{}'.format(ANSI['lightred'], ANSI['reset']), + *([' cd {};'.format(result.pwd)] if result.pwd else []), + ' {}'.format(quoted_cmd), + ] + print('\n'.join( + ' {}'.format(line) + for line in output_lines + if line + )) + print() + + +def log_list_started(filter_patterns: Optional[List[str]], filter_type: str): + print('{green}[*] Finding links in the archive index matching these {} patterns:{reset}'.format( + filter_type, + **ANSI, + )) + print(' {}'.format(' '.join(filter_patterns or ()))) + +def log_list_finished(links): + from .index.csv import links_to_csv + print() + print('---------------------------------------------------------------------------------------------------') + print(links_to_csv(links, cols=['timestamp', 'is_archived', 'num_outputs', 'url'], header=True, ljust=16, separator=' | ')) + print('---------------------------------------------------------------------------------------------------') + print() + + +def log_removal_started(links: List["Link"], yes: bool, delete: bool): + print('{lightyellow}[i] Found {} matching URLs to remove.{reset}'.format(len(links), **ANSI)) + if delete: + file_counts = [link.num_outputs for link in links if Path(link.link_dir).exists()] + print( + f' {len(links)} Links will be de-listed from the main index, and their archived content folders will be deleted from disk.\n' + f' ({len(file_counts)} data folders with {sum(file_counts)} archived files will be deleted!)' + ) + else: + print( + ' Matching links will be de-listed from the main index, but their archived content folders will remain in place on disk.\n' + ' (Pass --delete if you also want to permanently delete the data folders)' + ) + + if not yes: + print() + print('{lightyellow}[?] Do you want to proceed with removing these {} links?{reset}'.format(len(links), **ANSI)) + try: + assert input(' y/[n]: ').lower() == 'y' + except (KeyboardInterrupt, EOFError, AssertionError): + raise SystemExit(0) + +def log_removal_finished(all_links: int, to_remove: int): + if all_links == 0: + print() + print('{red}[X] No matching links found.{reset}'.format(**ANSI)) + else: + print() + print('{red}[√] Removed {} out of {} links from the archive index.{reset}'.format( + to_remove, + all_links, + **ANSI, + )) + print(' Index now contains {} links.'.format(all_links - to_remove)) + + +def log_shell_welcome_msg(): + from .cli import list_subcommands + + print('{green}# ArchiveBox Imports{reset}'.format(**ANSI)) + print('{green}from core.models import Snapshot, User{reset}'.format(**ANSI)) + print('{green}from archivebox import *\n {}{reset}'.format("\n ".join(list_subcommands().keys()), **ANSI)) + print() + print('[i] Welcome to the ArchiveBox Shell!') + print(' https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Shell-Usage') + print() + print(' {lightred}Hint:{reset} Example use:'.format(**ANSI)) + print(' print(Snapshot.objects.filter(is_archived=True).count())') + print(' Snapshot.objects.get(url="https://example.com").as_json()') + print(' add("https://example.com/some/new/url")') + + + +### Helpers + +@enforce_types +def pretty_path(path: Union[Path, str]) -> str: + """convert paths like .../ArchiveBox/archivebox/../output/abc into output/abc""" + pwd = Path('.').resolve() + # parent = os.path.abspath(os.path.join(pwd, os.path.pardir)) + return str(path).replace(str(pwd) + '/', './') + + +@enforce_types +def printable_filesize(num_bytes: Union[int, float]) -> str: + for count in ['Bytes','KB','MB','GB']: + if num_bytes > -1024.0 and num_bytes < 1024.0: + return '%3.1f %s' % (num_bytes, count) + num_bytes /= 1024.0 + return '%3.1f %s' % (num_bytes, 'TB') + + +@enforce_types +def printable_folders(folders: Dict[str, Optional["Link"]], + with_headers: bool=False) -> str: + return '\n'.join( + f'{folder} {link and link.url} "{link and link.title}"' + for folder, link in folders.items() + ) + + + +@enforce_types +def printable_config(config: ConfigDict, prefix: str='') -> str: + return f'\n{prefix}'.join( + f'{key}={val}' + for key, val in config.items() + if not (isinstance(val, dict) or callable(val)) + ) + + +@enforce_types +def printable_folder_status(name: str, folder: Dict) -> str: + if folder['enabled']: + if folder['is_valid']: + color, symbol, note = 'green', '√', 'valid' + else: + color, symbol, note, num_files = 'red', 'X', 'invalid', '?' + else: + color, symbol, note, num_files = 'lightyellow', '-', 'disabled', '-' + + if folder['path']: + if Path(folder['path']).exists(): + num_files = ( + f'{len(os.listdir(folder["path"]))} files' + if Path(folder['path']).is_dir() else + printable_filesize(Path(folder['path']).stat().st_size) + ) + else: + num_files = 'missing' + + path = str(folder['path']).replace(str(OUTPUT_DIR), '.') if folder['path'] else '' + if path and ' ' in path: + path = f'"{path}"' + + # if path is just a plain dot, replace it back with the full path for clarity + if path == '.': + path = str(OUTPUT_DIR) + + return ' '.join(( + ANSI[color], + symbol, + ANSI['reset'], + name.ljust(21), + num_files.ljust(14), + ANSI[color], + note.ljust(8), + ANSI['reset'], + path.ljust(76), + )) + + +@enforce_types +def printable_dependency_version(name: str, dependency: Dict) -> str: + version = None + if dependency['enabled']: + if dependency['is_valid']: + color, symbol, note, version = 'green', '√', 'valid', '' + + parsed_version_num = re.search(r'[\d\.]+', dependency['version']) + if parsed_version_num: + version = f'v{parsed_version_num[0]}' + + if not version: + color, symbol, note, version = 'red', 'X', 'invalid', '?' + else: + color, symbol, note, version = 'lightyellow', '-', 'disabled', '-' + + path = str(dependency["path"]).replace(str(OUTPUT_DIR), '.') if dependency["path"] else '' + if path and ' ' in path: + path = f'"{path}"' + + return ' '.join(( + ANSI[color], + symbol, + ANSI['reset'], + name.ljust(21), + version.ljust(14), + ANSI[color], + note.ljust(8), + ANSI['reset'], + path.ljust(76), + )) diff --git a/archivebox-0.5.3/archivebox/main.py b/archivebox-0.5.3/archivebox/main.py new file mode 100644 index 0000000..eb8cd6a --- /dev/null +++ b/archivebox-0.5.3/archivebox/main.py @@ -0,0 +1,1131 @@ +__package__ = 'archivebox' + +import os +import sys +import shutil +import platform +from pathlib import Path +from datetime import date + +from typing import Dict, List, Optional, Iterable, IO, Union +from crontab import CronTab, CronSlices +from django.db.models import QuerySet + +from .cli import ( + list_subcommands, + run_subcommand, + display_first, + meta_cmds, + main_cmds, + archive_cmds, +) +from .parsers import ( + save_text_as_source, + save_file_as_source, + parse_links_memory, +) +from .index.schema import Link +from .util import enforce_types # type: ignore +from .system import get_dir_size, dedupe_cron_jobs, CRON_COMMENT +from .index import ( + load_main_index, + parse_links_from_source, + dedupe_links, + write_main_index, + snapshot_filter, + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, + fix_invalid_folder_locations, + write_link_details, +) +from .index.json import ( + parse_json_main_index, + parse_json_links_details, + generate_json_index_from_links, +) +from .index.sql import ( + get_admins, + apply_migrations, + remove_from_sql_main_index, +) +from .index.html import ( + generate_index_from_links, +) +from .index.csv import links_to_csv +from .extractors import archive_links, archive_link, ignore_methods +from .config import ( + stderr, + hint, + ConfigDict, + ANSI, + IS_TTY, + IN_DOCKER, + USER, + ARCHIVEBOX_BINARY, + ONLY_NEW, + OUTPUT_DIR, + SOURCES_DIR, + ARCHIVE_DIR, + LOGS_DIR, + CONFIG_FILE, + ARCHIVE_DIR_NAME, + SOURCES_DIR_NAME, + LOGS_DIR_NAME, + STATIC_DIR_NAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, + SQL_INDEX_FILENAME, + ROBOTS_TXT_FILENAME, + FAVICON_FILENAME, + check_dependencies, + check_data_folder, + write_config_file, + VERSION, + CODE_LOCATIONS, + EXTERNAL_LOCATIONS, + DATA_LOCATIONS, + DEPENDENCIES, + load_all_config, + CONFIG, + USER_CONFIG, + get_real_name, +) +from .logging_util import ( + TERM_WIDTH, + TimedProgress, + log_importing_started, + log_crawl_started, + log_removal_started, + log_removal_finished, + log_list_started, + log_list_finished, + printable_config, + printable_folders, + printable_filesize, + printable_folder_status, + printable_dependency_version, +) + +from .search import flush_search_index, index_links + +ALLOWED_IN_OUTPUT_DIR = { + 'lost+found', + '.DS_Store', + '.venv', + 'venv', + 'virtualenv', + '.virtualenv', + 'node_modules', + 'package-lock.json', + ARCHIVE_DIR_NAME, + SOURCES_DIR_NAME, + LOGS_DIR_NAME, + STATIC_DIR_NAME, + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, + ROBOTS_TXT_FILENAME, + FAVICON_FILENAME, +} + +@enforce_types +def help(out_dir: Path=OUTPUT_DIR) -> None: + """Print the ArchiveBox help message and usage""" + + all_subcommands = list_subcommands() + COMMANDS_HELP_TEXT = '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in meta_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in main_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in archive_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd not in display_first + ) + + + if (Path(out_dir) / SQL_INDEX_FILENAME).exists(): + print('''{green}ArchiveBox v{}: The self-hosted internet archive.{reset} + +{lightred}Active data directory:{reset} + {} + +{lightred}Usage:{reset} + archivebox [command] [--help] [--version] [...args] + +{lightred}Commands:{reset} + {} + +{lightred}Example Use:{reset} + mkdir my-archive; cd my-archive/ + archivebox init + archivebox status + + archivebox add https://example.com/some/page + archivebox add --depth=1 ~/Downloads/bookmarks_export.html + + archivebox list --sort=timestamp --csv=timestamp,url,is_archived + archivebox schedule --every=day https://example.com/some/feed.rss + archivebox update --resume=15109948213.123 + +{lightred}Documentation:{reset} + https://github.com/ArchiveBox/ArchiveBox/wiki +'''.format(VERSION, out_dir, COMMANDS_HELP_TEXT, **ANSI)) + + else: + print('{green}Welcome to ArchiveBox v{}!{reset}'.format(VERSION, **ANSI)) + print() + if IN_DOCKER: + print('When using Docker, you need to mount a volume to use as your data dir:') + print(' docker run -v /some/path:/data archivebox ...') + print() + print('To import an existing archive (from a previous version of ArchiveBox):') + print(' 1. cd into your data dir OUTPUT_DIR (usually ArchiveBox/output) and run:') + print(' 2. archivebox init') + print() + print('To start a new archive:') + print(' 1. Create an empty directory, then cd into it and run:') + print(' 2. archivebox init') + print() + print('For more information, see the documentation here:') + print(' https://github.com/ArchiveBox/ArchiveBox/wiki') + + +@enforce_types +def version(quiet: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Print the ArchiveBox version and dependency information""" + + if quiet: + print(VERSION) + else: + print('ArchiveBox v{}'.format(VERSION)) + p = platform.uname() + print(sys.implementation.name.title(), p.system, platform.platform(), p.machine, '(in Docker)' if IN_DOCKER else '(not in Docker)') + print() + + print('{white}[i] Dependency versions:{reset}'.format(**ANSI)) + for name, dependency in DEPENDENCIES.items(): + print(printable_dependency_version(name, dependency)) + + print() + print('{white}[i] Source-code locations:{reset}'.format(**ANSI)) + for name, folder in CODE_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + + print() + print('{white}[i] Secrets locations:{reset}'.format(**ANSI)) + for name, folder in EXTERNAL_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + + print() + if DATA_LOCATIONS['OUTPUT_DIR']['is_valid']: + print('{white}[i] Data locations:{reset}'.format(**ANSI)) + for name, folder in DATA_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + else: + print() + print('{white}[i] Data locations:{reset}'.format(**ANSI)) + + print() + check_dependencies() + + +@enforce_types +def run(subcommand: str, + subcommand_args: Optional[List[str]], + stdin: Optional[IO]=None, + out_dir: Path=OUTPUT_DIR) -> None: + """Run a given ArchiveBox subcommand with the given list of args""" + run_subcommand( + subcommand=subcommand, + subcommand_args=subcommand_args, + stdin=stdin, + pwd=out_dir, + ) + + +@enforce_types +def init(force: bool=False, out_dir: Path=OUTPUT_DIR) -> None: + """Initialize a new ArchiveBox collection in the current directory""" + from core.models import Snapshot + Path(out_dir).mkdir(exist_ok=True) + is_empty = not len(set(os.listdir(out_dir)) - ALLOWED_IN_OUTPUT_DIR) + + if (Path(out_dir) / JSON_INDEX_FILENAME).exists(): + stderr("[!] This folder contains a JSON index. It is deprecated, and will no longer be kept up to date automatically.", color="lightyellow") + stderr(" You can run `archivebox list --json --with-headers > index.json` to manually generate it.", color="lightyellow") + + existing_index = (Path(out_dir) / SQL_INDEX_FILENAME).exists() + + if is_empty and not existing_index: + print('{green}[+] Initializing a new ArchiveBox collection in this folder...{reset}'.format(**ANSI)) + print(f' {out_dir}') + print('{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + elif existing_index: + print('{green}[*] Updating existing ArchiveBox collection in this folder...{reset}'.format(**ANSI)) + print(f' {out_dir}') + print('{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + else: + if force: + stderr('[!] This folder appears to already have files in it, but no index.sqlite3 is present.', color='lightyellow') + stderr(' Because --force was passed, ArchiveBox will initialize anyway (which may overwrite existing files).') + else: + stderr( + ("{red}[X] This folder appears to already have files in it, but no index.sqlite3 present.{reset}\n\n" + " You must run init in a completely empty directory, or an existing data folder.\n\n" + " {lightred}Hint:{reset} To import an existing data folder make sure to cd into the folder first, \n" + " then run and run 'archivebox init' to pick up where you left off.\n\n" + " (Always make sure your data folder is backed up first before updating ArchiveBox)" + ).format(out_dir, **ANSI) + ) + raise SystemExit(2) + + if existing_index: + print('\n{green}[*] Verifying archive folder structure...{reset}'.format(**ANSI)) + else: + print('\n{green}[+] Building archive folder structure...{reset}'.format(**ANSI)) + + Path(SOURCES_DIR).mkdir(exist_ok=True) + print(f' √ {SOURCES_DIR}') + + Path(ARCHIVE_DIR).mkdir(exist_ok=True) + print(f' √ {ARCHIVE_DIR}') + + Path(LOGS_DIR).mkdir(exist_ok=True) + print(f' √ {LOGS_DIR}') + + write_config_file({}, out_dir=out_dir) + print(f' √ {CONFIG_FILE}') + if (Path(out_dir) / SQL_INDEX_FILENAME).exists(): + print('\n{green}[*] Verifying main SQL index and running migrations...{reset}'.format(**ANSI)) + else: + print('\n{green}[+] Building main SQL index and running migrations...{reset}'.format(**ANSI)) + + DATABASE_FILE = Path(out_dir) / SQL_INDEX_FILENAME + print(f' √ {DATABASE_FILE}') + print() + for migration_line in apply_migrations(out_dir): + print(f' {migration_line}') + + + assert DATABASE_FILE.exists() + + # from django.contrib.auth.models import User + # if IS_TTY and not User.objects.filter(is_superuser=True).exists(): + # print('{green}[+] Creating admin user account...{reset}'.format(**ANSI)) + # call_command("createsuperuser", interactive=True) + + print() + print('{green}[*] Collecting links from any existing indexes and archive folders...{reset}'.format(**ANSI)) + + all_links = Snapshot.objects.none() + pending_links: Dict[str, Link] = {} + + if existing_index: + all_links = load_main_index(out_dir=out_dir, warn=False) + print(' √ Loaded {} links from existing main index.'.format(all_links.count())) + + # Links in data folders that dont match their timestamp + fixed, cant_fix = fix_invalid_folder_locations(out_dir=out_dir) + if fixed: + print(' {lightyellow}√ Fixed {} data directory locations that didn\'t match their link timestamps.{reset}'.format(len(fixed), **ANSI)) + if cant_fix: + print(' {lightyellow}! Could not fix {} data directory locations due to conflicts with existing folders.{reset}'.format(len(cant_fix), **ANSI)) + + # Links in JSON index but not in main index + orphaned_json_links = { + link.url: link + for link in parse_json_main_index(out_dir) + if not all_links.filter(url=link.url).exists() + } + if orphaned_json_links: + pending_links.update(orphaned_json_links) + print(' {lightyellow}√ Added {} orphaned links from existing JSON index...{reset}'.format(len(orphaned_json_links), **ANSI)) + + # Links in data dir indexes but not in main index + orphaned_data_dir_links = { + link.url: link + for link in parse_json_links_details(out_dir) + if not all_links.filter(url=link.url).exists() + } + if orphaned_data_dir_links: + pending_links.update(orphaned_data_dir_links) + print(' {lightyellow}√ Added {} orphaned links from existing archive directories.{reset}'.format(len(orphaned_data_dir_links), **ANSI)) + + # Links in invalid/duplicate data dirs + invalid_folders = { + folder: link + for folder, link in get_invalid_folders(all_links, out_dir=out_dir).items() + } + if invalid_folders: + print(' {lightyellow}! Skipped adding {} invalid link data directories.{reset}'.format(len(invalid_folders), **ANSI)) + print(' X ' + '\n X '.join(f'{folder} {link}' for folder, link in invalid_folders.items())) + print() + print(' {lightred}Hint:{reset} For more information about the link data directories that were skipped, run:'.format(**ANSI)) + print(' archivebox status') + print(' archivebox list --status=invalid') + + + write_main_index(list(pending_links.values()), out_dir=out_dir) + + print('\n{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + if existing_index: + print('{green}[√] Done. Verified and updated the existing ArchiveBox collection.{reset}'.format(**ANSI)) + else: + print('{green}[√] Done. A new ArchiveBox collection was initialized ({} links).{reset}'.format(len(all_links), **ANSI)) + print() + print(' {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI)) + print(' archivebox server # then visit http://127.0.0.1:8000') + print() + print(' To add new links, you can run:') + print(" archivebox add ~/some/path/or/url/to/list_of_links.txt") + print() + print(' For more usage and examples, run:') + print(' archivebox help') + + json_index = Path(out_dir) / JSON_INDEX_FILENAME + html_index = Path(out_dir) / HTML_INDEX_FILENAME + index_name = f"{date.today()}_index_old" + if json_index.exists(): + json_index.rename(f"{index_name}.json") + if html_index.exists(): + html_index.rename(f"{index_name}.html") + + + +@enforce_types +def status(out_dir: Path=OUTPUT_DIR) -> None: + """Print out some info and statistics about the archive collection""" + + check_data_folder(out_dir=out_dir) + + from core.models import Snapshot + from django.contrib.auth import get_user_model + User = get_user_model() + + print('{green}[*] Scanning archive main index...{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {out_dir}/*', ANSI['reset']) + num_bytes, num_dirs, num_files = get_dir_size(out_dir, recursive=False, pattern='index.') + size = printable_filesize(num_bytes) + print(f' Index size: {size} across {num_files} files') + print() + + links = load_main_index(out_dir=out_dir) + num_sql_links = links.count() + num_link_details = sum(1 for link in parse_json_links_details(out_dir=out_dir)) + print(f' > SQL Main Index: {num_sql_links} links'.ljust(36), f'(found in {SQL_INDEX_FILENAME})') + print(f' > JSON Link Details: {num_link_details} links'.ljust(36), f'(found in {ARCHIVE_DIR_NAME}/*/index.json)') + print() + print('{green}[*] Scanning archive data directories...{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {ARCHIVE_DIR}/*', ANSI['reset']) + num_bytes, num_dirs, num_files = get_dir_size(ARCHIVE_DIR) + size = printable_filesize(num_bytes) + print(f' Size: {size} across {num_files} files in {num_dirs} directories') + print(ANSI['black']) + num_indexed = len(get_indexed_folders(links, out_dir=out_dir)) + num_archived = len(get_archived_folders(links, out_dir=out_dir)) + num_unarchived = len(get_unarchived_folders(links, out_dir=out_dir)) + print(f' > indexed: {num_indexed}'.ljust(36), f'({get_indexed_folders.__doc__})') + print(f' > archived: {num_archived}'.ljust(36), f'({get_archived_folders.__doc__})') + print(f' > unarchived: {num_unarchived}'.ljust(36), f'({get_unarchived_folders.__doc__})') + + num_present = len(get_present_folders(links, out_dir=out_dir)) + num_valid = len(get_valid_folders(links, out_dir=out_dir)) + print() + print(f' > present: {num_present}'.ljust(36), f'({get_present_folders.__doc__})') + print(f' > valid: {num_valid}'.ljust(36), f'({get_valid_folders.__doc__})') + + duplicate = get_duplicate_folders(links, out_dir=out_dir) + orphaned = get_orphaned_folders(links, out_dir=out_dir) + corrupted = get_corrupted_folders(links, out_dir=out_dir) + unrecognized = get_unrecognized_folders(links, out_dir=out_dir) + num_invalid = len({**duplicate, **orphaned, **corrupted, **unrecognized}) + print(f' > invalid: {num_invalid}'.ljust(36), f'({get_invalid_folders.__doc__})') + print(f' > duplicate: {len(duplicate)}'.ljust(36), f'({get_duplicate_folders.__doc__})') + print(f' > orphaned: {len(orphaned)}'.ljust(36), f'({get_orphaned_folders.__doc__})') + print(f' > corrupted: {len(corrupted)}'.ljust(36), f'({get_corrupted_folders.__doc__})') + print(f' > unrecognized: {len(unrecognized)}'.ljust(36), f'({get_unrecognized_folders.__doc__})') + + print(ANSI['reset']) + + if num_indexed: + print(' {lightred}Hint:{reset} You can list link data directories by status like so:'.format(**ANSI)) + print(' archivebox list --status=<status> (e.g. indexed, corrupted, archived, etc.)') + + if orphaned: + print(' {lightred}Hint:{reset} To automatically import orphaned data directories into the main index, run:'.format(**ANSI)) + print(' archivebox init') + + if num_invalid: + print(' {lightred}Hint:{reset} You may need to manually remove or fix some invalid data directories, afterwards make sure to run:'.format(**ANSI)) + print(' archivebox init') + + print() + print('{green}[*] Scanning recent archive changes and user logins:{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {LOGS_DIR}/*', ANSI['reset']) + users = get_admins().values_list('username', flat=True) + print(f' UI users {len(users)}: {", ".join(users)}') + last_login = User.objects.order_by('last_login').last() + if last_login: + print(f' Last UI login: {last_login.username} @ {str(last_login.last_login)[:16]}') + last_updated = Snapshot.objects.order_by('updated').last() + if last_updated: + print(f' Last changes: {str(last_updated.updated)[:16]}') + + if not users: + print() + print(' {lightred}Hint:{reset} You can create an admin user by running:'.format(**ANSI)) + print(' archivebox manage createsuperuser') + + print() + for snapshot in links.order_by('-updated')[:10]: + if not snapshot.updated: + continue + print( + ANSI['black'], + ( + f' > {str(snapshot.updated)[:16]} ' + f'[{snapshot.num_outputs} {("X", "√")[snapshot.is_archived]} {printable_filesize(snapshot.archive_size)}] ' + f'"{snapshot.title}": {snapshot.url}' + )[:TERM_WIDTH()], + ANSI['reset'], + ) + print(ANSI['black'], ' ...', ANSI['reset']) + + +@enforce_types +def oneshot(url: str, extractors: str="", out_dir: Path=OUTPUT_DIR): + """ + Create a single URL archive folder with an index.json and index.html, and all the archive method outputs. + You can run this to archive single pages without needing to create a whole collection with archivebox init. + """ + oneshot_link, _ = parse_links_memory([url]) + if len(oneshot_link) > 1: + stderr( + '[X] You should pass a single url to the oneshot command', + color='red' + ) + raise SystemExit(2) + + methods = extractors.split(",") if extractors else ignore_methods(['title']) + archive_link(oneshot_link[0], out_dir=out_dir, methods=methods) + return oneshot_link + +@enforce_types +def add(urls: Union[str, List[str]], + depth: int=0, + update_all: bool=not ONLY_NEW, + index_only: bool=False, + overwrite: bool=False, + init: bool=False, + extractors: str="", + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Add a new URL or list of URLs to your archive""" + + assert depth in (0, 1), 'Depth must be 0 or 1 (depth >1 is not supported yet)' + + extractors = extractors.split(",") if extractors else [] + + if init: + run_subcommand('init', stdin=None, pwd=out_dir) + + # Load list of links from the existing index + check_data_folder(out_dir=out_dir) + check_dependencies() + new_links: List[Link] = [] + all_links = load_main_index(out_dir=out_dir) + + log_importing_started(urls=urls, depth=depth, index_only=index_only) + if isinstance(urls, str): + # save verbatim stdin to sources + write_ahead_log = save_text_as_source(urls, filename='{ts}-import.txt', out_dir=out_dir) + elif isinstance(urls, list): + # save verbatim args to sources + write_ahead_log = save_text_as_source('\n'.join(urls), filename='{ts}-import.txt', out_dir=out_dir) + + new_links += parse_links_from_source(write_ahead_log, root_url=None) + + # If we're going one level deeper, download each link and look for more links + new_links_depth = [] + if new_links and depth == 1: + log_crawl_started(new_links) + for new_link in new_links: + downloaded_file = save_file_as_source(new_link.url, filename=f'{new_link.timestamp}-crawl-{new_link.domain}.txt', out_dir=out_dir) + new_links_depth += parse_links_from_source(downloaded_file, root_url=new_link.url) + + imported_links = list({link.url: link for link in (new_links + new_links_depth)}.values()) + new_links = dedupe_links(all_links, imported_links) + + write_main_index(links=new_links, out_dir=out_dir) + all_links = load_main_index(out_dir=out_dir) + + if index_only: + return all_links + + # Run the archive methods for each link + archive_kwargs = { + "out_dir": out_dir, + } + if extractors: + archive_kwargs["methods"] = extractors + if update_all: + archive_links(all_links, overwrite=overwrite, **archive_kwargs) + elif overwrite: + archive_links(imported_links, overwrite=True, **archive_kwargs) + elif new_links: + archive_links(new_links, overwrite=False, **archive_kwargs) + + return all_links + +@enforce_types +def remove(filter_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + snapshots: Optional[QuerySet]=None, + after: Optional[float]=None, + before: Optional[float]=None, + yes: bool=False, + delete: bool=False, + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Remove the specified URLs from the archive""" + + check_data_folder(out_dir=out_dir) + + if snapshots is None: + if filter_str and filter_patterns: + stderr( + '[X] You should pass either a pattern as an argument, ' + 'or pass a list of patterns via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif not (filter_str or filter_patterns): + stderr( + '[X] You should pass either a pattern as an argument, ' + 'or pass a list of patterns via stdin.', + color='red', + ) + stderr() + hint(('To remove all urls you can run:', + 'archivebox remove --filter-type=regex ".*"')) + stderr() + raise SystemExit(2) + elif filter_str: + filter_patterns = [ptn.strip() for ptn in filter_str.split('\n')] + + list_kwargs = { + "filter_patterns": filter_patterns, + "filter_type": filter_type, + "after": after, + "before": before, + } + if snapshots: + list_kwargs["snapshots"] = snapshots + + log_list_started(filter_patterns, filter_type) + timer = TimedProgress(360, prefix=' ') + try: + snapshots = list_links(**list_kwargs) + finally: + timer.end() + + + if not snapshots.exists(): + log_removal_finished(0, 0) + raise SystemExit(1) + + + log_links = [link.as_link() for link in snapshots] + log_list_finished(log_links) + log_removal_started(log_links, yes=yes, delete=delete) + + timer = TimedProgress(360, prefix=' ') + try: + for snapshot in snapshots: + if delete: + shutil.rmtree(snapshot.as_link().link_dir, ignore_errors=True) + finally: + timer.end() + + to_remove = snapshots.count() + + flush_search_index(snapshots=snapshots) + remove_from_sql_main_index(snapshots=snapshots, out_dir=out_dir) + all_snapshots = load_main_index(out_dir=out_dir) + log_removal_finished(all_snapshots.count(), to_remove) + + return all_snapshots + +@enforce_types +def update(resume: Optional[float]=None, + only_new: bool=ONLY_NEW, + index_only: bool=False, + overwrite: bool=False, + filter_patterns_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: Optional[str]=None, + status: Optional[str]=None, + after: Optional[str]=None, + before: Optional[str]=None, + extractors: str="", + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Import any new links from subscriptions and retry any previously failed/skipped links""" + + check_data_folder(out_dir=out_dir) + check_dependencies() + new_links: List[Link] = [] # TODO: Remove input argument: only_new + + extractors = extractors.split(",") if extractors else [] + + # Step 1: Filter for selected_links + matching_snapshots = list_links( + filter_patterns=filter_patterns, + filter_type=filter_type, + before=before, + after=after, + ) + + matching_folders = list_folders( + links=matching_snapshots, + status=status, + out_dir=out_dir, + ) + all_links = [link for link in matching_folders.values() if link] + + if index_only: + for link in all_links: + write_link_details(link, out_dir=out_dir, skip_sql_index=True) + index_links(all_links, out_dir=out_dir) + return all_links + + # Step 2: Run the archive methods for each link + to_archive = new_links if only_new else all_links + if resume: + to_archive = [ + link for link in to_archive + if link.timestamp >= str(resume) + ] + if not to_archive: + stderr('') + stderr(f'[√] Nothing found to resume after {resume}', color='green') + return all_links + + archive_kwargs = { + "out_dir": out_dir, + } + if extractors: + archive_kwargs["methods"] = extractors + + archive_links(to_archive, overwrite=overwrite, **archive_kwargs) + + # Step 4: Re-write links index with updated titles, icons, and resources + all_links = load_main_index(out_dir=out_dir) + return all_links + +@enforce_types +def list_all(filter_patterns_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + status: Optional[str]=None, + after: Optional[float]=None, + before: Optional[float]=None, + sort: Optional[str]=None, + csv: Optional[str]=None, + json: bool=False, + html: bool=False, + with_headers: bool=False, + out_dir: Path=OUTPUT_DIR) -> Iterable[Link]: + """List, filter, and export information about archive entries""" + + check_data_folder(out_dir=out_dir) + + if filter_patterns and filter_patterns_str: + stderr( + '[X] You should either pass filter patterns as an arguments ' + 'or via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif filter_patterns_str: + filter_patterns = filter_patterns_str.split('\n') + + snapshots = list_links( + filter_patterns=filter_patterns, + filter_type=filter_type, + before=before, + after=after, + ) + + if sort: + snapshots = snapshots.order_by(sort) + + folders = list_folders( + links=snapshots, + status=status, + out_dir=out_dir, + ) + + if json: + output = generate_json_index_from_links(folders.values(), with_headers) + elif html: + output = generate_index_from_links(folders.values(), with_headers) + elif csv: + output = links_to_csv(folders.values(), cols=csv.split(','), header=with_headers) + else: + output = printable_folders(folders, with_headers=with_headers) + print(output) + return folders + + +@enforce_types +def list_links(snapshots: Optional[QuerySet]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + after: Optional[float]=None, + before: Optional[float]=None, + out_dir: Path=OUTPUT_DIR) -> Iterable[Link]: + + check_data_folder(out_dir=out_dir) + + if snapshots: + all_snapshots = snapshots + else: + all_snapshots = load_main_index(out_dir=out_dir) + + if after is not None: + all_snapshots = all_snapshots.filter(timestamp__lt=after) + if before is not None: + all_snapshots = all_snapshots.filter(timestamp__gt=before) + if filter_patterns: + all_snapshots = snapshot_filter(all_snapshots, filter_patterns, filter_type) + return all_snapshots + +@enforce_types +def list_folders(links: List[Link], + status: str, + out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + + check_data_folder(out_dir=out_dir) + + STATUS_FUNCTIONS = { + "indexed": get_indexed_folders, + "archived": get_archived_folders, + "unarchived": get_unarchived_folders, + "present": get_present_folders, + "valid": get_valid_folders, + "invalid": get_invalid_folders, + "duplicate": get_duplicate_folders, + "orphaned": get_orphaned_folders, + "corrupted": get_corrupted_folders, + "unrecognized": get_unrecognized_folders, + } + + try: + return STATUS_FUNCTIONS[status](links, out_dir=out_dir) + except KeyError: + raise ValueError('Status not recognized.') + + +@enforce_types +def config(config_options_str: Optional[str]=None, + config_options: Optional[List[str]]=None, + get: bool=False, + set: bool=False, + reset: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Get and set your ArchiveBox project configuration values""" + + check_data_folder(out_dir=out_dir) + + if config_options and config_options_str: + stderr( + '[X] You should either pass config values as an arguments ' + 'or via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif config_options_str: + config_options = config_options_str.split('\n') + + config_options = config_options or [] + + no_args = not (get or set or reset or config_options) + + matching_config: ConfigDict = {} + if get or no_args: + if config_options: + config_options = [get_real_name(key) for key in config_options] + matching_config = {key: CONFIG[key] for key in config_options if key in CONFIG} + failed_config = [key for key in config_options if key not in CONFIG] + if failed_config: + stderr() + stderr('[X] These options failed to get', color='red') + stderr(' {}'.format('\n '.join(config_options))) + raise SystemExit(1) + else: + matching_config = CONFIG + + print(printable_config(matching_config)) + raise SystemExit(not matching_config) + elif set: + new_config = {} + failed_options = [] + for line in config_options: + if line.startswith('#') or not line.strip(): + continue + if '=' not in line: + stderr('[X] Config KEY=VALUE must have an = sign in it', color='red') + stderr(f' {line}') + raise SystemExit(2) + + raw_key, val = line.split('=', 1) + raw_key = raw_key.upper().strip() + key = get_real_name(raw_key) + if key != raw_key: + stderr(f'[i] Note: The config option {raw_key} has been renamed to {key}, please use the new name going forwards.', color='lightyellow') + + if key in CONFIG: + new_config[key] = val.strip() + else: + failed_options.append(line) + + if new_config: + before = CONFIG + matching_config = write_config_file(new_config, out_dir=OUTPUT_DIR) + after = load_all_config() + print(printable_config(matching_config)) + + side_effect_changes: ConfigDict = {} + for key, val in after.items(): + if key in USER_CONFIG and (before[key] != after[key]) and (key not in matching_config): + side_effect_changes[key] = after[key] + + if side_effect_changes: + stderr() + stderr('[i] Note: This change also affected these other options that depended on it:', color='lightyellow') + print(' {}'.format(printable_config(side_effect_changes, prefix=' '))) + if failed_options: + stderr() + stderr('[X] These options failed to set (check for typos):', color='red') + stderr(' {}'.format('\n '.join(failed_options))) + raise SystemExit(bool(failed_options)) + elif reset: + stderr('[X] This command is not implemented yet.', color='red') + stderr(' Please manually remove the relevant lines from your config file:') + stderr(f' {CONFIG_FILE}') + raise SystemExit(2) + else: + stderr('[X] You must pass either --get or --set, or no arguments to get the whole config.', color='red') + stderr(' archivebox config') + stderr(' archivebox config --get SOME_KEY') + stderr(' archivebox config --set SOME_KEY=SOME_VALUE') + raise SystemExit(2) + + +@enforce_types +def schedule(add: bool=False, + show: bool=False, + clear: bool=False, + foreground: bool=False, + run_all: bool=False, + quiet: bool=False, + every: Optional[str]=None, + depth: int=0, + import_path: Optional[str]=None, + out_dir: Path=OUTPUT_DIR): + """Set ArchiveBox to regularly import URLs at specific times using cron""" + + check_data_folder(out_dir=out_dir) + + (Path(out_dir) / LOGS_DIR_NAME).mkdir(exist_ok=True) + + cron = CronTab(user=True) + cron = dedupe_cron_jobs(cron) + + if clear: + print(cron.remove_all(comment=CRON_COMMENT)) + cron.write() + raise SystemExit(0) + + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + if every or add: + every = every or 'day' + quoted = lambda s: f'"{s}"' if s and ' ' in str(s) else str(s) + cmd = [ + 'cd', + quoted(out_dir), + '&&', + quoted(ARCHIVEBOX_BINARY), + *(['add', f'--depth={depth}', f'"{import_path}"'] if import_path else ['update']), + '>', + quoted(Path(LOGS_DIR) / 'archivebox.log'), + '2>&1', + + ] + new_job = cron.new(command=' '.join(cmd), comment=CRON_COMMENT) + + if every in ('minute', 'hour', 'day', 'month', 'year'): + set_every = getattr(new_job.every(), every) + set_every() + elif CronSlices.is_valid(every): + new_job.setall(every) + else: + stderr('{red}[X] Got invalid timeperiod for cron task.{reset}'.format(**ANSI)) + stderr(' It must be one of minute/hour/day/month') + stderr(' or a quoted cron-format schedule like:') + stderr(' archivebox init --every=day https://example.com/some/rss/feed.xml') + stderr(' archivebox init --every="0/5 * * * *" https://example.com/some/rss/feed.xml') + raise SystemExit(1) + + cron = dedupe_cron_jobs(cron) + cron.write() + + total_runs = sum(j.frequency_per_year() for j in cron) + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + print() + print('{green}[√] Scheduled new ArchiveBox cron job for user: {} ({} jobs are active).{reset}'.format(USER, len(existing_jobs), **ANSI)) + print('\n'.join(f' > {cmd}' if str(cmd) == str(new_job) else f' {cmd}' for cmd in existing_jobs)) + if total_runs > 60 and not quiet: + stderr() + stderr('{lightyellow}[!] With the current cron config, ArchiveBox is estimated to run >{} times per year.{reset}'.format(total_runs, **ANSI)) + stderr(' Congrats on being an enthusiastic internet archiver! 👌') + stderr() + stderr(' Make sure you have enough storage space available to hold all the data.') + stderr(' Using a compressed/deduped filesystem like ZFS is recommended if you plan on archiving a lot.') + stderr('') + elif show: + if existing_jobs: + print('\n'.join(str(cmd) for cmd in existing_jobs)) + else: + stderr('{red}[X] There are no ArchiveBox cron jobs scheduled for your user ({}).{reset}'.format(USER, **ANSI)) + stderr(' To schedule a new job, run:') + stderr(' archivebox schedule --every=[timeperiod] https://example.com/some/rss/feed.xml') + raise SystemExit(0) + + cron = CronTab(user=True) + cron = dedupe_cron_jobs(cron) + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + if foreground or run_all: + if not existing_jobs: + stderr('{red}[X] You must schedule some jobs first before running in foreground mode.{reset}'.format(**ANSI)) + stderr(' archivebox schedule --every=hour https://example.com/some/rss/feed.xml') + raise SystemExit(1) + + print('{green}[*] Running {} ArchiveBox jobs in foreground task scheduler...{reset}'.format(len(existing_jobs), **ANSI)) + if run_all: + try: + for job in existing_jobs: + sys.stdout.write(f' > {job.command.split("/archivebox ")[0].split(" && ")[0]}\n') + sys.stdout.write(f' > {job.command.split("/archivebox ")[-1].split(" > ")[0]}') + sys.stdout.flush() + job.run() + sys.stdout.write(f'\r √ {job.command.split("/archivebox ")[-1]}\n') + except KeyboardInterrupt: + print('\n{green}[√] Stopped.{reset}'.format(**ANSI)) + raise SystemExit(1) + + if foreground: + try: + for job in existing_jobs: + print(f' > {job.command.split("/archivebox ")[-1].split(" > ")[0]}') + for result in cron.run_scheduler(): + print(result) + except KeyboardInterrupt: + print('\n{green}[√] Stopped.{reset}'.format(**ANSI)) + raise SystemExit(1) + + +@enforce_types +def server(runserver_args: Optional[List[str]]=None, + reload: bool=False, + debug: bool=False, + init: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Run the ArchiveBox HTTP server""" + + runserver_args = runserver_args or [] + + if init: + run_subcommand('init', stdin=None, pwd=out_dir) + + # setup config for django runserver + from . import config + config.SHOW_PROGRESS = False + config.DEBUG = config.DEBUG or debug + + check_data_folder(out_dir=out_dir) + + from django.core.management import call_command + from django.contrib.auth.models import User + + admin_user = User.objects.filter(is_superuser=True).order_by('date_joined').only('username').last() + + print('{green}[+] Starting ArchiveBox webserver...{reset}'.format(**ANSI)) + if admin_user: + hint('The admin username is{lightblue} {}{reset}\n'.format(admin_user.username, **ANSI)) + else: + print('{lightyellow}[!] No admin users exist yet, you will not be able to edit links in the UI.{reset}'.format(**ANSI)) + print() + print(' To create an admin user, run:') + print(' archivebox manage createsuperuser') + print() + + # fallback to serving staticfiles insecurely with django when DEBUG=False + if not config.DEBUG: + runserver_args.append('--insecure') # TODO: serve statics w/ nginx instead + + # toggle autoreloading when archivebox code changes (it's on by default) + if not reload: + runserver_args.append('--noreload') + + config.SHOW_PROGRESS = False + config.DEBUG = config.DEBUG or debug + + + call_command("runserver", *runserver_args) + + +@enforce_types +def manage(args: Optional[List[str]]=None, out_dir: Path=OUTPUT_DIR) -> None: + """Run an ArchiveBox Django management command""" + + check_data_folder(out_dir=out_dir) + from django.core.management import execute_from_command_line + + if (args and "createsuperuser" in args) and (IN_DOCKER and not IS_TTY): + stderr('[!] Warning: you need to pass -it to use interactive commands in docker', color='lightyellow') + stderr(' docker run -it archivebox manage {}'.format(' '.join(args or ['...'])), color='lightyellow') + stderr() + + execute_from_command_line([f'{ARCHIVEBOX_BINARY} manage', *(args or ['help'])]) + + +@enforce_types +def shell(out_dir: Path=OUTPUT_DIR) -> None: + """Enter an interactive ArchiveBox Django shell""" + + check_data_folder(out_dir=out_dir) + + from django.core.management import call_command + call_command("shell_plus") + diff --git a/archivebox-0.5.3/archivebox/manage.py b/archivebox-0.5.3/archivebox/manage.py new file mode 100755 index 0000000..1a9b297 --- /dev/null +++ b/archivebox-0.5.3/archivebox/manage.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python +import os +import sys + +if __name__ == '__main__': + # if you're a developer working on archivebox, still prefer the archivebox + # versions of ./manage.py commands whenever possible. When that's not possible + # (e.g. makemigrations), you can comment out this check temporarily + + if not ('makemigrations' in sys.argv or 'migrate' in sys.argv): + print("[X] Don't run ./manage.py directly (unless you are a developer running makemigrations):") + print() + print(' Hint: Use these archivebox CLI commands instead of the ./manage.py equivalents:') + print(' archivebox init (migrates the databse to latest version)') + print(' archivebox server (runs the Django web server)') + print(' archivebox shell (opens an iPython Django shell with all models imported)') + print(' archivebox manage [cmd] (any other management commands)') + raise SystemExit(2) + + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings') + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) diff --git a/archivebox-0.5.3/archivebox/mypy.ini b/archivebox-0.5.3/archivebox/mypy.ini new file mode 100644 index 0000000..b1b4489 --- /dev/null +++ b/archivebox-0.5.3/archivebox/mypy.ini @@ -0,0 +1,3 @@ +[mypy] +plugins = + mypy_django_plugin.main diff --git a/archivebox-0.5.3/archivebox/package.json b/archivebox-0.5.3/archivebox/package.json new file mode 100644 index 0000000..7f8bf66 --- /dev/null +++ b/archivebox-0.5.3/archivebox/package.json @@ -0,0 +1,21 @@ +{ + "name": "archivebox", + "version": "0.5.3", + "description": "ArchiveBox: The self-hosted internet archive", + "author": "Nick Sweeting <archivebox-npm@sweeting.me>", + "license": "MIT", + "scripts": { + "archivebox": "./bin/archive" + }, + "bin": { + "archivebox-node": "./bin/archive", + "single-file": "./node_modules/.bin/single-file", + "readability-extractor": "./node_modules/.bin/readability-extractor", + "mercury-parser": "./node_modules/.bin/mercury-parser" + }, + "dependencies": { + "@postlight/mercury-parser": "^2.2.0", + "readability-extractor": "git+https://github.com/pirate/readability-extractor.git", + "single-file": "git+https://github.com/gildas-lormeau/SingleFile.git" + } +} diff --git a/archivebox-0.5.3/archivebox/parsers/__init__.py b/archivebox-0.5.3/archivebox/parsers/__init__.py new file mode 100644 index 0000000..441c08a --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/__init__.py @@ -0,0 +1,203 @@ +""" +Everything related to parsing links from input sources. + +For a list of supported services, see the README.md. +For examples of supported import formats see tests/. +""" + +__package__ = 'archivebox.parsers' + +import re +from io import StringIO + +from typing import IO, Tuple, List, Optional +from datetime import datetime +from pathlib import Path + +from ..system import atomic_write +from ..config import ( + ANSI, + OUTPUT_DIR, + SOURCES_DIR_NAME, + TIMEOUT, +) +from ..util import ( + basename, + htmldecode, + download_url, + enforce_types, + URL_REGEX, +) +from ..index.schema import Link +from ..logging_util import TimedProgress, log_source_saved + +from .pocket_html import parse_pocket_html_export +from .pocket_api import parse_pocket_api_export +from .pinboard_rss import parse_pinboard_rss_export +from .wallabag_atom import parse_wallabag_atom_export +from .shaarli_rss import parse_shaarli_rss_export +from .medium_rss import parse_medium_rss_export +from .netscape_html import parse_netscape_html_export +from .generic_rss import parse_generic_rss_export +from .generic_json import parse_generic_json_export +from .generic_html import parse_generic_html_export +from .generic_txt import parse_generic_txt_export + +PARSERS = ( + # Specialized parsers + ('Pocket API', parse_pocket_api_export), + ('Wallabag ATOM', parse_wallabag_atom_export), + ('Pocket HTML', parse_pocket_html_export), + ('Pinboard RSS', parse_pinboard_rss_export), + ('Shaarli RSS', parse_shaarli_rss_export), + ('Medium RSS', parse_medium_rss_export), + + # General parsers + ('Netscape HTML', parse_netscape_html_export), + ('Generic RSS', parse_generic_rss_export), + ('Generic JSON', parse_generic_json_export), + ('Generic HTML', parse_generic_html_export), + + # Fallback parser + ('Plain Text', parse_generic_txt_export), +) + + +@enforce_types +def parse_links_memory(urls: List[str], root_url: Optional[str]=None): + """ + parse a list of URLS without touching the filesystem + """ + check_url_parsing_invariants() + + timer = TimedProgress(TIMEOUT * 4) + #urls = list(map(lambda x: x + "\n", urls)) + file = StringIO() + file.writelines(urls) + file.name = "io_string" + links, parser = run_parser_functions(file, timer, root_url=root_url) + timer.end() + + if parser is None: + return [], 'Failed to parse' + return links, parser + + +@enforce_types +def parse_links(source_file: str, root_url: Optional[str]=None) -> Tuple[List[Link], str]: + """parse a list of URLs with their metadata from an + RSS feed, bookmarks export, or text file + """ + + check_url_parsing_invariants() + + timer = TimedProgress(TIMEOUT * 4) + with open(source_file, 'r', encoding='utf-8') as file: + links, parser = run_parser_functions(file, timer, root_url=root_url) + + timer.end() + if parser is None: + return [], 'Failed to parse' + return links, parser + + +def run_parser_functions(to_parse: IO[str], timer, root_url: Optional[str]=None) -> Tuple[List[Link], Optional[str]]: + most_links: List[Link] = [] + best_parser_name = None + + for parser_name, parser_func in PARSERS: + try: + parsed_links = list(parser_func(to_parse, root_url=root_url)) + if not parsed_links: + raise Exception('no links found') + + # print(f'[√] Parser {parser_name} succeeded: {len(parsed_links)} links parsed') + if len(parsed_links) > len(most_links): + most_links = parsed_links + best_parser_name = parser_name + + except Exception as err: # noqa + # Parsers are tried one by one down the list, and the first one + # that succeeds is used. To see why a certain parser was not used + # due to error or format incompatibility, uncomment this line: + + # print('[!] Parser {} failed: {} {}'.format(parser_name, err.__class__.__name__, err)) + # raise + pass + timer.end() + return most_links, best_parser_name + + +@enforce_types +def save_text_as_source(raw_text: str, filename: str='{ts}-stdin.txt', out_dir: Path=OUTPUT_DIR) -> str: + ts = str(datetime.now().timestamp()).split('.', 1)[0] + source_path = str(out_dir / SOURCES_DIR_NAME / filename.format(ts=ts)) + atomic_write(source_path, raw_text) + log_source_saved(source_file=source_path) + return source_path + + +@enforce_types +def save_file_as_source(path: str, timeout: int=TIMEOUT, filename: str='{ts}-{basename}.txt', out_dir: Path=OUTPUT_DIR) -> str: + """download a given url's content into output/sources/domain-<timestamp>.txt""" + ts = str(datetime.now().timestamp()).split('.', 1)[0] + source_path = str(OUTPUT_DIR / SOURCES_DIR_NAME / filename.format(basename=basename(path), ts=ts)) + + if any(path.startswith(s) for s in ('http://', 'https://', 'ftp://')): + # Source is a URL that needs to be downloaded + print(f' > Downloading {path} contents') + timer = TimedProgress(timeout, prefix=' ') + try: + raw_source_text = download_url(path, timeout=timeout) + raw_source_text = htmldecode(raw_source_text) + timer.end() + except Exception as e: + timer.end() + print('{}[!] Failed to download {}{}\n'.format( + ANSI['red'], + path, + ANSI['reset'], + )) + print(' ', e) + raise SystemExit(1) + + else: + # Source is a path to a local file on the filesystem + with open(path, 'r') as f: + raw_source_text = f.read() + + atomic_write(source_path, raw_source_text) + + log_source_saved(source_file=source_path) + + return source_path + + +def check_url_parsing_invariants() -> None: + """Check that plain text regex URL parsing works as expected""" + + # this is last-line-of-defense to make sure the URL_REGEX isn't + # misbehaving, as the consequences could be disastrous and lead to many + # incorrect/badly parsed links being added to the archive + + test_urls = ''' + https://example1.com/what/is/happening.html?what=1#how-about-this=1 + https://example2.com/what/is/happening/?what=1#how-about-this=1 + HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f + https://example4.com/what/is/happening.html + https://example5.com/ + https://example6.com + + <test>http://example7.com</test> + [https://example8.com/what/is/this.php?what=1] + [and http://example9.com?what=1&other=3#and-thing=2] + <what>https://example10.com#and-thing=2 "</about> + abc<this["https://example11.com/what/is#and-thing=2?whoami=23&where=1"]that>def + sdflkf[what](https://example12.com/who/what.php?whoami=1#whatami=2)?am=hi + example13.bada + and example14.badb + <or>htt://example15.badc</that> + ''' + # print('\n'.join(re.findall(URL_REGEX, test_urls))) + assert len(re.findall(URL_REGEX, test_urls)) == 12 + diff --git a/archivebox-0.5.3/archivebox/parsers/generic_html.py b/archivebox-0.5.3/archivebox/parsers/generic_html.py new file mode 100644 index 0000000..74b3d1f --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/generic_html.py @@ -0,0 +1,53 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable, Optional +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + URL_REGEX, +) +from html.parser import HTMLParser +from urllib.parse import urljoin + + +class HrefParser(HTMLParser): + def __init__(self): + super().__init__() + self.urls = [] + + def handle_starttag(self, tag, attrs): + if tag == "a": + for attr, value in attrs: + if attr == "href": + self.urls.append(value) + + +@enforce_types +def parse_generic_html_export(html_file: IO[str], root_url: Optional[str]=None, **_kwargs) -> Iterable[Link]: + """Parse Generic HTML for href tags and use only the url (support for title coming later)""" + + html_file.seek(0) + for line in html_file: + parser = HrefParser() + # example line + # <li><a href="http://example.com/ time_added="1478739709" tags="tag1,tag2">example title</a></li> + parser.feed(line) + for url in parser.urls: + if root_url: + # resolve relative urls /home.html -> https://example.com/home.html + url = urljoin(root_url, url) + + for archivable_url in re.findall(URL_REGEX, url): + yield Link( + url=htmldecode(archivable_url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[html_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/parsers/generic_json.py b/archivebox-0.5.3/archivebox/parsers/generic_json.py new file mode 100644 index 0000000..e6ed677 --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/generic_json.py @@ -0,0 +1,65 @@ +__package__ = 'archivebox.parsers' + +import json + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_generic_json_export(json_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)""" + + json_file.seek(0) + links = json.load(json_file) + json_date = lambda s: datetime.strptime(s, '%Y-%m-%dT%H:%M:%S%z') + + for link in links: + # example line + # {"href":"http:\/\/www.reddit.com\/r\/example","description":"title here","extended":"","meta":"18a973f09c9cc0608c116967b64e0419","hash":"910293f019c2f4bb1a749fb937ba58e3","time":"2014-06-14T15:51:42Z","shared":"no","toread":"no","tags":"reddit android"}] + if link: + # Parse URL + url = link.get('href') or link.get('url') or link.get('URL') + if not url: + raise Exception('JSON must contain URL in each entry [{"url": "http://...", ...}, ...]') + + # Parse the timestamp + ts_str = str(datetime.now().timestamp()) + if link.get('timestamp'): + # chrome/ff histories use a very precise timestamp + ts_str = str(link['timestamp'] / 10000000) + elif link.get('time'): + ts_str = str(json_date(link['time'].split(',', 1)[0]).timestamp()) + elif link.get('created_at'): + ts_str = str(json_date(link['created_at']).timestamp()) + elif link.get('created'): + ts_str = str(json_date(link['created']).timestamp()) + elif link.get('date'): + ts_str = str(json_date(link['date']).timestamp()) + elif link.get('bookmarked'): + ts_str = str(json_date(link['bookmarked']).timestamp()) + elif link.get('saved'): + ts_str = str(json_date(link['saved']).timestamp()) + + # Parse the title + title = None + if link.get('title'): + title = link['title'].strip() + elif link.get('description'): + title = link['description'].replace(' — Readability', '').strip() + elif link.get('name'): + title = link['name'].strip() + + yield Link( + url=htmldecode(url), + timestamp=ts_str, + title=htmldecode(title) or None, + tags=htmldecode(link.get('tags')) or '', + sources=[json_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/parsers/generic_rss.py b/archivebox-0.5.3/archivebox/parsers/generic_rss.py new file mode 100644 index 0000000..2831844 --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/generic_rss.py @@ -0,0 +1,49 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + +@enforce_types +def parse_generic_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse RSS XML-format files into links""" + + rss_file.seek(0) + items = rss_file.read().split('<item>') + items = items[1:] if items else [] + for item in items: + # example item: + # <item> + # <title><![CDATA[How JavaScript works: inside the V8 engine]]> + # Unread + # https://blog.sessionstack.com/how-javascript-works-inside + # https://blog.sessionstack.com/how-javascript-works-inside + # Mon, 21 Aug 2017 14:21:58 -0500 + # + + trailing_removed = item.split('', 1)[0] + leading_removed = trailing_removed.split('', 1)[-1].strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r for r in rows if r.strip().startswith('<{}>'.format(key))][0] + + url = str_between(get_row('link'), '', '') + ts_str = str_between(get_row('pubDate'), '', '') + time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %z") + title = str_between(get_row('title'), ' Iterable[Link]: + """Parse raw links from each line in a text file""" + + text_file.seek(0) + for line in text_file.readlines(): + if not line.strip(): + continue + + # if the line is a local file path that resolves, then we can archive it + try: + if Path(line).exists(): + yield Link( + url=line, + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) + except (OSError, PermissionError): + # nvm, not a valid path... + pass + + # otherwise look for anything that looks like a URL in the line + for url in re.findall(URL_REGEX, line): + yield Link( + url=htmldecode(url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) + + # look inside the URL for any sub-urls, e.g. for archive.org links + # https://web.archive.org/web/20200531203453/https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/ + # -> https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/ + for url in re.findall(URL_REGEX, line[1:]): + yield Link( + url=htmldecode(url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/parsers/medium_rss.py b/archivebox-0.5.3/archivebox/parsers/medium_rss.py new file mode 100644 index 0000000..8f14f77 --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/medium_rss.py @@ -0,0 +1,35 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from xml.etree import ElementTree + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_medium_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Medium RSS feed files into links""" + + rss_file.seek(0) + root = ElementTree.parse(rss_file).getroot() + items = root.find("channel").findall("item") # type: ignore + for item in items: + url = item.find("link").text # type: ignore + title = item.find("title").text.strip() # type: ignore + ts_str = item.find("pubDate").text # type: ignore + time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %Z") # type: ignore + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/parsers/netscape_html.py b/archivebox-0.5.3/archivebox/parsers/netscape_html.py new file mode 100644 index 0000000..a063023 --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/netscape_html.py @@ -0,0 +1,39 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_netscape_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse netscape-format bookmarks export files (produced by all browsers)""" + + html_file.seek(0) + pattern = re.compile("]*>(.+)", re.UNICODE | re.IGNORECASE) + for line in html_file: + # example line + #
example bookmark title + + match = pattern.search(line) + if match: + url = match.group(1) + time = datetime.fromtimestamp(float(match.group(2))) + title = match.group(3).strip() + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[html_file.name], + ) + diff --git a/archivebox-0.5.3/archivebox/parsers/pinboard_rss.py b/archivebox-0.5.3/archivebox/parsers/pinboard_rss.py new file mode 100644 index 0000000..98ff14a --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/pinboard_rss.py @@ -0,0 +1,47 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from xml.etree import ElementTree + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_pinboard_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Pinboard RSS feed files into links""" + + rss_file.seek(0) + root = ElementTree.parse(rss_file).getroot() + items = root.findall("{http://purl.org/rss/1.0/}item") + for item in items: + find = lambda p: item.find(p).text.strip() if item.find(p) else None # type: ignore + + url = find("{http://purl.org/rss/1.0/}link") + tags = find("{http://purl.org/dc/elements/1.1/}subject") + title = find("{http://purl.org/rss/1.0/}title") + ts_str = find("{http://purl.org/dc/elements/1.1/}date") + + # Pinboard includes a colon in its date stamp timezone offsets, which + # Python can't parse. Remove it: + if ts_str and ts_str[-3:-2] == ":": + ts_str = ts_str[:-3]+ts_str[-2:] + + if ts_str: + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + else: + time = datetime.now() + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=htmldecode(tags) or None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/parsers/pocket_api.py b/archivebox-0.5.3/archivebox/parsers/pocket_api.py new file mode 100644 index 0000000..bf3a292 --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/pocket_api.py @@ -0,0 +1,113 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable, Optional +from configparser import ConfigParser + +from pathlib import Path +from ..vendor.pocket import Pocket + +from ..index.schema import Link +from ..util import enforce_types +from ..system import atomic_write +from ..config import ( + SOURCES_DIR, + POCKET_CONSUMER_KEY, + POCKET_ACCESS_TOKENS, +) + + +COUNT_PER_PAGE = 500 +API_DB_PATH = Path(SOURCES_DIR) / 'pocket_api.db' + +# search for broken protocols that sometimes come from the Pocket API +_BROKEN_PROTOCOL_RE = re.compile('^(http[s]?)(:/(?!/))') + + +def get_pocket_articles(api: Pocket, since=None, page=0): + body, headers = api.get( + state='archive', + sort='oldest', + since=since, + count=COUNT_PER_PAGE, + offset=page * COUNT_PER_PAGE, + ) + + articles = body['list'].values() if isinstance(body['list'], dict) else body['list'] + returned_count = len(articles) + + yield from articles + + if returned_count == COUNT_PER_PAGE: + yield from get_pocket_articles(api, since=since, page=page + 1) + else: + api.last_since = body['since'] + + +def link_from_article(article: dict, sources: list): + url: str = article['resolved_url'] or article['given_url'] + broken_protocol = _BROKEN_PROTOCOL_RE.match(url) + if broken_protocol: + url = url.replace(f'{broken_protocol.group(1)}:/', f'{broken_protocol.group(1)}://') + title = article['resolved_title'] or article['given_title'] or url + + return Link( + url=url, + timestamp=article['time_read'], + title=title, + tags=article.get('tags'), + sources=sources + ) + + +def write_since(username: str, since: str): + if not API_DB_PATH.exists(): + atomic_write(API_DB_PATH, '') + + since_file = ConfigParser() + since_file.optionxform = str + since_file.read(API_DB_PATH) + + since_file[username] = { + 'since': since + } + + with open(API_DB_PATH, 'w+') as new: + since_file.write(new) + + +def read_since(username: str) -> Optional[str]: + if not API_DB_PATH.exists(): + atomic_write(API_DB_PATH, '') + + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(API_DB_PATH) + + return config_file.get(username, 'since', fallback=None) + + +@enforce_types +def should_parse_as_pocket_api(text: str) -> bool: + return text.startswith('pocket://') + + +@enforce_types +def parse_pocket_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Link]: + """Parse bookmarks from the Pocket API""" + + input_buffer.seek(0) + pattern = re.compile(r"^pocket:\/\/(\w+)") + for line in input_buffer: + if should_parse_as_pocket_api(line): + + username = pattern.search(line).group(1) + api = Pocket(POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKENS[username]) + api.last_since = None + + for article in get_pocket_articles(api, since=read_since(username)): + yield link_from_article(article, sources=[line]) + + write_since(username, api.last_since) diff --git a/archivebox-0.5.3/archivebox/parsers/pocket_html.py b/archivebox-0.5.3/archivebox/parsers/pocket_html.py new file mode 100644 index 0000000..653f21b --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/pocket_html.py @@ -0,0 +1,38 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_pocket_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)""" + + html_file.seek(0) + pattern = re.compile("^\\s*
  • (.+)
    • ", re.UNICODE) + for line in html_file: + # example line + #
  • example title
    • + match = pattern.search(line) + if match: + url = match.group(1).replace('http://www.readability.com/read?url=', '') # remove old readability prefixes to get original url + time = datetime.fromtimestamp(float(match.group(2))) + tags = match.group(3) + title = match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=tags or '', + sources=[html_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/parsers/shaarli_rss.py b/archivebox-0.5.3/archivebox/parsers/shaarli_rss.py new file mode 100644 index 0000000..4a925f4 --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/shaarli_rss.py @@ -0,0 +1,50 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + + +@enforce_types +def parse_shaarli_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Shaarli-specific RSS XML-format files into links""" + + rss_file.seek(0) + entries = rss_file.read().split('')[1:] + for entry in entries: + # example entry: + # + # Aktuelle Trojaner-Welle: Emotet lauert in gefälschten Rechnungsmails | heise online + # + # https://demo.shaarli.org/?cEV4vw + # 2019-01-30T06:06:01+00:00 + # 2019-01-30T06:06:01+00:00 + #

    — Permalink

    ]]>     + #     + + trailing_removed = entry.split('    ', 1)[0] + leading_removed = trailing_removed.strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0] + + title = str_between(get_row('title'), '', '').strip() + url = str_between(get_row('link'), '') + ts_str = str_between(get_row('published'), '', '') + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/parsers/wallabag_atom.py b/archivebox-0.5.3/archivebox/parsers/wallabag_atom.py new file mode 100644 index 0000000..0d77869 --- /dev/null +++ b/archivebox-0.5.3/archivebox/parsers/wallabag_atom.py @@ -0,0 +1,57 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + + +@enforce_types +def parse_wallabag_atom_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Wallabag Atom files into links""" + + rss_file.seek(0) + entries = rss_file.read().split('')[1:] + for entry in entries: + # example entry: + # + # <![CDATA[Orient Ray vs Mako: Is There Much Difference? - iknowwatches.com]]> + # + # https://iknowwatches.com/orient-ray-vs-mako/ + # wallabag:wallabag.drycat.fr:milosh:entry:14041 + # 2020-10-18T09:14:02+02:00 + # 2020-10-18T09:13:56+02:00 + # + # + # + + trailing_removed = entry.split('', 1)[0] + leading_removed = trailing_removed.strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0] + + title = str_between(get_row('title'), '<![CDATA[', ']]>').strip() + url = str_between(get_row('link rel="via"'), '', '') + ts_str = str_between(get_row('published'), '', '') + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + try: + tags = str_between(get_row('category'), 'label="', '" />') + except: + tags = None + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=tags or '', + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/archivebox/search/__init__.py b/archivebox-0.5.3/archivebox/search/__init__.py new file mode 100644 index 0000000..6191ede --- /dev/null +++ b/archivebox-0.5.3/archivebox/search/__init__.py @@ -0,0 +1,108 @@ +from typing import List, Union +from pathlib import Path +from importlib import import_module + +from django.db.models import QuerySet + +from archivebox.index.schema import Link +from archivebox.util import enforce_types +from archivebox.config import stderr, OUTPUT_DIR, USE_INDEXING_BACKEND, USE_SEARCHING_BACKEND, SEARCH_BACKEND_ENGINE + +from .utils import get_indexable_content, log_index_started + +def indexing_enabled(): + return USE_INDEXING_BACKEND + +def search_backend_enabled(): + return USE_SEARCHING_BACKEND + +def get_backend(): + return f'search.backends.{SEARCH_BACKEND_ENGINE}' + +def import_backend(): + backend_string = get_backend() + try: + backend = import_module(backend_string) + except Exception as err: + raise Exception("Could not load '%s' as a backend: %s" % (backend_string, err)) + return backend + +@enforce_types +def write_search_index(link: Link, texts: Union[List[str], None]=None, out_dir: Path=OUTPUT_DIR, skip_text_index: bool=False) -> None: + if not indexing_enabled(): + return + + if not skip_text_index and texts: + from core.models import Snapshot + + snap = Snapshot.objects.filter(url=link.url).first() + backend = import_backend() + if snap: + try: + backend.index(snapshot_id=str(snap.id), texts=texts) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + +@enforce_types +def query_search_index(query: str, out_dir: Path=OUTPUT_DIR) -> QuerySet: + from core.models import Snapshot + + if search_backend_enabled(): + backend = import_backend() + try: + snapshot_ids = backend.search(query) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + raise + else: + # TODO preserve ordering from backend + qsearch = Snapshot.objects.filter(pk__in=snapshot_ids) + return qsearch + + return Snapshot.objects.none() + +@enforce_types +def flush_search_index(snapshots: QuerySet): + if not indexing_enabled() or not snapshots: + return + backend = import_backend() + snapshot_ids=(str(pk) for pk in snapshots.values_list('pk',flat=True)) + try: + backend.flush(snapshot_ids) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + +@enforce_types +def index_links(links: Union[List[Link],None], out_dir: Path=OUTPUT_DIR): + if not links: + return + + from core.models import Snapshot, ArchiveResult + + for link in links: + snap = Snapshot.objects.filter(url=link.url).first() + if snap: + results = ArchiveResult.objects.indexable().filter(snapshot=snap) + log_index_started(link.url) + try: + texts = get_indexable_content(results) + except Exception as err: + stderr() + stderr( + f'[X] An Exception ocurred reading the indexable content={err}:', + color='red', + ) + else: + write_search_index(link, texts, out_dir=out_dir) diff --git a/archivebox-0.5.3/archivebox/search/backends/__init__.py b/archivebox-0.5.3/archivebox/search/backends/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/archivebox/search/backends/ripgrep.py b/archivebox-0.5.3/archivebox/search/backends/ripgrep.py new file mode 100644 index 0000000..840d2d2 --- /dev/null +++ b/archivebox-0.5.3/archivebox/search/backends/ripgrep.py @@ -0,0 +1,45 @@ +import re +from subprocess import run, PIPE +from typing import List, Generator + +from archivebox.config import ARCHIVE_DIR, RIPGREP_VERSION +from archivebox.util import enforce_types + +RG_IGNORE_EXTENSIONS = ('css','js','orig','svg') + +RG_ADD_TYPE = '--type-add' +RG_IGNORE_ARGUMENTS = f"ignore:*.{{{','.join(RG_IGNORE_EXTENSIONS)}}}" +RG_DEFAULT_ARGUMENTS = "-ilTignore" # Case insensitive(i), matching files results(l) +RG_REGEX_ARGUMENT = '-e' + +TIMESTAMP_REGEX = r'\/([\d]+\.[\d]+)\/' + +ts_regex = re.compile(TIMESTAMP_REGEX) + +@enforce_types +def index(snapshot_id: str, texts: List[str]): + return + +@enforce_types +def flush(snapshot_ids: Generator[str, None, None]): + return + +@enforce_types +def search(text: str) -> List[str]: + if not RIPGREP_VERSION: + raise Exception("ripgrep binary not found, install ripgrep to use this search backend") + + from core.models import Snapshot + + rg_cmd = ['rg', RG_ADD_TYPE, RG_IGNORE_ARGUMENTS, RG_DEFAULT_ARGUMENTS, RG_REGEX_ARGUMENT, text, str(ARCHIVE_DIR)] + rg = run(rg_cmd, stdout=PIPE, stderr=PIPE, timeout=60) + file_paths = [p.decode() for p in rg.stdout.splitlines()] + timestamps = set() + for path in file_paths: + ts = ts_regex.findall(path) + if ts: + timestamps.add(ts[0]) + + snap_ids = [str(id) for id in Snapshot.objects.filter(timestamp__in=timestamps).values_list('pk', flat=True)] + + return snap_ids diff --git a/archivebox-0.5.3/archivebox/search/backends/sonic.py b/archivebox-0.5.3/archivebox/search/backends/sonic.py new file mode 100644 index 0000000..f0beadd --- /dev/null +++ b/archivebox-0.5.3/archivebox/search/backends/sonic.py @@ -0,0 +1,28 @@ +from typing import List, Generator + +from sonic import IngestClient, SearchClient + +from archivebox.util import enforce_types +from archivebox.config import SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD, SONIC_BUCKET, SONIC_COLLECTION + +MAX_SONIC_TEXT_LENGTH = 20000 + +@enforce_types +def index(snapshot_id: str, texts: List[str]): + with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl: + for text in texts: + chunks = [text[i:i+MAX_SONIC_TEXT_LENGTH] for i in range(0, len(text), MAX_SONIC_TEXT_LENGTH)] + for chunk in chunks: + ingestcl.push(SONIC_COLLECTION, SONIC_BUCKET, snapshot_id, str(chunk)) + +@enforce_types +def search(text: str) -> List[str]: + with SearchClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as querycl: + snap_ids = querycl.query(SONIC_COLLECTION, SONIC_BUCKET, text) + return snap_ids + +@enforce_types +def flush(snapshot_ids: Generator[str, None, None]): + with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl: + for id in snapshot_ids: + ingestcl.flush_object(SONIC_COLLECTION, SONIC_BUCKET, str(id)) diff --git a/archivebox-0.5.3/archivebox/search/utils.py b/archivebox-0.5.3/archivebox/search/utils.py new file mode 100644 index 0000000..55c97e7 --- /dev/null +++ b/archivebox-0.5.3/archivebox/search/utils.py @@ -0,0 +1,44 @@ +from django.db.models import QuerySet + +from archivebox.util import enforce_types +from archivebox.config import ANSI + +def log_index_started(url): + print('{green}[*] Indexing url: {} in the search index {reset}'.format(url, **ANSI)) + print( ) + +def get_file_result_content(res, extra_path, use_pwd=False): + if use_pwd: + fpath = f'{res.pwd}/{res.output}' + else: + fpath = f'{res.output}' + + if extra_path: + fpath = f'{fpath}/{extra_path}' + + with open(fpath, 'r') as file: + data = file.read() + if data: + return [data] + return [] + + +# This should be abstracted by a plugin interface for extractors +@enforce_types +def get_indexable_content(results: QuerySet): + if not results: + return [] + # Only use the first method available + res, method = results.first(), results.first().extractor + if method not in ('readability', 'singlefile', 'dom', 'wget'): + return [] + # This should come from a plugin interface + + if method == 'readability': + return get_file_result_content(res, 'content.txt') + elif method == 'singlefile': + return get_file_result_content(res, '') + elif method == 'dom': + return get_file_result_content(res,'',use_pwd=True) + elif method == 'wget': + return get_file_result_content(res,'',use_pwd=True) diff --git a/archivebox-0.5.3/archivebox/system.py b/archivebox-0.5.3/archivebox/system.py new file mode 100644 index 0000000..b27c5e4 --- /dev/null +++ b/archivebox-0.5.3/archivebox/system.py @@ -0,0 +1,163 @@ +__package__ = 'archivebox' + + +import os +import shutil + +from json import dump +from pathlib import Path +from typing import Optional, Union, Set, Tuple +from subprocess import run as subprocess_run + +from crontab import CronTab +from atomicwrites import atomic_write as lib_atomic_write + +from .util import enforce_types, ExtendedEncoder +from .config import OUTPUT_PERMISSIONS + + + +def run(*args, input=None, capture_output=True, text=False, **kwargs): + """Patched of subprocess.run to fix blocking io making timeout=innefective""" + + if input is not None: + if 'stdin' in kwargs: + raise ValueError('stdin and input arguments may not both be used.') + + if capture_output: + if ('stdout' in kwargs) or ('stderr' in kwargs): + raise ValueError('stdout and stderr arguments may not be used ' + 'with capture_output.') + + return subprocess_run(*args, input=input, capture_output=capture_output, text=text, **kwargs) + + +@enforce_types +def atomic_write(path: Union[Path, str], contents: Union[dict, str, bytes], overwrite: bool=True) -> None: + """Safe atomic write to filesystem by writing to temp file + atomic rename""" + + mode = 'wb+' if isinstance(contents, bytes) else 'w' + + # print('\n> Atomic Write:', mode, path, len(contents), f'overwrite={overwrite}') + try: + with lib_atomic_write(path, mode=mode, overwrite=overwrite) as f: + if isinstance(contents, dict): + dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder) + elif isinstance(contents, (bytes, str)): + f.write(contents) + except OSError as e: + print(f"[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC. ({e})") + print(" For data integrity, ArchiveBox requires a filesystem that supports atomic writes.") + print(" Filesystems and network drives that don't implement FSYNC are incompatible and require workarounds.") + raise SystemExit(1) + os.chmod(path, int(OUTPUT_PERMISSIONS, base=8)) + +@enforce_types +def chmod_file(path: str, cwd: str='.', permissions: str=OUTPUT_PERMISSIONS) -> None: + """chmod -R /""" + + root = Path(cwd) / path + if not root.exists(): + raise Exception('Failed to chmod: {} does not exist (did the previous step fail?)'.format(path)) + + if not root.is_dir(): + os.chmod(root, int(OUTPUT_PERMISSIONS, base=8)) + else: + for subpath in Path(path).glob('**/*'): + os.chmod(subpath, int(OUTPUT_PERMISSIONS, base=8)) + + +@enforce_types +def copy_and_overwrite(from_path: Union[str, Path], to_path: Union[str, Path]): + """copy a given file or directory to a given path, overwriting the destination""" + if Path(from_path).is_dir(): + shutil.rmtree(to_path, ignore_errors=True) + shutil.copytree(from_path, to_path) + else: + with open(from_path, 'rb') as src: + contents = src.read() + atomic_write(to_path, contents) + + +@enforce_types +def get_dir_size(path: Union[str, Path], recursive: bool=True, pattern: Optional[str]=None) -> Tuple[int, int, int]: + """get the total disk size of a given directory, optionally summing up + recursively and limiting to a given filter list + """ + num_bytes, num_dirs, num_files = 0, 0, 0 + for entry in os.scandir(path): + if (pattern is not None) and (pattern not in entry.path): + continue + if entry.is_dir(follow_symlinks=False): + if not recursive: + continue + num_dirs += 1 + bytes_inside, dirs_inside, files_inside = get_dir_size(entry.path) + num_bytes += bytes_inside + num_dirs += dirs_inside + num_files += files_inside + else: + num_bytes += entry.stat(follow_symlinks=False).st_size + num_files += 1 + return num_bytes, num_dirs, num_files + + +CRON_COMMENT = 'archivebox_schedule' + + +@enforce_types +def dedupe_cron_jobs(cron: CronTab) -> CronTab: + deduped: Set[Tuple[str, str]] = set() + + for job in list(cron): + unique_tuple = (str(job.slices), job.command) + if unique_tuple not in deduped: + deduped.add(unique_tuple) + cron.remove(job) + + for schedule, command in deduped: + job = cron.new(command=command, comment=CRON_COMMENT) + job.setall(schedule) + job.enable() + + return cron + + +class suppress_output(object): + ''' + A context manager for doing a "deep suppression" of stdout and stderr in + Python, i.e. will suppress all print, even if the print originates in a + compiled C/Fortran sub-function. + This will not suppress raised exceptions, since exceptions are printed + to stderr just before a script exits, and after the context manager has + exited (at least, I think that is why it lets exceptions through). + + with suppress_stdout_stderr(): + rogue_function() + ''' + def __init__(self, stdout=True, stderr=True): + # Open a pair of null files + # Save the actual stdout (1) and stderr (2) file descriptors. + self.stdout, self.stderr = stdout, stderr + if stdout: + self.null_stdout = os.open(os.devnull, os.O_RDWR) + self.real_stdout = os.dup(1) + if stderr: + self.null_stderr = os.open(os.devnull, os.O_RDWR) + self.real_stderr = os.dup(2) + + def __enter__(self): + # Assign the null pointers to stdout and stderr. + if self.stdout: + os.dup2(self.null_stdout, 1) + if self.stderr: + os.dup2(self.null_stderr, 2) + + def __exit__(self, *_): + # Re-assign the real stdout/stderr back to (1) and (2) + if self.stdout: + os.dup2(self.real_stdout, 1) + os.close(self.null_stdout) + if self.stderr: + os.dup2(self.real_stderr, 2) + os.close(self.null_stderr) diff --git a/archivebox-0.5.3/archivebox/themes/admin/actions_as_select.html b/archivebox-0.5.3/archivebox/themes/admin/actions_as_select.html new file mode 100644 index 0000000..86a7719 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/admin/actions_as_select.html @@ -0,0 +1 @@ +actions_as_select diff --git a/archivebox-0.5.3/archivebox/themes/admin/app_index.html b/archivebox-0.5.3/archivebox/themes/admin/app_index.html new file mode 100644 index 0000000..6868b49 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/admin/app_index.html @@ -0,0 +1,18 @@ +{% extends "admin/index.html" %} +{% load i18n %} + +{% block bodyclass %}{{ block.super }} app-{{ app_label }}{% endblock %} + +{% if not is_popup %} +{% block breadcrumbs %} +
    +{% trans 'Home' %} +› +{% for app in app_list %} +{{ app.name }} +{% endfor %} +
    +{% endblock %} +{% endif %} + +{% block sidebar %}{% endblock %} diff --git a/archivebox-0.5.3/archivebox/themes/admin/base.html b/archivebox-0.5.3/archivebox/themes/admin/base.html new file mode 100644 index 0000000..d8ad8d0 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/admin/base.html @@ -0,0 +1,246 @@ +{% load i18n static %} +{% get_current_language as LANGUAGE_CODE %}{% get_current_language_bidi as LANGUAGE_BIDI %} + + +{% block title %}{% endblock %} | ArchiveBox + +{% block extrastyle %}{% endblock %} +{% if LANGUAGE_BIDI %}{% endif %} +{% block extrahead %}{% endblock %} +{% block responsive %} + + + {% if LANGUAGE_BIDI %}{% endif %} +{% endblock %} +{% block blockbots %}{% endblock %} + + +{% load i18n %} + + + + + + + + + +
    + + {% if not is_popup %} + + + + {% block breadcrumbs %} +
    + {% trans 'Home' %} + {% if title %} › {{ title }}{% endif %} +
    + {% endblock %} + {% endif %} + + {% block messages %} + {% if messages %} +
      {% for message in messages %} + {{ message|capfirst }} + {% endfor %}
    + {% endif %} + {% endblock messages %} + + +
    + {% block pretitle %}{% endblock %} + {% block content_title %}{# {% if title %}

    {{ title }}

    {% endif %} #}{% endblock %} + {% block content %} + {% block object-tools %}{% endblock %} + {{ content }} + {% endblock %} + {% block sidebar %}{% endblock %} +
    +
    + + + {% block footer %}{% endblock %} +
    + + + + + diff --git a/archivebox-0.5.3/archivebox/themes/admin/grid_change_list.html b/archivebox-0.5.3/archivebox/themes/admin/grid_change_list.html new file mode 100644 index 0000000..6894efd --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/admin/grid_change_list.html @@ -0,0 +1,91 @@ +{% extends "admin/base_site.html" %} +{% load i18n admin_urls static admin_list %} +{% load core_tags %} + +{% block extrastyle %} + {{ block.super }} + + {% if cl.formset %} + + {% endif %} + {% if cl.formset or action_form %} + + {% endif %} + {{ media.css }} + {% if not actions_on_top and not actions_on_bottom %} + + {% endif %} +{% endblock %} + +{% block extrahead %} +{{ block.super }} +{{ media.js }} +{% endblock %} + +{% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-list{% endblock %} + +{% if not is_popup %} +{% block breadcrumbs %} +
    +{% translate 'Home' %} +› {{ cl.opts.app_config.verbose_name }} +› {{ cl.opts.verbose_name_plural|capfirst }} +
    +{% endblock %} +{% endif %} + +{% block coltype %}{% endblock %} + +{% block content %} +
    + {% block object-tools %} +
      + {% block object-tools-items %} + {% change_list_object_tools %} + {% endblock %} +
    + {% endblock %} + {% if cl.formset and cl.formset.errors %} +

    + {% if cl.formset.total_error_count == 1 %}{% translate "Please correct the error below." %}{% else %}{% translate "Please correct the errors below." %}{% endif %} +

    + {{ cl.formset.non_form_errors }} + {% endif %} +
    +
    + {% block search %}{% search_form cl %}{% endblock %} + {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %} + +
    {% csrf_token %} + {% if cl.formset %} +
    {{ cl.formset.management_form }}
    + {% endif %} + + {% block result_list %} + {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %} + {% comment %} + Table grid + {% result_list cl %} + {% endcomment %} + {% snapshots_grid cl %} + {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %} + {% endblock %} + {% block pagination %}{% pagination cl %}{% endblock %} +
    +
    + {% block filters %} + {% if cl.has_filters %} +
    +

    {% translate 'Filter' %}

    + {% if cl.has_active_filters %}

    + ✖ {% translate "Clear all filters" %} +

    {% endif %} + {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %} +
    + {% endif %} + {% endblock %} +
    +
    +{% endblock %} \ No newline at end of file diff --git a/archivebox-0.5.3/archivebox/themes/admin/login.html b/archivebox-0.5.3/archivebox/themes/admin/login.html new file mode 100644 index 0000000..98283f8 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/admin/login.html @@ -0,0 +1,100 @@ +{% extends "admin/base_site.html" %} +{% load i18n static %} + +{% block extrastyle %}{{ block.super }} +{{ form.media }} +{% endblock %} + +{% block bodyclass %}{{ block.super }} login{% endblock %} + +{% block branding %}

    ArchiveBox Admin

    {% endblock %} + +{% block usertools %} +
    + Back to Main Index +{% endblock %} + +{% block nav-global %}{% endblock %} + +{% block content_title %} +
    + Log in to add, edit, and remove links from your archive. +


    +
    +{% endblock %} + +{% block breadcrumbs %}{% endblock %} + +{% block content %} +{% if form.errors and not form.non_field_errors %} +

    +{% if form.errors.items|length == 1 %}{% trans "Please correct the error below." %}{% else %}{% trans "Please correct the errors below." %}{% endif %} +

    +{% endif %} + +{% if form.non_field_errors %} +{% for error in form.non_field_errors %} +

    + {{ error }} +

    +{% endfor %} +{% endif %} + +
    + +{% if user.is_authenticated %} +

    +{% blocktrans trimmed %} + You are authenticated as {{ username }}, but are not authorized to + access this page. Would you like to login to a different account? +{% endblocktrans %} +

    +{% endif %} + +
    +
    {% csrf_token %} +
    + {{ form.username.errors }} + {{ form.username.label_tag }} {{ form.username }} +
    +
    + {{ form.password.errors }} + {{ form.password.label_tag }} {{ form.password }} + +
    + {% url 'admin_password_reset' as password_reset_url %} + {% if password_reset_url %} +
    + {% trans 'Forgotten your password or username?' %} +
    + {% endif %} +
    + +
    +
    + +
    +

    +
    +
    + If you forgot your password, reset it here or run:
    +
    +archivebox manage changepassword USERNAME
+
    + +

    +
    +
    + To create a new admin user, run the following: +
    +archivebox manage createsuperuser
+
    +
    +
    + + (cd into your archive folder before running commands) +
    + + +
    +{% endblock %} diff --git a/archivebox-0.5.3/archivebox/themes/admin/snapshots_grid.html b/archivebox-0.5.3/archivebox/themes/admin/snapshots_grid.html new file mode 100644 index 0000000..a7a2d4f --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/admin/snapshots_grid.html @@ -0,0 +1,162 @@ +{% load i18n admin_urls static admin_list %} +{% load core_tags %} + +{% block extrastyle %} + + +{% endblock %} + +{% block content %} +
    + {% for obj in results %} +
    + + + + + +
    + {% if obj.tags_str %} +

    {{obj.tags_str}}

    + {% endif %} + {% if obj.title %} + +

    {{obj.title|truncatechars:55 }}

    +     + {% endif %} + {% comment %}

    TEXT If needed.

    {% endcomment %} +
    +
    +
    + + 🕑 {{obj.added}} + 📖{{obj.num_outputs}} + 🗄️{{ obj.archive_size | file_size }} +
    +
    +
    + {% endfor %} +
    + +{% endblock %} \ No newline at end of file diff --git a/archivebox-0.5.3/archivebox/themes/default/add_links.html b/archivebox-0.5.3/archivebox/themes/default/add_links.html new file mode 100644 index 0000000..0b384f5 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/add_links.html @@ -0,0 +1,71 @@ +{% extends "base.html" %} + +{% load static %} +{% load i18n %} + +{% block breadcrumbs %} +
    + {% trans 'Home' %} + {% if title %} › {{ title }}{% endif %} +
    +{% endblock %} + +{% block extra_head %} + +{% endblock %} + +{% block body %} +
    +

    + {% if stdout %} +

    Add new URLs to your archive: results

    + 
    +                {{ stdout | safe }}
+                

    
+
    +
    +
    +   Add more URLs ➕ +
    + {% else %} +
    {% csrf_token %} +

    Add new URLs to your archive

    +
    + {{ form.as_p }} +
    + +
    +
    +


    + + {% if absolute_add_path %} +
    +

    Bookmark this link to quickly add to your archive: + Add to ArchiveBox

    +
    + {% endif %} + + {% endif %} +
    +{% endblock %} + +{% block sidebar %}{% endblock %} diff --git a/archivebox-0.5.3/archivebox/themes/default/base.html b/archivebox-0.5.3/archivebox/themes/default/base.html new file mode 100644 index 0000000..a70430e --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/base.html @@ -0,0 +1,284 @@ +{% load static %} + + + + + + Archived Sites + + + + + + {% block extra_head %} + {% endblock %} + + + + + + +
    +
    +
    +
    + + Logo + ArchiveBox: Index + +
    +
    + Add Links   |   + Admin   |   + Docs +
    +
    +
    +
    + {% block body %} + {% endblock %} +
    + + + + diff --git a/archivebox-0.5.3/archivebox/themes/default/core/snapshot_list.html b/archivebox-0.5.3/archivebox/themes/default/core/snapshot_list.html new file mode 100644 index 0000000..ce2b2fa --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/core/snapshot_list.html @@ -0,0 +1,51 @@ +{% extends "base.html" %} +{% load static %} + +{% block body %} +
    +
    + + + +
    + + + + + + + + + + + {% for link in object_list %} + {% include 'main_index_row.html' with link=link %} + {% endfor %} + +
    BookmarkedSnapshot ({{object_list|length}})FilesOriginal URL
    +
    + + {% if page_obj.has_previous %} + « first + previous + {% endif %} + + + Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}. + + + {% if page_obj.has_next %} + next + last » + {% endif %} + + + {% if page_obj.has_next %} + next + last » + {% endif %} + +
    +
    +{% endblock %} diff --git a/archivebox-0.5.3/archivebox/themes/default/link_details.html b/archivebox-0.5.3/archivebox/themes/default/link_details.html new file mode 100644 index 0000000..b1edcfe --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/link_details.html @@ -0,0 +1,488 @@ + + + + {{title}} + + + + + +
    +
    +
    +
    + + + Archive Icon + ArchiveBox + +
    +
    + Favicon +    + {{title}} +    + ▾ +
    + + + {{url_str}} + + +
    +
    +
    +
    +
    +
    +
    +
    Added
    + {{bookmarked_date}} +
    +
    +
    First Archived
    + {{oldest_archive_date}} +
    +
    +
    Last Checked
    + {{updated_date}} +
    +
    +
    +
    +
    Type
    +
    {{extension}}
    +
    +
    +
    Tags
    +
    {{tags}}
    +
    +
    +
    Status
    +
    {{status}}
    +
    +
    +
    Saved
    + ✅ {{num_outputs}} +
    +
    +
    Errors
    + ❌ {{num_failures}} +
    +
    +
    Size
    + {{size}} +
    +
    +
    +
    +
    🗃 Files
    + JSON | + WARC | + Media | + Git | + Favicon | + See all... +
    +
    +
    +
    +
    +
    + +
    + + + +

    Wget > WARC

     +

    archive/{{domain}}

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > SingleFile

     +

    archive/singlefile.html

    +
    +
    +
    +
    +
    + +
    + + + +

    Archive.Org

     +

    web.archive.org/web/...

    +
    +
    +
    +
    +
    + +
    + + + +

    Original

     +

    {{domain}}

    +
    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > PDF

     +

    archive/output.pdf

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > Screenshot

     +

    archive/screenshot.png

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > HTML

     +

    archive/output.html

    +
    +
    +
    +
    +
    + +
    + + + +

    Readability

     +

    archive/readability/...

    +
    +
    +
    +
    +
    +
    + +
    + + + +

    mercury

     +

    archive/mercury/...

    +
    +
    +
    +
    +
    +
    + + + + + + + + diff --git a/archivebox-0.5.3/archivebox/themes/default/main_index.html b/archivebox-0.5.3/archivebox/themes/default/main_index.html new file mode 100644 index 0000000..95af196 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/main_index.html @@ -0,0 +1,255 @@ +{% load static %} + + + + + Archived Sites + + + + + + + + + +
    +
    +
    +
    + + Logo + ArchiveBox: Index + +
    +
    + Add Links   |   + Admin   |   + Docs +
    +
    +
    +
    + + + + + + + + + + + + {% for link in links %} + {% include 'main_index_row.html' with link=link %} + {% endfor %} + +
    BookmarkedSaved Link ({{num_links}})FilesOriginal URL
    + + + diff --git a/archivebox-0.5.3/archivebox/themes/default/main_index_minimal.html b/archivebox-0.5.3/archivebox/themes/default/main_index_minimal.html new file mode 100644 index 0000000..dcfaa23 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/main_index_minimal.html @@ -0,0 +1,24 @@ + + + + Archived Sites + + + + + + + + + + + + + + {% for link in links %} + {% include "main_index_row.html" with link=link %} + {% endfor %} + +
    BookmarkedSaved Link ({{num_links}})FilesOriginal URL
    + + \ No newline at end of file diff --git a/archivebox-0.5.3/archivebox/themes/default/main_index_row.html b/archivebox-0.5.3/archivebox/themes/default/main_index_row.html new file mode 100644 index 0000000..5e21a8c --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/main_index_row.html @@ -0,0 +1,22 @@ +{% load static %} + + + {% if link.bookmarked_date %} {{ link.bookmarked_date }} {% else %} {{ link.added }} {% endif %} + + {% if link.is_archived %} + + {% else %} + + {% endif %} + + {{link.title|default:'Loading...'}} + {% if link.tags_str != None %} {{link.tags_str|default:''}} {% else %} {{ link.tags|default:'' }} {% endif %} + + + + 📄 + {% if link.icons %} {{link.icons}} {% else %} {{ link.num_outputs}} {% endif %} + + + {{link.url}} + \ No newline at end of file diff --git a/archivebox-0.5.3/archivebox/themes/default/static/add.css b/archivebox-0.5.3/archivebox/themes/default/static/add.css new file mode 100644 index 0000000..b128bf4 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/static/add.css @@ -0,0 +1,62 @@ +.dashboard #content { + width: 100%; + margin-right: 0px; + margin-left: 0px; +} +#submit { + border: 1px solid rgba(0, 0, 0, 0.2); + padding: 10px; + border-radius: 4px; + background-color: #f5dd5d; + color: #333; + font-size: 18px; + font-weight: 800; +} +#add-form button[role="submit"]:hover { + background-color: #e5cd4d; +} +#add-form label { + display: block; + font-size: 16px; +} +#add-form textarea { + width: 100%; + min-height: 300px; +} +#delay-warning div { + border: 1px solid red; + border-radius: 4px; + margin: 10px; + padding: 10px; + font-size: 15px; + background-color: #f5dd5d; +} +#stdout { + background-color: #ded; + padding: 10px 10px; + border-radius: 4px; + white-space: normal; +} +ul#id_depth { + list-style-type: none; + padding: 0; +} + +@keyframes spin { + 0% { + transform: rotate(0deg); + } + 100% { + transform: rotate(360deg); + } +} + +.loader { + border: 16px solid #f3f3f3; /* Light grey */ + border-top: 16px solid #3498db; /* Blue */ + border-radius: 50%; + width: 30px; + height: 30px; + box-sizing: border-box; + animation: spin 2s linear infinite; +} diff --git a/archivebox-0.5.3/archivebox/themes/default/static/admin.css b/archivebox-0.5.3/archivebox/themes/default/static/admin.css new file mode 100644 index 0000000..181c06d --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/static/admin.css @@ -0,0 +1,234 @@ +#logo { + height: 30px; + vertical-align: -6px; + padding-right: 5px; +} +#site-name:hover a { + opacity: 0.9; +} +#site-name .loader { + height: 25px; + width: 25px; + display: inline-block; + border-width: 3px; + vertical-align: -3px; + margin-right: 5px; + margin-top: 2px; +} +#branding h1, #branding h1 a:link, #branding h1 a:visited { + color: mintcream; +} +#header { + background: #aa1e55; + padding: 6px 14px; +} +#content { + padding: 8px 8px; +} +#user-tools { + font-size: 13px; + +} + +div.breadcrumbs { + background: #772948; + color: #f5dd5d; + padding: 6px 15px; +} + +body.model-snapshot.change-list div.breadcrumbs, +body.model-snapshot.change-list #content .object-tools { + display: none; +} + +.module h2, .module caption, .inline-group h2 { + background: #772948; +} + +#content .object-tools { + margin-top: -35px; + margin-right: -10px; + float: right; +} + +#content .object-tools a:link, #content .object-tools a:visited { + border-radius: 0px; + background-color: #f5dd5d; + color: #333; + font-size: 12px; + font-weight: 800; +} + +#content .object-tools a.addlink { + background-blend-mode: difference; +} + +#content #changelist #toolbar { + padding: 0px; + background: none; + margin-bottom: 10px; + border-top: 0px; + border-bottom: 0px; +} + +#content #changelist #toolbar form input[type="submit"] { + border-color: #aa1e55; +} + +#content #changelist-filter li.selected a { + color: #aa1e55; +} + + +/*#content #changelist .actions { + position: fixed; + bottom: 0px; + z-index: 800; +}*/ +#content #changelist .actions { + float: right; + margin-top: -34px; + padding: 0px; + background: none; + margin-right: 0px; + width: auto; +} + +#content #changelist .actions .button { + border-radius: 2px; + background-color: #f5dd5d; + color: #333; + font-size: 12px; + font-weight: 800; + margin-right: 4px; + box-shadow: 4px 4px 4px rgba(0,0,0,0.02); + border: 1px solid rgba(0,0,0,0.08); +} +#content #changelist .actions .button:hover { + border: 1px solid rgba(0,0,0,0.2); + opacity: 0.9; +} +#content #changelist .actions .button[name=verify_snapshots], #content #changelist .actions .button[name=update_titles] { + background-color: #dedede; + color: #333; +} +#content #changelist .actions .button[name=update_snapshots] { + background-color:lightseagreen; + color: #333; +} +#content #changelist .actions .button[name=overwrite_snapshots] { + background-color: #ffaa31; + color: #333; +} +#content #changelist .actions .button[name=delete_snapshots] { + background-color: #f91f74; + color: rgb(255 248 252 / 64%); +} + + +#content #changelist-filter h2 { + border-radius: 4px 4px 0px 0px; +} + +@media (min-width: 767px) { + #content #changelist-filter { + top: 35px; + width: 110px; + margin-bottom: 35px; + } + + .change-list .filtered .results, + .change-list .filtered .paginator, + .filtered #toolbar, + .filtered div.xfull { + margin-right: 115px; + } +} + +@media (max-width: 1127px) { + #content #changelist .actions { + position: fixed; + bottom: 6px; + left: 10px; + float: left; + z-index: 1000; + } +} + +#content a img.favicon { + height: 20px; + width: 20px; + vertical-align: -5px; + padding-right: 6px; +} + +#content td, #content th { + vertical-align: middle; + padding: 4px; +} + +#content #changelist table input { + vertical-align: -2px; +} + +#content thead th .text a { + padding: 8px 4px; +} + +#content th.field-added, #content td.field-updated { + word-break: break-word; + min-width: 128px; + white-space: normal; +} + +#content th.field-title_str { + min-width: 300px; +} + +#content td.field-files { + white-space: nowrap; +} +#content td.field-files .exists-True { + opacity: 1; +} +#content td.field-files .exists-False { + opacity: 0.1; + filter: grayscale(100%); +} +#content td.field-size { + white-space: nowrap; +} + +#content td.field-url_str { + word-break: break-all; + min-width: 200px; +} + +#content tr b.status-pending { + font-weight: 200; + opacity: 0.6; +} + +.loader { + border: 16px solid #f3f3f3; /* Light grey */ + border-top: 16px solid #3498db; /* Blue */ + border-radius: 50%; + width: 30px; + height: 30px; + box-sizing: border-box; + animation: spin 2s linear infinite; +} + +@keyframes spin { + 0% { transform: rotate(0deg); } + 100% { transform: rotate(360deg); } +} + +.tags > a > .tag { + float: right; + border-radius: 5px; + background-color: #bfdfff; + padding: 2px 5px; + margin-left: 4px; + margin-top: 1px; +} diff --git a/archivebox-0.5.3/archivebox/themes/default/static/archive.png b/archivebox-0.5.3/archivebox/themes/default/static/archive.png new file mode 100644 index 0000000..307b450 Binary files /dev/null and b/archivebox-0.5.3/archivebox/themes/default/static/archive.png differ diff --git a/archivebox-0.5.3/archivebox/themes/default/static/bootstrap.min.css b/archivebox-0.5.3/archivebox/themes/default/static/bootstrap.min.css new file mode 100644 index 0000000..a8da074 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/static/bootstrap.min.css @@ -0,0 +1,6 @@ +/*! + * Bootstrap v4.0.0-alpha.6 (https://getbootstrap.com) + * Copyright 2011-2017 The Bootstrap Authors + * Copyright 2011-2017 Twitter, Inc. + * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE) + *//*! normalize.css v5.0.0 | MIT License | github.com/necolas/normalize.css */html{font-family:sans-serif;line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}figcaption,figure,main{display:block}figure{margin:1em 40px}hr{-webkit-box-sizing:content-box;box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1em}a{background-color:transparent;-webkit-text-decoration-skip:objects}a:active,a:hover{outline-width:0}abbr[title]{border-bottom:none;text-decoration:underline;text-decoration:underline dotted}b,strong{font-weight:inherit}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1em}dfn{font-style:italic}mark{background-color:#ff0;color:#000}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}audio,video{display:inline-block}audio:not([controls]){display:none;height:0}img{border-style:none}svg:not(:root){overflow:hidden}button,input,optgroup,select,textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}[type=reset],[type=submit],button,html [type=button]{-webkit-appearance:button}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}legend{-webkit-box-sizing:border-box;box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{display:inline-block;vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{-webkit-box-sizing:border-box;box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-cancel-button,[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details,menu{display:block}summary{display:list-item}canvas{display:inline-block}template{display:none}[hidden]{display:none}@media print{*,::after,::before,blockquote::first-letter,blockquote::first-line,div::first-letter,div::first-line,li::first-letter,li::first-line,p::first-letter,p::first-line{text-shadow:none!important;-webkit-box-shadow:none!important;box-shadow:none!important}a,a:visited{text-decoration:underline}abbr[title]::after{content:" (" attr(title) ")"}pre{white-space:pre-wrap!important}blockquote,pre{border:1px solid #999;page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}h2,h3,p{orphans:3;widows:3}h2,h3{page-break-after:avoid}.navbar{display:none}.badge{border:1px solid #000}.table{border-collapse:collapse!important}.table td,.table th{background-color:#fff!important}.table-bordered td,.table-bordered th{border:1px solid #ddd!important}}html{-webkit-box-sizing:border-box;box-sizing:border-box}*,::after,::before{-webkit-box-sizing:inherit;box-sizing:inherit}@-ms-viewport{width:device-width}html{-ms-overflow-style:scrollbar;-webkit-tap-highlight-color:transparent}body{font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-size:1rem;font-weight:400;line-height:1.5;color:#292b2c;background-color:#fff}[tabindex="-1"]:focus{outline:0!important}h1,h2,h3,h4,h5,h6{margin-top:0;margin-bottom:.5rem}p{margin-top:0;margin-bottom:1rem}abbr[data-original-title],abbr[title]{cursor:help}address{margin-bottom:1rem;font-style:normal;line-height:inherit}dl,ol,ul{margin-top:0;margin-bottom:1rem}ol ol,ol ul,ul ol,ul ul{margin-bottom:0}dt{font-weight:700}dd{margin-bottom:.5rem;margin-left:0}blockquote{margin:0 0 1rem}a{color:#0275d8;text-decoration:none}a:focus,a:hover{color:#014c8c;text-decoration:underline}a:not([href]):not([tabindex]){color:inherit;text-decoration:none}a:not([href]):not([tabindex]):focus,a:not([href]):not([tabindex]):hover{color:inherit;text-decoration:none}a:not([href]):not([tabindex]):focus{outline:0}pre{margin-top:0;margin-bottom:1rem;overflow:auto}figure{margin:0 0 1rem}img{vertical-align:middle}[role=button]{cursor:pointer}[role=button],a,area,button,input,label,select,summary,textarea{-ms-touch-action:manipulation;touch-action:manipulation}table{border-collapse:collapse;background-color:transparent}caption{padding-top:.75rem;padding-bottom:.75rem;color:#636c72;text-align:left;caption-side:bottom}th{text-align:left}label{display:inline-block;margin-bottom:.5rem}button:focus{outline:1px dotted;outline:5px auto -webkit-focus-ring-color}button,input,select,textarea{line-height:inherit}input[type=checkbox]:disabled,input[type=radio]:disabled{cursor:not-allowed}input[type=date],input[type=time],input[type=datetime-local],input[type=month]{-webkit-appearance:listbox}textarea{resize:vertical}fieldset{min-width:0;padding:0;margin:0;border:0}legend{display:block;width:100%;padding:0;margin-bottom:.5rem;font-size:1.5rem;line-height:inherit}input[type=search]{-webkit-appearance:none}output{display:inline-block}[hidden]{display:none!important}.h1,.h2,.h3,.h4,.h5,.h6,h1,h2,h3,h4,h5,h6{margin-bottom:.5rem;font-family:inherit;font-weight:500;line-height:1.1;color:inherit}.h1,h1{font-size:2.5rem}.h2,h2{font-size:2rem}.h3,h3{font-size:1.75rem}.h4,h4{font-size:1.5rem}.h5,h5{font-size:1.25rem}.h6,h6{font-size:1rem}.lead{font-size:1.25rem;font-weight:300}.display-1{font-size:6rem;font-weight:300;line-height:1.1}.display-2{font-size:5.5rem;font-weight:300;line-height:1.1}.display-3{font-size:4.5rem;font-weight:300;line-height:1.1}.display-4{font-size:3.5rem;font-weight:300;line-height:1.1}hr{margin-top:1rem;margin-bottom:1rem;border:0;border-top:1px solid rgba(0,0,0,.1)}.small,small{font-size:80%;font-weight:400}.mark,mark{padding:.2em;background-color:#fcf8e3}.list-unstyled{padding-left:0;list-style:none}.list-inline{padding-left:0;list-style:none}.list-inline-item{display:inline-block}.list-inline-item:not(:last-child){margin-right:5px}.initialism{font-size:90%;text-transform:uppercase}.blockquote{padding:.5rem 1rem;margin-bottom:1rem;font-size:1.25rem;border-left:.25rem solid #eceeef}.blockquote-footer{display:block;font-size:80%;color:#636c72}.blockquote-footer::before{content:"\2014 \00A0"}.blockquote-reverse{padding-right:1rem;padding-left:0;text-align:right;border-right:.25rem solid #eceeef;border-left:0}.blockquote-reverse .blockquote-footer::before{content:""}.blockquote-reverse .blockquote-footer::after{content:"\00A0 \2014"}.img-fluid{max-width:100%;height:auto}.img-thumbnail{padding:.25rem;background-color:#fff;border:1px solid #ddd;border-radius:.25rem;-webkit-transition:all .2s ease-in-out;-o-transition:all .2s ease-in-out;transition:all .2s ease-in-out;max-width:100%;height:auto}.figure{display:inline-block}.figure-img{margin-bottom:.5rem;line-height:1}.figure-caption{font-size:90%;color:#636c72}code,kbd,pre,samp{font-family:Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace}code{padding:.2rem .4rem;font-size:90%;color:#bd4147;background-color:#f7f7f9;border-radius:.25rem}a>code{padding:0;color:inherit;background-color:inherit}kbd{padding:.2rem .4rem;font-size:90%;color:#fff;background-color:#292b2c;border-radius:.2rem}kbd kbd{padding:0;font-size:100%;font-weight:700}pre{display:block;margin-top:0;margin-bottom:1rem;font-size:90%;color:#292b2c}pre code{padding:0;font-size:inherit;color:inherit;background-color:transparent;border-radius:0}.pre-scrollable{max-height:340px;overflow-y:scroll}.container{position:relative;margin-left:auto;margin-right:auto;padding-right:15px;padding-left:15px}@media (min-width:576px){.container{padding-right:15px;padding-left:15px}}@media (min-width:768px){.container{padding-right:15px;padding-left:15px}}@media (min-width:992px){.container{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.container{padding-right:15px;padding-left:15px}}@media (min-width:576px){.container{width:540px;max-width:100%}}@media (min-width:768px){.container{width:720px;max-width:100%}}@media (min-width:992px){.container{width:960px;max-width:100%}}@media (min-width:1200px){.container{width:1140px;max-width:100%}}.container-fluid{position:relative;margin-left:auto;margin-right:auto;padding-right:15px;padding-left:15px}@media (min-width:576px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:768px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:992px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.container-fluid{padding-right:15px;padding-left:15px}}.row{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;margin-right:-15px;margin-left:-15px}@media (min-width:576px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:768px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:992px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:1200px){.row{margin-right:-15px;margin-left:-15px}}.no-gutters{margin-right:0;margin-left:0}.no-gutters>.col,.no-gutters>[class*=col-]{padding-right:0;padding-left:0}.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{position:relative;width:100%;min-height:1px;padding-right:15px;padding-left:15px}@media (min-width:576px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:768px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:992px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}.col{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-0{right:auto}.pull-1{right:8.333333%}.pull-2{right:16.666667%}.pull-3{right:25%}.pull-4{right:33.333333%}.pull-5{right:41.666667%}.pull-6{right:50%}.pull-7{right:58.333333%}.pull-8{right:66.666667%}.pull-9{right:75%}.pull-10{right:83.333333%}.pull-11{right:91.666667%}.pull-12{right:100%}.push-0{left:auto}.push-1{left:8.333333%}.push-2{left:16.666667%}.push-3{left:25%}.push-4{left:33.333333%}.push-5{left:41.666667%}.push-6{left:50%}.push-7{left:58.333333%}.push-8{left:66.666667%}.push-9{left:75%}.push-10{left:83.333333%}.push-11{left:91.666667%}.push-12{left:100%}.offset-1{margin-left:8.333333%}.offset-2{margin-left:16.666667%}.offset-3{margin-left:25%}.offset-4{margin-left:33.333333%}.offset-5{margin-left:41.666667%}.offset-6{margin-left:50%}.offset-7{margin-left:58.333333%}.offset-8{margin-left:66.666667%}.offset-9{margin-left:75%}.offset-10{margin-left:83.333333%}.offset-11{margin-left:91.666667%}@media (min-width:576px){.col-sm{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-sm-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-sm-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-sm-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-sm-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-sm-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-sm-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-sm-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-sm-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-sm-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-sm-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-sm-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-sm-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-sm-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-sm-0{right:auto}.pull-sm-1{right:8.333333%}.pull-sm-2{right:16.666667%}.pull-sm-3{right:25%}.pull-sm-4{right:33.333333%}.pull-sm-5{right:41.666667%}.pull-sm-6{right:50%}.pull-sm-7{right:58.333333%}.pull-sm-8{right:66.666667%}.pull-sm-9{right:75%}.pull-sm-10{right:83.333333%}.pull-sm-11{right:91.666667%}.pull-sm-12{right:100%}.push-sm-0{left:auto}.push-sm-1{left:8.333333%}.push-sm-2{left:16.666667%}.push-sm-3{left:25%}.push-sm-4{left:33.333333%}.push-sm-5{left:41.666667%}.push-sm-6{left:50%}.push-sm-7{left:58.333333%}.push-sm-8{left:66.666667%}.push-sm-9{left:75%}.push-sm-10{left:83.333333%}.push-sm-11{left:91.666667%}.push-sm-12{left:100%}.offset-sm-0{margin-left:0}.offset-sm-1{margin-left:8.333333%}.offset-sm-2{margin-left:16.666667%}.offset-sm-3{margin-left:25%}.offset-sm-4{margin-left:33.333333%}.offset-sm-5{margin-left:41.666667%}.offset-sm-6{margin-left:50%}.offset-sm-7{margin-left:58.333333%}.offset-sm-8{margin-left:66.666667%}.offset-sm-9{margin-left:75%}.offset-sm-10{margin-left:83.333333%}.offset-sm-11{margin-left:91.666667%}}@media (min-width:768px){.col-md{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-md-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-md-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-md-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-md-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-md-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-md-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-md-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-md-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-md-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-md-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-md-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-md-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-md-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-md-0{right:auto}.pull-md-1{right:8.333333%}.pull-md-2{right:16.666667%}.pull-md-3{right:25%}.pull-md-4{right:33.333333%}.pull-md-5{right:41.666667%}.pull-md-6{right:50%}.pull-md-7{right:58.333333%}.pull-md-8{right:66.666667%}.pull-md-9{right:75%}.pull-md-10{right:83.333333%}.pull-md-11{right:91.666667%}.pull-md-12{right:100%}.push-md-0{left:auto}.push-md-1{left:8.333333%}.push-md-2{left:16.666667%}.push-md-3{left:25%}.push-md-4{left:33.333333%}.push-md-5{left:41.666667%}.push-md-6{left:50%}.push-md-7{left:58.333333%}.push-md-8{left:66.666667%}.push-md-9{left:75%}.push-md-10{left:83.333333%}.push-md-11{left:91.666667%}.push-md-12{left:100%}.offset-md-0{margin-left:0}.offset-md-1{margin-left:8.333333%}.offset-md-2{margin-left:16.666667%}.offset-md-3{margin-left:25%}.offset-md-4{margin-left:33.333333%}.offset-md-5{margin-left:41.666667%}.offset-md-6{margin-left:50%}.offset-md-7{margin-left:58.333333%}.offset-md-8{margin-left:66.666667%}.offset-md-9{margin-left:75%}.offset-md-10{margin-left:83.333333%}.offset-md-11{margin-left:91.666667%}}@media (min-width:992px){.col-lg{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-lg-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-lg-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-lg-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-lg-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-lg-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-lg-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-lg-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-lg-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-lg-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-lg-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-lg-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-lg-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-lg-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-lg-0{right:auto}.pull-lg-1{right:8.333333%}.pull-lg-2{right:16.666667%}.pull-lg-3{right:25%}.pull-lg-4{right:33.333333%}.pull-lg-5{right:41.666667%}.pull-lg-6{right:50%}.pull-lg-7{right:58.333333%}.pull-lg-8{right:66.666667%}.pull-lg-9{right:75%}.pull-lg-10{right:83.333333%}.pull-lg-11{right:91.666667%}.pull-lg-12{right:100%}.push-lg-0{left:auto}.push-lg-1{left:8.333333%}.push-lg-2{left:16.666667%}.push-lg-3{left:25%}.push-lg-4{left:33.333333%}.push-lg-5{left:41.666667%}.push-lg-6{left:50%}.push-lg-7{left:58.333333%}.push-lg-8{left:66.666667%}.push-lg-9{left:75%}.push-lg-10{left:83.333333%}.push-lg-11{left:91.666667%}.push-lg-12{left:100%}.offset-lg-0{margin-left:0}.offset-lg-1{margin-left:8.333333%}.offset-lg-2{margin-left:16.666667%}.offset-lg-3{margin-left:25%}.offset-lg-4{margin-left:33.333333%}.offset-lg-5{margin-left:41.666667%}.offset-lg-6{margin-left:50%}.offset-lg-7{margin-left:58.333333%}.offset-lg-8{margin-left:66.666667%}.offset-lg-9{margin-left:75%}.offset-lg-10{margin-left:83.333333%}.offset-lg-11{margin-left:91.666667%}}@media (min-width:1200px){.col-xl{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-xl-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-xl-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-xl-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-xl-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-xl-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-xl-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-xl-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-xl-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-xl-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-xl-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-xl-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-xl-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-xl-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-xl-0{right:auto}.pull-xl-1{right:8.333333%}.pull-xl-2{right:16.666667%}.pull-xl-3{right:25%}.pull-xl-4{right:33.333333%}.pull-xl-5{right:41.666667%}.pull-xl-6{right:50%}.pull-xl-7{right:58.333333%}.pull-xl-8{right:66.666667%}.pull-xl-9{right:75%}.pull-xl-10{right:83.333333%}.pull-xl-11{right:91.666667%}.pull-xl-12{right:100%}.push-xl-0{left:auto}.push-xl-1{left:8.333333%}.push-xl-2{left:16.666667%}.push-xl-3{left:25%}.push-xl-4{left:33.333333%}.push-xl-5{left:41.666667%}.push-xl-6{left:50%}.push-xl-7{left:58.333333%}.push-xl-8{left:66.666667%}.push-xl-9{left:75%}.push-xl-10{left:83.333333%}.push-xl-11{left:91.666667%}.push-xl-12{left:100%}.offset-xl-0{margin-left:0}.offset-xl-1{margin-left:8.333333%}.offset-xl-2{margin-left:16.666667%}.offset-xl-3{margin-left:25%}.offset-xl-4{margin-left:33.333333%}.offset-xl-5{margin-left:41.666667%}.offset-xl-6{margin-left:50%}.offset-xl-7{margin-left:58.333333%}.offset-xl-8{margin-left:66.666667%}.offset-xl-9{margin-left:75%}.offset-xl-10{margin-left:83.333333%}.offset-xl-11{margin-left:91.666667%}}.table{width:100%;max-width:100%;margin-bottom:1rem}.table td,.table th{padding:.75rem;vertical-align:top;border-top:1px solid #eceeef}.table thead th{vertical-align:bottom;border-bottom:2px solid #eceeef}.table tbody+tbody{border-top:2px solid #eceeef}.table .table{background-color:#fff}.table-sm td,.table-sm th{padding:.3rem}.table-bordered{border:1px solid #eceeef}.table-bordered td,.table-bordered th{border:1px solid #eceeef}.table-bordered thead td,.table-bordered thead th{border-bottom-width:2px}.table-striped tbody tr:nth-of-type(odd){background-color:rgba(0,0,0,.05)}.table-hover tbody tr:hover{background-color:rgba(0,0,0,.075)}.table-active,.table-active>td,.table-active>th{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover>td,.table-hover .table-active:hover>th{background-color:rgba(0,0,0,.075)}.table-success,.table-success>td,.table-success>th{background-color:#dff0d8}.table-hover .table-success:hover{background-color:#d0e9c6}.table-hover .table-success:hover>td,.table-hover .table-success:hover>th{background-color:#d0e9c6}.table-info,.table-info>td,.table-info>th{background-color:#d9edf7}.table-hover .table-info:hover{background-color:#c4e3f3}.table-hover .table-info:hover>td,.table-hover .table-info:hover>th{background-color:#c4e3f3}.table-warning,.table-warning>td,.table-warning>th{background-color:#fcf8e3}.table-hover .table-warning:hover{background-color:#faf2cc}.table-hover .table-warning:hover>td,.table-hover .table-warning:hover>th{background-color:#faf2cc}.table-danger,.table-danger>td,.table-danger>th{background-color:#f2dede}.table-hover .table-danger:hover{background-color:#ebcccc}.table-hover .table-danger:hover>td,.table-hover .table-danger:hover>th{background-color:#ebcccc}.thead-inverse th{color:#fff;background-color:#292b2c}.thead-default th{color:#464a4c;background-color:#eceeef}.table-inverse{color:#fff;background-color:#292b2c}.table-inverse td,.table-inverse th,.table-inverse thead th{border-color:#fff}.table-inverse.table-bordered{border:0}.table-responsive{display:block;width:100%;overflow-x:auto;-ms-overflow-style:-ms-autohiding-scrollbar}.table-responsive.table-bordered{border:0}.form-control{display:block;width:100%;padding:.5rem .75rem;font-size:1rem;line-height:1.25;color:#464a4c;background-color:#fff;background-image:none;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.15);border-radius:.25rem;-webkit-transition:border-color ease-in-out .15s,-webkit-box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,-webkit-box-shadow ease-in-out .15s;-o-transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s,-webkit-box-shadow ease-in-out .15s}.form-control::-ms-expand{background-color:transparent;border:0}.form-control:focus{color:#464a4c;background-color:#fff;border-color:#5cb3fd;outline:0}.form-control::-webkit-input-placeholder{color:#636c72;opacity:1}.form-control::-moz-placeholder{color:#636c72;opacity:1}.form-control:-ms-input-placeholder{color:#636c72;opacity:1}.form-control::placeholder{color:#636c72;opacity:1}.form-control:disabled,.form-control[readonly]{background-color:#eceeef;opacity:1}.form-control:disabled{cursor:not-allowed}select.form-control:not([size]):not([multiple]){height:calc(2.25rem + 2px)}select.form-control:focus::-ms-value{color:#464a4c;background-color:#fff}.form-control-file,.form-control-range{display:block}.col-form-label{padding-top:calc(.5rem - 1px * 2);padding-bottom:calc(.5rem - 1px * 2);margin-bottom:0}.col-form-label-lg{padding-top:calc(.75rem - 1px * 2);padding-bottom:calc(.75rem - 1px * 2);font-size:1.25rem}.col-form-label-sm{padding-top:calc(.25rem - 1px * 2);padding-bottom:calc(.25rem - 1px * 2);font-size:.875rem}.col-form-legend{padding-top:.5rem;padding-bottom:.5rem;margin-bottom:0;font-size:1rem}.form-control-static{padding-top:.5rem;padding-bottom:.5rem;margin-bottom:0;line-height:1.25;border:solid transparent;border-width:1px 0}.form-control-static.form-control-lg,.form-control-static.form-control-sm,.input-group-lg>.form-control-static.form-control,.input-group-lg>.form-control-static.input-group-addon,.input-group-lg>.input-group-btn>.form-control-static.btn,.input-group-sm>.form-control-static.form-control,.input-group-sm>.form-control-static.input-group-addon,.input-group-sm>.input-group-btn>.form-control-static.btn{padding-right:0;padding-left:0}.form-control-sm,.input-group-sm>.form-control,.input-group-sm>.input-group-addon,.input-group-sm>.input-group-btn>.btn{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.input-group-sm>.input-group-btn>select.btn:not([size]):not([multiple]),.input-group-sm>select.form-control:not([size]):not([multiple]),.input-group-sm>select.input-group-addon:not([size]):not([multiple]),select.form-control-sm:not([size]):not([multiple]){height:1.8125rem}.form-control-lg,.input-group-lg>.form-control,.input-group-lg>.input-group-addon,.input-group-lg>.input-group-btn>.btn{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.input-group-lg>.input-group-btn>select.btn:not([size]):not([multiple]),.input-group-lg>select.form-control:not([size]):not([multiple]),.input-group-lg>select.input-group-addon:not([size]):not([multiple]),select.form-control-lg:not([size]):not([multiple]){height:3.166667rem}.form-group{margin-bottom:1rem}.form-text{display:block;margin-top:.25rem}.form-check{position:relative;display:block;margin-bottom:.5rem}.form-check.disabled .form-check-label{color:#636c72;cursor:not-allowed}.form-check-label{padding-left:1.25rem;margin-bottom:0;cursor:pointer}.form-check-input{position:absolute;margin-top:.25rem;margin-left:-1.25rem}.form-check-input:only-child{position:static}.form-check-inline{display:inline-block}.form-check-inline .form-check-label{vertical-align:middle}.form-check-inline+.form-check-inline{margin-left:.75rem}.form-control-feedback{margin-top:.25rem}.form-control-danger,.form-control-success,.form-control-warning{padding-right:2.25rem;background-repeat:no-repeat;background-position:center right .5625rem;-webkit-background-size:1.125rem 1.125rem;background-size:1.125rem 1.125rem}.has-success .col-form-label,.has-success .custom-control,.has-success .form-check-label,.has-success .form-control-feedback,.has-success .form-control-label{color:#5cb85c}.has-success .form-control{border-color:#5cb85c}.has-success .input-group-addon{color:#5cb85c;border-color:#5cb85c;background-color:#eaf6ea}.has-success .form-control-success{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%235cb85c' d='M2.3 6.73L.6 4.53c-.4-1.04.46-1.4 1.1-.8l1.1 1.4 3.4-3.8c.6-.63 1.6-.27 1.2.7l-4 4.6c-.43.5-.8.4-1.1.1z'/%3E%3C/svg%3E")}.has-warning .col-form-label,.has-warning .custom-control,.has-warning .form-check-label,.has-warning .form-control-feedback,.has-warning .form-control-label{color:#f0ad4e}.has-warning .form-control{border-color:#f0ad4e}.has-warning .input-group-addon{color:#f0ad4e;border-color:#f0ad4e;background-color:#fff}.has-warning .form-control-warning{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%23f0ad4e' d='M4.4 5.324h-.8v-2.46h.8zm0 1.42h-.8V5.89h.8zM3.76.63L.04 7.075c-.115.2.016.425.26.426h7.397c.242 0 .372-.226.258-.426C6.726 4.924 5.47 2.79 4.253.63c-.113-.174-.39-.174-.494 0z'/%3E%3C/svg%3E")}.has-danger .col-form-label,.has-danger .custom-control,.has-danger .form-check-label,.has-danger .form-control-feedback,.has-danger .form-control-label{color:#d9534f}.has-danger .form-control{border-color:#d9534f}.has-danger .input-group-addon{color:#d9534f;border-color:#d9534f;background-color:#fdf7f7}.has-danger .form-control-danger{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23d9534f' viewBox='-2 -2 7 7'%3E%3Cpath stroke='%23d9534f' d='M0 0l3 3m0-3L0 3'/%3E%3Ccircle r='.5'/%3E%3Ccircle cx='3' r='.5'/%3E%3Ccircle cy='3' r='.5'/%3E%3Ccircle cx='3' cy='3' r='.5'/%3E%3C/svg%3E")}.form-inline{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.form-inline .form-check{width:100%}@media (min-width:576px){.form-inline label{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;margin-bottom:0}.form-inline .form-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;margin-bottom:0}.form-inline .form-control{display:inline-block;width:auto;vertical-align:middle}.form-inline .form-control-static{display:inline-block}.form-inline .input-group{width:auto}.form-inline .form-control-label{margin-bottom:0;vertical-align:middle}.form-inline .form-check{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;width:auto;margin-top:0;margin-bottom:0}.form-inline .form-check-label{padding-left:0}.form-inline .form-check-input{position:relative;margin-top:0;margin-right:.25rem;margin-left:0}.form-inline .custom-control{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;padding-left:0}.form-inline .custom-control-indicator{position:static;display:inline-block;margin-right:.25rem;vertical-align:text-bottom}.form-inline .has-feedback .form-control-feedback{top:0}}.btn{display:inline-block;font-weight:400;line-height:1.25;text-align:center;white-space:nowrap;vertical-align:middle;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;border:1px solid transparent;padding:.5rem 1rem;font-size:1rem;border-radius:.25rem;-webkit-transition:all .2s ease-in-out;-o-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.btn:focus,.btn:hover{text-decoration:none}.btn.focus,.btn:focus{outline:0;-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.25);box-shadow:0 0 0 2px rgba(2,117,216,.25)}.btn.disabled,.btn:disabled{cursor:not-allowed;opacity:.65}.btn.active,.btn:active{background-image:none}a.btn.disabled,fieldset[disabled] a.btn{pointer-events:none}.btn-primary{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-primary:hover{color:#fff;background-color:#025aa5;border-color:#01549b}.btn-primary.focus,.btn-primary:focus{-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.5);box-shadow:0 0 0 2px rgba(2,117,216,.5)}.btn-primary.disabled,.btn-primary:disabled{background-color:#0275d8;border-color:#0275d8}.btn-primary.active,.btn-primary:active,.show>.btn-primary.dropdown-toggle{color:#fff;background-color:#025aa5;background-image:none;border-color:#01549b}.btn-secondary{color:#292b2c;background-color:#fff;border-color:#ccc}.btn-secondary:hover{color:#292b2c;background-color:#e6e6e6;border-color:#adadad}.btn-secondary.focus,.btn-secondary:focus{-webkit-box-shadow:0 0 0 2px rgba(204,204,204,.5);box-shadow:0 0 0 2px rgba(204,204,204,.5)}.btn-secondary.disabled,.btn-secondary:disabled{background-color:#fff;border-color:#ccc}.btn-secondary.active,.btn-secondary:active,.show>.btn-secondary.dropdown-toggle{color:#292b2c;background-color:#e6e6e6;background-image:none;border-color:#adadad}.btn-info{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-info:hover{color:#fff;background-color:#31b0d5;border-color:#2aabd2}.btn-info.focus,.btn-info:focus{-webkit-box-shadow:0 0 0 2px rgba(91,192,222,.5);box-shadow:0 0 0 2px rgba(91,192,222,.5)}.btn-info.disabled,.btn-info:disabled{background-color:#5bc0de;border-color:#5bc0de}.btn-info.active,.btn-info:active,.show>.btn-info.dropdown-toggle{color:#fff;background-color:#31b0d5;background-image:none;border-color:#2aabd2}.btn-success{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-success:hover{color:#fff;background-color:#449d44;border-color:#419641}.btn-success.focus,.btn-success:focus{-webkit-box-shadow:0 0 0 2px rgba(92,184,92,.5);box-shadow:0 0 0 2px rgba(92,184,92,.5)}.btn-success.disabled,.btn-success:disabled{background-color:#5cb85c;border-color:#5cb85c}.btn-success.active,.btn-success:active,.show>.btn-success.dropdown-toggle{color:#fff;background-color:#449d44;background-image:none;border-color:#419641}.btn-warning{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-warning:hover{color:#fff;background-color:#ec971f;border-color:#eb9316}.btn-warning.focus,.btn-warning:focus{-webkit-box-shadow:0 0 0 2px rgba(240,173,78,.5);box-shadow:0 0 0 2px rgba(240,173,78,.5)}.btn-warning.disabled,.btn-warning:disabled{background-color:#f0ad4e;border-color:#f0ad4e}.btn-warning.active,.btn-warning:active,.show>.btn-warning.dropdown-toggle{color:#fff;background-color:#ec971f;background-image:none;border-color:#eb9316}.btn-danger{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-danger:hover{color:#fff;background-color:#c9302c;border-color:#c12e2a}.btn-danger.focus,.btn-danger:focus{-webkit-box-shadow:0 0 0 2px rgba(217,83,79,.5);box-shadow:0 0 0 2px rgba(217,83,79,.5)}.btn-danger.disabled,.btn-danger:disabled{background-color:#d9534f;border-color:#d9534f}.btn-danger.active,.btn-danger:active,.show>.btn-danger.dropdown-toggle{color:#fff;background-color:#c9302c;background-image:none;border-color:#c12e2a}.btn-outline-primary{color:#0275d8;background-image:none;background-color:transparent;border-color:#0275d8}.btn-outline-primary:hover{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-outline-primary.focus,.btn-outline-primary:focus{-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.5);box-shadow:0 0 0 2px rgba(2,117,216,.5)}.btn-outline-primary.disabled,.btn-outline-primary:disabled{color:#0275d8;background-color:transparent}.btn-outline-primary.active,.btn-outline-primary:active,.show>.btn-outline-primary.dropdown-toggle{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-outline-secondary{color:#ccc;background-image:none;background-color:transparent;border-color:#ccc}.btn-outline-secondary:hover{color:#fff;background-color:#ccc;border-color:#ccc}.btn-outline-secondary.focus,.btn-outline-secondary:focus{-webkit-box-shadow:0 0 0 2px rgba(204,204,204,.5);box-shadow:0 0 0 2px rgba(204,204,204,.5)}.btn-outline-secondary.disabled,.btn-outline-secondary:disabled{color:#ccc;background-color:transparent}.btn-outline-secondary.active,.btn-outline-secondary:active,.show>.btn-outline-secondary.dropdown-toggle{color:#fff;background-color:#ccc;border-color:#ccc}.btn-outline-info{color:#5bc0de;background-image:none;background-color:transparent;border-color:#5bc0de}.btn-outline-info:hover{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-outline-info.focus,.btn-outline-info:focus{-webkit-box-shadow:0 0 0 2px rgba(91,192,222,.5);box-shadow:0 0 0 2px rgba(91,192,222,.5)}.btn-outline-info.disabled,.btn-outline-info:disabled{color:#5bc0de;background-color:transparent}.btn-outline-info.active,.btn-outline-info:active,.show>.btn-outline-info.dropdown-toggle{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-outline-success{color:#5cb85c;background-image:none;background-color:transparent;border-color:#5cb85c}.btn-outline-success:hover{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-outline-success.focus,.btn-outline-success:focus{-webkit-box-shadow:0 0 0 2px rgba(92,184,92,.5);box-shadow:0 0 0 2px rgba(92,184,92,.5)}.btn-outline-success.disabled,.btn-outline-success:disabled{color:#5cb85c;background-color:transparent}.btn-outline-success.active,.btn-outline-success:active,.show>.btn-outline-success.dropdown-toggle{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-outline-warning{color:#f0ad4e;background-image:none;background-color:transparent;border-color:#f0ad4e}.btn-outline-warning:hover{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-outline-warning.focus,.btn-outline-warning:focus{-webkit-box-shadow:0 0 0 2px rgba(240,173,78,.5);box-shadow:0 0 0 2px rgba(240,173,78,.5)}.btn-outline-warning.disabled,.btn-outline-warning:disabled{color:#f0ad4e;background-color:transparent}.btn-outline-warning.active,.btn-outline-warning:active,.show>.btn-outline-warning.dropdown-toggle{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-outline-danger{color:#d9534f;background-image:none;background-color:transparent;border-color:#d9534f}.btn-outline-danger:hover{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-outline-danger.focus,.btn-outline-danger:focus{-webkit-box-shadow:0 0 0 2px rgba(217,83,79,.5);box-shadow:0 0 0 2px rgba(217,83,79,.5)}.btn-outline-danger.disabled,.btn-outline-danger:disabled{color:#d9534f;background-color:transparent}.btn-outline-danger.active,.btn-outline-danger:active,.show>.btn-outline-danger.dropdown-toggle{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-link{font-weight:400;color:#0275d8;border-radius:0}.btn-link,.btn-link.active,.btn-link:active,.btn-link:disabled{background-color:transparent}.btn-link,.btn-link:active,.btn-link:focus{border-color:transparent}.btn-link:hover{border-color:transparent}.btn-link:focus,.btn-link:hover{color:#014c8c;text-decoration:underline;background-color:transparent}.btn-link:disabled{color:#636c72}.btn-link:disabled:focus,.btn-link:disabled:hover{text-decoration:none}.btn-group-lg>.btn,.btn-lg{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.btn-group-sm>.btn,.btn-sm{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.btn-block{display:block;width:100%}.btn-block+.btn-block{margin-top:.5rem}input[type=button].btn-block,input[type=reset].btn-block,input[type=submit].btn-block{width:100%}.fade{opacity:0;-webkit-transition:opacity .15s linear;-o-transition:opacity .15s linear;transition:opacity .15s linear}.fade.show{opacity:1}.collapse{display:none}.collapse.show{display:block}tr.collapse.show{display:table-row}tbody.collapse.show{display:table-row-group}.collapsing{position:relative;height:0;overflow:hidden;-webkit-transition:height .35s ease;-o-transition:height .35s ease;transition:height .35s ease}.dropdown,.dropup{position:relative}.dropdown-toggle::after{display:inline-block;width:0;height:0;margin-left:.3em;vertical-align:middle;content:"";border-top:.3em solid;border-right:.3em solid transparent;border-left:.3em solid transparent}.dropdown-toggle:focus{outline:0}.dropup .dropdown-toggle::after{border-top:0;border-bottom:.3em solid}.dropdown-menu{position:absolute;top:100%;left:0;z-index:1000;display:none;float:left;min-width:10rem;padding:.5rem 0;margin:.125rem 0 0;font-size:1rem;color:#292b2c;text-align:left;list-style:none;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.dropdown-divider{height:1px;margin:.5rem 0;overflow:hidden;background-color:#eceeef}.dropdown-item{display:block;width:100%;padding:3px 1.5rem;clear:both;font-weight:400;color:#292b2c;text-align:inherit;white-space:nowrap;background:0 0;border:0}.dropdown-item:focus,.dropdown-item:hover{color:#1d1e1f;text-decoration:none;background-color:#f7f7f9}.dropdown-item.active,.dropdown-item:active{color:#fff;text-decoration:none;background-color:#0275d8}.dropdown-item.disabled,.dropdown-item:disabled{color:#636c72;cursor:not-allowed;background-color:transparent}.show>.dropdown-menu{display:block}.show>a{outline:0}.dropdown-menu-right{right:0;left:auto}.dropdown-menu-left{right:auto;left:0}.dropdown-header{display:block;padding:.5rem 1.5rem;margin-bottom:0;font-size:.875rem;color:#636c72;white-space:nowrap}.dropdown-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;z-index:990}.dropup .dropdown-menu{top:auto;bottom:100%;margin-bottom:.125rem}.btn-group,.btn-group-vertical{position:relative;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;vertical-align:middle}.btn-group-vertical>.btn,.btn-group>.btn{position:relative;-webkit-box-flex:0;-webkit-flex:0 1 auto;-ms-flex:0 1 auto;flex:0 1 auto}.btn-group-vertical>.btn:hover,.btn-group>.btn:hover{z-index:2}.btn-group-vertical>.btn.active,.btn-group-vertical>.btn:active,.btn-group-vertical>.btn:focus,.btn-group>.btn.active,.btn-group>.btn:active,.btn-group>.btn:focus{z-index:2}.btn-group .btn+.btn,.btn-group .btn+.btn-group,.btn-group .btn-group+.btn,.btn-group .btn-group+.btn-group,.btn-group-vertical .btn+.btn,.btn-group-vertical .btn+.btn-group,.btn-group-vertical .btn-group+.btn,.btn-group-vertical .btn-group+.btn-group{margin-left:-1px}.btn-toolbar{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-pack:start;-webkit-justify-content:flex-start;-ms-flex-pack:start;justify-content:flex-start}.btn-toolbar .input-group{width:auto}.btn-group>.btn:not(:first-child):not(:last-child):not(.dropdown-toggle){border-radius:0}.btn-group>.btn:first-child{margin-left:0}.btn-group>.btn:first-child:not(:last-child):not(.dropdown-toggle){border-bottom-right-radius:0;border-top-right-radius:0}.btn-group>.btn:last-child:not(:first-child),.btn-group>.dropdown-toggle:not(:first-child){border-bottom-left-radius:0;border-top-left-radius:0}.btn-group>.btn-group{float:left}.btn-group>.btn-group:not(:first-child):not(:last-child)>.btn{border-radius:0}.btn-group>.btn-group:first-child:not(:last-child)>.btn:last-child,.btn-group>.btn-group:first-child:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-top-right-radius:0}.btn-group>.btn-group:last-child:not(:first-child)>.btn:first-child{border-bottom-left-radius:0;border-top-left-radius:0}.btn-group .dropdown-toggle:active,.btn-group.open .dropdown-toggle{outline:0}.btn+.dropdown-toggle-split{padding-right:.75rem;padding-left:.75rem}.btn+.dropdown-toggle-split::after{margin-left:0}.btn-group-sm>.btn+.dropdown-toggle-split,.btn-sm+.dropdown-toggle-split{padding-right:.375rem;padding-left:.375rem}.btn-group-lg>.btn+.dropdown-toggle-split,.btn-lg+.dropdown-toggle-split{padding-right:1.125rem;padding-left:1.125rem}.btn-group-vertical{display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;-webkit-box-align:start;-webkit-align-items:flex-start;-ms-flex-align:start;align-items:flex-start;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center}.btn-group-vertical .btn,.btn-group-vertical .btn-group{width:100%}.btn-group-vertical>.btn+.btn,.btn-group-vertical>.btn+.btn-group,.btn-group-vertical>.btn-group+.btn,.btn-group-vertical>.btn-group+.btn-group{margin-top:-1px;margin-left:0}.btn-group-vertical>.btn:not(:first-child):not(:last-child){border-radius:0}.btn-group-vertical>.btn:first-child:not(:last-child){border-bottom-right-radius:0;border-bottom-left-radius:0}.btn-group-vertical>.btn:last-child:not(:first-child){border-top-right-radius:0;border-top-left-radius:0}.btn-group-vertical>.btn-group:not(:first-child):not(:last-child)>.btn{border-radius:0}.btn-group-vertical>.btn-group:first-child:not(:last-child)>.btn:last-child,.btn-group-vertical>.btn-group:first-child:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-bottom-left-radius:0}.btn-group-vertical>.btn-group:last-child:not(:first-child)>.btn:first-child{border-top-right-radius:0;border-top-left-radius:0}[data-toggle=buttons]>.btn input[type=checkbox],[data-toggle=buttons]>.btn input[type=radio],[data-toggle=buttons]>.btn-group>.btn input[type=checkbox],[data-toggle=buttons]>.btn-group>.btn input[type=radio]{position:absolute;clip:rect(0,0,0,0);pointer-events:none}.input-group{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;width:100%}.input-group .form-control{position:relative;z-index:2;-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;width:1%;margin-bottom:0}.input-group .form-control:active,.input-group .form-control:focus,.input-group .form-control:hover{z-index:3}.input-group .form-control,.input-group-addon,.input-group-btn{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center}.input-group .form-control:not(:first-child):not(:last-child),.input-group-addon:not(:first-child):not(:last-child),.input-group-btn:not(:first-child):not(:last-child){border-radius:0}.input-group-addon,.input-group-btn{white-space:nowrap;vertical-align:middle}.input-group-addon{padding:.5rem .75rem;margin-bottom:0;font-size:1rem;font-weight:400;line-height:1.25;color:#464a4c;text-align:center;background-color:#eceeef;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.input-group-addon.form-control-sm,.input-group-sm>.input-group-addon,.input-group-sm>.input-group-btn>.input-group-addon.btn{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.input-group-addon.form-control-lg,.input-group-lg>.input-group-addon,.input-group-lg>.input-group-btn>.input-group-addon.btn{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.input-group-addon input[type=checkbox],.input-group-addon input[type=radio]{margin-top:0}.input-group .form-control:not(:last-child),.input-group-addon:not(:last-child),.input-group-btn:not(:first-child)>.btn-group:not(:last-child)>.btn,.input-group-btn:not(:first-child)>.btn:not(:last-child):not(.dropdown-toggle),.input-group-btn:not(:last-child)>.btn,.input-group-btn:not(:last-child)>.btn-group>.btn,.input-group-btn:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-top-right-radius:0}.input-group-addon:not(:last-child){border-right:0}.input-group .form-control:not(:first-child),.input-group-addon:not(:first-child),.input-group-btn:not(:first-child)>.btn,.input-group-btn:not(:first-child)>.btn-group>.btn,.input-group-btn:not(:first-child)>.dropdown-toggle,.input-group-btn:not(:last-child)>.btn-group:not(:first-child)>.btn,.input-group-btn:not(:last-child)>.btn:not(:first-child){border-bottom-left-radius:0;border-top-left-radius:0}.form-control+.input-group-addon:not(:first-child){border-left:0}.input-group-btn{position:relative;font-size:0;white-space:nowrap}.input-group-btn>.btn{position:relative;-webkit-box-flex:1;-webkit-flex:1 1 0%;-ms-flex:1 1 0%;flex:1 1 0%}.input-group-btn>.btn+.btn{margin-left:-1px}.input-group-btn>.btn:active,.input-group-btn>.btn:focus,.input-group-btn>.btn:hover{z-index:3}.input-group-btn:not(:last-child)>.btn,.input-group-btn:not(:last-child)>.btn-group{margin-right:-1px}.input-group-btn:not(:first-child)>.btn,.input-group-btn:not(:first-child)>.btn-group{z-index:2;margin-left:-1px}.input-group-btn:not(:first-child)>.btn-group:active,.input-group-btn:not(:first-child)>.btn-group:focus,.input-group-btn:not(:first-child)>.btn-group:hover,.input-group-btn:not(:first-child)>.btn:active,.input-group-btn:not(:first-child)>.btn:focus,.input-group-btn:not(:first-child)>.btn:hover{z-index:3}.custom-control{position:relative;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;min-height:1.5rem;padding-left:1.5rem;margin-right:1rem;cursor:pointer}.custom-control-input{position:absolute;z-index:-1;opacity:0}.custom-control-input:checked~.custom-control-indicator{color:#fff;background-color:#0275d8}.custom-control-input:focus~.custom-control-indicator{-webkit-box-shadow:0 0 0 1px #fff,0 0 0 3px #0275d8;box-shadow:0 0 0 1px #fff,0 0 0 3px #0275d8}.custom-control-input:active~.custom-control-indicator{color:#fff;background-color:#8fcafe}.custom-control-input:disabled~.custom-control-indicator{cursor:not-allowed;background-color:#eceeef}.custom-control-input:disabled~.custom-control-description{color:#636c72;cursor:not-allowed}.custom-control-indicator{position:absolute;top:.25rem;left:0;display:block;width:1rem;height:1rem;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;background-color:#ddd;background-repeat:no-repeat;background-position:center center;-webkit-background-size:50% 50%;background-size:50% 50%}.custom-checkbox .custom-control-indicator{border-radius:.25rem}.custom-checkbox .custom-control-input:checked~.custom-control-indicator{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%23fff' d='M6.564.75l-3.59 3.612-1.538-1.55L0 4.26 2.974 7.25 8 2.193z'/%3E%3C/svg%3E")}.custom-checkbox .custom-control-input:indeterminate~.custom-control-indicator{background-color:#0275d8;background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 4 4'%3E%3Cpath stroke='%23fff' d='M0 2h4'/%3E%3C/svg%3E")}.custom-radio .custom-control-indicator{border-radius:50%}.custom-radio .custom-control-input:checked~.custom-control-indicator{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3E%3Ccircle r='3' fill='%23fff'/%3E%3C/svg%3E")}.custom-controls-stacked{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column}.custom-controls-stacked .custom-control{margin-bottom:.25rem}.custom-controls-stacked .custom-control+.custom-control{margin-left:0}.custom-select{display:inline-block;max-width:100%;height:calc(2.25rem + 2px);padding:.375rem 1.75rem .375rem .75rem;line-height:1.25;color:#464a4c;vertical-align:middle;background:#fff url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 4 5'%3E%3Cpath fill='%23333' d='M2 0L0 2h4zm0 5L0 3h4z'/%3E%3C/svg%3E") no-repeat right .75rem center;-webkit-background-size:8px 10px;background-size:8px 10px;border:1px solid rgba(0,0,0,.15);border-radius:.25rem;-moz-appearance:none;-webkit-appearance:none}.custom-select:focus{border-color:#5cb3fd;outline:0}.custom-select:focus::-ms-value{color:#464a4c;background-color:#fff}.custom-select:disabled{color:#636c72;cursor:not-allowed;background-color:#eceeef}.custom-select::-ms-expand{opacity:0}.custom-select-sm{padding-top:.375rem;padding-bottom:.375rem;font-size:75%}.custom-file{position:relative;display:inline-block;max-width:100%;height:2.5rem;margin-bottom:0;cursor:pointer}.custom-file-input{min-width:14rem;max-width:100%;height:2.5rem;margin:0;filter:alpha(opacity=0);opacity:0}.custom-file-control{position:absolute;top:0;right:0;left:0;z-index:5;height:2.5rem;padding:.5rem 1rem;line-height:1.5;color:#464a4c;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;background-color:#fff;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.custom-file-control:lang(en)::after{content:"Choose file..."}.custom-file-control::before{position:absolute;top:-1px;right:-1px;bottom:-1px;z-index:6;display:block;height:2.5rem;padding:.5rem 1rem;line-height:1.5;color:#464a4c;background-color:#eceeef;border:1px solid rgba(0,0,0,.15);border-radius:0 .25rem .25rem 0}.custom-file-control:lang(en)::before{content:"Browse"}.nav{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;padding-left:0;margin-bottom:0;list-style:none}.nav-link{display:block;padding:.5em 1em}.nav-link:focus,.nav-link:hover{text-decoration:none}.nav-link.disabled{color:#636c72;cursor:not-allowed}.nav-tabs{border-bottom:1px solid #ddd}.nav-tabs .nav-item{margin-bottom:-1px}.nav-tabs .nav-link{border:1px solid transparent;border-top-right-radius:.25rem;border-top-left-radius:.25rem}.nav-tabs .nav-link:focus,.nav-tabs .nav-link:hover{border-color:#eceeef #eceeef #ddd}.nav-tabs .nav-link.disabled{color:#636c72;background-color:transparent;border-color:transparent}.nav-tabs .nav-item.show .nav-link,.nav-tabs .nav-link.active{color:#464a4c;background-color:#fff;border-color:#ddd #ddd #fff}.nav-tabs .dropdown-menu{margin-top:-1px;border-top-right-radius:0;border-top-left-radius:0}.nav-pills .nav-link{border-radius:.25rem}.nav-pills .nav-item.show .nav-link,.nav-pills .nav-link.active{color:#fff;cursor:default;background-color:#0275d8}.nav-fill .nav-item{-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;text-align:center}.nav-justified .nav-item{-webkit-box-flex:1;-webkit-flex:1 1 100%;-ms-flex:1 1 100%;flex:1 1 100%;text-align:center}.tab-content>.tab-pane{display:none}.tab-content>.active{display:block}.navbar{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding:.5rem 1rem}.navbar-brand{display:inline-block;padding-top:.25rem;padding-bottom:.25rem;margin-right:1rem;font-size:1.25rem;line-height:inherit;white-space:nowrap}.navbar-brand:focus,.navbar-brand:hover{text-decoration:none}.navbar-nav{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding-left:0;margin-bottom:0;list-style:none}.navbar-nav .nav-link{padding-right:0;padding-left:0}.navbar-text{display:inline-block;padding-top:.425rem;padding-bottom:.425rem}.navbar-toggler{-webkit-align-self:flex-start;-ms-flex-item-align:start;align-self:flex-start;padding:.25rem .75rem;font-size:1.25rem;line-height:1;background:0 0;border:1px solid transparent;border-radius:.25rem}.navbar-toggler:focus,.navbar-toggler:hover{text-decoration:none}.navbar-toggler-icon{display:inline-block;width:1.5em;height:1.5em;vertical-align:middle;content:"";background:no-repeat center center;-webkit-background-size:100% 100%;background-size:100% 100%}.navbar-toggler-left{position:absolute;left:1rem}.navbar-toggler-right{position:absolute;right:1rem}@media (max-width:575px){.navbar-toggleable .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable>.container{padding-right:0;padding-left:0}}@media (min-width:576px){.navbar-toggleable{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable .navbar-toggler{display:none}}@media (max-width:767px){.navbar-toggleable-sm .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-sm>.container{padding-right:0;padding-left:0}}@media (min-width:768px){.navbar-toggleable-sm{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-sm .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-sm .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-sm>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-sm .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-sm .navbar-toggler{display:none}}@media (max-width:991px){.navbar-toggleable-md .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-md>.container{padding-right:0;padding-left:0}}@media (min-width:992px){.navbar-toggleable-md{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-md .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-md .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-md>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-md .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-md .navbar-toggler{display:none}}@media (max-width:1199px){.navbar-toggleable-lg .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-lg>.container{padding-right:0;padding-left:0}}@media (min-width:1200px){.navbar-toggleable-lg{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-lg .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-lg .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-lg>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-lg .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-lg .navbar-toggler{display:none}}.navbar-toggleable-xl{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-xl .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-xl>.container{padding-right:0;padding-left:0}.navbar-toggleable-xl .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-xl .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-xl>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-xl .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-xl .navbar-toggler{display:none}.navbar-light .navbar-brand,.navbar-light .navbar-toggler{color:rgba(0,0,0,.9)}.navbar-light .navbar-brand:focus,.navbar-light .navbar-brand:hover,.navbar-light .navbar-toggler:focus,.navbar-light .navbar-toggler:hover{color:rgba(0,0,0,.9)}.navbar-light .navbar-nav .nav-link{color:rgba(0,0,0,.5)}.navbar-light .navbar-nav .nav-link:focus,.navbar-light .navbar-nav .nav-link:hover{color:rgba(0,0,0,.7)}.navbar-light .navbar-nav .nav-link.disabled{color:rgba(0,0,0,.3)}.navbar-light .navbar-nav .active>.nav-link,.navbar-light .navbar-nav .nav-link.active,.navbar-light .navbar-nav .nav-link.open,.navbar-light .navbar-nav .open>.nav-link{color:rgba(0,0,0,.9)}.navbar-light .navbar-toggler{border-color:rgba(0,0,0,.1)}.navbar-light .navbar-toggler-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg viewBox='0 0 32 32' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath stroke='rgba(0, 0, 0, 0.5)' stroke-width='2' stroke-linecap='round' stroke-miterlimit='10' d='M4 8h24M4 16h24M4 24h24'/%3E%3C/svg%3E")}.navbar-light .navbar-text{color:rgba(0,0,0,.5)}.navbar-inverse .navbar-brand,.navbar-inverse .navbar-toggler{color:#fff}.navbar-inverse .navbar-brand:focus,.navbar-inverse .navbar-brand:hover,.navbar-inverse .navbar-toggler:focus,.navbar-inverse .navbar-toggler:hover{color:#fff}.navbar-inverse .navbar-nav .nav-link{color:rgba(255,255,255,.5)}.navbar-inverse .navbar-nav .nav-link:focus,.navbar-inverse .navbar-nav .nav-link:hover{color:rgba(255,255,255,.75)}.navbar-inverse .navbar-nav .nav-link.disabled{color:rgba(255,255,255,.25)}.navbar-inverse .navbar-nav .active>.nav-link,.navbar-inverse .navbar-nav .nav-link.active,.navbar-inverse .navbar-nav .nav-link.open,.navbar-inverse .navbar-nav .open>.nav-link{color:#fff}.navbar-inverse .navbar-toggler{border-color:rgba(255,255,255,.1)}.navbar-inverse .navbar-toggler-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg viewBox='0 0 32 32' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath stroke='rgba(255, 255, 255, 0.5)' stroke-width='2' stroke-linecap='round' stroke-miterlimit='10' d='M4 8h24M4 16h24M4 24h24'/%3E%3C/svg%3E")}.navbar-inverse .navbar-text{color:rgba(255,255,255,.5)}.card{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;background-color:#fff;border:1px solid rgba(0,0,0,.125);border-radius:.25rem}.card-block{-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;padding:1.25rem}.card-title{margin-bottom:.75rem}.card-subtitle{margin-top:-.375rem;margin-bottom:0}.card-text:last-child{margin-bottom:0}.card-link:hover{text-decoration:none}.card-link+.card-link{margin-left:1.25rem}.card>.list-group:first-child .list-group-item:first-child{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.card>.list-group:last-child .list-group-item:last-child{border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.card-header{padding:.75rem 1.25rem;margin-bottom:0;background-color:#f7f7f9;border-bottom:1px solid rgba(0,0,0,.125)}.card-header:first-child{border-radius:calc(.25rem - 1px) calc(.25rem - 1px) 0 0}.card-footer{padding:.75rem 1.25rem;background-color:#f7f7f9;border-top:1px solid rgba(0,0,0,.125)}.card-footer:last-child{border-radius:0 0 calc(.25rem - 1px) calc(.25rem - 1px)}.card-header-tabs{margin-right:-.625rem;margin-bottom:-.75rem;margin-left:-.625rem;border-bottom:0}.card-header-pills{margin-right:-.625rem;margin-left:-.625rem}.card-primary{background-color:#0275d8;border-color:#0275d8}.card-primary .card-footer,.card-primary .card-header{background-color:transparent}.card-success{background-color:#5cb85c;border-color:#5cb85c}.card-success .card-footer,.card-success .card-header{background-color:transparent}.card-info{background-color:#5bc0de;border-color:#5bc0de}.card-info .card-footer,.card-info .card-header{background-color:transparent}.card-warning{background-color:#f0ad4e;border-color:#f0ad4e}.card-warning .card-footer,.card-warning .card-header{background-color:transparent}.card-danger{background-color:#d9534f;border-color:#d9534f}.card-danger .card-footer,.card-danger .card-header{background-color:transparent}.card-outline-primary{background-color:transparent;border-color:#0275d8}.card-outline-secondary{background-color:transparent;border-color:#ccc}.card-outline-info{background-color:transparent;border-color:#5bc0de}.card-outline-success{background-color:transparent;border-color:#5cb85c}.card-outline-warning{background-color:transparent;border-color:#f0ad4e}.card-outline-danger{background-color:transparent;border-color:#d9534f}.card-inverse{color:rgba(255,255,255,.65)}.card-inverse .card-footer,.card-inverse .card-header{background-color:transparent;border-color:rgba(255,255,255,.2)}.card-inverse .card-blockquote,.card-inverse .card-footer,.card-inverse .card-header,.card-inverse .card-title{color:#fff}.card-inverse .card-blockquote .blockquote-footer,.card-inverse .card-link,.card-inverse .card-subtitle,.card-inverse .card-text{color:rgba(255,255,255,.65)}.card-inverse .card-link:focus,.card-inverse .card-link:hover{color:#fff}.card-blockquote{padding:0;margin-bottom:0;border-left:0}.card-img{border-radius:calc(.25rem - 1px)}.card-img-overlay{position:absolute;top:0;right:0;bottom:0;left:0;padding:1.25rem}.card-img-top{border-top-right-radius:calc(.25rem - 1px);border-top-left-radius:calc(.25rem - 1px)}.card-img-bottom{border-bottom-right-radius:calc(.25rem - 1px);border-bottom-left-radius:calc(.25rem - 1px)}@media (min-width:576px){.card-deck{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap}.card-deck .card{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-flex:1;-webkit-flex:1 0 0%;-ms-flex:1 0 0%;flex:1 0 0%;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column}.card-deck .card:not(:first-child){margin-left:15px}.card-deck .card:not(:last-child){margin-right:15px}}@media (min-width:576px){.card-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap}.card-group .card{-webkit-box-flex:1;-webkit-flex:1 0 0%;-ms-flex:1 0 0%;flex:1 0 0%}.card-group .card+.card{margin-left:0;border-left:0}.card-group .card:first-child{border-bottom-right-radius:0;border-top-right-radius:0}.card-group .card:first-child .card-img-top{border-top-right-radius:0}.card-group .card:first-child .card-img-bottom{border-bottom-right-radius:0}.card-group .card:last-child{border-bottom-left-radius:0;border-top-left-radius:0}.card-group .card:last-child .card-img-top{border-top-left-radius:0}.card-group .card:last-child .card-img-bottom{border-bottom-left-radius:0}.card-group .card:not(:first-child):not(:last-child){border-radius:0}.card-group .card:not(:first-child):not(:last-child) .card-img-bottom,.card-group .card:not(:first-child):not(:last-child) .card-img-top{border-radius:0}}@media (min-width:576px){.card-columns{-webkit-column-count:3;-moz-column-count:3;column-count:3;-webkit-column-gap:1.25rem;-moz-column-gap:1.25rem;column-gap:1.25rem}.card-columns .card{display:inline-block;width:100%;margin-bottom:.75rem}}.breadcrumb{padding:.75rem 1rem;margin-bottom:1rem;list-style:none;background-color:#eceeef;border-radius:.25rem}.breadcrumb::after{display:block;content:"";clear:both}.breadcrumb-item{float:left}.breadcrumb-item+.breadcrumb-item::before{display:inline-block;padding-right:.5rem;padding-left:.5rem;color:#636c72;content:"/"}.breadcrumb-item+.breadcrumb-item:hover::before{text-decoration:underline}.breadcrumb-item+.breadcrumb-item:hover::before{text-decoration:none}.breadcrumb-item.active{color:#636c72}.pagination{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;padding-left:0;list-style:none;border-radius:.25rem}.page-item:first-child .page-link{margin-left:0;border-bottom-left-radius:.25rem;border-top-left-radius:.25rem}.page-item:last-child .page-link{border-bottom-right-radius:.25rem;border-top-right-radius:.25rem}.page-item.active .page-link{z-index:2;color:#fff;background-color:#0275d8;border-color:#0275d8}.page-item.disabled .page-link{color:#636c72;pointer-events:none;cursor:not-allowed;background-color:#fff;border-color:#ddd}.page-link{position:relative;display:block;padding:.5rem .75rem;margin-left:-1px;line-height:1.25;color:#0275d8;background-color:#fff;border:1px solid #ddd}.page-link:focus,.page-link:hover{color:#014c8c;text-decoration:none;background-color:#eceeef;border-color:#ddd}.pagination-lg .page-link{padding:.75rem 1.5rem;font-size:1.25rem}.pagination-lg .page-item:first-child .page-link{border-bottom-left-radius:.3rem;border-top-left-radius:.3rem}.pagination-lg .page-item:last-child .page-link{border-bottom-right-radius:.3rem;border-top-right-radius:.3rem}.pagination-sm .page-link{padding:.25rem .5rem;font-size:.875rem}.pagination-sm .page-item:first-child .page-link{border-bottom-left-radius:.2rem;border-top-left-radius:.2rem}.pagination-sm .page-item:last-child .page-link{border-bottom-right-radius:.2rem;border-top-right-radius:.2rem}.badge{display:inline-block;padding:.25em .4em;font-size:75%;font-weight:700;line-height:1;color:#fff;text-align:center;white-space:nowrap;vertical-align:baseline;border-radius:.25rem}.badge:empty{display:none}.btn .badge{position:relative;top:-1px}a.badge:focus,a.badge:hover{color:#fff;text-decoration:none;cursor:pointer}.badge-pill{padding-right:.6em;padding-left:.6em;border-radius:10rem}.badge-default{background-color:#636c72}.badge-default[href]:focus,.badge-default[href]:hover{background-color:#4b5257}.badge-primary{background-color:#0275d8}.badge-primary[href]:focus,.badge-primary[href]:hover{background-color:#025aa5}.badge-success{background-color:#5cb85c}.badge-success[href]:focus,.badge-success[href]:hover{background-color:#449d44}.badge-info{background-color:#5bc0de}.badge-info[href]:focus,.badge-info[href]:hover{background-color:#31b0d5}.badge-warning{background-color:#f0ad4e}.badge-warning[href]:focus,.badge-warning[href]:hover{background-color:#ec971f}.badge-danger{background-color:#d9534f}.badge-danger[href]:focus,.badge-danger[href]:hover{background-color:#c9302c}.jumbotron{padding:2rem 1rem;margin-bottom:2rem;background-color:#eceeef;border-radius:.3rem}@media (min-width:576px){.jumbotron{padding:4rem 2rem}}.jumbotron-hr{border-top-color:#d0d5d8}.jumbotron-fluid{padding-right:0;padding-left:0;border-radius:0}.alert{padding:.75rem 1.25rem;margin-bottom:1rem;border:1px solid transparent;border-radius:.25rem}.alert-heading{color:inherit}.alert-link{font-weight:700}.alert-dismissible .close{position:relative;top:-.75rem;right:-1.25rem;padding:.75rem 1.25rem;color:inherit}.alert-success{background-color:#dff0d8;border-color:#d0e9c6;color:#3c763d}.alert-success hr{border-top-color:#c1e2b3}.alert-success .alert-link{color:#2b542c}.alert-info{background-color:#d9edf7;border-color:#bcdff1;color:#31708f}.alert-info hr{border-top-color:#a6d5ec}.alert-info .alert-link{color:#245269}.alert-warning{background-color:#fcf8e3;border-color:#faf2cc;color:#8a6d3b}.alert-warning hr{border-top-color:#f7ecb5}.alert-warning .alert-link{color:#66512c}.alert-danger{background-color:#f2dede;border-color:#ebcccc;color:#a94442}.alert-danger hr{border-top-color:#e4b9b9}.alert-danger .alert-link{color:#843534}@-webkit-keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}@-o-keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}@keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}.progress{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;overflow:hidden;font-size:.75rem;line-height:1rem;text-align:center;background-color:#eceeef;border-radius:.25rem}.progress-bar{height:1rem;color:#fff;background-color:#0275d8}.progress-bar-striped{background-image:-webkit-linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);background-image:-o-linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);background-image:linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);-webkit-background-size:1rem 1rem;background-size:1rem 1rem}.progress-bar-animated{-webkit-animation:progress-bar-stripes 1s linear infinite;-o-animation:progress-bar-stripes 1s linear infinite;animation:progress-bar-stripes 1s linear infinite}.media{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:start;-webkit-align-items:flex-start;-ms-flex-align:start;align-items:flex-start}.media-body{-webkit-box-flex:1;-webkit-flex:1 1 0%;-ms-flex:1 1 0%;flex:1 1 0%}.list-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding-left:0;margin-bottom:0}.list-group-item-action{width:100%;color:#464a4c;text-align:inherit}.list-group-item-action .list-group-item-heading{color:#292b2c}.list-group-item-action:focus,.list-group-item-action:hover{color:#464a4c;text-decoration:none;background-color:#f7f7f9}.list-group-item-action:active{color:#292b2c;background-color:#eceeef}.list-group-item{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;padding:.75rem 1.25rem;margin-bottom:-1px;background-color:#fff;border:1px solid rgba(0,0,0,.125)}.list-group-item:first-child{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.list-group-item:last-child{margin-bottom:0;border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.list-group-item:focus,.list-group-item:hover{text-decoration:none}.list-group-item.disabled,.list-group-item:disabled{color:#636c72;cursor:not-allowed;background-color:#fff}.list-group-item.disabled .list-group-item-heading,.list-group-item:disabled .list-group-item-heading{color:inherit}.list-group-item.disabled .list-group-item-text,.list-group-item:disabled .list-group-item-text{color:#636c72}.list-group-item.active{z-index:2;color:#fff;background-color:#0275d8;border-color:#0275d8}.list-group-item.active .list-group-item-heading,.list-group-item.active .list-group-item-heading>.small,.list-group-item.active .list-group-item-heading>small{color:inherit}.list-group-item.active .list-group-item-text{color:#daeeff}.list-group-flush .list-group-item{border-right:0;border-left:0;border-radius:0}.list-group-flush:first-child .list-group-item:first-child{border-top:0}.list-group-flush:last-child .list-group-item:last-child{border-bottom:0}.list-group-item-success{color:#3c763d;background-color:#dff0d8}a.list-group-item-success,button.list-group-item-success{color:#3c763d}a.list-group-item-success .list-group-item-heading,button.list-group-item-success .list-group-item-heading{color:inherit}a.list-group-item-success:focus,a.list-group-item-success:hover,button.list-group-item-success:focus,button.list-group-item-success:hover{color:#3c763d;background-color:#d0e9c6}a.list-group-item-success.active,button.list-group-item-success.active{color:#fff;background-color:#3c763d;border-color:#3c763d}.list-group-item-info{color:#31708f;background-color:#d9edf7}a.list-group-item-info,button.list-group-item-info{color:#31708f}a.list-group-item-info .list-group-item-heading,button.list-group-item-info .list-group-item-heading{color:inherit}a.list-group-item-info:focus,a.list-group-item-info:hover,button.list-group-item-info:focus,button.list-group-item-info:hover{color:#31708f;background-color:#c4e3f3}a.list-group-item-info.active,button.list-group-item-info.active{color:#fff;background-color:#31708f;border-color:#31708f}.list-group-item-warning{color:#8a6d3b;background-color:#fcf8e3}a.list-group-item-warning,button.list-group-item-warning{color:#8a6d3b}a.list-group-item-warning .list-group-item-heading,button.list-group-item-warning .list-group-item-heading{color:inherit}a.list-group-item-warning:focus,a.list-group-item-warning:hover,button.list-group-item-warning:focus,button.list-group-item-warning:hover{color:#8a6d3b;background-color:#faf2cc}a.list-group-item-warning.active,button.list-group-item-warning.active{color:#fff;background-color:#8a6d3b;border-color:#8a6d3b}.list-group-item-danger{color:#a94442;background-color:#f2dede}a.list-group-item-danger,button.list-group-item-danger{color:#a94442}a.list-group-item-danger .list-group-item-heading,button.list-group-item-danger .list-group-item-heading{color:inherit}a.list-group-item-danger:focus,a.list-group-item-danger:hover,button.list-group-item-danger:focus,button.list-group-item-danger:hover{color:#a94442;background-color:#ebcccc}a.list-group-item-danger.active,button.list-group-item-danger.active{color:#fff;background-color:#a94442;border-color:#a94442}.embed-responsive{position:relative;display:block;width:100%;padding:0;overflow:hidden}.embed-responsive::before{display:block;content:""}.embed-responsive .embed-responsive-item,.embed-responsive embed,.embed-responsive iframe,.embed-responsive object,.embed-responsive video{position:absolute;top:0;bottom:0;left:0;width:100%;height:100%;border:0}.embed-responsive-21by9::before{padding-top:42.857143%}.embed-responsive-16by9::before{padding-top:56.25%}.embed-responsive-4by3::before{padding-top:75%}.embed-responsive-1by1::before{padding-top:100%}.close{float:right;font-size:1.5rem;font-weight:700;line-height:1;color:#000;text-shadow:0 1px 0 #fff;opacity:.5}.close:focus,.close:hover{color:#000;text-decoration:none;cursor:pointer;opacity:.75}button.close{padding:0;cursor:pointer;background:0 0;border:0;-webkit-appearance:none}.modal-open{overflow:hidden}.modal{position:fixed;top:0;right:0;bottom:0;left:0;z-index:1050;display:none;overflow:hidden;outline:0}.modal.fade .modal-dialog{-webkit-transition:-webkit-transform .3s ease-out;transition:-webkit-transform .3s ease-out;-o-transition:-o-transform .3s ease-out;transition:transform .3s ease-out;transition:transform .3s ease-out,-webkit-transform .3s ease-out,-o-transform .3s ease-out;-webkit-transform:translate(0,-25%);-o-transform:translate(0,-25%);transform:translate(0,-25%)}.modal.show .modal-dialog{-webkit-transform:translate(0,0);-o-transform:translate(0,0);transform:translate(0,0)}.modal-open .modal{overflow-x:hidden;overflow-y:auto}.modal-dialog{position:relative;width:auto;margin:10px}.modal-content{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.2);border-radius:.3rem;outline:0}.modal-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;z-index:1040;background-color:#000}.modal-backdrop.fade{opacity:0}.modal-backdrop.show{opacity:.5}.modal-header{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:justify;-webkit-justify-content:space-between;-ms-flex-pack:justify;justify-content:space-between;padding:15px;border-bottom:1px solid #eceeef}.modal-title{margin-bottom:0;line-height:1.5}.modal-body{position:relative;-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;padding:15px}.modal-footer{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:end;-webkit-justify-content:flex-end;-ms-flex-pack:end;justify-content:flex-end;padding:15px;border-top:1px solid #eceeef}.modal-footer>:not(:first-child){margin-left:.25rem}.modal-footer>:not(:last-child){margin-right:.25rem}.modal-scrollbar-measure{position:absolute;top:-9999px;width:50px;height:50px;overflow:scroll}@media (min-width:576px){.modal-dialog{max-width:500px;margin:30px auto}.modal-sm{max-width:300px}}@media (min-width:992px){.modal-lg{max-width:800px}}.tooltip{position:absolute;z-index:1070;display:block;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-style:normal;font-weight:400;letter-spacing:normal;line-break:auto;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;white-space:normal;word-break:normal;word-spacing:normal;font-size:.875rem;word-wrap:break-word;opacity:0}.tooltip.show{opacity:.9}.tooltip.bs-tether-element-attached-bottom,.tooltip.tooltip-top{padding:5px 0;margin-top:-3px}.tooltip.bs-tether-element-attached-bottom .tooltip-inner::before,.tooltip.tooltip-top .tooltip-inner::before{bottom:0;left:50%;margin-left:-5px;content:"";border-width:5px 5px 0;border-top-color:#000}.tooltip.bs-tether-element-attached-left,.tooltip.tooltip-right{padding:0 5px;margin-left:3px}.tooltip.bs-tether-element-attached-left .tooltip-inner::before,.tooltip.tooltip-right .tooltip-inner::before{top:50%;left:0;margin-top:-5px;content:"";border-width:5px 5px 5px 0;border-right-color:#000}.tooltip.bs-tether-element-attached-top,.tooltip.tooltip-bottom{padding:5px 0;margin-top:3px}.tooltip.bs-tether-element-attached-top .tooltip-inner::before,.tooltip.tooltip-bottom .tooltip-inner::before{top:0;left:50%;margin-left:-5px;content:"";border-width:0 5px 5px;border-bottom-color:#000}.tooltip.bs-tether-element-attached-right,.tooltip.tooltip-left{padding:0 5px;margin-left:-3px}.tooltip.bs-tether-element-attached-right .tooltip-inner::before,.tooltip.tooltip-left .tooltip-inner::before{top:50%;right:0;margin-top:-5px;content:"";border-width:5px 0 5px 5px;border-left-color:#000}.tooltip-inner{max-width:200px;padding:3px 8px;color:#fff;text-align:center;background-color:#000;border-radius:.25rem}.tooltip-inner::before{position:absolute;width:0;height:0;border-color:transparent;border-style:solid}.popover{position:absolute;top:0;left:0;z-index:1060;display:block;max-width:276px;padding:1px;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-style:normal;font-weight:400;letter-spacing:normal;line-break:auto;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;white-space:normal;word-break:normal;word-spacing:normal;font-size:.875rem;word-wrap:break-word;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.2);border-radius:.3rem}.popover.bs-tether-element-attached-bottom,.popover.popover-top{margin-top:-10px}.popover.bs-tether-element-attached-bottom::after,.popover.bs-tether-element-attached-bottom::before,.popover.popover-top::after,.popover.popover-top::before{left:50%;border-bottom-width:0}.popover.bs-tether-element-attached-bottom::before,.popover.popover-top::before{bottom:-11px;margin-left:-11px;border-top-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-bottom::after,.popover.popover-top::after{bottom:-10px;margin-left:-10px;border-top-color:#fff}.popover.bs-tether-element-attached-left,.popover.popover-right{margin-left:10px}.popover.bs-tether-element-attached-left::after,.popover.bs-tether-element-attached-left::before,.popover.popover-right::after,.popover.popover-right::before{top:50%;border-left-width:0}.popover.bs-tether-element-attached-left::before,.popover.popover-right::before{left:-11px;margin-top:-11px;border-right-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-left::after,.popover.popover-right::after{left:-10px;margin-top:-10px;border-right-color:#fff}.popover.bs-tether-element-attached-top,.popover.popover-bottom{margin-top:10px}.popover.bs-tether-element-attached-top::after,.popover.bs-tether-element-attached-top::before,.popover.popover-bottom::after,.popover.popover-bottom::before{left:50%;border-top-width:0}.popover.bs-tether-element-attached-top::before,.popover.popover-bottom::before{top:-11px;margin-left:-11px;border-bottom-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-top::after,.popover.popover-bottom::after{top:-10px;margin-left:-10px;border-bottom-color:#f7f7f7}.popover.bs-tether-element-attached-top .popover-title::before,.popover.popover-bottom .popover-title::before{position:absolute;top:0;left:50%;display:block;width:20px;margin-left:-10px;content:"";border-bottom:1px solid #f7f7f7}.popover.bs-tether-element-attached-right,.popover.popover-left{margin-left:-10px}.popover.bs-tether-element-attached-right::after,.popover.bs-tether-element-attached-right::before,.popover.popover-left::after,.popover.popover-left::before{top:50%;border-right-width:0}.popover.bs-tether-element-attached-right::before,.popover.popover-left::before{right:-11px;margin-top:-11px;border-left-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-right::after,.popover.popover-left::after{right:-10px;margin-top:-10px;border-left-color:#fff}.popover-title{padding:8px 14px;margin-bottom:0;font-size:1rem;background-color:#f7f7f7;border-bottom:1px solid #ebebeb;border-top-right-radius:calc(.3rem - 1px);border-top-left-radius:calc(.3rem - 1px)}.popover-title:empty{display:none}.popover-content{padding:9px 14px}.popover::after,.popover::before{position:absolute;display:block;width:0;height:0;border-color:transparent;border-style:solid}.popover::before{content:"";border-width:11px}.popover::after{content:"";border-width:10px}.carousel{position:relative}.carousel-inner{position:relative;width:100%;overflow:hidden}.carousel-item{position:relative;display:none;width:100%}@media (-webkit-transform-3d){.carousel-item{-webkit-transition:-webkit-transform .6s ease-in-out;transition:-webkit-transform .6s ease-in-out;-o-transition:-o-transform .6s ease-in-out;transition:transform .6s ease-in-out;transition:transform .6s ease-in-out,-webkit-transform .6s ease-in-out,-o-transform .6s ease-in-out;-webkit-backface-visibility:hidden;backface-visibility:hidden;-webkit-perspective:1000px;perspective:1000px}}@supports ((-webkit-transform:translate3d(0,0,0)) or (transform:translate3d(0,0,0))){.carousel-item{-webkit-transition:-webkit-transform .6s ease-in-out;transition:-webkit-transform .6s ease-in-out;-o-transition:-o-transform .6s ease-in-out;transition:transform .6s ease-in-out;transition:transform .6s ease-in-out,-webkit-transform .6s ease-in-out,-o-transform .6s ease-in-out;-webkit-backface-visibility:hidden;backface-visibility:hidden;-webkit-perspective:1000px;perspective:1000px}}.carousel-item-next,.carousel-item-prev,.carousel-item.active{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex}.carousel-item-next,.carousel-item-prev{position:absolute;top:0}@media (-webkit-transform-3d){.carousel-item-next.carousel-item-left,.carousel-item-prev.carousel-item-right{-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0)}.active.carousel-item-right,.carousel-item-next{-webkit-transform:translate3d(100%,0,0);transform:translate3d(100%,0,0)}.active.carousel-item-left,.carousel-item-prev{-webkit-transform:translate3d(-100%,0,0);transform:translate3d(-100%,0,0)}}@supports ((-webkit-transform:translate3d(0,0,0)) or (transform:translate3d(0,0,0))){.carousel-item-next.carousel-item-left,.carousel-item-prev.carousel-item-right{-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0)}.active.carousel-item-right,.carousel-item-next{-webkit-transform:translate3d(100%,0,0);transform:translate3d(100%,0,0)}.active.carousel-item-left,.carousel-item-prev{-webkit-transform:translate3d(-100%,0,0);transform:translate3d(-100%,0,0)}}.carousel-control-next,.carousel-control-prev{position:absolute;top:0;bottom:0;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;width:15%;color:#fff;text-align:center;opacity:.5}.carousel-control-next:focus,.carousel-control-next:hover,.carousel-control-prev:focus,.carousel-control-prev:hover{color:#fff;text-decoration:none;outline:0;opacity:.9}.carousel-control-prev{left:0}.carousel-control-next{right:0}.carousel-control-next-icon,.carousel-control-prev-icon{display:inline-block;width:20px;height:20px;background:transparent no-repeat center center;-webkit-background-size:100% 100%;background-size:100% 100%}.carousel-control-prev-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23fff' viewBox='0 0 8 8'%3E%3Cpath d='M4 0l-4 4 4 4 1.5-1.5-2.5-2.5 2.5-2.5-1.5-1.5z'/%3E%3C/svg%3E")}.carousel-control-next-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23fff' viewBox='0 0 8 8'%3E%3Cpath d='M1.5 0l-1.5 1.5 2.5 2.5-2.5 2.5 1.5 1.5 4-4-4-4z'/%3E%3C/svg%3E")}.carousel-indicators{position:absolute;right:0;bottom:10px;left:0;z-index:15;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;padding-left:0;margin-right:15%;margin-left:15%;list-style:none}.carousel-indicators li{position:relative;-webkit-box-flex:1;-webkit-flex:1 0 auto;-ms-flex:1 0 auto;flex:1 0 auto;max-width:30px;height:3px;margin-right:3px;margin-left:3px;text-indent:-999px;cursor:pointer;background-color:rgba(255,255,255,.5)}.carousel-indicators li::before{position:absolute;top:-10px;left:0;display:inline-block;width:100%;height:10px;content:""}.carousel-indicators li::after{position:absolute;bottom:-10px;left:0;display:inline-block;width:100%;height:10px;content:""}.carousel-indicators .active{background-color:#fff}.carousel-caption{position:absolute;right:15%;bottom:20px;left:15%;z-index:10;padding-top:20px;padding-bottom:20px;color:#fff;text-align:center}.align-baseline{vertical-align:baseline!important}.align-top{vertical-align:top!important}.align-middle{vertical-align:middle!important}.align-bottom{vertical-align:bottom!important}.align-text-bottom{vertical-align:text-bottom!important}.align-text-top{vertical-align:text-top!important}.bg-faded{background-color:#f7f7f7}.bg-primary{background-color:#0275d8!important}a.bg-primary:focus,a.bg-primary:hover{background-color:#025aa5!important}.bg-success{background-color:#5cb85c!important}a.bg-success:focus,a.bg-success:hover{background-color:#449d44!important}.bg-info{background-color:#5bc0de!important}a.bg-info:focus,a.bg-info:hover{background-color:#31b0d5!important}.bg-warning{background-color:#f0ad4e!important}a.bg-warning:focus,a.bg-warning:hover{background-color:#ec971f!important}.bg-danger{background-color:#d9534f!important}a.bg-danger:focus,a.bg-danger:hover{background-color:#c9302c!important}.bg-inverse{background-color:#292b2c!important}a.bg-inverse:focus,a.bg-inverse:hover{background-color:#101112!important}.border-0{border:0!important}.border-top-0{border-top:0!important}.border-right-0{border-right:0!important}.border-bottom-0{border-bottom:0!important}.border-left-0{border-left:0!important}.rounded{border-radius:.25rem}.rounded-top{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.rounded-right{border-bottom-right-radius:.25rem;border-top-right-radius:.25rem}.rounded-bottom{border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.rounded-left{border-bottom-left-radius:.25rem;border-top-left-radius:.25rem}.rounded-circle{border-radius:50%}.rounded-0{border-radius:0}.clearfix::after{display:block;content:"";clear:both}.d-none{display:none!important}.d-inline{display:inline!important}.d-inline-block{display:inline-block!important}.d-block{display:block!important}.d-table{display:table!important}.d-table-cell{display:table-cell!important}.d-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}@media (min-width:576px){.d-sm-none{display:none!important}.d-sm-inline{display:inline!important}.d-sm-inline-block{display:inline-block!important}.d-sm-block{display:block!important}.d-sm-table{display:table!important}.d-sm-table-cell{display:table-cell!important}.d-sm-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-sm-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:768px){.d-md-none{display:none!important}.d-md-inline{display:inline!important}.d-md-inline-block{display:inline-block!important}.d-md-block{display:block!important}.d-md-table{display:table!important}.d-md-table-cell{display:table-cell!important}.d-md-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-md-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:992px){.d-lg-none{display:none!important}.d-lg-inline{display:inline!important}.d-lg-inline-block{display:inline-block!important}.d-lg-block{display:block!important}.d-lg-table{display:table!important}.d-lg-table-cell{display:table-cell!important}.d-lg-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-lg-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:1200px){.d-xl-none{display:none!important}.d-xl-inline{display:inline!important}.d-xl-inline-block{display:inline-block!important}.d-xl-block{display:block!important}.d-xl-table{display:table!important}.d-xl-table-cell{display:table-cell!important}.d-xl-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-xl-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}.flex-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}@media (min-width:576px){.flex-sm-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-sm-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-sm-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-sm-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-sm-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-sm-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-sm-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-sm-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-sm-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-sm-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-sm-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-sm-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-sm-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-sm-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-sm-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-sm-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-sm-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-sm-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-sm-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-sm-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-sm-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-sm-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-sm-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-sm-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-sm-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-sm-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-sm-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-sm-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-sm-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-sm-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-sm-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-sm-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:768px){.flex-md-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-md-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-md-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-md-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-md-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-md-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-md-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-md-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-md-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-md-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-md-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-md-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-md-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-md-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-md-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-md-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-md-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-md-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-md-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-md-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-md-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-md-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-md-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-md-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-md-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-md-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-md-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-md-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-md-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-md-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-md-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-md-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:992px){.flex-lg-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-lg-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-lg-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-lg-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-lg-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-lg-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-lg-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-lg-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-lg-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-lg-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-lg-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-lg-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-lg-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-lg-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-lg-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-lg-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-lg-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-lg-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-lg-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-lg-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-lg-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-lg-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-lg-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-lg-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-lg-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-lg-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-lg-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-lg-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-lg-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-lg-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-lg-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-lg-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:1200px){.flex-xl-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-xl-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-xl-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-xl-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-xl-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-xl-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-xl-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-xl-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-xl-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-xl-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-xl-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-xl-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-xl-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-xl-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-xl-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-xl-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-xl-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-xl-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-xl-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-xl-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-xl-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-xl-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-xl-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-xl-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-xl-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-xl-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-xl-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-xl-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-xl-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-xl-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-xl-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-xl-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}.float-left{float:left!important}.float-right{float:right!important}.float-none{float:none!important}@media (min-width:576px){.float-sm-left{float:left!important}.float-sm-right{float:right!important}.float-sm-none{float:none!important}}@media (min-width:768px){.float-md-left{float:left!important}.float-md-right{float:right!important}.float-md-none{float:none!important}}@media (min-width:992px){.float-lg-left{float:left!important}.float-lg-right{float:right!important}.float-lg-none{float:none!important}}@media (min-width:1200px){.float-xl-left{float:left!important}.float-xl-right{float:right!important}.float-xl-none{float:none!important}}.fixed-top{position:fixed;top:0;right:0;left:0;z-index:1030}.fixed-bottom{position:fixed;right:0;bottom:0;left:0;z-index:1030}.sticky-top{position:-webkit-sticky;position:sticky;top:0;z-index:1030}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.w-25{width:25%!important}.w-50{width:50%!important}.w-75{width:75%!important}.w-100{width:100%!important}.h-25{height:25%!important}.h-50{height:50%!important}.h-75{height:75%!important}.h-100{height:100%!important}.mw-100{max-width:100%!important}.mh-100{max-height:100%!important}.m-0{margin:0 0!important}.mt-0{margin-top:0!important}.mr-0{margin-right:0!important}.mb-0{margin-bottom:0!important}.ml-0{margin-left:0!important}.mx-0{margin-right:0!important;margin-left:0!important}.my-0{margin-top:0!important;margin-bottom:0!important}.m-1{margin:.25rem .25rem!important}.mt-1{margin-top:.25rem!important}.mr-1{margin-right:.25rem!important}.mb-1{margin-bottom:.25rem!important}.ml-1{margin-left:.25rem!important}.mx-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-2{margin:.5rem .5rem!important}.mt-2{margin-top:.5rem!important}.mr-2{margin-right:.5rem!important}.mb-2{margin-bottom:.5rem!important}.ml-2{margin-left:.5rem!important}.mx-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-3{margin:1rem 1rem!important}.mt-3{margin-top:1rem!important}.mr-3{margin-right:1rem!important}.mb-3{margin-bottom:1rem!important}.ml-3{margin-left:1rem!important}.mx-3{margin-right:1rem!important;margin-left:1rem!important}.my-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-4{margin:1.5rem 1.5rem!important}.mt-4{margin-top:1.5rem!important}.mr-4{margin-right:1.5rem!important}.mb-4{margin-bottom:1.5rem!important}.ml-4{margin-left:1.5rem!important}.mx-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-5{margin:3rem 3rem!important}.mt-5{margin-top:3rem!important}.mr-5{margin-right:3rem!important}.mb-5{margin-bottom:3rem!important}.ml-5{margin-left:3rem!important}.mx-5{margin-right:3rem!important;margin-left:3rem!important}.my-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-0{padding:0 0!important}.pt-0{padding-top:0!important}.pr-0{padding-right:0!important}.pb-0{padding-bottom:0!important}.pl-0{padding-left:0!important}.px-0{padding-right:0!important;padding-left:0!important}.py-0{padding-top:0!important;padding-bottom:0!important}.p-1{padding:.25rem .25rem!important}.pt-1{padding-top:.25rem!important}.pr-1{padding-right:.25rem!important}.pb-1{padding-bottom:.25rem!important}.pl-1{padding-left:.25rem!important}.px-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-2{padding:.5rem .5rem!important}.pt-2{padding-top:.5rem!important}.pr-2{padding-right:.5rem!important}.pb-2{padding-bottom:.5rem!important}.pl-2{padding-left:.5rem!important}.px-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-3{padding:1rem 1rem!important}.pt-3{padding-top:1rem!important}.pr-3{padding-right:1rem!important}.pb-3{padding-bottom:1rem!important}.pl-3{padding-left:1rem!important}.px-3{padding-right:1rem!important;padding-left:1rem!important}.py-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-4{padding:1.5rem 1.5rem!important}.pt-4{padding-top:1.5rem!important}.pr-4{padding-right:1.5rem!important}.pb-4{padding-bottom:1.5rem!important}.pl-4{padding-left:1.5rem!important}.px-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-5{padding:3rem 3rem!important}.pt-5{padding-top:3rem!important}.pr-5{padding-right:3rem!important}.pb-5{padding-bottom:3rem!important}.pl-5{padding-left:3rem!important}.px-5{padding-right:3rem!important;padding-left:3rem!important}.py-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-auto{margin:auto!important}.mt-auto{margin-top:auto!important}.mr-auto{margin-right:auto!important}.mb-auto{margin-bottom:auto!important}.ml-auto{margin-left:auto!important}.mx-auto{margin-right:auto!important;margin-left:auto!important}.my-auto{margin-top:auto!important;margin-bottom:auto!important}@media (min-width:576px){.m-sm-0{margin:0 0!important}.mt-sm-0{margin-top:0!important}.mr-sm-0{margin-right:0!important}.mb-sm-0{margin-bottom:0!important}.ml-sm-0{margin-left:0!important}.mx-sm-0{margin-right:0!important;margin-left:0!important}.my-sm-0{margin-top:0!important;margin-bottom:0!important}.m-sm-1{margin:.25rem .25rem!important}.mt-sm-1{margin-top:.25rem!important}.mr-sm-1{margin-right:.25rem!important}.mb-sm-1{margin-bottom:.25rem!important}.ml-sm-1{margin-left:.25rem!important}.mx-sm-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-sm-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-sm-2{margin:.5rem .5rem!important}.mt-sm-2{margin-top:.5rem!important}.mr-sm-2{margin-right:.5rem!important}.mb-sm-2{margin-bottom:.5rem!important}.ml-sm-2{margin-left:.5rem!important}.mx-sm-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-sm-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-sm-3{margin:1rem 1rem!important}.mt-sm-3{margin-top:1rem!important}.mr-sm-3{margin-right:1rem!important}.mb-sm-3{margin-bottom:1rem!important}.ml-sm-3{margin-left:1rem!important}.mx-sm-3{margin-right:1rem!important;margin-left:1rem!important}.my-sm-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-sm-4{margin:1.5rem 1.5rem!important}.mt-sm-4{margin-top:1.5rem!important}.mr-sm-4{margin-right:1.5rem!important}.mb-sm-4{margin-bottom:1.5rem!important}.ml-sm-4{margin-left:1.5rem!important}.mx-sm-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-sm-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-sm-5{margin:3rem 3rem!important}.mt-sm-5{margin-top:3rem!important}.mr-sm-5{margin-right:3rem!important}.mb-sm-5{margin-bottom:3rem!important}.ml-sm-5{margin-left:3rem!important}.mx-sm-5{margin-right:3rem!important;margin-left:3rem!important}.my-sm-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-sm-0{padding:0 0!important}.pt-sm-0{padding-top:0!important}.pr-sm-0{padding-right:0!important}.pb-sm-0{padding-bottom:0!important}.pl-sm-0{padding-left:0!important}.px-sm-0{padding-right:0!important;padding-left:0!important}.py-sm-0{padding-top:0!important;padding-bottom:0!important}.p-sm-1{padding:.25rem .25rem!important}.pt-sm-1{padding-top:.25rem!important}.pr-sm-1{padding-right:.25rem!important}.pb-sm-1{padding-bottom:.25rem!important}.pl-sm-1{padding-left:.25rem!important}.px-sm-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-sm-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-sm-2{padding:.5rem .5rem!important}.pt-sm-2{padding-top:.5rem!important}.pr-sm-2{padding-right:.5rem!important}.pb-sm-2{padding-bottom:.5rem!important}.pl-sm-2{padding-left:.5rem!important}.px-sm-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-sm-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-sm-3{padding:1rem 1rem!important}.pt-sm-3{padding-top:1rem!important}.pr-sm-3{padding-right:1rem!important}.pb-sm-3{padding-bottom:1rem!important}.pl-sm-3{padding-left:1rem!important}.px-sm-3{padding-right:1rem!important;padding-left:1rem!important}.py-sm-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-sm-4{padding:1.5rem 1.5rem!important}.pt-sm-4{padding-top:1.5rem!important}.pr-sm-4{padding-right:1.5rem!important}.pb-sm-4{padding-bottom:1.5rem!important}.pl-sm-4{padding-left:1.5rem!important}.px-sm-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-sm-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-sm-5{padding:3rem 3rem!important}.pt-sm-5{padding-top:3rem!important}.pr-sm-5{padding-right:3rem!important}.pb-sm-5{padding-bottom:3rem!important}.pl-sm-5{padding-left:3rem!important}.px-sm-5{padding-right:3rem!important;padding-left:3rem!important}.py-sm-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-sm-auto{margin:auto!important}.mt-sm-auto{margin-top:auto!important}.mr-sm-auto{margin-right:auto!important}.mb-sm-auto{margin-bottom:auto!important}.ml-sm-auto{margin-left:auto!important}.mx-sm-auto{margin-right:auto!important;margin-left:auto!important}.my-sm-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:768px){.m-md-0{margin:0 0!important}.mt-md-0{margin-top:0!important}.mr-md-0{margin-right:0!important}.mb-md-0{margin-bottom:0!important}.ml-md-0{margin-left:0!important}.mx-md-0{margin-right:0!important;margin-left:0!important}.my-md-0{margin-top:0!important;margin-bottom:0!important}.m-md-1{margin:.25rem .25rem!important}.mt-md-1{margin-top:.25rem!important}.mr-md-1{margin-right:.25rem!important}.mb-md-1{margin-bottom:.25rem!important}.ml-md-1{margin-left:.25rem!important}.mx-md-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-md-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-md-2{margin:.5rem .5rem!important}.mt-md-2{margin-top:.5rem!important}.mr-md-2{margin-right:.5rem!important}.mb-md-2{margin-bottom:.5rem!important}.ml-md-2{margin-left:.5rem!important}.mx-md-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-md-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-md-3{margin:1rem 1rem!important}.mt-md-3{margin-top:1rem!important}.mr-md-3{margin-right:1rem!important}.mb-md-3{margin-bottom:1rem!important}.ml-md-3{margin-left:1rem!important}.mx-md-3{margin-right:1rem!important;margin-left:1rem!important}.my-md-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-md-4{margin:1.5rem 1.5rem!important}.mt-md-4{margin-top:1.5rem!important}.mr-md-4{margin-right:1.5rem!important}.mb-md-4{margin-bottom:1.5rem!important}.ml-md-4{margin-left:1.5rem!important}.mx-md-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-md-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-md-5{margin:3rem 3rem!important}.mt-md-5{margin-top:3rem!important}.mr-md-5{margin-right:3rem!important}.mb-md-5{margin-bottom:3rem!important}.ml-md-5{margin-left:3rem!important}.mx-md-5{margin-right:3rem!important;margin-left:3rem!important}.my-md-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-md-0{padding:0 0!important}.pt-md-0{padding-top:0!important}.pr-md-0{padding-right:0!important}.pb-md-0{padding-bottom:0!important}.pl-md-0{padding-left:0!important}.px-md-0{padding-right:0!important;padding-left:0!important}.py-md-0{padding-top:0!important;padding-bottom:0!important}.p-md-1{padding:.25rem .25rem!important}.pt-md-1{padding-top:.25rem!important}.pr-md-1{padding-right:.25rem!important}.pb-md-1{padding-bottom:.25rem!important}.pl-md-1{padding-left:.25rem!important}.px-md-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-md-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-md-2{padding:.5rem .5rem!important}.pt-md-2{padding-top:.5rem!important}.pr-md-2{padding-right:.5rem!important}.pb-md-2{padding-bottom:.5rem!important}.pl-md-2{padding-left:.5rem!important}.px-md-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-md-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-md-3{padding:1rem 1rem!important}.pt-md-3{padding-top:1rem!important}.pr-md-3{padding-right:1rem!important}.pb-md-3{padding-bottom:1rem!important}.pl-md-3{padding-left:1rem!important}.px-md-3{padding-right:1rem!important;padding-left:1rem!important}.py-md-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-md-4{padding:1.5rem 1.5rem!important}.pt-md-4{padding-top:1.5rem!important}.pr-md-4{padding-right:1.5rem!important}.pb-md-4{padding-bottom:1.5rem!important}.pl-md-4{padding-left:1.5rem!important}.px-md-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-md-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-md-5{padding:3rem 3rem!important}.pt-md-5{padding-top:3rem!important}.pr-md-5{padding-right:3rem!important}.pb-md-5{padding-bottom:3rem!important}.pl-md-5{padding-left:3rem!important}.px-md-5{padding-right:3rem!important;padding-left:3rem!important}.py-md-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-md-auto{margin:auto!important}.mt-md-auto{margin-top:auto!important}.mr-md-auto{margin-right:auto!important}.mb-md-auto{margin-bottom:auto!important}.ml-md-auto{margin-left:auto!important}.mx-md-auto{margin-right:auto!important;margin-left:auto!important}.my-md-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:992px){.m-lg-0{margin:0 0!important}.mt-lg-0{margin-top:0!important}.mr-lg-0{margin-right:0!important}.mb-lg-0{margin-bottom:0!important}.ml-lg-0{margin-left:0!important}.mx-lg-0{margin-right:0!important;margin-left:0!important}.my-lg-0{margin-top:0!important;margin-bottom:0!important}.m-lg-1{margin:.25rem .25rem!important}.mt-lg-1{margin-top:.25rem!important}.mr-lg-1{margin-right:.25rem!important}.mb-lg-1{margin-bottom:.25rem!important}.ml-lg-1{margin-left:.25rem!important}.mx-lg-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-lg-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-lg-2{margin:.5rem .5rem!important}.mt-lg-2{margin-top:.5rem!important}.mr-lg-2{margin-right:.5rem!important}.mb-lg-2{margin-bottom:.5rem!important}.ml-lg-2{margin-left:.5rem!important}.mx-lg-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-lg-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-lg-3{margin:1rem 1rem!important}.mt-lg-3{margin-top:1rem!important}.mr-lg-3{margin-right:1rem!important}.mb-lg-3{margin-bottom:1rem!important}.ml-lg-3{margin-left:1rem!important}.mx-lg-3{margin-right:1rem!important;margin-left:1rem!important}.my-lg-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-lg-4{margin:1.5rem 1.5rem!important}.mt-lg-4{margin-top:1.5rem!important}.mr-lg-4{margin-right:1.5rem!important}.mb-lg-4{margin-bottom:1.5rem!important}.ml-lg-4{margin-left:1.5rem!important}.mx-lg-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-lg-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-lg-5{margin:3rem 3rem!important}.mt-lg-5{margin-top:3rem!important}.mr-lg-5{margin-right:3rem!important}.mb-lg-5{margin-bottom:3rem!important}.ml-lg-5{margin-left:3rem!important}.mx-lg-5{margin-right:3rem!important;margin-left:3rem!important}.my-lg-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-lg-0{padding:0 0!important}.pt-lg-0{padding-top:0!important}.pr-lg-0{padding-right:0!important}.pb-lg-0{padding-bottom:0!important}.pl-lg-0{padding-left:0!important}.px-lg-0{padding-right:0!important;padding-left:0!important}.py-lg-0{padding-top:0!important;padding-bottom:0!important}.p-lg-1{padding:.25rem .25rem!important}.pt-lg-1{padding-top:.25rem!important}.pr-lg-1{padding-right:.25rem!important}.pb-lg-1{padding-bottom:.25rem!important}.pl-lg-1{padding-left:.25rem!important}.px-lg-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-lg-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-lg-2{padding:.5rem .5rem!important}.pt-lg-2{padding-top:.5rem!important}.pr-lg-2{padding-right:.5rem!important}.pb-lg-2{padding-bottom:.5rem!important}.pl-lg-2{padding-left:.5rem!important}.px-lg-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-lg-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-lg-3{padding:1rem 1rem!important}.pt-lg-3{padding-top:1rem!important}.pr-lg-3{padding-right:1rem!important}.pb-lg-3{padding-bottom:1rem!important}.pl-lg-3{padding-left:1rem!important}.px-lg-3{padding-right:1rem!important;padding-left:1rem!important}.py-lg-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-lg-4{padding:1.5rem 1.5rem!important}.pt-lg-4{padding-top:1.5rem!important}.pr-lg-4{padding-right:1.5rem!important}.pb-lg-4{padding-bottom:1.5rem!important}.pl-lg-4{padding-left:1.5rem!important}.px-lg-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-lg-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-lg-5{padding:3rem 3rem!important}.pt-lg-5{padding-top:3rem!important}.pr-lg-5{padding-right:3rem!important}.pb-lg-5{padding-bottom:3rem!important}.pl-lg-5{padding-left:3rem!important}.px-lg-5{padding-right:3rem!important;padding-left:3rem!important}.py-lg-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-lg-auto{margin:auto!important}.mt-lg-auto{margin-top:auto!important}.mr-lg-auto{margin-right:auto!important}.mb-lg-auto{margin-bottom:auto!important}.ml-lg-auto{margin-left:auto!important}.mx-lg-auto{margin-right:auto!important;margin-left:auto!important}.my-lg-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:1200px){.m-xl-0{margin:0 0!important}.mt-xl-0{margin-top:0!important}.mr-xl-0{margin-right:0!important}.mb-xl-0{margin-bottom:0!important}.ml-xl-0{margin-left:0!important}.mx-xl-0{margin-right:0!important;margin-left:0!important}.my-xl-0{margin-top:0!important;margin-bottom:0!important}.m-xl-1{margin:.25rem .25rem!important}.mt-xl-1{margin-top:.25rem!important}.mr-xl-1{margin-right:.25rem!important}.mb-xl-1{margin-bottom:.25rem!important}.ml-xl-1{margin-left:.25rem!important}.mx-xl-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-xl-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-xl-2{margin:.5rem .5rem!important}.mt-xl-2{margin-top:.5rem!important}.mr-xl-2{margin-right:.5rem!important}.mb-xl-2{margin-bottom:.5rem!important}.ml-xl-2{margin-left:.5rem!important}.mx-xl-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-xl-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-xl-3{margin:1rem 1rem!important}.mt-xl-3{margin-top:1rem!important}.mr-xl-3{margin-right:1rem!important}.mb-xl-3{margin-bottom:1rem!important}.ml-xl-3{margin-left:1rem!important}.mx-xl-3{margin-right:1rem!important;margin-left:1rem!important}.my-xl-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-xl-4{margin:1.5rem 1.5rem!important}.mt-xl-4{margin-top:1.5rem!important}.mr-xl-4{margin-right:1.5rem!important}.mb-xl-4{margin-bottom:1.5rem!important}.ml-xl-4{margin-left:1.5rem!important}.mx-xl-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-xl-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-xl-5{margin:3rem 3rem!important}.mt-xl-5{margin-top:3rem!important}.mr-xl-5{margin-right:3rem!important}.mb-xl-5{margin-bottom:3rem!important}.ml-xl-5{margin-left:3rem!important}.mx-xl-5{margin-right:3rem!important;margin-left:3rem!important}.my-xl-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-xl-0{padding:0 0!important}.pt-xl-0{padding-top:0!important}.pr-xl-0{padding-right:0!important}.pb-xl-0{padding-bottom:0!important}.pl-xl-0{padding-left:0!important}.px-xl-0{padding-right:0!important;padding-left:0!important}.py-xl-0{padding-top:0!important;padding-bottom:0!important}.p-xl-1{padding:.25rem .25rem!important}.pt-xl-1{padding-top:.25rem!important}.pr-xl-1{padding-right:.25rem!important}.pb-xl-1{padding-bottom:.25rem!important}.pl-xl-1{padding-left:.25rem!important}.px-xl-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-xl-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-xl-2{padding:.5rem .5rem!important}.pt-xl-2{padding-top:.5rem!important}.pr-xl-2{padding-right:.5rem!important}.pb-xl-2{padding-bottom:.5rem!important}.pl-xl-2{padding-left:.5rem!important}.px-xl-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-xl-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-xl-3{padding:1rem 1rem!important}.pt-xl-3{padding-top:1rem!important}.pr-xl-3{padding-right:1rem!important}.pb-xl-3{padding-bottom:1rem!important}.pl-xl-3{padding-left:1rem!important}.px-xl-3{padding-right:1rem!important;padding-left:1rem!important}.py-xl-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-xl-4{padding:1.5rem 1.5rem!important}.pt-xl-4{padding-top:1.5rem!important}.pr-xl-4{padding-right:1.5rem!important}.pb-xl-4{padding-bottom:1.5rem!important}.pl-xl-4{padding-left:1.5rem!important}.px-xl-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-xl-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-xl-5{padding:3rem 3rem!important}.pt-xl-5{padding-top:3rem!important}.pr-xl-5{padding-right:3rem!important}.pb-xl-5{padding-bottom:3rem!important}.pl-xl-5{padding-left:3rem!important}.px-xl-5{padding-right:3rem!important;padding-left:3rem!important}.py-xl-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-xl-auto{margin:auto!important}.mt-xl-auto{margin-top:auto!important}.mr-xl-auto{margin-right:auto!important}.mb-xl-auto{margin-bottom:auto!important}.ml-xl-auto{margin-left:auto!important}.mx-xl-auto{margin-right:auto!important;margin-left:auto!important}.my-xl-auto{margin-top:auto!important;margin-bottom:auto!important}}.text-justify{text-align:justify!important}.text-nowrap{white-space:nowrap!important}.text-truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.text-left{text-align:left!important}.text-right{text-align:right!important}.text-center{text-align:center!important}@media (min-width:576px){.text-sm-left{text-align:left!important}.text-sm-right{text-align:right!important}.text-sm-center{text-align:center!important}}@media (min-width:768px){.text-md-left{text-align:left!important}.text-md-right{text-align:right!important}.text-md-center{text-align:center!important}}@media (min-width:992px){.text-lg-left{text-align:left!important}.text-lg-right{text-align:right!important}.text-lg-center{text-align:center!important}}@media (min-width:1200px){.text-xl-left{text-align:left!important}.text-xl-right{text-align:right!important}.text-xl-center{text-align:center!important}}.text-lowercase{text-transform:lowercase!important}.text-uppercase{text-transform:uppercase!important}.text-capitalize{text-transform:capitalize!important}.font-weight-normal{font-weight:400}.font-weight-bold{font-weight:700}.font-italic{font-style:italic}.text-white{color:#fff!important}.text-muted{color:#636c72!important}a.text-muted:focus,a.text-muted:hover{color:#4b5257!important}.text-primary{color:#0275d8!important}a.text-primary:focus,a.text-primary:hover{color:#025aa5!important}.text-success{color:#5cb85c!important}a.text-success:focus,a.text-success:hover{color:#449d44!important}.text-info{color:#5bc0de!important}a.text-info:focus,a.text-info:hover{color:#31b0d5!important}.text-warning{color:#f0ad4e!important}a.text-warning:focus,a.text-warning:hover{color:#ec971f!important}.text-danger{color:#d9534f!important}a.text-danger:focus,a.text-danger:hover{color:#c9302c!important}.text-gray-dark{color:#292b2c!important}a.text-gray-dark:focus,a.text-gray-dark:hover{color:#101112!important}.text-hide{font:0/0 a;color:transparent;text-shadow:none;background-color:transparent;border:0}.invisible{visibility:hidden!important}.hidden-xs-up{display:none!important}@media (max-width:575px){.hidden-xs-down{display:none!important}}@media (min-width:576px){.hidden-sm-up{display:none!important}}@media (max-width:767px){.hidden-sm-down{display:none!important}}@media (min-width:768px){.hidden-md-up{display:none!important}}@media (max-width:991px){.hidden-md-down{display:none!important}}@media (min-width:992px){.hidden-lg-up{display:none!important}}@media (max-width:1199px){.hidden-lg-down{display:none!important}}@media (min-width:1200px){.hidden-xl-up{display:none!important}}.hidden-xl-down{display:none!important}.visible-print-block{display:none!important}@media print{.visible-print-block{display:block!important}}.visible-print-inline{display:none!important}@media print{.visible-print-inline{display:inline!important}}.visible-print-inline-block{display:none!important}@media print{.visible-print-inline-block{display:inline-block!important}}@media print{.hidden-print{display:none!important}}/*# sourceMappingURL=bootstrap.min.css.map */ \ No newline at end of file diff --git a/archivebox-0.5.3/archivebox/themes/default/static/external.png b/archivebox-0.5.3/archivebox/themes/default/static/external.png new file mode 100755 index 0000000..7e1a5f0 Binary files /dev/null and b/archivebox-0.5.3/archivebox/themes/default/static/external.png differ diff --git a/archivebox-0.5.3/archivebox/themes/default/static/jquery.dataTables.min.css b/archivebox-0.5.3/archivebox/themes/default/static/jquery.dataTables.min.css new file mode 100644 index 0000000..4303138 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/static/jquery.dataTables.min.css @@ -0,0 +1 @@ +table.dataTable{width:100%;margin:0 auto;clear:both;border-collapse:separate;border-spacing:0}table.dataTable thead th,table.dataTable tfoot th{font-weight:bold}table.dataTable thead th,table.dataTable thead td{padding:10px 18px;border-bottom:1px solid #111}table.dataTable thead th:active,table.dataTable thead td:active{outline:none}table.dataTable tfoot th,table.dataTable tfoot td{padding:10px 18px 6px 18px;border-top:1px solid #111}table.dataTable thead .sorting,table.dataTable thead .sorting_asc,table.dataTable thead .sorting_desc,table.dataTable thead .sorting_asc_disabled,table.dataTable thead .sorting_desc_disabled{cursor:pointer;*cursor:hand;background-repeat:no-repeat;background-position:center right}table.dataTable thead .sorting{background-image:url("sort_both.png")}table.dataTable thead .sorting_asc{background-image:url("sort_asc.png")}table.dataTable thead .sorting_desc{background-image:url("sort_desc.png")}table.dataTable thead .sorting_asc_disabled{background-image:url("sort_asc_disabled.png")}table.dataTable thead .sorting_desc_disabled{background-image:url("sort_desc_disabled.png")}table.dataTable tbody tr{background-color:#ffffff}table.dataTable tbody tr.selected{background-color:#B0BED9}table.dataTable tbody th,table.dataTable tbody td{padding:8px 10px}table.dataTable.row-border tbody th,table.dataTable.row-border tbody td,table.dataTable.display tbody th,table.dataTable.display tbody td{border-top:1px solid #ddd}table.dataTable.row-border tbody tr:first-child th,table.dataTable.row-border tbody tr:first-child td,table.dataTable.display tbody tr:first-child th,table.dataTable.display tbody tr:first-child td{border-top:none}table.dataTable.cell-border tbody th,table.dataTable.cell-border tbody td{border-top:1px solid #ddd;border-right:1px solid #ddd}table.dataTable.cell-border tbody tr th:first-child,table.dataTable.cell-border tbody tr td:first-child{border-left:1px solid #ddd}table.dataTable.cell-border tbody tr:first-child th,table.dataTable.cell-border tbody tr:first-child td{border-top:none}table.dataTable.stripe tbody tr.odd,table.dataTable.display tbody tr.odd{background-color:#f9f9f9}table.dataTable.stripe tbody tr.odd.selected,table.dataTable.display tbody tr.odd.selected{background-color:#acbad4}table.dataTable.hover tbody tr:hover,table.dataTable.display tbody tr:hover{background-color:#f6f6f6}table.dataTable.hover tbody tr:hover.selected,table.dataTable.display tbody tr:hover.selected{background-color:#aab7d1}table.dataTable.order-column tbody tr>.sorting_1,table.dataTable.order-column tbody tr>.sorting_2,table.dataTable.order-column tbody tr>.sorting_3,table.dataTable.display tbody tr>.sorting_1,table.dataTable.display tbody tr>.sorting_2,table.dataTable.display tbody tr>.sorting_3{background-color:#fafafa}table.dataTable.order-column tbody tr.selected>.sorting_1,table.dataTable.order-column tbody tr.selected>.sorting_2,table.dataTable.order-column tbody tr.selected>.sorting_3,table.dataTable.display tbody tr.selected>.sorting_1,table.dataTable.display tbody tr.selected>.sorting_2,table.dataTable.display tbody tr.selected>.sorting_3{background-color:#acbad5}table.dataTable.display tbody tr.odd>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd>.sorting_1{background-color:#f1f1f1}table.dataTable.display tbody tr.odd>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd>.sorting_2{background-color:#f3f3f3}table.dataTable.display tbody tr.odd>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd>.sorting_3{background-color:whitesmoke}table.dataTable.display tbody tr.odd.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_1{background-color:#a6b4cd}table.dataTable.display tbody tr.odd.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_2{background-color:#a8b5cf}table.dataTable.display tbody tr.odd.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_3{background-color:#a9b7d1}table.dataTable.display tbody tr.even>.sorting_1,table.dataTable.order-column.stripe tbody tr.even>.sorting_1{background-color:#fafafa}table.dataTable.display tbody tr.even>.sorting_2,table.dataTable.order-column.stripe tbody tr.even>.sorting_2{background-color:#fcfcfc}table.dataTable.display tbody tr.even>.sorting_3,table.dataTable.order-column.stripe tbody tr.even>.sorting_3{background-color:#fefefe}table.dataTable.display tbody tr.even.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_1{background-color:#acbad5}table.dataTable.display tbody tr.even.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_2{background-color:#aebcd6}table.dataTable.display tbody tr.even.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_3{background-color:#afbdd8}table.dataTable.display tbody tr:hover>.sorting_1,table.dataTable.order-column.hover tbody tr:hover>.sorting_1{background-color:#eaeaea}table.dataTable.display tbody tr:hover>.sorting_2,table.dataTable.order-column.hover tbody tr:hover>.sorting_2{background-color:#ececec}table.dataTable.display tbody tr:hover>.sorting_3,table.dataTable.order-column.hover tbody tr:hover>.sorting_3{background-color:#efefef}table.dataTable.display tbody tr:hover.selected>.sorting_1,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_1{background-color:#a2aec7}table.dataTable.display tbody tr:hover.selected>.sorting_2,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_2{background-color:#a3b0c9}table.dataTable.display tbody tr:hover.selected>.sorting_3,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_3{background-color:#a5b2cb}table.dataTable.no-footer{border-bottom:1px solid #111}table.dataTable.nowrap th,table.dataTable.nowrap td{white-space:nowrap}table.dataTable.compact thead th,table.dataTable.compact thead td{padding:4px 17px 4px 4px}table.dataTable.compact tfoot th,table.dataTable.compact tfoot td{padding:4px}table.dataTable.compact tbody th,table.dataTable.compact tbody td{padding:4px}table.dataTable th.dt-left,table.dataTable td.dt-left{text-align:left}table.dataTable th.dt-center,table.dataTable td.dt-center,table.dataTable td.dataTables_empty{text-align:center}table.dataTable th.dt-right,table.dataTable td.dt-right{text-align:right}table.dataTable th.dt-justify,table.dataTable td.dt-justify{text-align:justify}table.dataTable th.dt-nowrap,table.dataTable td.dt-nowrap{white-space:nowrap}table.dataTable thead th.dt-head-left,table.dataTable thead td.dt-head-left,table.dataTable tfoot th.dt-head-left,table.dataTable tfoot td.dt-head-left{text-align:left}table.dataTable thead th.dt-head-center,table.dataTable thead td.dt-head-center,table.dataTable tfoot th.dt-head-center,table.dataTable tfoot td.dt-head-center{text-align:center}table.dataTable thead th.dt-head-right,table.dataTable thead td.dt-head-right,table.dataTable tfoot th.dt-head-right,table.dataTable tfoot td.dt-head-right{text-align:right}table.dataTable thead th.dt-head-justify,table.dataTable thead td.dt-head-justify,table.dataTable tfoot th.dt-head-justify,table.dataTable tfoot td.dt-head-justify{text-align:justify}table.dataTable thead th.dt-head-nowrap,table.dataTable thead td.dt-head-nowrap,table.dataTable tfoot th.dt-head-nowrap,table.dataTable tfoot td.dt-head-nowrap{white-space:nowrap}table.dataTable tbody th.dt-body-left,table.dataTable tbody td.dt-body-left{text-align:left}table.dataTable tbody th.dt-body-center,table.dataTable tbody td.dt-body-center{text-align:center}table.dataTable tbody th.dt-body-right,table.dataTable tbody td.dt-body-right{text-align:right}table.dataTable tbody th.dt-body-justify,table.dataTable tbody td.dt-body-justify{text-align:justify}table.dataTable tbody th.dt-body-nowrap,table.dataTable tbody td.dt-body-nowrap{white-space:nowrap}table.dataTable,table.dataTable th,table.dataTable td{box-sizing:content-box}.dataTables_wrapper{position:relative;clear:both;*zoom:1;zoom:1}.dataTables_wrapper .dataTables_length{float:left}.dataTables_wrapper .dataTables_filter{float:right;text-align:right}.dataTables_wrapper .dataTables_filter input{margin-left:0.5em}.dataTables_wrapper .dataTables_info{clear:both;float:left;padding-top:0.755em}.dataTables_wrapper .dataTables_paginate{float:right;text-align:right;padding-top:0.25em}.dataTables_wrapper .dataTables_paginate .paginate_button{box-sizing:border-box;display:inline-block;min-width:1.5em;padding:0.5em 1em;margin-left:2px;text-align:center;text-decoration:none !important;cursor:pointer;*cursor:hand;color:#333 !important;border:1px solid transparent;border-radius:2px}.dataTables_wrapper .dataTables_paginate .paginate_button.current,.dataTables_wrapper .dataTables_paginate .paginate_button.current:hover{color:#333 !important;border:1px solid #979797;background-color:white;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #fff), color-stop(100%, #dcdcdc));background:-webkit-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-moz-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-ms-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-o-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:linear-gradient(to bottom, #fff 0%, #dcdcdc 100%)}.dataTables_wrapper .dataTables_paginate .paginate_button.disabled,.dataTables_wrapper .dataTables_paginate .paginate_button.disabled:hover,.dataTables_wrapper .dataTables_paginate .paginate_button.disabled:active{cursor:default;color:#666 !important;border:1px solid transparent;background:transparent;box-shadow:none}.dataTables_wrapper .dataTables_paginate .paginate_button:hover{color:white !important;border:1px solid #111;background-color:#585858;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #585858), color-stop(100%, #111));background:-webkit-linear-gradient(top, #585858 0%, #111 100%);background:-moz-linear-gradient(top, #585858 0%, #111 100%);background:-ms-linear-gradient(top, #585858 0%, #111 100%);background:-o-linear-gradient(top, #585858 0%, #111 100%);background:linear-gradient(to bottom, #585858 0%, #111 100%)}.dataTables_wrapper .dataTables_paginate .paginate_button:active{outline:none;background-color:#2b2b2b;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #2b2b2b), color-stop(100%, #0c0c0c));background:-webkit-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-moz-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-ms-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-o-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:linear-gradient(to bottom, #2b2b2b 0%, #0c0c0c 100%);box-shadow:inset 0 0 3px #111}.dataTables_wrapper .dataTables_paginate .ellipsis{padding:0 1em}.dataTables_wrapper .dataTables_processing{position:absolute;top:50%;left:50%;width:100%;height:40px;margin-left:-50%;margin-top:-25px;padding-top:20px;text-align:center;font-size:1.2em;background-color:white;background:-webkit-gradient(linear, left top, right top, color-stop(0%, rgba(255,255,255,0)), color-stop(25%, rgba(255,255,255,0.9)), color-stop(75%, rgba(255,255,255,0.9)), color-stop(100%, rgba(255,255,255,0)));background:-webkit-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-moz-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-ms-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-o-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:linear-gradient(to right, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%)}.dataTables_wrapper .dataTables_length,.dataTables_wrapper .dataTables_filter,.dataTables_wrapper .dataTables_info,.dataTables_wrapper .dataTables_processing,.dataTables_wrapper .dataTables_paginate{color:#333}.dataTables_wrapper .dataTables_scroll{clear:both}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody{*margin-top:-1px;-webkit-overflow-scrolling:touch}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>th,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>td,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>th,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>td{vertical-align:middle}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>th>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>td>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>th>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>td>div.dataTables_sizing{height:0;overflow:hidden;margin:0 !important;padding:0 !important}.dataTables_wrapper.no-footer .dataTables_scrollBody{border-bottom:1px solid #111}.dataTables_wrapper.no-footer div.dataTables_scrollHead table.dataTable,.dataTables_wrapper.no-footer div.dataTables_scrollBody>table{border-bottom:none}.dataTables_wrapper:after{visibility:hidden;display:block;content:"";clear:both;height:0}@media screen and (max-width: 767px){.dataTables_wrapper .dataTables_info,.dataTables_wrapper .dataTables_paginate{float:none;text-align:center}.dataTables_wrapper .dataTables_paginate{margin-top:0.5em}}@media screen and (max-width: 640px){.dataTables_wrapper .dataTables_length,.dataTables_wrapper .dataTables_filter{float:none;text-align:center}.dataTables_wrapper .dataTables_filter{margin-top:0.5em}} diff --git a/archivebox-0.5.3/archivebox/themes/default/static/jquery.dataTables.min.js b/archivebox-0.5.3/archivebox/themes/default/static/jquery.dataTables.min.js new file mode 100644 index 0000000..07af1c3 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/static/jquery.dataTables.min.js @@ -0,0 +1,166 @@ +/*! + DataTables 1.10.19 + ©2008-2018 SpryMedia Ltd - datatables.net/license +*/ +(function(h){"function"===typeof define&&define.amd?define(["jquery"],function(E){return h(E,window,document)}):"object"===typeof exports?module.exports=function(E,H){E||(E=window);H||(H="undefined"!==typeof window?require("jquery"):require("jquery")(E));return h(H,E,E.document)}:h(jQuery,window,document)})(function(h,E,H,k){function Z(a){var b,c,d={};h.each(a,function(e){if((b=e.match(/^([^A-Z]+?)([A-Z])/))&&-1!=="a aa ai ao as b fn i m o s ".indexOf(b[1]+" "))c=e.replace(b[0],b[2].toLowerCase()), +d[c]=e,"o"===b[1]&&Z(a[e])});a._hungarianMap=d}function J(a,b,c){a._hungarianMap||Z(a);var d;h.each(b,function(e){d=a._hungarianMap[e];if(d!==k&&(c||b[d]===k))"o"===d.charAt(0)?(b[d]||(b[d]={}),h.extend(!0,b[d],b[e]),J(a[d],b[d],c)):b[d]=b[e]})}function Ca(a){var b=n.defaults.oLanguage,c=b.sDecimal;c&&Da(c);if(a){var d=a.sZeroRecords;!a.sEmptyTable&&(d&&"No data available in table"===b.sEmptyTable)&&F(a,a,"sZeroRecords","sEmptyTable");!a.sLoadingRecords&&(d&&"Loading..."===b.sLoadingRecords)&&F(a, +a,"sZeroRecords","sLoadingRecords");a.sInfoThousands&&(a.sThousands=a.sInfoThousands);(a=a.sDecimal)&&c!==a&&Da(a)}}function fb(a){A(a,"ordering","bSort");A(a,"orderMulti","bSortMulti");A(a,"orderClasses","bSortClasses");A(a,"orderCellsTop","bSortCellsTop");A(a,"order","aaSorting");A(a,"orderFixed","aaSortingFixed");A(a,"paging","bPaginate");A(a,"pagingType","sPaginationType");A(a,"pageLength","iDisplayLength");A(a,"searching","bFilter");"boolean"===typeof a.sScrollX&&(a.sScrollX=a.sScrollX?"100%": +"");"boolean"===typeof a.scrollX&&(a.scrollX=a.scrollX?"100%":"");if(a=a.aoSearchCols)for(var b=0,c=a.length;b").css({position:"fixed",top:0,left:-1*h(E).scrollLeft(),height:1,width:1, +overflow:"hidden"}).append(h("
    ").css({position:"absolute",top:1,left:1,width:100,overflow:"scroll"}).append(h("
    ").css({width:"100%",height:10}))).appendTo("body"),d=c.children(),e=d.children();b.barWidth=d[0].offsetWidth-d[0].clientWidth;b.bScrollOversize=100===e[0].offsetWidth&&100!==d[0].clientWidth;b.bScrollbarLeft=1!==Math.round(e.offset().left);b.bBounding=c[0].getBoundingClientRect().width?!0:!1;c.remove()}h.extend(a.oBrowser,n.__browser);a.oScroll.iBarWidth=n.__browser.barWidth} +function ib(a,b,c,d,e,f){var g,j=!1;c!==k&&(g=c,j=!0);for(;d!==e;)a.hasOwnProperty(d)&&(g=j?b(g,a[d],d,a):a[d],j=!0,d+=f);return g}function Ea(a,b){var c=n.defaults.column,d=a.aoColumns.length,c=h.extend({},n.models.oColumn,c,{nTh:b?b:H.createElement("th"),sTitle:c.sTitle?c.sTitle:b?b.innerHTML:"",aDataSort:c.aDataSort?c.aDataSort:[d],mData:c.mData?c.mData:d,idx:d});a.aoColumns.push(c);c=a.aoPreSearchCols;c[d]=h.extend({},n.models.oSearch,c[d]);ka(a,d,h(b).data())}function ka(a,b,c){var b=a.aoColumns[b], +d=a.oClasses,e=h(b.nTh);if(!b.sWidthOrig){b.sWidthOrig=e.attr("width")||null;var f=(e.attr("style")||"").match(/width:\s*(\d+[pxem%]+)/);f&&(b.sWidthOrig=f[1])}c!==k&&null!==c&&(gb(c),J(n.defaults.column,c),c.mDataProp!==k&&!c.mData&&(c.mData=c.mDataProp),c.sType&&(b._sManualType=c.sType),c.className&&!c.sClass&&(c.sClass=c.className),c.sClass&&e.addClass(c.sClass),h.extend(b,c),F(b,c,"sWidth","sWidthOrig"),c.iDataSort!==k&&(b.aDataSort=[c.iDataSort]),F(b,c,"aDataSort"));var g=b.mData,j=S(g),i=b.mRender? +S(b.mRender):null,c=function(a){return"string"===typeof a&&-1!==a.indexOf("@")};b._bAttrSrc=h.isPlainObject(g)&&(c(g.sort)||c(g.type)||c(g.filter));b._setter=null;b.fnGetData=function(a,b,c){var d=j(a,b,k,c);return i&&b?i(d,b,a,c):d};b.fnSetData=function(a,b,c){return N(g)(a,b,c)};"number"!==typeof g&&(a._rowReadObject=!0);a.oFeatures.bSort||(b.bSortable=!1,e.addClass(d.sSortableNone));a=-1!==h.inArray("asc",b.asSorting);c=-1!==h.inArray("desc",b.asSorting);!b.bSortable||!a&&!c?(b.sSortingClass=d.sSortableNone, +b.sSortingClassJUI=""):a&&!c?(b.sSortingClass=d.sSortableAsc,b.sSortingClassJUI=d.sSortJUIAscAllowed):!a&&c?(b.sSortingClass=d.sSortableDesc,b.sSortingClassJUI=d.sSortJUIDescAllowed):(b.sSortingClass=d.sSortable,b.sSortingClassJUI=d.sSortJUI)}function $(a){if(!1!==a.oFeatures.bAutoWidth){var b=a.aoColumns;Fa(a);for(var c=0,d=b.length;cq[f])d(l.length+q[f],m);else if("string"=== +typeof q[f]){j=0;for(i=l.length;jb&&a[e]--; -1!=d&&c===k&&a.splice(d, +1)}function da(a,b,c,d){var e=a.aoData[b],f,g=function(c,d){for(;c.childNodes.length;)c.removeChild(c.firstChild);c.innerHTML=B(a,b,d,"display")};if("dom"===c||(!c||"auto"===c)&&"dom"===e.src)e._aData=Ia(a,e,d,d===k?k:e._aData).data;else{var j=e.anCells;if(j)if(d!==k)g(j[d],d);else{c=0;for(f=j.length;c").appendTo(g));b=0;for(c=l.length;btr").attr("role","row");h(g).find(">tr>th, >tr>td").addClass(m.sHeaderTH);h(j).find(">tr>th, >tr>td").addClass(m.sFooterTH);if(null!==j){a=a.aoFooter[0];b=0;for(c=a.length;b=a.fnRecordsDisplay()?0:g,a.iInitDisplayStart=-1);var g=a._iDisplayStart,m=a.fnDisplayEnd();if(a.bDeferLoading)a.bDeferLoading=!1,a.iDraw++,C(a,!1);else if(j){if(!a.bDestroying&&!mb(a))return}else a.iDraw++;if(0!==i.length){f=j?a.aoData.length:m;for(j=j?0:g;j",{"class":e?d[0]:""}).append(h("",{valign:"top",colSpan:V(a),"class":a.oClasses.sRowEmpty}).html(c))[0];r(a,"aoHeaderCallback","header",[h(a.nTHead).children("tr")[0],Ka(a),g,m,i]);r(a,"aoFooterCallback","footer",[h(a.nTFoot).children("tr")[0],Ka(a),g,m,i]);d=h(a.nTBody);d.children().detach(); +d.append(h(b));r(a,"aoDrawCallback","draw",[a]);a.bSorted=!1;a.bFiltered=!1;a.bDrawing=!1}}function T(a,b){var c=a.oFeatures,d=c.bFilter;c.bSort&&nb(a);d?ga(a,a.oPreviousSearch):a.aiDisplay=a.aiDisplayMaster.slice();!0!==b&&(a._iDisplayStart=0);a._drawHold=b;P(a);a._drawHold=!1}function ob(a){var b=a.oClasses,c=h(a.nTable),c=h("
    ").insertBefore(c),d=a.oFeatures,e=h("
    ",{id:a.sTableId+"_wrapper","class":b.sWrapper+(a.nTFoot?"":" "+b.sNoFooter)});a.nHolding=c[0];a.nTableWrapper=e[0];a.nTableReinsertBefore= +a.nTable.nextSibling;for(var f=a.sDom.split(""),g,j,i,m,l,q,k=0;k")[0];m=f[k+1];if("'"==m||'"'==m){l="";for(q=2;f[k+q]!=m;)l+=f[k+q],q++;"H"==l?l=b.sJUIHeader:"F"==l&&(l=b.sJUIFooter);-1!=l.indexOf(".")?(m=l.split("."),i.id=m[0].substr(1,m[0].length-1),i.className=m[1]):"#"==l.charAt(0)?i.id=l.substr(1,l.length-1):i.className=l;k+=q}e.append(i);e=h(i)}else if(">"==j)e=e.parent();else if("l"==j&&d.bPaginate&&d.bLengthChange)g=pb(a);else if("f"==j&& +d.bFilter)g=qb(a);else if("r"==j&&d.bProcessing)g=rb(a);else if("t"==j)g=sb(a);else if("i"==j&&d.bInfo)g=tb(a);else if("p"==j&&d.bPaginate)g=ub(a);else if(0!==n.ext.feature.length){i=n.ext.feature;q=0;for(m=i.length;q',j=d.sSearch,j=j.match(/_INPUT_/)?j.replace("_INPUT_", +g):j+g,b=h("
    ",{id:!f.f?c+"_filter":null,"class":b.sFilter}).append(h("
    ").addClass(b.sLength);a.aanFeatures.l||(i[0].id=c+"_length");i.children().append(a.oLanguage.sLengthMenu.replace("_MENU_",e[0].outerHTML));h("select",i).val(a._iDisplayLength).on("change.DT",function(){Ra(a,h(this).val());P(a)});h(a.nTable).on("length.dt.DT",function(b,c,d){a=== +c&&h("select",i).val(d)});return i[0]}function ub(a){var b=a.sPaginationType,c=n.ext.pager[b],d="function"===typeof c,e=function(a){P(a)},b=h("
    ").addClass(a.oClasses.sPaging+b)[0],f=a.aanFeatures;d||c.fnInit(a,b,e);f.p||(b.id=a.sTableId+"_paginate",a.aoDrawCallback.push({fn:function(a){if(d){var b=a._iDisplayStart,i=a._iDisplayLength,h=a.fnRecordsDisplay(),l=-1===i,b=l?0:Math.ceil(b/i),i=l?1:Math.ceil(h/i),h=c(b,i),k,l=0;for(k=f.p.length;lf&&(d=0)):"first"==b?d=0:"previous"==b?(d=0<=e?d-e:0,0>d&&(d=0)):"next"==b?d+e",{id:!a.aanFeatures.r?a.sTableId+"_processing":null,"class":a.oClasses.sProcessing}).html(a.oLanguage.sProcessing).insertBefore(a.nTable)[0]} +function C(a,b){a.oFeatures.bProcessing&&h(a.aanFeatures.r).css("display",b?"block":"none");r(a,null,"processing",[a,b])}function sb(a){var b=h(a.nTable);b.attr("role","grid");var c=a.oScroll;if(""===c.sX&&""===c.sY)return a.nTable;var d=c.sX,e=c.sY,f=a.oClasses,g=b.children("caption"),j=g.length?g[0]._captionSide:null,i=h(b[0].cloneNode(!1)),m=h(b[0].cloneNode(!1)),l=b.children("tfoot");l.length||(l=null);i=h("
    ",{"class":f.sScrollWrapper}).append(h("
    ",{"class":f.sScrollHead}).css({overflow:"hidden", +position:"relative",border:0,width:d?!d?null:v(d):"100%"}).append(h("
    ",{"class":f.sScrollHeadInner}).css({"box-sizing":"content-box",width:c.sXInner||"100%"}).append(i.removeAttr("id").css("margin-left",0).append("top"===j?g:null).append(b.children("thead"))))).append(h("
    ",{"class":f.sScrollBody}).css({position:"relative",overflow:"auto",width:!d?null:v(d)}).append(b));l&&i.append(h("
    ",{"class":f.sScrollFoot}).css({overflow:"hidden",border:0,width:d?!d?null:v(d):"100%"}).append(h("
    ", +{"class":f.sScrollFootInner}).append(m.removeAttr("id").css("margin-left",0).append("bottom"===j?g:null).append(b.children("tfoot")))));var b=i.children(),k=b[0],f=b[1],t=l?b[2]:null;if(d)h(f).on("scroll.DT",function(){var a=this.scrollLeft;k.scrollLeft=a;l&&(t.scrollLeft=a)});h(f).css(e&&c.bCollapse?"max-height":"height",e);a.nScrollHead=k;a.nScrollBody=f;a.nScrollFoot=t;a.aoDrawCallback.push({fn:la,sName:"scrolling"});return i[0]}function la(a){var b=a.oScroll,c=b.sX,d=b.sXInner,e=b.sY,b=b.iBarWidth, +f=h(a.nScrollHead),g=f[0].style,j=f.children("div"),i=j[0].style,m=j.children("table"),j=a.nScrollBody,l=h(j),q=j.style,t=h(a.nScrollFoot).children("div"),n=t.children("table"),o=h(a.nTHead),p=h(a.nTable),s=p[0],r=s.style,u=a.nTFoot?h(a.nTFoot):null,x=a.oBrowser,U=x.bScrollOversize,Xb=D(a.aoColumns,"nTh"),Q,L,R,w,Ua=[],y=[],z=[],A=[],B,C=function(a){a=a.style;a.paddingTop="0";a.paddingBottom="0";a.borderTopWidth="0";a.borderBottomWidth="0";a.height=0};L=j.scrollHeight>j.clientHeight;if(a.scrollBarVis!== +L&&a.scrollBarVis!==k)a.scrollBarVis=L,$(a);else{a.scrollBarVis=L;p.children("thead, tfoot").remove();u&&(R=u.clone().prependTo(p),Q=u.find("tr"),R=R.find("tr"));w=o.clone().prependTo(p);o=o.find("tr");L=w.find("tr");w.find("th, td").removeAttr("tabindex");c||(q.width="100%",f[0].style.width="100%");h.each(ra(a,w),function(b,c){B=aa(a,b);c.style.width=a.aoColumns[B].sWidth});u&&I(function(a){a.style.width=""},R);f=p.outerWidth();if(""===c){r.width="100%";if(U&&(p.find("tbody").height()>j.offsetHeight|| +"scroll"==l.css("overflow-y")))r.width=v(p.outerWidth()-b);f=p.outerWidth()}else""!==d&&(r.width=v(d),f=p.outerWidth());I(C,L);I(function(a){z.push(a.innerHTML);Ua.push(v(h(a).css("width")))},L);I(function(a,b){if(h.inArray(a,Xb)!==-1)a.style.width=Ua[b]},o);h(L).height(0);u&&(I(C,R),I(function(a){A.push(a.innerHTML);y.push(v(h(a).css("width")))},R),I(function(a,b){a.style.width=y[b]},Q),h(R).height(0));I(function(a,b){a.innerHTML='
    '+z[b]+"
    ";a.childNodes[0].style.height= +"0";a.childNodes[0].style.overflow="hidden";a.style.width=Ua[b]},L);u&&I(function(a,b){a.innerHTML='
    '+A[b]+"
    ";a.childNodes[0].style.height="0";a.childNodes[0].style.overflow="hidden";a.style.width=y[b]},R);if(p.outerWidth()j.offsetHeight||"scroll"==l.css("overflow-y")?f+b:f;if(U&&(j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=v(Q-b);(""===c||""!==d)&&K(a,1,"Possible column misalignment",6)}else Q="100%";q.width=v(Q); +g.width=v(Q);u&&(a.nScrollFoot.style.width=v(Q));!e&&U&&(q.height=v(s.offsetHeight+b));c=p.outerWidth();m[0].style.width=v(c);i.width=v(c);d=p.height()>j.clientHeight||"scroll"==l.css("overflow-y");e="padding"+(x.bScrollbarLeft?"Left":"Right");i[e]=d?b+"px":"0px";u&&(n[0].style.width=v(c),t[0].style.width=v(c),t[0].style[e]=d?b+"px":"0px");p.children("colgroup").insertBefore(p.children("thead"));l.scroll();if((a.bSorted||a.bFiltered)&&!a._drawHold)j.scrollTop=0}}function I(a,b,c){for(var d=0,e=0, +f=b.length,g,j;e").appendTo(j.find("tbody"));j.find("thead, tfoot").remove();j.append(h(a.nTHead).clone()).append(h(a.nTFoot).clone());j.find("tfoot th, tfoot td").css("width","");m=ra(a,j.find("thead")[0]);for(n=0;n").css({width:o.sWidthOrig,margin:0,padding:0,border:0,height:1}));if(a.aoData.length)for(n=0;n").css(f||e?{position:"absolute",top:0,left:0,height:1,right:0,overflow:"hidden"}:{}).append(j).appendTo(k);f&&g?j.width(g):f?(j.css("width","auto"),j.removeAttr("width"),j.width()").css("width",v(a)).appendTo(b||H.body),d=c[0].offsetWidth;c.remove();return d}function Gb(a, +b){var c=Hb(a,b);if(0>c)return null;var d=a.aoData[c];return!d.nTr?h("").html(B(a,c,b,"display"))[0]:d.anCells[b]}function Hb(a,b){for(var c,d=-1,e=-1,f=0,g=a.aoData.length;fd&&(d=c.length,e=f);return e}function v(a){return null===a?"0px":"number"==typeof a?0>a?"0px":a+"px":a.match(/\d$/)?a+"px":a}function X(a){var b,c,d=[],e=a.aoColumns,f,g,j,i;b=a.aaSortingFixed;c=h.isPlainObject(b);var m=[];f=function(a){a.length&& +!h.isArray(a[0])?m.push(a):h.merge(m,a)};h.isArray(b)&&f(b);c&&b.pre&&f(b.pre);f(a.aaSorting);c&&b.post&&f(b.post);for(a=0;ae?1:0,0!==c)return"asc"===j.dir?c:-c;c=d[a];e=d[b];return ce?1:0}):i.sort(function(a,b){var c,g,j,i,k=h.length,n=f[a]._aSortData,o=f[b]._aSortData;for(j=0;jg?1:0})}a.bSorted=!0}function Jb(a){for(var b,c,d=a.aoColumns,e=X(a),a=a.oLanguage.oAria,f=0,g=d.length;f/g,"");var i=c.nTh;i.removeAttribute("aria-sort");c.bSortable&&(0e?e+1:3));e=0;for(f=d.length;ee?e+1:3))}a.aLastSort=d}function Ib(a,b){var c=a.aoColumns[b],d=n.ext.order[c.sSortDataType],e;d&&(e=d.call(a.oInstance,a,b,ba(a,b)));for(var f,g=n.ext.type.order[c.sType+"-pre"],j=0,i=a.aoData.length;j=f.length?[0,c[1]]:c)}));b.search!==k&&h.extend(a.oPreviousSearch,Cb(b.search));if(b.columns){d=0;for(e=b.columns.length;d=c&&(b=c-d);b-=b%d;if(-1===d||0>b)b=0;a._iDisplayStart=b}function Na(a,b){var c=a.renderer,d=n.ext.renderer[b];return h.isPlainObject(c)&&c[b]?d[c[b]]||d._:"string"=== +typeof c?d[c]||d._:d._}function y(a){return a.oFeatures.bServerSide?"ssp":a.ajax||a.sAjaxSource?"ajax":"dom"}function ia(a,b){var c=[],c=Lb.numbers_length,d=Math.floor(c/2);b<=c?c=Y(0,b):a<=d?(c=Y(0,c-2),c.push("ellipsis"),c.push(b-1)):(a>=b-1-d?c=Y(b-(c-2),b):(c=Y(a-d+2,a+d-1),c.push("ellipsis"),c.push(b-1)),c.splice(0,0,"ellipsis"),c.splice(0,0,0));c.DT_el="span";return c}function Da(a){h.each({num:function(b){return za(b,a)},"num-fmt":function(b){return za(b,a,Ya)},"html-num":function(b){return za(b, +a,Aa)},"html-num-fmt":function(b){return za(b,a,Aa,Ya)}},function(b,c){x.type.order[b+a+"-pre"]=c;b.match(/^html\-/)&&(x.type.search[b+a]=x.type.search.html)})}function Mb(a){return function(){var b=[ya(this[n.ext.iApiIndex])].concat(Array.prototype.slice.call(arguments));return n.ext.internal[a].apply(this,b)}}var n=function(a){this.$=function(a,b){return this.api(!0).$(a,b)};this._=function(a,b){return this.api(!0).rows(a,b).data()};this.api=function(a){return a?new s(ya(this[x.iApiIndex])):new s(this)}; +this.fnAddData=function(a,b){var c=this.api(!0),d=h.isArray(a)&&(h.isArray(a[0])||h.isPlainObject(a[0]))?c.rows.add(a):c.row.add(a);(b===k||b)&&c.draw();return d.flatten().toArray()};this.fnAdjustColumnSizing=function(a){var b=this.api(!0).columns.adjust(),c=b.settings()[0],d=c.oScroll;a===k||a?b.draw(!1):(""!==d.sX||""!==d.sY)&&la(c)};this.fnClearTable=function(a){var b=this.api(!0).clear();(a===k||a)&&b.draw()};this.fnClose=function(a){this.api(!0).row(a).child.hide()};this.fnDeleteRow=function(a, +b,c){var d=this.api(!0),a=d.rows(a),e=a.settings()[0],h=e.aoData[a[0][0]];a.remove();b&&b.call(this,e,h);(c===k||c)&&d.draw();return h};this.fnDestroy=function(a){this.api(!0).destroy(a)};this.fnDraw=function(a){this.api(!0).draw(a)};this.fnFilter=function(a,b,c,d,e,h){e=this.api(!0);null===b||b===k?e.search(a,c,d,h):e.column(b).search(a,c,d,h);e.draw()};this.fnGetData=function(a,b){var c=this.api(!0);if(a!==k){var d=a.nodeName?a.nodeName.toLowerCase():"";return b!==k||"td"==d||"th"==d?c.cell(a,b).data(): +c.row(a).data()||null}return c.data().toArray()};this.fnGetNodes=function(a){var b=this.api(!0);return a!==k?b.row(a).node():b.rows().nodes().flatten().toArray()};this.fnGetPosition=function(a){var b=this.api(!0),c=a.nodeName.toUpperCase();return"TR"==c?b.row(a).index():"TD"==c||"TH"==c?(a=b.cell(a).index(),[a.row,a.columnVisible,a.column]):null};this.fnIsOpen=function(a){return this.api(!0).row(a).child.isShown()};this.fnOpen=function(a,b,c){return this.api(!0).row(a).child(b,c).show().child()[0]}; +this.fnPageChange=function(a,b){var c=this.api(!0).page(a);(b===k||b)&&c.draw(!1)};this.fnSetColumnVis=function(a,b,c){a=this.api(!0).column(a).visible(b);(c===k||c)&&a.columns.adjust().draw()};this.fnSettings=function(){return ya(this[x.iApiIndex])};this.fnSort=function(a){this.api(!0).order(a).draw()};this.fnSortListener=function(a,b,c){this.api(!0).order.listener(a,b,c)};this.fnUpdate=function(a,b,c,d,e){var h=this.api(!0);c===k||null===c?h.row(b).data(a):h.cell(b,c).data(a);(e===k||e)&&h.columns.adjust(); +(d===k||d)&&h.draw();return 0};this.fnVersionCheck=x.fnVersionCheck;var b=this,c=a===k,d=this.length;c&&(a={});this.oApi=this.internal=x.internal;for(var e in n.ext.internal)e&&(this[e]=Mb(e));this.each(function(){var e={},g=1").appendTo(q)); +p.nTHead=b[0];b=q.children("tbody");b.length===0&&(b=h("").appendTo(q));p.nTBody=b[0];b=q.children("tfoot");if(b.length===0&&a.length>0&&(p.oScroll.sX!==""||p.oScroll.sY!==""))b=h("").appendTo(q);if(b.length===0||b.children().length===0)q.addClass(u.sNoFooter);else if(b.length>0){p.nTFoot=b[0];ea(p.aoFooter,p.nTFoot)}if(g.aaData)for(j=0;j/g,Zb=/^\d{2,4}[\.\/\-]\d{1,2}[\.\/\-]\d{1,2}([T ]{1}\d{1,2}[:\.]\d{2}([\.:]\d{2})?)?$/,$b=RegExp("(\\/|\\.|\\*|\\+|\\?|\\||\\(|\\)|\\[|\\]|\\{|\\}|\\\\|\\$|\\^|\\-)","g"),Ya=/[',$£€¥%\u2009\u202F\u20BD\u20a9\u20BArfkɃΞ]/gi,M=function(a){return!a||!0===a||"-"===a?!0:!1},Ob=function(a){var b=parseInt(a,10);return!isNaN(b)&& +isFinite(a)?b:null},Pb=function(a,b){Za[b]||(Za[b]=RegExp(Qa(b),"g"));return"string"===typeof a&&"."!==b?a.replace(/\./g,"").replace(Za[b],"."):a},$a=function(a,b,c){var d="string"===typeof a;if(M(a))return!0;b&&d&&(a=Pb(a,b));c&&d&&(a=a.replace(Ya,""));return!isNaN(parseFloat(a))&&isFinite(a)},Qb=function(a,b,c){return M(a)?!0:!(M(a)||"string"===typeof a)?null:$a(a.replace(Aa,""),b,c)?!0:null},D=function(a,b,c){var d=[],e=0,f=a.length;if(c!==k)for(;ea.length)){b=a.slice().sort();for(var c=b[0],d=1,e=b.length;d")[0],Wb=va.textContent!==k,Yb= +/<.*?>/g,Oa=n.util.throttle,Sb=[],w=Array.prototype,ac=function(a){var b,c,d=n.settings,e=h.map(d,function(a){return a.nTable});if(a){if(a.nTable&&a.oApi)return[a];if(a.nodeName&&"table"===a.nodeName.toLowerCase())return b=h.inArray(a,e),-1!==b?[d[b]]:null;if(a&&"function"===typeof a.settings)return a.settings().toArray();"string"===typeof a?c=h(a):a instanceof h&&(c=a)}else return[];if(c)return c.map(function(){b=h.inArray(this,e);return-1!==b?d[b]:null}).toArray()};s=function(a,b){if(!(this instanceof +s))return new s(a,b);var c=[],d=function(a){(a=ac(a))&&(c=c.concat(a))};if(h.isArray(a))for(var e=0,f=a.length;ea?new s(b[a],this[a]):null},filter:function(a){var b=[];if(w.filter)b=w.filter.call(this,a,this);else for(var c=0,d=this.length;c").addClass(b),h("td",c).addClass(b).html(a)[0].colSpan=V(d),e.push(c[0]))};f(a,b);c._details&&c._details.detach();c._details=h(e); +c._detailsShow&&c._details.insertAfter(c.nTr)}return this});o(["row().child.show()","row().child().show()"],function(){Ub(this,!0);return this});o(["row().child.hide()","row().child().hide()"],function(){Ub(this,!1);return this});o(["row().child.remove()","row().child().remove()"],function(){db(this);return this});o("row().child.isShown()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]]._detailsShow||!1:!1});var bc=/^([^:]+):(name|visIdx|visible)$/,Vb=function(a,b, +c,d,e){for(var c=[],d=0,f=e.length;d=0?b:g.length+b];if(typeof a==="function"){var e=Ba(c,f);return h.map(g,function(b,f){return a(f,Vb(c,f,0,0,e),i[f])?f:null})}var k=typeof a==="string"?a.match(bc): +"";if(k)switch(k[2]){case "visIdx":case "visible":b=parseInt(k[1],10);if(b<0){var n=h.map(g,function(a,b){return a.bVisible?b:null});return[n[n.length+b]]}return[aa(c,b)];case "name":return h.map(j,function(a,b){return a===k[1]?b:null});default:return[]}if(a.nodeName&&a._DT_CellIndex)return[a._DT_CellIndex.column];b=h(i).filter(a).map(function(){return h.inArray(this,i)}).toArray();if(b.length||!a.nodeName)return b;b=h(a).closest("*[data-dt-column]");return b.length?[b.data("dt-column")]:[]},c,f)}, +1);c.selector.cols=a;c.selector.opts=b;return c});u("columns().header()","column().header()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTh},1)});u("columns().footer()","column().footer()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTf},1)});u("columns().data()","column().data()",function(){return this.iterator("column-rows",Vb,1)});u("columns().dataSrc()","column().dataSrc()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].mData}, +1)});u("columns().cache()","column().cache()",function(a){return this.iterator("column-rows",function(b,c,d,e,f){return ja(b.aoData,f,"search"===a?"_aFilterData":"_aSortData",c)},1)});u("columns().nodes()","column().nodes()",function(){return this.iterator("column-rows",function(a,b,c,d,e){return ja(a.aoData,e,"anCells",b)},1)});u("columns().visible()","column().visible()",function(a,b){var c=this.iterator("column",function(b,c){if(a===k)return b.aoColumns[c].bVisible;var f=b.aoColumns,g=f[c],j=b.aoData, +i,m,l;if(a!==k&&g.bVisible!==a){if(a){var n=h.inArray(!0,D(f,"bVisible"),c+1);i=0;for(m=j.length;id;return!0};n.isDataTable= +n.fnIsDataTable=function(a){var b=h(a).get(0),c=!1;if(a instanceof n.Api)return!0;h.each(n.settings,function(a,e){var f=e.nScrollHead?h("table",e.nScrollHead)[0]:null,g=e.nScrollFoot?h("table",e.nScrollFoot)[0]:null;if(e.nTable===b||f===b||g===b)c=!0});return c};n.tables=n.fnTables=function(a){var b=!1;h.isPlainObject(a)&&(b=a.api,a=a.visible);var c=h.map(n.settings,function(b){if(!a||a&&h(b.nTable).is(":visible"))return b.nTable});return b?new s(c):c};n.camelToHungarian=J;o("$()",function(a,b){var c= +this.rows(b).nodes(),c=h(c);return h([].concat(c.filter(a).toArray(),c.find(a).toArray()))});h.each(["on","one","off"],function(a,b){o(b+"()",function(){var a=Array.prototype.slice.call(arguments);a[0]=h.map(a[0].split(/\s/),function(a){return!a.match(/\.dt\b/)?a+".dt":a}).join(" ");var d=h(this.tables().nodes());d[b].apply(d,a);return this})});o("clear()",function(){return this.iterator("table",function(a){oa(a)})});o("settings()",function(){return new s(this.context,this.context)});o("init()",function(){var a= +this.context;return a.length?a[0].oInit:null});o("data()",function(){return this.iterator("table",function(a){return D(a.aoData,"_aData")}).flatten()});o("destroy()",function(a){a=a||!1;return this.iterator("table",function(b){var c=b.nTableWrapper.parentNode,d=b.oClasses,e=b.nTable,f=b.nTBody,g=b.nTHead,j=b.nTFoot,i=h(e),f=h(f),k=h(b.nTableWrapper),l=h.map(b.aoData,function(a){return a.nTr}),o;b.bDestroying=!0;r(b,"aoDestroyCallback","destroy",[b]);a||(new s(b)).columns().visible(!0);k.off(".DT").find(":not(tbody *)").off(".DT"); +h(E).off(".DT-"+b.sInstance);e!=g.parentNode&&(i.children("thead").detach(),i.append(g));j&&e!=j.parentNode&&(i.children("tfoot").detach(),i.append(j));b.aaSorting=[];b.aaSortingFixed=[];wa(b);h(l).removeClass(b.asStripeClasses.join(" "));h("th, td",g).removeClass(d.sSortable+" "+d.sSortableAsc+" "+d.sSortableDesc+" "+d.sSortableNone);f.children().detach();f.append(l);g=a?"remove":"detach";i[g]();k[g]();!a&&c&&(c.insertBefore(e,b.nTableReinsertBefore),i.css("width",b.sDestroyWidth).removeClass(d.sTable), +(o=b.asDestroyStripes.length)&&f.children().each(function(a){h(this).addClass(b.asDestroyStripes[a%o])}));c=h.inArray(b,n.settings);-1!==c&&n.settings.splice(c,1)})});h.each(["column","row","cell"],function(a,b){o(b+"s().every()",function(a){var d=this.selector.opts,e=this;return this.iterator(b,function(f,g,h,i,m){a.call(e[b](g,"cell"===b?h:d,"cell"===b?d:k),g,h,i,m)})})});o("i18n()",function(a,b,c){var d=this.context[0],a=S(a)(d.oLanguage);a===k&&(a=b);c!==k&&h.isPlainObject(a)&&(a=a[c]!==k?a[c]: +a._);return a.replace("%d",c)});n.version="1.10.19";n.settings=[];n.models={};n.models.oSearch={bCaseInsensitive:!0,sSearch:"",bRegex:!1,bSmart:!0};n.models.oRow={nTr:null,anCells:null,_aData:[],_aSortData:null,_aFilterData:null,_sFilterRow:null,_sRowStripe:"",src:null,idx:-1};n.models.oColumn={idx:null,aDataSort:null,asSorting:null,bSearchable:null,bSortable:null,bVisible:null,_sManualType:null,_bAttrSrc:!1,fnCreatedCell:null,fnGetData:null,fnSetData:null,mData:null,mRender:null,nTh:null,nTf:null, +sClass:null,sContentPadding:null,sDefaultContent:null,sName:null,sSortDataType:"std",sSortingClass:null,sSortingClassJUI:null,sTitle:null,sType:null,sWidth:null,sWidthOrig:null};n.defaults={aaData:null,aaSorting:[[0,"asc"]],aaSortingFixed:[],ajax:null,aLengthMenu:[10,25,50,100],aoColumns:null,aoColumnDefs:null,aoSearchCols:[],asStripeClasses:null,bAutoWidth:!0,bDeferRender:!1,bDestroy:!1,bFilter:!0,bInfo:!0,bLengthChange:!0,bPaginate:!0,bProcessing:!1,bRetrieve:!1,bScrollCollapse:!1,bServerSide:!1, +bSort:!0,bSortMulti:!0,bSortCellsTop:!1,bSortClasses:!0,bStateSave:!1,fnCreatedRow:null,fnDrawCallback:null,fnFooterCallback:null,fnFormatNumber:function(a){return a.toString().replace(/\B(?=(\d{3})+(?!\d))/g,this.oLanguage.sThousands)},fnHeaderCallback:null,fnInfoCallback:null,fnInitComplete:null,fnPreDrawCallback:null,fnRowCallback:null,fnServerData:null,fnServerParams:null,fnStateLoadCallback:function(a){try{return JSON.parse((-1===a.iStateDuration?sessionStorage:localStorage).getItem("DataTables_"+ +a.sInstance+"_"+location.pathname))}catch(b){}},fnStateLoadParams:null,fnStateLoaded:null,fnStateSaveCallback:function(a,b){try{(-1===a.iStateDuration?sessionStorage:localStorage).setItem("DataTables_"+a.sInstance+"_"+location.pathname,JSON.stringify(b))}catch(c){}},fnStateSaveParams:null,iStateDuration:7200,iDeferLoading:null,iDisplayLength:10,iDisplayStart:0,iTabIndex:0,oClasses:{},oLanguage:{oAria:{sSortAscending:": activate to sort column ascending",sSortDescending:": activate to sort column descending"}, +oPaginate:{sFirst:"First",sLast:"Last",sNext:"Next",sPrevious:"Previous"},sEmptyTable:"No data available in table",sInfo:"Showing _START_ to _END_ of _TOTAL_ entries",sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({}, +n.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",renderer:null,rowId:"DT_RowId"};Z(n.defaults);n.defaults.column={aDataSort:null,iDataSort:-1,asSorting:["asc","desc"],bSearchable:!0,bSortable:!0,bVisible:!0,fnCreatedCell:null,mData:null,mRender:null,sCellType:"td",sClass:"",sContentPadding:"",sDefaultContent:null,sName:"",sSortDataType:"std",sTitle:null,sType:null,sWidth:null}; +Z(n.defaults.column);n.models.oSettings={oFeatures:{bAutoWidth:null,bDeferRender:null,bFilter:null,bInfo:null,bLengthChange:null,bPaginate:null,bProcessing:null,bServerSide:null,bSort:null,bSortMulti:null,bSortClasses:null,bStateSave:null},oScroll:{bCollapse:null,iBarWidth:0,sX:null,sXInner:null,sY:null},oLanguage:{fnInfoCallback:null},oBrowser:{bScrollOversize:!1,bScrollbarLeft:!1,bBounding:!1,barWidth:0},ajax:null,aanFeatures:[],aoData:[],aiDisplay:[],aiDisplayMaster:[],aIds:{},aoColumns:[],aoHeader:[], +aoFooter:[],oPreviousSearch:{},aoPreSearchCols:[],aaSorting:null,aaSortingFixed:[],asStripeClasses:null,asDestroyStripes:[],sDestroyWidth:0,aoRowCallback:[],aoHeaderCallback:[],aoFooterCallback:[],aoDrawCallback:[],aoRowCreatedCallback:[],aoPreDrawCallback:[],aoInitComplete:[],aoStateSaveParams:[],aoStateLoadParams:[],aoStateLoaded:[],sTableId:"",nTable:null,nTHead:null,nTFoot:null,nTBody:null,nTableWrapper:null,bDeferLoading:!1,bInitialised:!1,aoOpenRows:[],sDom:null,searchDelay:null,sPaginationType:"two_button", +iStateDuration:0,aoStateSave:[],aoStateLoad:[],oSavedState:null,oLoadedState:null,sAjaxSource:null,sAjaxDataProp:null,bAjaxDataGet:!0,jqXHR:null,json:k,oAjaxData:k,fnServerData:null,aoServerParams:[],sServerMethod:null,fnFormatNumber:null,aLengthMenu:null,iDraw:0,bDrawing:!1,iDrawError:-1,_iDisplayLength:10,_iDisplayStart:0,_iRecordsTotal:0,_iRecordsDisplay:0,oClasses:{},bFiltered:!1,bSorted:!1,bSortCellsTop:null,oInit:null,aoDestroyCallback:[],fnRecordsTotal:function(){return"ssp"==y(this)?1*this._iRecordsTotal: +this.aiDisplayMaster.length},fnRecordsDisplay:function(){return"ssp"==y(this)?1*this._iRecordsDisplay:this.aiDisplay.length},fnDisplayEnd:function(){var a=this._iDisplayLength,b=this._iDisplayStart,c=b+a,d=this.aiDisplay.length,e=this.oFeatures,f=e.bPaginate;return e.bServerSide?!1===f||-1===a?b+d:Math.min(b+a,this._iRecordsDisplay):!f||c>d||-1===a?d:c},oInstance:null,sInstance:null,iTabIndex:0,nScrollHead:null,nScrollFoot:null,aLastSort:[],oPlugins:{},rowIdFn:null,rowId:null};n.ext=x={buttons:{}, +classes:{},builder:"-source-",errMode:"alert",feature:[],search:[],selector:{cell:[],column:[],row:[]},internal:{},legacy:{ajax:null},pager:{},renderer:{pageButton:{},header:{}},order:{},type:{detect:[],search:{},order:{}},_unique:0,fnVersionCheck:n.fnVersionCheck,iApiIndex:0,oJUIClasses:{},sVersion:n.version};h.extend(x,{afnFiltering:x.search,aTypes:x.type.detect,ofnSearch:x.type.search,oSort:x.type.order,afnSortData:x.order,aoFeatures:x.feature,oApi:x.internal,oStdClasses:x.classes,oPagination:x.pager}); +h.extend(n.ext.classes,{sTable:"dataTable",sNoFooter:"no-footer",sPageButton:"paginate_button",sPageButtonActive:"current",sPageButtonDisabled:"disabled",sStripeOdd:"odd",sStripeEven:"even",sRowEmpty:"dataTables_empty",sWrapper:"dataTables_wrapper",sFilter:"dataTables_filter",sInfo:"dataTables_info",sPaging:"dataTables_paginate paging_",sLength:"dataTables_length",sProcessing:"dataTables_processing",sSortAsc:"sorting_asc",sSortDesc:"sorting_desc",sSortable:"sorting",sSortableAsc:"sorting_asc_disabled", +sSortableDesc:"sorting_desc_disabled",sSortableNone:"sorting_disabled",sSortColumn:"sorting_",sFilterInput:"",sLengthSelect:"",sScrollWrapper:"dataTables_scroll",sScrollHead:"dataTables_scrollHead",sScrollHeadInner:"dataTables_scrollHeadInner",sScrollBody:"dataTables_scrollBody",sScrollFoot:"dataTables_scrollFoot",sScrollFootInner:"dataTables_scrollFootInner",sHeaderTH:"",sFooterTH:"",sSortJUIAsc:"",sSortJUIDesc:"",sSortJUI:"",sSortJUIAscAllowed:"",sSortJUIDescAllowed:"",sSortJUIWrapper:"",sSortIcon:"", +sJUIHeader:"",sJUIFooter:""});var Lb=n.ext.pager;h.extend(Lb,{simple:function(){return["previous","next"]},full:function(){return["first","previous","next","last"]},numbers:function(a,b){return[ia(a,b)]},simple_numbers:function(a,b){return["previous",ia(a,b),"next"]},full_numbers:function(a,b){return["first","previous",ia(a,b),"next","last"]},first_last_numbers:function(a,b){return["first",ia(a,b),"last"]},_numbers:ia,numbers_length:7});h.extend(!0,n.ext.renderer,{pageButton:{_:function(a,b,c,d,e, +f){var g=a.oClasses,j=a.oLanguage.oPaginate,i=a.oLanguage.oAria.paginate||{},m,l,n=0,o=function(b,d){var k,s,u,r,v=function(b){Ta(a,b.data.action,true)};k=0;for(s=d.length;k").appendTo(b);o(u,r)}else{m=null;l="";switch(r){case "ellipsis":b.append('…');break;case "first":m=j.sFirst;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "previous":m=j.sPrevious;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "next":m= +j.sNext;l=r+(e",{"class":g.sPageButton+" "+l,"aria-controls":a.sTableId,"aria-label":i[r],"data-dt-idx":n,tabindex:a.iTabIndex,id:c===0&&typeof r==="string"?a.sTableId+"_"+r:null}).html(m).appendTo(b);Wa(u,{action:r},v);n++}}}},s;try{s=h(b).find(H.activeElement).data("dt-idx")}catch(u){}o(h(b).empty(),d);s!==k&&h(b).find("[data-dt-idx="+ +s+"]").focus()}}});h.extend(n.ext.type.detect,[function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c)?"num"+c:null},function(a){if(a&&!(a instanceof Date)&&!Zb.test(a))return null;var b=Date.parse(a);return null!==b&&!isNaN(b)||M(a)?"date":null},function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c,!0)?"num-fmt"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c)?"html-num"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c,!0)?"html-num-fmt"+c:null},function(a){return M(a)|| +"string"===typeof a&&-1!==a.indexOf("<")?"html":null}]);h.extend(n.ext.type.search,{html:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," ").replace(Aa,""):""},string:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," "):a}});var za=function(a,b,c,d){if(0!==a&&(!a||"-"===a))return-Infinity;b&&(a=Pb(a,b));a.replace&&(c&&(a=a.replace(c,"")),d&&(a=a.replace(d,"")));return 1*a};h.extend(x.type.order,{"date-pre":function(a){a=Date.parse(a);return isNaN(a)?-Infinity:a},"html-pre":function(a){return M(a)? +"":a.replace?a.replace(/<.*?>/g,"").toLowerCase():a+""},"string-pre":function(a){return M(a)?"":"string"===typeof a?a.toLowerCase():!a.toString?"":a.toString()},"string-asc":function(a,b){return ab?1:0},"string-desc":function(a,b){return ab?-1:0}});Da("");h.extend(!0,n.ext.renderer,{header:{_:function(a,b,c,d){h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(c.sSortingClass+" "+d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc: +c.sSortingClass)}})},jqueryui:function(a,b,c,d){h("
    ").addClass(d.sSortJUIWrapper).append(b.contents()).append(h("").addClass(d.sSortIcon+" "+c.sSortingClassJUI)).appendTo(b);h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass);b.find("span."+d.sSortIcon).removeClass(d.sSortJUIAsc+" "+d.sSortJUIDesc+" "+d.sSortJUI+" "+d.sSortJUIAscAllowed+" "+d.sSortJUIDescAllowed).addClass(h[e]== +"asc"?d.sSortJUIAsc:h[e]=="desc"?d.sSortJUIDesc:c.sSortingClassJUI)}})}}});var eb=function(a){return"string"===typeof a?a.replace(//g,">").replace(/"/g,"""):a};n.render={number:function(a,b,c,d,e){return{display:function(f){if("number"!==typeof f&&"string"!==typeof f)return f;var g=0>f?"-":"",h=parseFloat(f);if(isNaN(h))return eb(f);h=h.toFixed(c);f=Math.abs(h);h=parseInt(f,10);f=c?b+(f-h).toFixed(c).substring(2):"";return g+(d||"")+h.toString().replace(/\B(?=(\d{3})+(?!\d))/g, +a)+f+(e||"")}}},text:function(){return{display:eb,filter:eb}}};h.extend(n.ext.internal,{_fnExternApiFunc:Mb,_fnBuildAjax:sa,_fnAjaxUpdate:mb,_fnAjaxParameters:vb,_fnAjaxUpdateDraw:wb,_fnAjaxDataSrc:ta,_fnAddColumn:Ea,_fnColumnOptions:ka,_fnAdjustColumnSizing:$,_fnVisibleToColumnIndex:aa,_fnColumnIndexToVisible:ba,_fnVisbleColumns:V,_fnGetColumns:ma,_fnColumnTypes:Ga,_fnApplyColumnDefs:jb,_fnHungarianMap:Z,_fnCamelToHungarian:J,_fnLanguageCompat:Ca,_fnBrowserDetect:hb,_fnAddData:O,_fnAddTr:na,_fnNodeToDataIndex:function(a, +b){return b._DT_RowIndex!==k?b._DT_RowIndex:null},_fnNodeToColumnIndex:function(a,b,c){return h.inArray(c,a.aoData[b].anCells)},_fnGetCellData:B,_fnSetCellData:kb,_fnSplitObjNotation:Ja,_fnGetObjectDataFn:S,_fnSetObjectDataFn:N,_fnGetDataMaster:Ka,_fnClearTable:oa,_fnDeleteIndex:pa,_fnInvalidate:da,_fnGetRowElements:Ia,_fnCreateTr:Ha,_fnBuildHead:lb,_fnDrawHead:fa,_fnDraw:P,_fnReDraw:T,_fnAddOptionsHtml:ob,_fnDetectHeader:ea,_fnGetUniqueThs:ra,_fnFeatureHtmlFilter:qb,_fnFilterComplete:ga,_fnFilterCustom:zb, +_fnFilterColumn:yb,_fnFilter:xb,_fnFilterCreateSearch:Pa,_fnEscapeRegex:Qa,_fnFilterData:Ab,_fnFeatureHtmlInfo:tb,_fnUpdateInfo:Db,_fnInfoMacros:Eb,_fnInitialise:ha,_fnInitComplete:ua,_fnLengthChange:Ra,_fnFeatureHtmlLength:pb,_fnFeatureHtmlPaginate:ub,_fnPageChange:Ta,_fnFeatureHtmlProcessing:rb,_fnProcessingDisplay:C,_fnFeatureHtmlTable:sb,_fnScrollDraw:la,_fnApplyToChildren:I,_fnCalculateColumnWidths:Fa,_fnThrottle:Oa,_fnConvertToWidth:Fb,_fnGetWidestNode:Gb,_fnGetMaxLenString:Hb,_fnStringToCss:v, +_fnSortFlatten:X,_fnSort:nb,_fnSortAria:Jb,_fnSortListener:Va,_fnSortAttachListener:Ma,_fnSortingClasses:wa,_fnSortData:Ib,_fnSaveState:xa,_fnLoadState:Kb,_fnSettingsFromNode:ya,_fnLog:K,_fnMap:F,_fnBindAction:Wa,_fnCallbackReg:z,_fnCallbackFire:r,_fnLengthOverflow:Sa,_fnRenderer:Na,_fnDataSource:y,_fnRowAttributes:La,_fnExtend:Xa,_fnCalculateEnd:function(){}});h.fn.dataTable=n;n.$=h;h.fn.dataTableSettings=n.settings;h.fn.dataTableExt=n.ext;h.fn.DataTable=function(a){return h(this).dataTable(a).api()}; +h.each(n,function(a,b){h.fn.DataTable[a]=b});return h.fn.dataTable}); diff --git a/archivebox-0.5.3/archivebox/themes/default/static/jquery.min.js b/archivebox-0.5.3/archivebox/themes/default/static/jquery.min.js new file mode 100644 index 0000000..4d9b3a2 --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/default/static/jquery.min.js @@ -0,0 +1,2 @@ +/*! jQuery v3.3.1 | (c) JS Foundation and other contributors | jquery.org/license */ +!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(e,t){"use strict";var n=[],r=e.document,i=Object.getPrototypeOf,o=n.slice,a=n.concat,s=n.push,u=n.indexOf,l={},c=l.toString,f=l.hasOwnProperty,p=f.toString,d=p.call(Object),h={},g=function e(t){return"function"==typeof t&&"number"!=typeof t.nodeType},y=function e(t){return null!=t&&t===t.window},v={type:!0,src:!0,noModule:!0};function m(e,t,n){var i,o=(t=t||r).createElement("script");if(o.text=e,n)for(i in v)n[i]&&(o[i]=n[i]);t.head.appendChild(o).parentNode.removeChild(o)}function x(e){return null==e?e+"":"object"==typeof e||"function"==typeof e?l[c.call(e)]||"object":typeof e}var b="3.3.1",w=function(e,t){return new w.fn.init(e,t)},T=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g;w.fn=w.prototype={jquery:"3.3.1",constructor:w,length:0,toArray:function(){return o.call(this)},get:function(e){return null==e?o.call(this):e<0?this[e+this.length]:this[e]},pushStack:function(e){var t=w.merge(this.constructor(),e);return t.prevObject=this,t},each:function(e){return w.each(this,e)},map:function(e){return this.pushStack(w.map(this,function(t,n){return e.call(t,n,t)}))},slice:function(){return this.pushStack(o.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(e){var t=this.length,n=+e+(e<0?t:0);return this.pushStack(n>=0&&n0&&t-1 in e)}var E=function(e){var t,n,r,i,o,a,s,u,l,c,f,p,d,h,g,y,v,m,x,b="sizzle"+1*new Date,w=e.document,T=0,C=0,E=ae(),k=ae(),S=ae(),D=function(e,t){return e===t&&(f=!0),0},N={}.hasOwnProperty,A=[],j=A.pop,q=A.push,L=A.push,H=A.slice,O=function(e,t){for(var n=0,r=e.length;n+~]|"+M+")"+M+"*"),z=new RegExp("="+M+"*([^\\]'\"]*?)"+M+"*\\]","g"),X=new RegExp(W),U=new RegExp("^"+R+"$"),V={ID:new RegExp("^#("+R+")"),CLASS:new RegExp("^\\.("+R+")"),TAG:new RegExp("^("+R+"|[*])"),ATTR:new RegExp("^"+I),PSEUDO:new RegExp("^"+W),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+M+"*(even|odd|(([+-]|)(\\d*)n|)"+M+"*(?:([+-]|)"+M+"*(\\d+)|))"+M+"*\\)|)","i"),bool:new RegExp("^(?:"+P+")$","i"),needsContext:new RegExp("^"+M+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+M+"*((?:-\\d)?\\d*)"+M+"*\\)|)(?=[^-]|$)","i")},G=/^(?:input|select|textarea|button)$/i,Y=/^h\d$/i,Q=/^[^{]+\{\s*\[native \w/,J=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,K=/[+~]/,Z=new RegExp("\\\\([\\da-f]{1,6}"+M+"?|("+M+")|.)","ig"),ee=function(e,t,n){var r="0x"+t-65536;return r!==r||n?t:r<0?String.fromCharCode(r+65536):String.fromCharCode(r>>10|55296,1023&r|56320)},te=/([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g,ne=function(e,t){return t?"\0"===e?"\ufffd":e.slice(0,-1)+"\\"+e.charCodeAt(e.length-1).toString(16)+" ":"\\"+e},re=function(){p()},ie=me(function(e){return!0===e.disabled&&("form"in e||"label"in e)},{dir:"parentNode",next:"legend"});try{L.apply(A=H.call(w.childNodes),w.childNodes),A[w.childNodes.length].nodeType}catch(e){L={apply:A.length?function(e,t){q.apply(e,H.call(t))}:function(e,t){var n=e.length,r=0;while(e[n++]=t[r++]);e.length=n-1}}}function oe(e,t,r,i){var o,s,l,c,f,h,v,m=t&&t.ownerDocument,T=t?t.nodeType:9;if(r=r||[],"string"!=typeof e||!e||1!==T&&9!==T&&11!==T)return r;if(!i&&((t?t.ownerDocument||t:w)!==d&&p(t),t=t||d,g)){if(11!==T&&(f=J.exec(e)))if(o=f[1]){if(9===T){if(!(l=t.getElementById(o)))return r;if(l.id===o)return r.push(l),r}else if(m&&(l=m.getElementById(o))&&x(t,l)&&l.id===o)return r.push(l),r}else{if(f[2])return L.apply(r,t.getElementsByTagName(e)),r;if((o=f[3])&&n.getElementsByClassName&&t.getElementsByClassName)return L.apply(r,t.getElementsByClassName(o)),r}if(n.qsa&&!S[e+" "]&&(!y||!y.test(e))){if(1!==T)m=t,v=e;else if("object"!==t.nodeName.toLowerCase()){(c=t.getAttribute("id"))?c=c.replace(te,ne):t.setAttribute("id",c=b),s=(h=a(e)).length;while(s--)h[s]="#"+c+" "+ve(h[s]);v=h.join(","),m=K.test(e)&&ge(t.parentNode)||t}if(v)try{return L.apply(r,m.querySelectorAll(v)),r}catch(e){}finally{c===b&&t.removeAttribute("id")}}}return u(e.replace(B,"$1"),t,r,i)}function ae(){var e=[];function t(n,i){return e.push(n+" ")>r.cacheLength&&delete t[e.shift()],t[n+" "]=i}return t}function se(e){return e[b]=!0,e}function ue(e){var t=d.createElement("fieldset");try{return!!e(t)}catch(e){return!1}finally{t.parentNode&&t.parentNode.removeChild(t),t=null}}function le(e,t){var n=e.split("|"),i=n.length;while(i--)r.attrHandle[n[i]]=t}function ce(e,t){var n=t&&e,r=n&&1===e.nodeType&&1===t.nodeType&&e.sourceIndex-t.sourceIndex;if(r)return r;if(n)while(n=n.nextSibling)if(n===t)return-1;return e?1:-1}function fe(e){return function(t){return"input"===t.nodeName.toLowerCase()&&t.type===e}}function pe(e){return function(t){var n=t.nodeName.toLowerCase();return("input"===n||"button"===n)&&t.type===e}}function de(e){return function(t){return"form"in t?t.parentNode&&!1===t.disabled?"label"in t?"label"in t.parentNode?t.parentNode.disabled===e:t.disabled===e:t.isDisabled===e||t.isDisabled!==!e&&ie(t)===e:t.disabled===e:"label"in t&&t.disabled===e}}function he(e){return se(function(t){return t=+t,se(function(n,r){var i,o=e([],n.length,t),a=o.length;while(a--)n[i=o[a]]&&(n[i]=!(r[i]=n[i]))})})}function ge(e){return e&&"undefined"!=typeof e.getElementsByTagName&&e}n=oe.support={},o=oe.isXML=function(e){var t=e&&(e.ownerDocument||e).documentElement;return!!t&&"HTML"!==t.nodeName},p=oe.setDocument=function(e){var t,i,a=e?e.ownerDocument||e:w;return a!==d&&9===a.nodeType&&a.documentElement?(d=a,h=d.documentElement,g=!o(d),w!==d&&(i=d.defaultView)&&i.top!==i&&(i.addEventListener?i.addEventListener("unload",re,!1):i.attachEvent&&i.attachEvent("onunload",re)),n.attributes=ue(function(e){return e.className="i",!e.getAttribute("className")}),n.getElementsByTagName=ue(function(e){return e.appendChild(d.createComment("")),!e.getElementsByTagName("*").length}),n.getElementsByClassName=Q.test(d.getElementsByClassName),n.getById=ue(function(e){return h.appendChild(e).id=b,!d.getElementsByName||!d.getElementsByName(b).length}),n.getById?(r.filter.ID=function(e){var t=e.replace(Z,ee);return function(e){return e.getAttribute("id")===t}},r.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&g){var n=t.getElementById(e);return n?[n]:[]}}):(r.filter.ID=function(e){var t=e.replace(Z,ee);return function(e){var n="undefined"!=typeof e.getAttributeNode&&e.getAttributeNode("id");return n&&n.value===t}},r.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&g){var n,r,i,o=t.getElementById(e);if(o){if((n=o.getAttributeNode("id"))&&n.value===e)return[o];i=t.getElementsByName(e),r=0;while(o=i[r++])if((n=o.getAttributeNode("id"))&&n.value===e)return[o]}return[]}}),r.find.TAG=n.getElementsByTagName?function(e,t){return"undefined"!=typeof t.getElementsByTagName?t.getElementsByTagName(e):n.qsa?t.querySelectorAll(e):void 0}:function(e,t){var n,r=[],i=0,o=t.getElementsByTagName(e);if("*"===e){while(n=o[i++])1===n.nodeType&&r.push(n);return r}return o},r.find.CLASS=n.getElementsByClassName&&function(e,t){if("undefined"!=typeof t.getElementsByClassName&&g)return t.getElementsByClassName(e)},v=[],y=[],(n.qsa=Q.test(d.querySelectorAll))&&(ue(function(e){h.appendChild(e).innerHTML="",e.querySelectorAll("[msallowcapture^='']").length&&y.push("[*^$]="+M+"*(?:''|\"\")"),e.querySelectorAll("[selected]").length||y.push("\\["+M+"*(?:value|"+P+")"),e.querySelectorAll("[id~="+b+"-]").length||y.push("~="),e.querySelectorAll(":checked").length||y.push(":checked"),e.querySelectorAll("a#"+b+"+*").length||y.push(".#.+[+~]")}),ue(function(e){e.innerHTML="";var t=d.createElement("input");t.setAttribute("type","hidden"),e.appendChild(t).setAttribute("name","D"),e.querySelectorAll("[name=d]").length&&y.push("name"+M+"*[*^$|!~]?="),2!==e.querySelectorAll(":enabled").length&&y.push(":enabled",":disabled"),h.appendChild(e).disabled=!0,2!==e.querySelectorAll(":disabled").length&&y.push(":enabled",":disabled"),e.querySelectorAll("*,:x"),y.push(",.*:")})),(n.matchesSelector=Q.test(m=h.matches||h.webkitMatchesSelector||h.mozMatchesSelector||h.oMatchesSelector||h.msMatchesSelector))&&ue(function(e){n.disconnectedMatch=m.call(e,"*"),m.call(e,"[s!='']:x"),v.push("!=",W)}),y=y.length&&new RegExp(y.join("|")),v=v.length&&new RegExp(v.join("|")),t=Q.test(h.compareDocumentPosition),x=t||Q.test(h.contains)?function(e,t){var n=9===e.nodeType?e.documentElement:e,r=t&&t.parentNode;return e===r||!(!r||1!==r.nodeType||!(n.contains?n.contains(r):e.compareDocumentPosition&&16&e.compareDocumentPosition(r)))}:function(e,t){if(t)while(t=t.parentNode)if(t===e)return!0;return!1},D=t?function(e,t){if(e===t)return f=!0,0;var r=!e.compareDocumentPosition-!t.compareDocumentPosition;return r||(1&(r=(e.ownerDocument||e)===(t.ownerDocument||t)?e.compareDocumentPosition(t):1)||!n.sortDetached&&t.compareDocumentPosition(e)===r?e===d||e.ownerDocument===w&&x(w,e)?-1:t===d||t.ownerDocument===w&&x(w,t)?1:c?O(c,e)-O(c,t):0:4&r?-1:1)}:function(e,t){if(e===t)return f=!0,0;var n,r=0,i=e.parentNode,o=t.parentNode,a=[e],s=[t];if(!i||!o)return e===d?-1:t===d?1:i?-1:o?1:c?O(c,e)-O(c,t):0;if(i===o)return ce(e,t);n=e;while(n=n.parentNode)a.unshift(n);n=t;while(n=n.parentNode)s.unshift(n);while(a[r]===s[r])r++;return r?ce(a[r],s[r]):a[r]===w?-1:s[r]===w?1:0},d):d},oe.matches=function(e,t){return oe(e,null,null,t)},oe.matchesSelector=function(e,t){if((e.ownerDocument||e)!==d&&p(e),t=t.replace(z,"='$1']"),n.matchesSelector&&g&&!S[t+" "]&&(!v||!v.test(t))&&(!y||!y.test(t)))try{var r=m.call(e,t);if(r||n.disconnectedMatch||e.document&&11!==e.document.nodeType)return r}catch(e){}return oe(t,d,null,[e]).length>0},oe.contains=function(e,t){return(e.ownerDocument||e)!==d&&p(e),x(e,t)},oe.attr=function(e,t){(e.ownerDocument||e)!==d&&p(e);var i=r.attrHandle[t.toLowerCase()],o=i&&N.call(r.attrHandle,t.toLowerCase())?i(e,t,!g):void 0;return void 0!==o?o:n.attributes||!g?e.getAttribute(t):(o=e.getAttributeNode(t))&&o.specified?o.value:null},oe.escape=function(e){return(e+"").replace(te,ne)},oe.error=function(e){throw new Error("Syntax error, unrecognized expression: "+e)},oe.uniqueSort=function(e){var t,r=[],i=0,o=0;if(f=!n.detectDuplicates,c=!n.sortStable&&e.slice(0),e.sort(D),f){while(t=e[o++])t===e[o]&&(i=r.push(o));while(i--)e.splice(r[i],1)}return c=null,e},i=oe.getText=function(e){var t,n="",r=0,o=e.nodeType;if(o){if(1===o||9===o||11===o){if("string"==typeof e.textContent)return e.textContent;for(e=e.firstChild;e;e=e.nextSibling)n+=i(e)}else if(3===o||4===o)return e.nodeValue}else while(t=e[r++])n+=i(t);return n},(r=oe.selectors={cacheLength:50,createPseudo:se,match:V,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(e){return e[1]=e[1].replace(Z,ee),e[3]=(e[3]||e[4]||e[5]||"").replace(Z,ee),"~="===e[2]&&(e[3]=" "+e[3]+" "),e.slice(0,4)},CHILD:function(e){return e[1]=e[1].toLowerCase(),"nth"===e[1].slice(0,3)?(e[3]||oe.error(e[0]),e[4]=+(e[4]?e[5]+(e[6]||1):2*("even"===e[3]||"odd"===e[3])),e[5]=+(e[7]+e[8]||"odd"===e[3])):e[3]&&oe.error(e[0]),e},PSEUDO:function(e){var t,n=!e[6]&&e[2];return V.CHILD.test(e[0])?null:(e[3]?e[2]=e[4]||e[5]||"":n&&X.test(n)&&(t=a(n,!0))&&(t=n.indexOf(")",n.length-t)-n.length)&&(e[0]=e[0].slice(0,t),e[2]=n.slice(0,t)),e.slice(0,3))}},filter:{TAG:function(e){var t=e.replace(Z,ee).toLowerCase();return"*"===e?function(){return!0}:function(e){return e.nodeName&&e.nodeName.toLowerCase()===t}},CLASS:function(e){var t=E[e+" "];return t||(t=new RegExp("(^|"+M+")"+e+"("+M+"|$)"))&&E(e,function(e){return t.test("string"==typeof e.className&&e.className||"undefined"!=typeof e.getAttribute&&e.getAttribute("class")||"")})},ATTR:function(e,t,n){return function(r){var i=oe.attr(r,e);return null==i?"!="===t:!t||(i+="","="===t?i===n:"!="===t?i!==n:"^="===t?n&&0===i.indexOf(n):"*="===t?n&&i.indexOf(n)>-1:"$="===t?n&&i.slice(-n.length)===n:"~="===t?(" "+i.replace($," ")+" ").indexOf(n)>-1:"|="===t&&(i===n||i.slice(0,n.length+1)===n+"-"))}},CHILD:function(e,t,n,r,i){var o="nth"!==e.slice(0,3),a="last"!==e.slice(-4),s="of-type"===t;return 1===r&&0===i?function(e){return!!e.parentNode}:function(t,n,u){var l,c,f,p,d,h,g=o!==a?"nextSibling":"previousSibling",y=t.parentNode,v=s&&t.nodeName.toLowerCase(),m=!u&&!s,x=!1;if(y){if(o){while(g){p=t;while(p=p[g])if(s?p.nodeName.toLowerCase()===v:1===p.nodeType)return!1;h=g="only"===e&&!h&&"nextSibling"}return!0}if(h=[a?y.firstChild:y.lastChild],a&&m){x=(d=(l=(c=(f=(p=y)[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]||[])[0]===T&&l[1])&&l[2],p=d&&y.childNodes[d];while(p=++d&&p&&p[g]||(x=d=0)||h.pop())if(1===p.nodeType&&++x&&p===t){c[e]=[T,d,x];break}}else if(m&&(x=d=(l=(c=(f=(p=t)[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]||[])[0]===T&&l[1]),!1===x)while(p=++d&&p&&p[g]||(x=d=0)||h.pop())if((s?p.nodeName.toLowerCase()===v:1===p.nodeType)&&++x&&(m&&((c=(f=p[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]=[T,x]),p===t))break;return(x-=i)===r||x%r==0&&x/r>=0}}},PSEUDO:function(e,t){var n,i=r.pseudos[e]||r.setFilters[e.toLowerCase()]||oe.error("unsupported pseudo: "+e);return i[b]?i(t):i.length>1?(n=[e,e,"",t],r.setFilters.hasOwnProperty(e.toLowerCase())?se(function(e,n){var r,o=i(e,t),a=o.length;while(a--)e[r=O(e,o[a])]=!(n[r]=o[a])}):function(e){return i(e,0,n)}):i}},pseudos:{not:se(function(e){var t=[],n=[],r=s(e.replace(B,"$1"));return r[b]?se(function(e,t,n,i){var o,a=r(e,null,i,[]),s=e.length;while(s--)(o=a[s])&&(e[s]=!(t[s]=o))}):function(e,i,o){return t[0]=e,r(t,null,o,n),t[0]=null,!n.pop()}}),has:se(function(e){return function(t){return oe(e,t).length>0}}),contains:se(function(e){return e=e.replace(Z,ee),function(t){return(t.textContent||t.innerText||i(t)).indexOf(e)>-1}}),lang:se(function(e){return U.test(e||"")||oe.error("unsupported lang: "+e),e=e.replace(Z,ee).toLowerCase(),function(t){var n;do{if(n=g?t.lang:t.getAttribute("xml:lang")||t.getAttribute("lang"))return(n=n.toLowerCase())===e||0===n.indexOf(e+"-")}while((t=t.parentNode)&&1===t.nodeType);return!1}}),target:function(t){var n=e.location&&e.location.hash;return n&&n.slice(1)===t.id},root:function(e){return e===h},focus:function(e){return e===d.activeElement&&(!d.hasFocus||d.hasFocus())&&!!(e.type||e.href||~e.tabIndex)},enabled:de(!1),disabled:de(!0),checked:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&!!e.checked||"option"===t&&!!e.selected},selected:function(e){return e.parentNode&&e.parentNode.selectedIndex,!0===e.selected},empty:function(e){for(e=e.firstChild;e;e=e.nextSibling)if(e.nodeType<6)return!1;return!0},parent:function(e){return!r.pseudos.empty(e)},header:function(e){return Y.test(e.nodeName)},input:function(e){return G.test(e.nodeName)},button:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&"button"===e.type||"button"===t},text:function(e){var t;return"input"===e.nodeName.toLowerCase()&&"text"===e.type&&(null==(t=e.getAttribute("type"))||"text"===t.toLowerCase())},first:he(function(){return[0]}),last:he(function(e,t){return[t-1]}),eq:he(function(e,t,n){return[n<0?n+t:n]}),even:he(function(e,t){for(var n=0;n=0;)e.push(r);return e}),gt:he(function(e,t,n){for(var r=n<0?n+t:n;++r1?function(t,n,r){var i=e.length;while(i--)if(!e[i](t,n,r))return!1;return!0}:e[0]}function be(e,t,n){for(var r=0,i=t.length;r-1&&(o[l]=!(a[l]=f))}}else v=we(v===a?v.splice(h,v.length):v),i?i(null,a,v,u):L.apply(a,v)})}function Ce(e){for(var t,n,i,o=e.length,a=r.relative[e[0].type],s=a||r.relative[" "],u=a?1:0,c=me(function(e){return e===t},s,!0),f=me(function(e){return O(t,e)>-1},s,!0),p=[function(e,n,r){var i=!a&&(r||n!==l)||((t=n).nodeType?c(e,n,r):f(e,n,r));return t=null,i}];u1&&xe(p),u>1&&ve(e.slice(0,u-1).concat({value:" "===e[u-2].type?"*":""})).replace(B,"$1"),n,u0,i=e.length>0,o=function(o,a,s,u,c){var f,h,y,v=0,m="0",x=o&&[],b=[],w=l,C=o||i&&r.find.TAG("*",c),E=T+=null==w?1:Math.random()||.1,k=C.length;for(c&&(l=a===d||a||c);m!==k&&null!=(f=C[m]);m++){if(i&&f){h=0,a||f.ownerDocument===d||(p(f),s=!g);while(y=e[h++])if(y(f,a||d,s)){u.push(f);break}c&&(T=E)}n&&((f=!y&&f)&&v--,o&&x.push(f))}if(v+=m,n&&m!==v){h=0;while(y=t[h++])y(x,b,a,s);if(o){if(v>0)while(m--)x[m]||b[m]||(b[m]=j.call(u));b=we(b)}L.apply(u,b),c&&!o&&b.length>0&&v+t.length>1&&oe.uniqueSort(u)}return c&&(T=E,l=w),x};return n?se(o):o}return s=oe.compile=function(e,t){var n,r=[],i=[],o=S[e+" "];if(!o){t||(t=a(e)),n=t.length;while(n--)(o=Ce(t[n]))[b]?r.push(o):i.push(o);(o=S(e,Ee(i,r))).selector=e}return o},u=oe.select=function(e,t,n,i){var o,u,l,c,f,p="function"==typeof e&&e,d=!i&&a(e=p.selector||e);if(n=n||[],1===d.length){if((u=d[0]=d[0].slice(0)).length>2&&"ID"===(l=u[0]).type&&9===t.nodeType&&g&&r.relative[u[1].type]){if(!(t=(r.find.ID(l.matches[0].replace(Z,ee),t)||[])[0]))return n;p&&(t=t.parentNode),e=e.slice(u.shift().value.length)}o=V.needsContext.test(e)?0:u.length;while(o--){if(l=u[o],r.relative[c=l.type])break;if((f=r.find[c])&&(i=f(l.matches[0].replace(Z,ee),K.test(u[0].type)&&ge(t.parentNode)||t))){if(u.splice(o,1),!(e=i.length&&ve(u)))return L.apply(n,i),n;break}}}return(p||s(e,d))(i,t,!g,n,!t||K.test(e)&&ge(t.parentNode)||t),n},n.sortStable=b.split("").sort(D).join("")===b,n.detectDuplicates=!!f,p(),n.sortDetached=ue(function(e){return 1&e.compareDocumentPosition(d.createElement("fieldset"))}),ue(function(e){return e.innerHTML="","#"===e.firstChild.getAttribute("href")})||le("type|href|height|width",function(e,t,n){if(!n)return e.getAttribute(t,"type"===t.toLowerCase()?1:2)}),n.attributes&&ue(function(e){return e.innerHTML="",e.firstChild.setAttribute("value",""),""===e.firstChild.getAttribute("value")})||le("value",function(e,t,n){if(!n&&"input"===e.nodeName.toLowerCase())return e.defaultValue}),ue(function(e){return null==e.getAttribute("disabled")})||le(P,function(e,t,n){var r;if(!n)return!0===e[t]?t.toLowerCase():(r=e.getAttributeNode(t))&&r.specified?r.value:null}),oe}(e);w.find=E,w.expr=E.selectors,w.expr[":"]=w.expr.pseudos,w.uniqueSort=w.unique=E.uniqueSort,w.text=E.getText,w.isXMLDoc=E.isXML,w.contains=E.contains,w.escapeSelector=E.escape;var k=function(e,t,n){var r=[],i=void 0!==n;while((e=e[t])&&9!==e.nodeType)if(1===e.nodeType){if(i&&w(e).is(n))break;r.push(e)}return r},S=function(e,t){for(var n=[];e;e=e.nextSibling)1===e.nodeType&&e!==t&&n.push(e);return n},D=w.expr.match.needsContext;function N(e,t){return e.nodeName&&e.nodeName.toLowerCase()===t.toLowerCase()}var A=/^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i;function j(e,t,n){return g(t)?w.grep(e,function(e,r){return!!t.call(e,r,e)!==n}):t.nodeType?w.grep(e,function(e){return e===t!==n}):"string"!=typeof t?w.grep(e,function(e){return u.call(t,e)>-1!==n}):w.filter(t,e,n)}w.filter=function(e,t,n){var r=t[0];return n&&(e=":not("+e+")"),1===t.length&&1===r.nodeType?w.find.matchesSelector(r,e)?[r]:[]:w.find.matches(e,w.grep(t,function(e){return 1===e.nodeType}))},w.fn.extend({find:function(e){var t,n,r=this.length,i=this;if("string"!=typeof e)return this.pushStack(w(e).filter(function(){for(t=0;t1?w.uniqueSort(n):n},filter:function(e){return this.pushStack(j(this,e||[],!1))},not:function(e){return this.pushStack(j(this,e||[],!0))},is:function(e){return!!j(this,"string"==typeof e&&D.test(e)?w(e):e||[],!1).length}});var q,L=/^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/;(w.fn.init=function(e,t,n){var i,o;if(!e)return this;if(n=n||q,"string"==typeof e){if(!(i="<"===e[0]&&">"===e[e.length-1]&&e.length>=3?[null,e,null]:L.exec(e))||!i[1]&&t)return!t||t.jquery?(t||n).find(e):this.constructor(t).find(e);if(i[1]){if(t=t instanceof w?t[0]:t,w.merge(this,w.parseHTML(i[1],t&&t.nodeType?t.ownerDocument||t:r,!0)),A.test(i[1])&&w.isPlainObject(t))for(i in t)g(this[i])?this[i](t[i]):this.attr(i,t[i]);return this}return(o=r.getElementById(i[2]))&&(this[0]=o,this.length=1),this}return e.nodeType?(this[0]=e,this.length=1,this):g(e)?void 0!==n.ready?n.ready(e):e(w):w.makeArray(e,this)}).prototype=w.fn,q=w(r);var H=/^(?:parents|prev(?:Until|All))/,O={children:!0,contents:!0,next:!0,prev:!0};w.fn.extend({has:function(e){var t=w(e,this),n=t.length;return this.filter(function(){for(var e=0;e-1:1===n.nodeType&&w.find.matchesSelector(n,e))){o.push(n);break}return this.pushStack(o.length>1?w.uniqueSort(o):o)},index:function(e){return e?"string"==typeof e?u.call(w(e),this[0]):u.call(this,e.jquery?e[0]:e):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(e,t){return this.pushStack(w.uniqueSort(w.merge(this.get(),w(e,t))))},addBack:function(e){return this.add(null==e?this.prevObject:this.prevObject.filter(e))}});function P(e,t){while((e=e[t])&&1!==e.nodeType);return e}w.each({parent:function(e){var t=e.parentNode;return t&&11!==t.nodeType?t:null},parents:function(e){return k(e,"parentNode")},parentsUntil:function(e,t,n){return k(e,"parentNode",n)},next:function(e){return P(e,"nextSibling")},prev:function(e){return P(e,"previousSibling")},nextAll:function(e){return k(e,"nextSibling")},prevAll:function(e){return k(e,"previousSibling")},nextUntil:function(e,t,n){return k(e,"nextSibling",n)},prevUntil:function(e,t,n){return k(e,"previousSibling",n)},siblings:function(e){return S((e.parentNode||{}).firstChild,e)},children:function(e){return S(e.firstChild)},contents:function(e){return N(e,"iframe")?e.contentDocument:(N(e,"template")&&(e=e.content||e),w.merge([],e.childNodes))}},function(e,t){w.fn[e]=function(n,r){var i=w.map(this,t,n);return"Until"!==e.slice(-5)&&(r=n),r&&"string"==typeof r&&(i=w.filter(r,i)),this.length>1&&(O[e]||w.uniqueSort(i),H.test(e)&&i.reverse()),this.pushStack(i)}});var M=/[^\x20\t\r\n\f]+/g;function R(e){var t={};return w.each(e.match(M)||[],function(e,n){t[n]=!0}),t}w.Callbacks=function(e){e="string"==typeof e?R(e):w.extend({},e);var t,n,r,i,o=[],a=[],s=-1,u=function(){for(i=i||e.once,r=t=!0;a.length;s=-1){n=a.shift();while(++s-1)o.splice(n,1),n<=s&&s--}),this},has:function(e){return e?w.inArray(e,o)>-1:o.length>0},empty:function(){return o&&(o=[]),this},disable:function(){return i=a=[],o=n="",this},disabled:function(){return!o},lock:function(){return i=a=[],n||t||(o=n=""),this},locked:function(){return!!i},fireWith:function(e,n){return i||(n=[e,(n=n||[]).slice?n.slice():n],a.push(n),t||u()),this},fire:function(){return l.fireWith(this,arguments),this},fired:function(){return!!r}};return l};function I(e){return e}function W(e){throw e}function $(e,t,n,r){var i;try{e&&g(i=e.promise)?i.call(e).done(t).fail(n):e&&g(i=e.then)?i.call(e,t,n):t.apply(void 0,[e].slice(r))}catch(e){n.apply(void 0,[e])}}w.extend({Deferred:function(t){var n=[["notify","progress",w.Callbacks("memory"),w.Callbacks("memory"),2],["resolve","done",w.Callbacks("once memory"),w.Callbacks("once memory"),0,"resolved"],["reject","fail",w.Callbacks("once memory"),w.Callbacks("once memory"),1,"rejected"]],r="pending",i={state:function(){return r},always:function(){return o.done(arguments).fail(arguments),this},"catch":function(e){return i.then(null,e)},pipe:function(){var e=arguments;return w.Deferred(function(t){w.each(n,function(n,r){var i=g(e[r[4]])&&e[r[4]];o[r[1]](function(){var e=i&&i.apply(this,arguments);e&&g(e.promise)?e.promise().progress(t.notify).done(t.resolve).fail(t.reject):t[r[0]+"With"](this,i?[e]:arguments)})}),e=null}).promise()},then:function(t,r,i){var o=0;function a(t,n,r,i){return function(){var s=this,u=arguments,l=function(){var e,l;if(!(t=o&&(r!==W&&(s=void 0,u=[e]),n.rejectWith(s,u))}};t?c():(w.Deferred.getStackHook&&(c.stackTrace=w.Deferred.getStackHook()),e.setTimeout(c))}}return w.Deferred(function(e){n[0][3].add(a(0,e,g(i)?i:I,e.notifyWith)),n[1][3].add(a(0,e,g(t)?t:I)),n[2][3].add(a(0,e,g(r)?r:W))}).promise()},promise:function(e){return null!=e?w.extend(e,i):i}},o={};return w.each(n,function(e,t){var a=t[2],s=t[5];i[t[1]]=a.add,s&&a.add(function(){r=s},n[3-e][2].disable,n[3-e][3].disable,n[0][2].lock,n[0][3].lock),a.add(t[3].fire),o[t[0]]=function(){return o[t[0]+"With"](this===o?void 0:this,arguments),this},o[t[0]+"With"]=a.fireWith}),i.promise(o),t&&t.call(o,o),o},when:function(e){var t=arguments.length,n=t,r=Array(n),i=o.call(arguments),a=w.Deferred(),s=function(e){return function(n){r[e]=this,i[e]=arguments.length>1?o.call(arguments):n,--t||a.resolveWith(r,i)}};if(t<=1&&($(e,a.done(s(n)).resolve,a.reject,!t),"pending"===a.state()||g(i[n]&&i[n].then)))return a.then();while(n--)$(i[n],s(n),a.reject);return a.promise()}});var B=/^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/;w.Deferred.exceptionHook=function(t,n){e.console&&e.console.warn&&t&&B.test(t.name)&&e.console.warn("jQuery.Deferred exception: "+t.message,t.stack,n)},w.readyException=function(t){e.setTimeout(function(){throw t})};var F=w.Deferred();w.fn.ready=function(e){return F.then(e)["catch"](function(e){w.readyException(e)}),this},w.extend({isReady:!1,readyWait:1,ready:function(e){(!0===e?--w.readyWait:w.isReady)||(w.isReady=!0,!0!==e&&--w.readyWait>0||F.resolveWith(r,[w]))}}),w.ready.then=F.then;function _(){r.removeEventListener("DOMContentLoaded",_),e.removeEventListener("load",_),w.ready()}"complete"===r.readyState||"loading"!==r.readyState&&!r.documentElement.doScroll?e.setTimeout(w.ready):(r.addEventListener("DOMContentLoaded",_),e.addEventListener("load",_));var z=function(e,t,n,r,i,o,a){var s=0,u=e.length,l=null==n;if("object"===x(n)){i=!0;for(s in n)z(e,t,s,n[s],!0,o,a)}else if(void 0!==r&&(i=!0,g(r)||(a=!0),l&&(a?(t.call(e,r),t=null):(l=t,t=function(e,t,n){return l.call(w(e),n)})),t))for(;s1,null,!0)},removeData:function(e){return this.each(function(){K.remove(this,e)})}}),w.extend({queue:function(e,t,n){var r;if(e)return t=(t||"fx")+"queue",r=J.get(e,t),n&&(!r||Array.isArray(n)?r=J.access(e,t,w.makeArray(n)):r.push(n)),r||[]},dequeue:function(e,t){t=t||"fx";var n=w.queue(e,t),r=n.length,i=n.shift(),o=w._queueHooks(e,t),a=function(){w.dequeue(e,t)};"inprogress"===i&&(i=n.shift(),r--),i&&("fx"===t&&n.unshift("inprogress"),delete o.stop,i.call(e,a,o)),!r&&o&&o.empty.fire()},_queueHooks:function(e,t){var n=t+"queueHooks";return J.get(e,n)||J.access(e,n,{empty:w.Callbacks("once memory").add(function(){J.remove(e,[t+"queue",n])})})}}),w.fn.extend({queue:function(e,t){var n=2;return"string"!=typeof e&&(t=e,e="fx",n--),arguments.length\x20\t\r\n\f]+)/i,he=/^$|^module$|\/(?:java|ecma)script/i,ge={option:[1,""],thead:[1,"","
    "],col:[2,"","
    "],tr:[2,"","
    "],td:[3,"","
    "],_default:[0,"",""]};ge.optgroup=ge.option,ge.tbody=ge.tfoot=ge.colgroup=ge.caption=ge.thead,ge.th=ge.td;function ye(e,t){var n;return n="undefined"!=typeof e.getElementsByTagName?e.getElementsByTagName(t||"*"):"undefined"!=typeof e.querySelectorAll?e.querySelectorAll(t||"*"):[],void 0===t||t&&N(e,t)?w.merge([e],n):n}function ve(e,t){for(var n=0,r=e.length;n-1)i&&i.push(o);else if(l=w.contains(o.ownerDocument,o),a=ye(f.appendChild(o),"script"),l&&ve(a),n){c=0;while(o=a[c++])he.test(o.type||"")&&n.push(o)}return f}!function(){var e=r.createDocumentFragment().appendChild(r.createElement("div")),t=r.createElement("input");t.setAttribute("type","radio"),t.setAttribute("checked","checked"),t.setAttribute("name","t"),e.appendChild(t),h.checkClone=e.cloneNode(!0).cloneNode(!0).lastChild.checked,e.innerHTML="",h.noCloneChecked=!!e.cloneNode(!0).lastChild.defaultValue}();var be=r.documentElement,we=/^key/,Te=/^(?:mouse|pointer|contextmenu|drag|drop)|click/,Ce=/^([^.]*)(?:\.(.+)|)/;function Ee(){return!0}function ke(){return!1}function Se(){try{return r.activeElement}catch(e){}}function De(e,t,n,r,i,o){var a,s;if("object"==typeof t){"string"!=typeof n&&(r=r||n,n=void 0);for(s in t)De(e,s,n,r,t[s],o);return e}if(null==r&&null==i?(i=n,r=n=void 0):null==i&&("string"==typeof n?(i=r,r=void 0):(i=r,r=n,n=void 0)),!1===i)i=ke;else if(!i)return e;return 1===o&&(a=i,(i=function(e){return w().off(e),a.apply(this,arguments)}).guid=a.guid||(a.guid=w.guid++)),e.each(function(){w.event.add(this,t,i,r,n)})}w.event={global:{},add:function(e,t,n,r,i){var o,a,s,u,l,c,f,p,d,h,g,y=J.get(e);if(y){n.handler&&(n=(o=n).handler,i=o.selector),i&&w.find.matchesSelector(be,i),n.guid||(n.guid=w.guid++),(u=y.events)||(u=y.events={}),(a=y.handle)||(a=y.handle=function(t){return"undefined"!=typeof w&&w.event.triggered!==t.type?w.event.dispatch.apply(e,arguments):void 0}),l=(t=(t||"").match(M)||[""]).length;while(l--)d=g=(s=Ce.exec(t[l])||[])[1],h=(s[2]||"").split(".").sort(),d&&(f=w.event.special[d]||{},d=(i?f.delegateType:f.bindType)||d,f=w.event.special[d]||{},c=w.extend({type:d,origType:g,data:r,handler:n,guid:n.guid,selector:i,needsContext:i&&w.expr.match.needsContext.test(i),namespace:h.join(".")},o),(p=u[d])||((p=u[d]=[]).delegateCount=0,f.setup&&!1!==f.setup.call(e,r,h,a)||e.addEventListener&&e.addEventListener(d,a)),f.add&&(f.add.call(e,c),c.handler.guid||(c.handler.guid=n.guid)),i?p.splice(p.delegateCount++,0,c):p.push(c),w.event.global[d]=!0)}},remove:function(e,t,n,r,i){var o,a,s,u,l,c,f,p,d,h,g,y=J.hasData(e)&&J.get(e);if(y&&(u=y.events)){l=(t=(t||"").match(M)||[""]).length;while(l--)if(s=Ce.exec(t[l])||[],d=g=s[1],h=(s[2]||"").split(".").sort(),d){f=w.event.special[d]||{},p=u[d=(r?f.delegateType:f.bindType)||d]||[],s=s[2]&&new RegExp("(^|\\.)"+h.join("\\.(?:.*\\.|)")+"(\\.|$)"),a=o=p.length;while(o--)c=p[o],!i&&g!==c.origType||n&&n.guid!==c.guid||s&&!s.test(c.namespace)||r&&r!==c.selector&&("**"!==r||!c.selector)||(p.splice(o,1),c.selector&&p.delegateCount--,f.remove&&f.remove.call(e,c));a&&!p.length&&(f.teardown&&!1!==f.teardown.call(e,h,y.handle)||w.removeEvent(e,d,y.handle),delete u[d])}else for(d in u)w.event.remove(e,d+t[l],n,r,!0);w.isEmptyObject(u)&&J.remove(e,"handle events")}},dispatch:function(e){var t=w.event.fix(e),n,r,i,o,a,s,u=new Array(arguments.length),l=(J.get(this,"events")||{})[t.type]||[],c=w.event.special[t.type]||{};for(u[0]=t,n=1;n=1))for(;l!==this;l=l.parentNode||this)if(1===l.nodeType&&("click"!==e.type||!0!==l.disabled)){for(o=[],a={},n=0;n-1:w.find(i,this,null,[l]).length),a[i]&&o.push(r);o.length&&s.push({elem:l,handlers:o})}return l=this,u\x20\t\r\n\f]*)[^>]*)\/>/gi,Ae=/\s*$/g;function Le(e,t){return N(e,"table")&&N(11!==t.nodeType?t:t.firstChild,"tr")?w(e).children("tbody")[0]||e:e}function He(e){return e.type=(null!==e.getAttribute("type"))+"/"+e.type,e}function Oe(e){return"true/"===(e.type||"").slice(0,5)?e.type=e.type.slice(5):e.removeAttribute("type"),e}function Pe(e,t){var n,r,i,o,a,s,u,l;if(1===t.nodeType){if(J.hasData(e)&&(o=J.access(e),a=J.set(t,o),l=o.events)){delete a.handle,a.events={};for(i in l)for(n=0,r=l[i].length;n1&&"string"==typeof y&&!h.checkClone&&je.test(y))return e.each(function(i){var o=e.eq(i);v&&(t[0]=y.call(this,i,o.html())),Re(o,t,n,r)});if(p&&(i=xe(t,e[0].ownerDocument,!1,e,r),o=i.firstChild,1===i.childNodes.length&&(i=o),o||r)){for(u=(s=w.map(ye(i,"script"),He)).length;f")},clone:function(e,t,n){var r,i,o,a,s=e.cloneNode(!0),u=w.contains(e.ownerDocument,e);if(!(h.noCloneChecked||1!==e.nodeType&&11!==e.nodeType||w.isXMLDoc(e)))for(a=ye(s),r=0,i=(o=ye(e)).length;r0&&ve(a,!u&&ye(e,"script")),s},cleanData:function(e){for(var t,n,r,i=w.event.special,o=0;void 0!==(n=e[o]);o++)if(Y(n)){if(t=n[J.expando]){if(t.events)for(r in t.events)i[r]?w.event.remove(n,r):w.removeEvent(n,r,t.handle);n[J.expando]=void 0}n[K.expando]&&(n[K.expando]=void 0)}}}),w.fn.extend({detach:function(e){return Ie(this,e,!0)},remove:function(e){return Ie(this,e)},text:function(e){return z(this,function(e){return void 0===e?w.text(this):this.empty().each(function(){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||(this.textContent=e)})},null,e,arguments.length)},append:function(){return Re(this,arguments,function(e){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||Le(this,e).appendChild(e)})},prepend:function(){return Re(this,arguments,function(e){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var t=Le(this,e);t.insertBefore(e,t.firstChild)}})},before:function(){return Re(this,arguments,function(e){this.parentNode&&this.parentNode.insertBefore(e,this)})},after:function(){return Re(this,arguments,function(e){this.parentNode&&this.parentNode.insertBefore(e,this.nextSibling)})},empty:function(){for(var e,t=0;null!=(e=this[t]);t++)1===e.nodeType&&(w.cleanData(ye(e,!1)),e.textContent="");return this},clone:function(e,t){return e=null!=e&&e,t=null==t?e:t,this.map(function(){return w.clone(this,e,t)})},html:function(e){return z(this,function(e){var t=this[0]||{},n=0,r=this.length;if(void 0===e&&1===t.nodeType)return t.innerHTML;if("string"==typeof e&&!Ae.test(e)&&!ge[(de.exec(e)||["",""])[1].toLowerCase()]){e=w.htmlPrefilter(e);try{for(;n=0&&(u+=Math.max(0,Math.ceil(e["offset"+t[0].toUpperCase()+t.slice(1)]-o-u-s-.5))),u}function et(e,t,n){var r=$e(e),i=Fe(e,t,r),o="border-box"===w.css(e,"boxSizing",!1,r),a=o;if(We.test(i)){if(!n)return i;i="auto"}return a=a&&(h.boxSizingReliable()||i===e.style[t]),("auto"===i||!parseFloat(i)&&"inline"===w.css(e,"display",!1,r))&&(i=e["offset"+t[0].toUpperCase()+t.slice(1)],a=!0),(i=parseFloat(i)||0)+Ze(e,t,n||(o?"border":"content"),a,r,i)+"px"}w.extend({cssHooks:{opacity:{get:function(e,t){if(t){var n=Fe(e,"opacity");return""===n?"1":n}}}},cssNumber:{animationIterationCount:!0,columnCount:!0,fillOpacity:!0,flexGrow:!0,flexShrink:!0,fontWeight:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{},style:function(e,t,n,r){if(e&&3!==e.nodeType&&8!==e.nodeType&&e.style){var i,o,a,s=G(t),u=Xe.test(t),l=e.style;if(u||(t=Je(s)),a=w.cssHooks[t]||w.cssHooks[s],void 0===n)return a&&"get"in a&&void 0!==(i=a.get(e,!1,r))?i:l[t];"string"==(o=typeof n)&&(i=ie.exec(n))&&i[1]&&(n=ue(e,t,i),o="number"),null!=n&&n===n&&("number"===o&&(n+=i&&i[3]||(w.cssNumber[s]?"":"px")),h.clearCloneStyle||""!==n||0!==t.indexOf("background")||(l[t]="inherit"),a&&"set"in a&&void 0===(n=a.set(e,n,r))||(u?l.setProperty(t,n):l[t]=n))}},css:function(e,t,n,r){var i,o,a,s=G(t);return Xe.test(t)||(t=Je(s)),(a=w.cssHooks[t]||w.cssHooks[s])&&"get"in a&&(i=a.get(e,!0,n)),void 0===i&&(i=Fe(e,t,r)),"normal"===i&&t in Ve&&(i=Ve[t]),""===n||n?(o=parseFloat(i),!0===n||isFinite(o)?o||0:i):i}}),w.each(["height","width"],function(e,t){w.cssHooks[t]={get:function(e,n,r){if(n)return!ze.test(w.css(e,"display"))||e.getClientRects().length&&e.getBoundingClientRect().width?et(e,t,r):se(e,Ue,function(){return et(e,t,r)})},set:function(e,n,r){var i,o=$e(e),a="border-box"===w.css(e,"boxSizing",!1,o),s=r&&Ze(e,t,r,a,o);return a&&h.scrollboxSize()===o.position&&(s-=Math.ceil(e["offset"+t[0].toUpperCase()+t.slice(1)]-parseFloat(o[t])-Ze(e,t,"border",!1,o)-.5)),s&&(i=ie.exec(n))&&"px"!==(i[3]||"px")&&(e.style[t]=n,n=w.css(e,t)),Ke(e,n,s)}}}),w.cssHooks.marginLeft=_e(h.reliableMarginLeft,function(e,t){if(t)return(parseFloat(Fe(e,"marginLeft"))||e.getBoundingClientRect().left-se(e,{marginLeft:0},function(){return e.getBoundingClientRect().left}))+"px"}),w.each({margin:"",padding:"",border:"Width"},function(e,t){w.cssHooks[e+t]={expand:function(n){for(var r=0,i={},o="string"==typeof n?n.split(" "):[n];r<4;r++)i[e+oe[r]+t]=o[r]||o[r-2]||o[0];return i}},"margin"!==e&&(w.cssHooks[e+t].set=Ke)}),w.fn.extend({css:function(e,t){return z(this,function(e,t,n){var r,i,o={},a=0;if(Array.isArray(t)){for(r=$e(e),i=t.length;a1)}});function tt(e,t,n,r,i){return new tt.prototype.init(e,t,n,r,i)}w.Tween=tt,tt.prototype={constructor:tt,init:function(e,t,n,r,i,o){this.elem=e,this.prop=n,this.easing=i||w.easing._default,this.options=t,this.start=this.now=this.cur(),this.end=r,this.unit=o||(w.cssNumber[n]?"":"px")},cur:function(){var e=tt.propHooks[this.prop];return e&&e.get?e.get(this):tt.propHooks._default.get(this)},run:function(e){var t,n=tt.propHooks[this.prop];return this.options.duration?this.pos=t=w.easing[this.easing](e,this.options.duration*e,0,1,this.options.duration):this.pos=t=e,this.now=(this.end-this.start)*t+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),n&&n.set?n.set(this):tt.propHooks._default.set(this),this}},tt.prototype.init.prototype=tt.prototype,tt.propHooks={_default:{get:function(e){var t;return 1!==e.elem.nodeType||null!=e.elem[e.prop]&&null==e.elem.style[e.prop]?e.elem[e.prop]:(t=w.css(e.elem,e.prop,""))&&"auto"!==t?t:0},set:function(e){w.fx.step[e.prop]?w.fx.step[e.prop](e):1!==e.elem.nodeType||null==e.elem.style[w.cssProps[e.prop]]&&!w.cssHooks[e.prop]?e.elem[e.prop]=e.now:w.style(e.elem,e.prop,e.now+e.unit)}}},tt.propHooks.scrollTop=tt.propHooks.scrollLeft={set:function(e){e.elem.nodeType&&e.elem.parentNode&&(e.elem[e.prop]=e.now)}},w.easing={linear:function(e){return e},swing:function(e){return.5-Math.cos(e*Math.PI)/2},_default:"swing"},w.fx=tt.prototype.init,w.fx.step={};var nt,rt,it=/^(?:toggle|show|hide)$/,ot=/queueHooks$/;function at(){rt&&(!1===r.hidden&&e.requestAnimationFrame?e.requestAnimationFrame(at):e.setTimeout(at,w.fx.interval),w.fx.tick())}function st(){return e.setTimeout(function(){nt=void 0}),nt=Date.now()}function ut(e,t){var n,r=0,i={height:e};for(t=t?1:0;r<4;r+=2-t)i["margin"+(n=oe[r])]=i["padding"+n]=e;return t&&(i.opacity=i.width=e),i}function lt(e,t,n){for(var r,i=(pt.tweeners[t]||[]).concat(pt.tweeners["*"]),o=0,a=i.length;o1)},removeAttr:function(e){return this.each(function(){w.removeAttr(this,e)})}}),w.extend({attr:function(e,t,n){var r,i,o=e.nodeType;if(3!==o&&8!==o&&2!==o)return"undefined"==typeof e.getAttribute?w.prop(e,t,n):(1===o&&w.isXMLDoc(e)||(i=w.attrHooks[t.toLowerCase()]||(w.expr.match.bool.test(t)?dt:void 0)),void 0!==n?null===n?void w.removeAttr(e,t):i&&"set"in i&&void 0!==(r=i.set(e,n,t))?r:(e.setAttribute(t,n+""),n):i&&"get"in i&&null!==(r=i.get(e,t))?r:null==(r=w.find.attr(e,t))?void 0:r)},attrHooks:{type:{set:function(e,t){if(!h.radioValue&&"radio"===t&&N(e,"input")){var n=e.value;return e.setAttribute("type",t),n&&(e.value=n),t}}}},removeAttr:function(e,t){var n,r=0,i=t&&t.match(M);if(i&&1===e.nodeType)while(n=i[r++])e.removeAttribute(n)}}),dt={set:function(e,t,n){return!1===t?w.removeAttr(e,n):e.setAttribute(n,n),n}},w.each(w.expr.match.bool.source.match(/\w+/g),function(e,t){var n=ht[t]||w.find.attr;ht[t]=function(e,t,r){var i,o,a=t.toLowerCase();return r||(o=ht[a],ht[a]=i,i=null!=n(e,t,r)?a:null,ht[a]=o),i}});var gt=/^(?:input|select|textarea|button)$/i,yt=/^(?:a|area)$/i;w.fn.extend({prop:function(e,t){return z(this,w.prop,e,t,arguments.length>1)},removeProp:function(e){return this.each(function(){delete this[w.propFix[e]||e]})}}),w.extend({prop:function(e,t,n){var r,i,o=e.nodeType;if(3!==o&&8!==o&&2!==o)return 1===o&&w.isXMLDoc(e)||(t=w.propFix[t]||t,i=w.propHooks[t]),void 0!==n?i&&"set"in i&&void 0!==(r=i.set(e,n,t))?r:e[t]=n:i&&"get"in i&&null!==(r=i.get(e,t))?r:e[t]},propHooks:{tabIndex:{get:function(e){var t=w.find.attr(e,"tabindex");return t?parseInt(t,10):gt.test(e.nodeName)||yt.test(e.nodeName)&&e.href?0:-1}}},propFix:{"for":"htmlFor","class":"className"}}),h.optSelected||(w.propHooks.selected={get:function(e){var t=e.parentNode;return t&&t.parentNode&&t.parentNode.selectedIndex,null},set:function(e){var t=e.parentNode;t&&(t.selectedIndex,t.parentNode&&t.parentNode.selectedIndex)}}),w.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){w.propFix[this.toLowerCase()]=this});function vt(e){return(e.match(M)||[]).join(" ")}function mt(e){return e.getAttribute&&e.getAttribute("class")||""}function xt(e){return Array.isArray(e)?e:"string"==typeof e?e.match(M)||[]:[]}w.fn.extend({addClass:function(e){var t,n,r,i,o,a,s,u=0;if(g(e))return this.each(function(t){w(this).addClass(e.call(this,t,mt(this)))});if((t=xt(e)).length)while(n=this[u++])if(i=mt(n),r=1===n.nodeType&&" "+vt(i)+" "){a=0;while(o=t[a++])r.indexOf(" "+o+" ")<0&&(r+=o+" ");i!==(s=vt(r))&&n.setAttribute("class",s)}return this},removeClass:function(e){var t,n,r,i,o,a,s,u=0;if(g(e))return this.each(function(t){w(this).removeClass(e.call(this,t,mt(this)))});if(!arguments.length)return this.attr("class","");if((t=xt(e)).length)while(n=this[u++])if(i=mt(n),r=1===n.nodeType&&" "+vt(i)+" "){a=0;while(o=t[a++])while(r.indexOf(" "+o+" ")>-1)r=r.replace(" "+o+" "," ");i!==(s=vt(r))&&n.setAttribute("class",s)}return this},toggleClass:function(e,t){var n=typeof e,r="string"===n||Array.isArray(e);return"boolean"==typeof t&&r?t?this.addClass(e):this.removeClass(e):g(e)?this.each(function(n){w(this).toggleClass(e.call(this,n,mt(this),t),t)}):this.each(function(){var t,i,o,a;if(r){i=0,o=w(this),a=xt(e);while(t=a[i++])o.hasClass(t)?o.removeClass(t):o.addClass(t)}else void 0!==e&&"boolean"!==n||((t=mt(this))&&J.set(this,"__className__",t),this.setAttribute&&this.setAttribute("class",t||!1===e?"":J.get(this,"__className__")||""))})},hasClass:function(e){var t,n,r=0;t=" "+e+" ";while(n=this[r++])if(1===n.nodeType&&(" "+vt(mt(n))+" ").indexOf(t)>-1)return!0;return!1}});var bt=/\r/g;w.fn.extend({val:function(e){var t,n,r,i=this[0];{if(arguments.length)return r=g(e),this.each(function(n){var i;1===this.nodeType&&(null==(i=r?e.call(this,n,w(this).val()):e)?i="":"number"==typeof i?i+="":Array.isArray(i)&&(i=w.map(i,function(e){return null==e?"":e+""})),(t=w.valHooks[this.type]||w.valHooks[this.nodeName.toLowerCase()])&&"set"in t&&void 0!==t.set(this,i,"value")||(this.value=i))});if(i)return(t=w.valHooks[i.type]||w.valHooks[i.nodeName.toLowerCase()])&&"get"in t&&void 0!==(n=t.get(i,"value"))?n:"string"==typeof(n=i.value)?n.replace(bt,""):null==n?"":n}}}),w.extend({valHooks:{option:{get:function(e){var t=w.find.attr(e,"value");return null!=t?t:vt(w.text(e))}},select:{get:function(e){var t,n,r,i=e.options,o=e.selectedIndex,a="select-one"===e.type,s=a?null:[],u=a?o+1:i.length;for(r=o<0?u:a?o:0;r-1)&&(n=!0);return n||(e.selectedIndex=-1),o}}}}),w.each(["radio","checkbox"],function(){w.valHooks[this]={set:function(e,t){if(Array.isArray(t))return e.checked=w.inArray(w(e).val(),t)>-1}},h.checkOn||(w.valHooks[this].get=function(e){return null===e.getAttribute("value")?"on":e.value})}),h.focusin="onfocusin"in e;var wt=/^(?:focusinfocus|focusoutblur)$/,Tt=function(e){e.stopPropagation()};w.extend(w.event,{trigger:function(t,n,i,o){var a,s,u,l,c,p,d,h,v=[i||r],m=f.call(t,"type")?t.type:t,x=f.call(t,"namespace")?t.namespace.split("."):[];if(s=h=u=i=i||r,3!==i.nodeType&&8!==i.nodeType&&!wt.test(m+w.event.triggered)&&(m.indexOf(".")>-1&&(m=(x=m.split(".")).shift(),x.sort()),c=m.indexOf(":")<0&&"on"+m,t=t[w.expando]?t:new w.Event(m,"object"==typeof t&&t),t.isTrigger=o?2:3,t.namespace=x.join("."),t.rnamespace=t.namespace?new RegExp("(^|\\.)"+x.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,t.result=void 0,t.target||(t.target=i),n=null==n?[t]:w.makeArray(n,[t]),d=w.event.special[m]||{},o||!d.trigger||!1!==d.trigger.apply(i,n))){if(!o&&!d.noBubble&&!y(i)){for(l=d.delegateType||m,wt.test(l+m)||(s=s.parentNode);s;s=s.parentNode)v.push(s),u=s;u===(i.ownerDocument||r)&&v.push(u.defaultView||u.parentWindow||e)}a=0;while((s=v[a++])&&!t.isPropagationStopped())h=s,t.type=a>1?l:d.bindType||m,(p=(J.get(s,"events")||{})[t.type]&&J.get(s,"handle"))&&p.apply(s,n),(p=c&&s[c])&&p.apply&&Y(s)&&(t.result=p.apply(s,n),!1===t.result&&t.preventDefault());return t.type=m,o||t.isDefaultPrevented()||d._default&&!1!==d._default.apply(v.pop(),n)||!Y(i)||c&&g(i[m])&&!y(i)&&((u=i[c])&&(i[c]=null),w.event.triggered=m,t.isPropagationStopped()&&h.addEventListener(m,Tt),i[m](),t.isPropagationStopped()&&h.removeEventListener(m,Tt),w.event.triggered=void 0,u&&(i[c]=u)),t.result}},simulate:function(e,t,n){var r=w.extend(new w.Event,n,{type:e,isSimulated:!0});w.event.trigger(r,null,t)}}),w.fn.extend({trigger:function(e,t){return this.each(function(){w.event.trigger(e,t,this)})},triggerHandler:function(e,t){var n=this[0];if(n)return w.event.trigger(e,t,n,!0)}}),h.focusin||w.each({focus:"focusin",blur:"focusout"},function(e,t){var n=function(e){w.event.simulate(t,e.target,w.event.fix(e))};w.event.special[t]={setup:function(){var r=this.ownerDocument||this,i=J.access(r,t);i||r.addEventListener(e,n,!0),J.access(r,t,(i||0)+1)},teardown:function(){var r=this.ownerDocument||this,i=J.access(r,t)-1;i?J.access(r,t,i):(r.removeEventListener(e,n,!0),J.remove(r,t))}}});var Ct=e.location,Et=Date.now(),kt=/\?/;w.parseXML=function(t){var n;if(!t||"string"!=typeof t)return null;try{n=(new e.DOMParser).parseFromString(t,"text/xml")}catch(e){n=void 0}return n&&!n.getElementsByTagName("parsererror").length||w.error("Invalid XML: "+t),n};var St=/\[\]$/,Dt=/\r?\n/g,Nt=/^(?:submit|button|image|reset|file)$/i,At=/^(?:input|select|textarea|keygen)/i;function jt(e,t,n,r){var i;if(Array.isArray(t))w.each(t,function(t,i){n||St.test(e)?r(e,i):jt(e+"["+("object"==typeof i&&null!=i?t:"")+"]",i,n,r)});else if(n||"object"!==x(t))r(e,t);else for(i in t)jt(e+"["+i+"]",t[i],n,r)}w.param=function(e,t){var n,r=[],i=function(e,t){var n=g(t)?t():t;r[r.length]=encodeURIComponent(e)+"="+encodeURIComponent(null==n?"":n)};if(Array.isArray(e)||e.jquery&&!w.isPlainObject(e))w.each(e,function(){i(this.name,this.value)});else for(n in e)jt(n,e[n],t,i);return r.join("&")},w.fn.extend({serialize:function(){return w.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var e=w.prop(this,"elements");return e?w.makeArray(e):this}).filter(function(){var e=this.type;return this.name&&!w(this).is(":disabled")&&At.test(this.nodeName)&&!Nt.test(e)&&(this.checked||!pe.test(e))}).map(function(e,t){var n=w(this).val();return null==n?null:Array.isArray(n)?w.map(n,function(e){return{name:t.name,value:e.replace(Dt,"\r\n")}}):{name:t.name,value:n.replace(Dt,"\r\n")}}).get()}});var qt=/%20/g,Lt=/#.*$/,Ht=/([?&])_=[^&]*/,Ot=/^(.*?):[ \t]*([^\r\n]*)$/gm,Pt=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Mt=/^(?:GET|HEAD)$/,Rt=/^\/\//,It={},Wt={},$t="*/".concat("*"),Bt=r.createElement("a");Bt.href=Ct.href;function Ft(e){return function(t,n){"string"!=typeof t&&(n=t,t="*");var r,i=0,o=t.toLowerCase().match(M)||[];if(g(n))while(r=o[i++])"+"===r[0]?(r=r.slice(1)||"*",(e[r]=e[r]||[]).unshift(n)):(e[r]=e[r]||[]).push(n)}}function _t(e,t,n,r){var i={},o=e===Wt;function a(s){var u;return i[s]=!0,w.each(e[s]||[],function(e,s){var l=s(t,n,r);return"string"!=typeof l||o||i[l]?o?!(u=l):void 0:(t.dataTypes.unshift(l),a(l),!1)}),u}return a(t.dataTypes[0])||!i["*"]&&a("*")}function zt(e,t){var n,r,i=w.ajaxSettings.flatOptions||{};for(n in t)void 0!==t[n]&&((i[n]?e:r||(r={}))[n]=t[n]);return r&&w.extend(!0,e,r),e}function Xt(e,t,n){var r,i,o,a,s=e.contents,u=e.dataTypes;while("*"===u[0])u.shift(),void 0===r&&(r=e.mimeType||t.getResponseHeader("Content-Type"));if(r)for(i in s)if(s[i]&&s[i].test(r)){u.unshift(i);break}if(u[0]in n)o=u[0];else{for(i in n){if(!u[0]||e.converters[i+" "+u[0]]){o=i;break}a||(a=i)}o=o||a}if(o)return o!==u[0]&&u.unshift(o),n[o]}function Ut(e,t,n,r){var i,o,a,s,u,l={},c=e.dataTypes.slice();if(c[1])for(a in e.converters)l[a.toLowerCase()]=e.converters[a];o=c.shift();while(o)if(e.responseFields[o]&&(n[e.responseFields[o]]=t),!u&&r&&e.dataFilter&&(t=e.dataFilter(t,e.dataType)),u=o,o=c.shift())if("*"===o)o=u;else if("*"!==u&&u!==o){if(!(a=l[u+" "+o]||l["* "+o]))for(i in l)if((s=i.split(" "))[1]===o&&(a=l[u+" "+s[0]]||l["* "+s[0]])){!0===a?a=l[i]:!0!==l[i]&&(o=s[0],c.unshift(s[1]));break}if(!0!==a)if(a&&e["throws"])t=a(t);else try{t=a(t)}catch(e){return{state:"parsererror",error:a?e:"No conversion from "+u+" to "+o}}}return{state:"success",data:t}}w.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:Ct.href,type:"GET",isLocal:Pt.test(Ct.protocol),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":$t,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/\bxml\b/,html:/\bhtml/,json:/\bjson\b/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":JSON.parse,"text xml":w.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(e,t){return t?zt(zt(e,w.ajaxSettings),t):zt(w.ajaxSettings,e)},ajaxPrefilter:Ft(It),ajaxTransport:Ft(Wt),ajax:function(t,n){"object"==typeof t&&(n=t,t=void 0),n=n||{};var i,o,a,s,u,l,c,f,p,d,h=w.ajaxSetup({},n),g=h.context||h,y=h.context&&(g.nodeType||g.jquery)?w(g):w.event,v=w.Deferred(),m=w.Callbacks("once memory"),x=h.statusCode||{},b={},T={},C="canceled",E={readyState:0,getResponseHeader:function(e){var t;if(c){if(!s){s={};while(t=Ot.exec(a))s[t[1].toLowerCase()]=t[2]}t=s[e.toLowerCase()]}return null==t?null:t},getAllResponseHeaders:function(){return c?a:null},setRequestHeader:function(e,t){return null==c&&(e=T[e.toLowerCase()]=T[e.toLowerCase()]||e,b[e]=t),this},overrideMimeType:function(e){return null==c&&(h.mimeType=e),this},statusCode:function(e){var t;if(e)if(c)E.always(e[E.status]);else for(t in e)x[t]=[x[t],e[t]];return this},abort:function(e){var t=e||C;return i&&i.abort(t),k(0,t),this}};if(v.promise(E),h.url=((t||h.url||Ct.href)+"").replace(Rt,Ct.protocol+"//"),h.type=n.method||n.type||h.method||h.type,h.dataTypes=(h.dataType||"*").toLowerCase().match(M)||[""],null==h.crossDomain){l=r.createElement("a");try{l.href=h.url,l.href=l.href,h.crossDomain=Bt.protocol+"//"+Bt.host!=l.protocol+"//"+l.host}catch(e){h.crossDomain=!0}}if(h.data&&h.processData&&"string"!=typeof h.data&&(h.data=w.param(h.data,h.traditional)),_t(It,h,n,E),c)return E;(f=w.event&&h.global)&&0==w.active++&&w.event.trigger("ajaxStart"),h.type=h.type.toUpperCase(),h.hasContent=!Mt.test(h.type),o=h.url.replace(Lt,""),h.hasContent?h.data&&h.processData&&0===(h.contentType||"").indexOf("application/x-www-form-urlencoded")&&(h.data=h.data.replace(qt,"+")):(d=h.url.slice(o.length),h.data&&(h.processData||"string"==typeof h.data)&&(o+=(kt.test(o)?"&":"?")+h.data,delete h.data),!1===h.cache&&(o=o.replace(Ht,"$1"),d=(kt.test(o)?"&":"?")+"_="+Et+++d),h.url=o+d),h.ifModified&&(w.lastModified[o]&&E.setRequestHeader("If-Modified-Since",w.lastModified[o]),w.etag[o]&&E.setRequestHeader("If-None-Match",w.etag[o])),(h.data&&h.hasContent&&!1!==h.contentType||n.contentType)&&E.setRequestHeader("Content-Type",h.contentType),E.setRequestHeader("Accept",h.dataTypes[0]&&h.accepts[h.dataTypes[0]]?h.accepts[h.dataTypes[0]]+("*"!==h.dataTypes[0]?", "+$t+"; q=0.01":""):h.accepts["*"]);for(p in h.headers)E.setRequestHeader(p,h.headers[p]);if(h.beforeSend&&(!1===h.beforeSend.call(g,E,h)||c))return E.abort();if(C="abort",m.add(h.complete),E.done(h.success),E.fail(h.error),i=_t(Wt,h,n,E)){if(E.readyState=1,f&&y.trigger("ajaxSend",[E,h]),c)return E;h.async&&h.timeout>0&&(u=e.setTimeout(function(){E.abort("timeout")},h.timeout));try{c=!1,i.send(b,k)}catch(e){if(c)throw e;k(-1,e)}}else k(-1,"No Transport");function k(t,n,r,s){var l,p,d,b,T,C=n;c||(c=!0,u&&e.clearTimeout(u),i=void 0,a=s||"",E.readyState=t>0?4:0,l=t>=200&&t<300||304===t,r&&(b=Xt(h,E,r)),b=Ut(h,b,E,l),l?(h.ifModified&&((T=E.getResponseHeader("Last-Modified"))&&(w.lastModified[o]=T),(T=E.getResponseHeader("etag"))&&(w.etag[o]=T)),204===t||"HEAD"===h.type?C="nocontent":304===t?C="notmodified":(C=b.state,p=b.data,l=!(d=b.error))):(d=C,!t&&C||(C="error",t<0&&(t=0))),E.status=t,E.statusText=(n||C)+"",l?v.resolveWith(g,[p,C,E]):v.rejectWith(g,[E,C,d]),E.statusCode(x),x=void 0,f&&y.trigger(l?"ajaxSuccess":"ajaxError",[E,h,l?p:d]),m.fireWith(g,[E,C]),f&&(y.trigger("ajaxComplete",[E,h]),--w.active||w.event.trigger("ajaxStop")))}return E},getJSON:function(e,t,n){return w.get(e,t,n,"json")},getScript:function(e,t){return w.get(e,void 0,t,"script")}}),w.each(["get","post"],function(e,t){w[t]=function(e,n,r,i){return g(n)&&(i=i||r,r=n,n=void 0),w.ajax(w.extend({url:e,type:t,dataType:i,data:n,success:r},w.isPlainObject(e)&&e))}}),w._evalUrl=function(e){return w.ajax({url:e,type:"GET",dataType:"script",cache:!0,async:!1,global:!1,"throws":!0})},w.fn.extend({wrapAll:function(e){var t;return this[0]&&(g(e)&&(e=e.call(this[0])),t=w(e,this[0].ownerDocument).eq(0).clone(!0),this[0].parentNode&&t.insertBefore(this[0]),t.map(function(){var e=this;while(e.firstElementChild)e=e.firstElementChild;return e}).append(this)),this},wrapInner:function(e){return g(e)?this.each(function(t){w(this).wrapInner(e.call(this,t))}):this.each(function(){var t=w(this),n=t.contents();n.length?n.wrapAll(e):t.append(e)})},wrap:function(e){var t=g(e);return this.each(function(n){w(this).wrapAll(t?e.call(this,n):e)})},unwrap:function(e){return this.parent(e).not("body").each(function(){w(this).replaceWith(this.childNodes)}),this}}),w.expr.pseudos.hidden=function(e){return!w.expr.pseudos.visible(e)},w.expr.pseudos.visible=function(e){return!!(e.offsetWidth||e.offsetHeight||e.getClientRects().length)},w.ajaxSettings.xhr=function(){try{return new e.XMLHttpRequest}catch(e){}};var Vt={0:200,1223:204},Gt=w.ajaxSettings.xhr();h.cors=!!Gt&&"withCredentials"in Gt,h.ajax=Gt=!!Gt,w.ajaxTransport(function(t){var n,r;if(h.cors||Gt&&!t.crossDomain)return{send:function(i,o){var a,s=t.xhr();if(s.open(t.type,t.url,t.async,t.username,t.password),t.xhrFields)for(a in t.xhrFields)s[a]=t.xhrFields[a];t.mimeType&&s.overrideMimeType&&s.overrideMimeType(t.mimeType),t.crossDomain||i["X-Requested-With"]||(i["X-Requested-With"]="XMLHttpRequest");for(a in i)s.setRequestHeader(a,i[a]);n=function(e){return function(){n&&(n=r=s.onload=s.onerror=s.onabort=s.ontimeout=s.onreadystatechange=null,"abort"===e?s.abort():"error"===e?"number"!=typeof s.status?o(0,"error"):o(s.status,s.statusText):o(Vt[s.status]||s.status,s.statusText,"text"!==(s.responseType||"text")||"string"!=typeof s.responseText?{binary:s.response}:{text:s.responseText},s.getAllResponseHeaders()))}},s.onload=n(),r=s.onerror=s.ontimeout=n("error"),void 0!==s.onabort?s.onabort=r:s.onreadystatechange=function(){4===s.readyState&&e.setTimeout(function(){n&&r()})},n=n("abort");try{s.send(t.hasContent&&t.data||null)}catch(e){if(n)throw e}},abort:function(){n&&n()}}}),w.ajaxPrefilter(function(e){e.crossDomain&&(e.contents.script=!1)}),w.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/\b(?:java|ecma)script\b/},converters:{"text script":function(e){return w.globalEval(e),e}}}),w.ajaxPrefilter("script",function(e){void 0===e.cache&&(e.cache=!1),e.crossDomain&&(e.type="GET")}),w.ajaxTransport("script",function(e){if(e.crossDomain){var t,n;return{send:function(i,o){t=w(" + + + + +
    +
    +
    +
    + + Logo + ArchiveBox + +
    +
    + Documentation   |   + Source   |   + Website +
    +
    +
    +
    + + + + + + + + + + $rows +
    BookmarkedSnapshot ($num_links)FilesOriginal URL
    + + + diff --git a/archivebox-0.5.3/archivebox/themes/legacy/main_index_row.html b/archivebox-0.5.3/archivebox/themes/legacy/main_index_row.html new file mode 100644 index 0000000..9112eac --- /dev/null +++ b/archivebox-0.5.3/archivebox/themes/legacy/main_index_row.html @@ -0,0 +1,16 @@ + + $bookmarked_date + + + + $title + $tags + + + + 📄 + $num_outputs + + + $url + diff --git a/archivebox-0.5.3/archivebox/util.py b/archivebox-0.5.3/archivebox/util.py new file mode 100644 index 0000000..5530ab4 --- /dev/null +++ b/archivebox-0.5.3/archivebox/util.py @@ -0,0 +1,318 @@ +__package__ = 'archivebox' + +import re +import requests +import json as pyjson + +from typing import List, Optional, Any +from pathlib import Path +from inspect import signature +from functools import wraps +from hashlib import sha256 +from urllib.parse import urlparse, quote, unquote +from html import escape, unescape +from datetime import datetime +from dateparser import parse as dateparser +from requests.exceptions import RequestException, ReadTimeout + +from .vendor.base32_crockford import encode as base32_encode # type: ignore +from w3lib.encoding import html_body_declared_encoding, http_content_type_encoding + +try: + import chardet + detect_encoding = lambda rawdata: chardet.detect(rawdata)["encoding"] +except ImportError: + detect_encoding = lambda rawdata: "utf-8" + +### Parsing Helpers + +# All of these are (str) -> str +# shortcuts to: https://docs.python.org/3/library/urllib.parse.html#url-parsing +scheme = lambda url: urlparse(url).scheme.lower() +without_scheme = lambda url: urlparse(url)._replace(scheme='').geturl().strip('//') +without_query = lambda url: urlparse(url)._replace(query='').geturl().strip('//') +without_fragment = lambda url: urlparse(url)._replace(fragment='').geturl().strip('//') +without_path = lambda url: urlparse(url)._replace(path='', fragment='', query='').geturl().strip('//') +path = lambda url: urlparse(url).path +basename = lambda url: urlparse(url).path.rsplit('/', 1)[-1] +domain = lambda url: urlparse(url).netloc +query = lambda url: urlparse(url).query +fragment = lambda url: urlparse(url).fragment +extension = lambda url: basename(url).rsplit('.', 1)[-1].lower() if '.' in basename(url) else '' +base_url = lambda url: without_scheme(url) # uniq base url used to dedupe links + +without_www = lambda url: url.replace('://www.', '://', 1) +without_trailing_slash = lambda url: url[:-1] if url[-1] == '/' else url.replace('/?', '?') +hashurl = lambda url: base32_encode(int(sha256(base_url(url).encode('utf-8')).hexdigest(), 16))[:20] + +urlencode = lambda s: s and quote(s, encoding='utf-8', errors='replace') +urldecode = lambda s: s and unquote(s) +htmlencode = lambda s: s and escape(s, quote=True) +htmldecode = lambda s: s and unescape(s) + +short_ts = lambda ts: str(parse_date(ts).timestamp()).split('.')[0] +ts_to_date = lambda ts: ts and parse_date(ts).strftime('%Y-%m-%d %H:%M') +ts_to_iso = lambda ts: ts and parse_date(ts).isoformat() + + +URL_REGEX = re.compile( + r'http[s]?://' # start matching from allowed schemes + r'(?:[a-zA-Z]|[0-9]' # followed by allowed alphanum characters + r'|[$-_@.&+]|[!*\(\),]' # or allowed symbols + r'|(?:%[0-9a-fA-F][0-9a-fA-F]))' # or allowed unicode bytes + r'[^\]\[\(\)<>"\'\s]+', # stop parsing at these symbols + re.IGNORECASE, +) + +COLOR_REGEX = re.compile(r'\[(?P\d+)(;(?P\d+)(;(?P\d+))?)?m') + +def is_static_file(url: str): + # TODO: the proper way is with MIME type detection + ext, not only extension + from .config import STATICFILE_EXTENSIONS + return extension(url).lower() in STATICFILE_EXTENSIONS + + +def enforce_types(func): + """ + Enforce function arg and kwarg types at runtime using its python3 type hints + """ + # TODO: check return type as well + + @wraps(func) + def typechecked_function(*args, **kwargs): + sig = signature(func) + + def check_argument_type(arg_key, arg_val): + try: + annotation = sig.parameters[arg_key].annotation + except KeyError: + annotation = None + + if annotation is not None and annotation.__class__ is type: + if not isinstance(arg_val, annotation): + raise TypeError( + '{}(..., {}: {}) got unexpected {} argument {}={}'.format( + func.__name__, + arg_key, + annotation.__name__, + type(arg_val).__name__, + arg_key, + str(arg_val)[:64], + ) + ) + + # check args + for arg_val, arg_key in zip(args, sig.parameters): + check_argument_type(arg_key, arg_val) + + # check kwargs + for arg_key, arg_val in kwargs.items(): + check_argument_type(arg_key, arg_val) + + return func(*args, **kwargs) + + return typechecked_function + + +def docstring(text: Optional[str]): + """attach the given docstring to the decorated function""" + def decorator(func): + if text: + func.__doc__ = text + return func + return decorator + + +@enforce_types +def str_between(string: str, start: str, end: str=None) -> str: + """(12345, , ) -> 12345""" + + content = string.split(start, 1)[-1] + if end is not None: + content = content.rsplit(end, 1)[0] + + return content + + +@enforce_types +def parse_date(date: Any) -> Optional[datetime]: + """Parse unix timestamps, iso format, and human-readable strings""" + + if date is None: + return None + + if isinstance(date, datetime): + return date + + if isinstance(date, (float, int)): + date = str(date) + + if isinstance(date, str): + return dateparser(date) + + raise ValueError('Tried to parse invalid date! {}'.format(date)) + + +@enforce_types +def download_url(url: str, timeout: int=None) -> str: + """Download the contents of a remote url and return the text""" + from .config import TIMEOUT, CHECK_SSL_VALIDITY, WGET_USER_AGENT + timeout = timeout or TIMEOUT + response = requests.get( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + ) + + content_type = response.headers.get('Content-Type', '') + encoding = http_content_type_encoding(content_type) or html_body_declared_encoding(response.text) + + if encoding is not None: + response.encoding = encoding + + return response.text + +@enforce_types +def get_headers(url: str, timeout: int=None) -> str: + """Download the contents of a remote url and return the headers""" + from .config import TIMEOUT, CHECK_SSL_VALIDITY, WGET_USER_AGENT + timeout = timeout or TIMEOUT + + try: + response = requests.head( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + allow_redirects=True, + ) + if response.status_code >= 400: + raise RequestException + except ReadTimeout: + raise + except RequestException: + response = requests.get( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + stream=True + ) + + return pyjson.dumps(dict(response.headers), indent=4) + + +@enforce_types +def chrome_args(**options) -> List[str]: + """helper to build up a chrome shell command with arguments""" + + from .config import CHROME_OPTIONS + + options = {**CHROME_OPTIONS, **options} + + cmd_args = [options['CHROME_BINARY']] + + if options['CHROME_HEADLESS']: + cmd_args += ('--headless',) + + if not options['CHROME_SANDBOX']: + # assume this means we are running inside a docker container + # in docker, GPU support is limited, sandboxing is unecessary, + # and SHM is limited to 64MB by default (which is too low to be usable). + cmd_args += ( + '--no-sandbox', + '--disable-gpu', + '--disable-dev-shm-usage', + '--disable-software-rasterizer', + ) + + + if not options['CHECK_SSL_VALIDITY']: + cmd_args += ('--disable-web-security', '--ignore-certificate-errors') + + if options['CHROME_USER_AGENT']: + cmd_args += ('--user-agent={}'.format(options['CHROME_USER_AGENT']),) + + if options['RESOLUTION']: + cmd_args += ('--window-size={}'.format(options['RESOLUTION']),) + + if options['TIMEOUT']: + cmd_args += ('--timeout={}'.format((options['TIMEOUT']) * 1000),) + + if options['CHROME_USER_DATA_DIR']: + cmd_args.append('--user-data-dir={}'.format(options['CHROME_USER_DATA_DIR'])) + + return cmd_args + + +def ansi_to_html(text): + """ + Based on: https://stackoverflow.com/questions/19212665/python-converting-ansi-color-codes-to-html + """ + from .config import COLOR_DICT + + TEMPLATE = '
    ' + text = text.replace('[m', '    ') + + def single_sub(match): + argsdict = match.groupdict() + if argsdict['arg_3'] is None: + if argsdict['arg_2'] is None: + _, color = 0, argsdict['arg_1'] + else: + _, color = argsdict['arg_1'], argsdict['arg_2'] + else: + _, color = argsdict['arg_3'], argsdict['arg_2'] + + return TEMPLATE.format(COLOR_DICT[color][0]) + + return COLOR_REGEX.sub(single_sub, text) + + +class AttributeDict(dict): + """Helper to allow accessing dict values via Example.key or Example['key']""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # Recursively convert nested dicts to AttributeDicts (optional): + # for key, val in self.items(): + # if isinstance(val, dict) and type(val) is not AttributeDict: + # self[key] = AttributeDict(val) + + def __getattr__(self, attr: str) -> Any: + return dict.__getitem__(self, attr) + + def __setattr__(self, attr: str, value: Any) -> None: + return dict.__setitem__(self, attr, value) + + +class ExtendedEncoder(pyjson.JSONEncoder): + """ + Extended json serializer that supports serializing several model + fields and objects + """ + + def default(self, obj): + cls_name = obj.__class__.__name__ + + if hasattr(obj, '_asdict'): + return obj._asdict() + + elif isinstance(obj, bytes): + return obj.decode() + + elif isinstance(obj, datetime): + return obj.isoformat() + + elif isinstance(obj, Exception): + return '{}: {}'.format(obj.__class__.__name__, obj) + + elif isinstance(obj, Path): + return str(obj) + + elif cls_name in ('dict_items', 'dict_keys', 'dict_values'): + return tuple(obj) + + return pyjson.JSONEncoder.default(self, obj) + diff --git a/archivebox-0.5.3/archivebox/vendor/__init__.py b/archivebox-0.5.3/archivebox/vendor/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/build/lib/archivebox/.flake8 b/archivebox-0.5.3/build/lib/archivebox/.flake8 new file mode 100644 index 0000000..dd6ba8e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/.flake8 @@ -0,0 +1,6 @@ +[flake8] +ignore = D100,D101,D102,D103,D104,D105,D202,D203,D205,D400,E131,E241,E252,E266,E272,E701,E731,W293,W503,W291,W391 +select = F,E9,W +max-line-length = 130 +max-complexity = 10 +exclude = migrations,tests,node_modules,vendor,static,venv,.venv,.venv2,.docker-venv diff --git a/archivebox-0.5.3/build/lib/archivebox/LICENSE b/archivebox-0.5.3/build/lib/archivebox/LICENSE new file mode 100644 index 0000000..ea201f9 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2020 Nick Sweeting + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/archivebox-0.5.3/build/lib/archivebox/README.md b/archivebox-0.5.3/build/lib/archivebox/README.md new file mode 100644 index 0000000..2e35783 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/README.md @@ -0,0 +1,545 @@ +
    + +

    ArchiveBox
    The open-source self-hosted web archive.

    + +▶️ Quickstart | +Demo | +Github | +Documentation | +Info & Motivation | +Community | +Roadmap + +
    +"Your own personal internet archive" (网站存档 / 爬虫)
+
    + + + + + + + + + + +
    +
    + +ArchiveBox is a powerful self-hosted internet archiving solution written in Python 3. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on the configuration and the content it detects. + +Your archive can be managed through the command line with commands like `archivebox add`, through the built-in Web UI `archivebox server`, or via the Python library API (beta). It can ingest bookmarks from a browser or service like Pocket/Pinboard, your entire browsing history, RSS feeds, or URLs one at a time. You can also schedule regular/realtime imports with `archivebox schedule`. + +The main index is a self-contained `index.sqlite3` file, and each snapshot is stored as a folder `data/archive//`, with an easy-to-read `index.html` and `index.json` within. For each page, ArchiveBox auto-extracts many types of assets/media and saves them in standard formats, with out-of-the-box support for: several types of HTML snapshots (wget, Chrome headless, singlefile), PDF snapshotting, screenshotting, WARC archiving, git repositories, images, audio, video, subtitles, article text, and more. The snapshots are browseable and managable offline through the filesystem, the built-in webserver, or the Python library API. + +### Quickstart + +It works on Linux/BSD (Intel and ARM CPUs with `docker`/`apt`/`pip3`), macOS (with `docker`/`brew`/`pip3`), and Windows (beta with `docker`/`pip3`). + +```bash +pip3 install archivebox +archivebox --version +# install extras as-needed, or use one of full setup methods below to get everything out-of-the-box + +mkdir ~/archivebox && cd ~/archivebox # this can be anywhere +archivebox init + +archivebox add 'https://example.com' +archivebox add --depth=1 'https://example.com' +archivebox schedule --every=day https://getpocket.com/users/USERNAME/feed/all +archivebox oneshot --extract=title,favicon,media https://www.youtube.com/watch?v=dQw4w9WgXcQ +archivebox help # to see more options +``` + +*(click to expand the sections below for full setup instructions)* + +
    +Get ArchiveBox with docker-compose on any platform (recommended, everything included out-of-the-box) + +First make sure you have Docker installed: https://docs.docker.com/get-docker/ +

    +This is the recommended way to run ArchiveBox because it includes *all* the extractors like chrome, wget, youtube-dl, git, etc., as well as full-text search with sonic, and many other great features. + +```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml +docker-compose run archivebox init +docker-compose run archivebox --version + +# start the webserver and open the UI (optional) +docker-compose run archivebox manage createsuperuser +docker-compose up -d +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker-compose run archivebox add 'https://example.com' +docker-compose run archivebox status +docker-compose run archivebox help # to see more options +``` + +
    + +
    +Get ArchiveBox with docker on any platform + +First make sure you have Docker installed: https://docs.docker.com/get-docker/
    +```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +docker run -v $PWD:/data -it archivebox/archivebox init +docker run -v $PWD:/data -it archivebox/archivebox --version + +# start the webserver and open the UI (optional) +docker run -v $PWD:/data -it archivebox/archivebox manage createsuperuser +docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com' +docker run -v $PWD:/data -it archivebox/archivebox status +docker run -v $PWD:/data -it archivebox/archivebox help # to see more options +``` + +
    + +
    +Get ArchiveBox with apt on Ubuntu >=20.04 + +```bash +sudo add-apt-repository -u ppa:archivebox/archivebox +sudo apt install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`: +```bash +deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +``` +(you may need to install some other dependencies manually however) + +
    + +
    +Get ArchiveBox with brew on macOS >=10.13 + +```bash +brew install archivebox/archivebox/archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
    + +
    +Get ArchiveBox with pip on any platform + +```bash +pip3 install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version +# Install any missing extras like wget/git/chrome/etc. manually as needed + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
    + +--- + +
    + +
    + +DEMO: archivebox.zervice.io/ +For more information, see the full Quickstart guide, Usage, and Configuration docs. +
    + +--- + + +# Overview + +ArchiveBox is a command line tool, self-hostable web-archiving server, and Python library all-in-one. It can be installed on Docker, macOS, and Linux/BSD, and Windows. You can download and install it as a Debian/Ubuntu package, Homebrew package, Python3 package, or a Docker image. No matter which install method you choose, they all provide the same CLI, Web UI, and on-disk data format. + +To use ArchiveBox you start by creating a folder for your data to live in (it can be anywhere on your system), and running `archivebox init` inside of it. That will create a sqlite3 index and an `ArchiveBox.conf` file. After that, you can continue to add/export/manage/etc using the CLI `archivebox help`, or you can run the Web UI (recommended). If you only want to archive a single site, you can run `archivebox oneshot` to avoid having to create a whole collection. + +The CLI is considered "stable", the ArchiveBox Python API and REST APIs are "beta", and the [desktop app](https://github.com/ArchiveBox/desktop) is "alpha". + +At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer). You can also self-host your archivebox server on a public domain to provide archive.org-style public access to your site snapshots. + +
    +CLI Screenshot +Desktop index screenshot +Desktop details page Screenshot +Desktop details page Screenshot
    +Demo | Usage | Screenshots +
    +. . . . . . . . . . . . . . . . . . . . . . . . . . . . +

    + + +## Key Features + +- [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally +- [**Few dependencies**](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) +- [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) +- Easy to set up **[scheduled importing](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources** +- Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC +- ~~**Suitable for paywalled / [authenticated content](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes) +- **Doesn't require a constantly-running daemon**, proxy, or native app +- Provides a CLI, Python API, self-hosted web UI, and REST API (WIP) +- Architected to be able to run [**many varieties of scripts during archiving**](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. to extract media, summarize articles, [scroll pages](https://github.com/ArchiveBox/ArchiveBox/issues/80), [close modals](https://github.com/ArchiveBox/ArchiveBox/issues/175), expand comment threads, etc. +- Can also [**mirror content to 3rd-party archiving services**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy + +## Input formats + +ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more! + +```bash +echo 'http://example.com' | archivebox add +archivebox add 'https://example.com/some/page' +archivebox add < ~/Downloads/firefox_bookmarks_export.html +archivebox add < any_text_with_urls_in_it.txt +archivebox add --depth=1 'https://example.com/some/downloads.html' +archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12' +``` + +- Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more) +- RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format +- Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more + +See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples. + +It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand. + +## Output formats + +All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`. + +The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.) + +```bash + ls ./archive// +``` + +- **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details +- **Title:** `title` title of the site +- **Favicon:** `favicon.ico` favicon of the site +- **Headers:** `headers.json` Any HTTP headers the site returns are saved in a json file +- **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile +- **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present +- **WARC:** `warc/.gz` gzipped WARC of all the resources fetched while archiving +- **PDF:** `output.pdf` Printed PDF of site using headless chrome +- **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome +- **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome +- **Readability:** `article.html/json` Article text extraction using Readability +- **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org +- **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl +- **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links +- _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._ + +It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file. + +## Dependencies + +You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled. + +If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) or the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install). + +ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more. + +## Caveats + +If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process. +```bash +# don't do this: +archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere' +archivebox add 'https://example.com/any/url/you/want/to/keep/secret/' + +# without first disabling share the URL with 3rd party APIs: +archivebox config --set SAVE_ARCHIVE_DOT_ORG=False # disable saving all URLs in Archive.org +archivebox config --set SAVE_FAVICON=False # optional: only the domain is leaked, not full URL +archivebox config --set CHROME_BINARY=chromium # optional: switch to chromium to avoid Chrome phoning home to Google +``` + +Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details. +```bash +# visiting an archived page with malicious JS: +https://127.0.0.1:8000/archive/1602401954/example.com/index.html + +# example.com/index.js can now make a request to read everything: +https://127.0.0.1:8000/index.html +https://127.0.0.1:8000/archive/* +# then example.com/index.js can send it off to some evil server +``` + +Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash: +```bash +archivebox add 'https://example.com#2020-10-24' +... +archivebox add 'https://example.com#2020-10-25' +``` + +--- + +
    + +
    + +--- + +# Background & Motivation + +Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity. + +Whether it's to resist censorship by saving articles before they get taken down or edited, or +just to save a collection of early 2010's flash games you love to play, having the tools to +archive internet content enables to you save the stuff you care most about before it disappears. + +
    +
    + Image from WTF is Link Rot?...
     +
    + +The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. +I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about. + +Because modern websites are complicated and often rely on dynamic content, +ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats. + +All the archived links are stored by date bookmarked in `./archive/`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM. + +## Comparison to Other Projects + +▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.** + +comparison The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations. + +#### User Interface & Intended Purpose + +ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend. + +#### Private Local Archives vs Centralized Public Archives + +Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle. + +#### Storage Requirements + +Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files. + +## Learn more + +Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web! + + + +- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists) + _Community-maintained indexes of archiving tools and institutions._ + - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) + _Open source tools and projects in the internet archiving space._ + - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List) + _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._ + - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities) + _A collection of the most active internet archiving communities and initiatives._ +- Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post. +- Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter. + +--- + +# Documentation + + + +We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation. + +You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder. + +## Getting Started + +- [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) +- [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) +- [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + +## Reference + +- [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) +- [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) +- [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) +- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site) +- [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) +- [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) +- [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) +- [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) +- [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) +- [Python API](https://docs.archivebox.io/en/latest/modules.html) +- REST API (coming soon...) + +## More Info + +- [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues) +- [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) +- [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) +- [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation) +- [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + +--- + +# ArchiveBox Development + +All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap. + +### Setup the dev environment + +First, install the system dependencies from the "Bare Metal" section above. +Then you can clone the ArchiveBox repo and install +```python3 +git clone https://github.com/ArchiveBox/ArchiveBox && cd ArchiveBox +git checkout master # or the branch you want to test +git submodule update --init --recursive +git pull --recurse-submodules + +# Install ArchiveBox + python dependencies +python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev] +# or with pipenv: pipenv install --dev && pipenv shell + +# Install node dependencies +npm install + +# Optional: install extractor dependencies manually or with helper script +./bin/setup.sh + +# Optional: develop via docker by mounting the code dir into the container +# if you edit e.g. ./archivebox/core/models.py on the docker host, runserver +# inside the container will reload and pick up your changes +docker build . -t archivebox +docker run -it -p 8000:8000 \ + -v $PWD/data:/data \ + -v $PWD/archivebox:/app/archivebox \ + archivebox server 0.0.0.0:8000 --debug --reload +``` + +### Common development tasks + +See the `./bin/` folder and read the source of the bash scripts within. +You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`. + +#### Run the linters + +```bash +./bin/lint.sh +``` +(uses `flake8` and `mypy`) + +#### Run the integration tests + +```bash +./bin/test.sh +``` +(uses `pytest -s`) + +#### Make migrations or enter a django shell + +```bash +cd archivebox/ +./manage.py makemigrations + +cd data/ +archivebox shell +``` +(uses `pytest -s`) + +#### Build the docs, pip package, and docker image + +```bash +./bin/build.sh + +# or individually: +./bin/build_docs.sh +./bin/build_pip.sh +./bin/build_deb.sh +./bin/build_brew.sh +./bin/build_docker.sh +``` + +#### Roll a release + +```bash +./bin/release.sh +``` +(bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages) + + +--- + +
    +

    + +
    +This project is maintained mostly in my spare time with the help from generous contributors and Monadical.com. +

    + +
    +Sponsor us on Github +
    +
    + +
    + + + + +

    + +
    diff --git a/archivebox-0.5.3/build/lib/archivebox/__init__.py b/archivebox-0.5.3/build/lib/archivebox/__init__.py new file mode 100644 index 0000000..b0c00b6 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/__init__.py @@ -0,0 +1 @@ +__package__ = 'archivebox' diff --git a/archivebox-0.5.3/build/lib/archivebox/__main__.py b/archivebox-0.5.3/build/lib/archivebox/__main__.py new file mode 100644 index 0000000..8afaa27 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/__main__.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox' + +import sys + +from .cli import main + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/__init__.py b/archivebox-0.5.3/build/lib/archivebox/cli/__init__.py new file mode 100644 index 0000000..f9a55ef --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/__init__.py @@ -0,0 +1,144 @@ +__package__ = 'archivebox.cli' +__command__ = 'archivebox' + +import os +import sys +import argparse + +from typing import Optional, Dict, List, IO, Union +from pathlib import Path + +from ..config import OUTPUT_DIR + +from importlib import import_module + +CLI_DIR = Path(__file__).resolve().parent + +# these common commands will appear sorted before any others for ease-of-use +meta_cmds = ('help', 'version') +main_cmds = ('init', 'info', 'config') +archive_cmds = ('add', 'remove', 'update', 'list', 'status') + +fake_db = ("oneshot",) + +display_first = (*meta_cmds, *main_cmds, *archive_cmds) + +# every imported command module must have these properties in order to be valid +required_attrs = ('__package__', '__command__', 'main') + +# basic checks to make sure imported files are valid subcommands +is_cli_module = lambda fname: fname.startswith('archivebox_') and fname.endswith('.py') +is_valid_cli_module = lambda module, subcommand: ( + all(hasattr(module, attr) for attr in required_attrs) + and module.__command__.split(' ')[-1] == subcommand +) + + +def list_subcommands() -> Dict[str, str]: + """find and import all valid archivebox_.py files in CLI_DIR""" + + COMMANDS = [] + for filename in os.listdir(CLI_DIR): + if is_cli_module(filename): + subcommand = filename.replace('archivebox_', '').replace('.py', '') + module = import_module('.archivebox_{}'.format(subcommand), __package__) + assert is_valid_cli_module(module, subcommand) + COMMANDS.append((subcommand, module.main.__doc__)) + globals()[subcommand] = module.main + + display_order = lambda cmd: ( + display_first.index(cmd[0]) + if cmd[0] in display_first else + 100 + len(cmd[0]) + ) + + return dict(sorted(COMMANDS, key=display_order)) + + +def run_subcommand(subcommand: str, + subcommand_args: List[str]=None, + stdin: Optional[IO]=None, + pwd: Union[Path, str, None]=None) -> None: + """Run a given ArchiveBox subcommand with the given list of args""" + + if subcommand not in meta_cmds: + from ..config import setup_django + setup_django(in_memory_db=subcommand in fake_db, check_db=subcommand in archive_cmds) + + module = import_module('.archivebox_{}'.format(subcommand), __package__) + module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore + + +SUBCOMMANDS = list_subcommands() + +class NotProvided: + pass + + +def main(args: Optional[List[str]]=NotProvided, stdin: Optional[IO]=NotProvided, pwd: Optional[str]=None) -> None: + args = sys.argv[1:] if args is NotProvided else args + stdin = sys.stdin if stdin is NotProvided else stdin + + subcommands = list_subcommands() + parser = argparse.ArgumentParser( + prog=__command__, + description='ArchiveBox: The self-hosted internet archive', + add_help=False, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--help', '-h', + action='store_true', + help=subcommands['help'], + ) + group.add_argument( + '--version', + action='store_true', + help=subcommands['version'], + ) + group.add_argument( + "subcommand", + type=str, + help= "The name of the subcommand to run", + nargs='?', + choices=subcommands.keys(), + default=None, + ) + parser.add_argument( + "subcommand_args", + help="Arguments for the subcommand", + nargs=argparse.REMAINDER, + ) + command = parser.parse_args(args or ()) + + if command.version: + command.subcommand = 'version' + elif command.help or command.subcommand is None: + command.subcommand = 'help' + + if command.subcommand not in ('help', 'version', 'status'): + from ..logging_util import log_cli_command + + log_cli_command( + subcommand=command.subcommand, + subcommand_args=command.subcommand_args, + stdin=stdin, + pwd=pwd or OUTPUT_DIR + ) + + run_subcommand( + subcommand=command.subcommand, + subcommand_args=command.subcommand_args, + stdin=stdin, + pwd=pwd or OUTPUT_DIR, + ) + + +__all__ = ( + 'SUBCOMMANDS', + 'list_subcommands', + 'run_subcommand', + *SUBCOMMANDS.keys(), +) + + diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_add.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_add.py new file mode 100644 index 0000000..41c7554 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_add.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox add' + +import sys +import argparse + +from typing import List, Optional, IO + +from ..main import add +from ..util import docstring +from ..config import OUTPUT_DIR, ONLY_NEW +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(add.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=add.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--update-all', #'-n', + action='store_true', + default=not ONLY_NEW, # when ONLY_NEW=True we skip updating old links + help="Also retry previously skipped/failed links when adding new links", + ) + parser.add_argument( + '--index-only', #'-o', + action='store_true', + help="Add the links to the main index without archiving them", + ) + parser.add_argument( + 'urls', + nargs='*', + type=str, + default=None, + help=( + 'URLs or paths to archive e.g.:\n' + ' https://getpocket.com/users/USERNAME/feed/all\n' + ' https://example.com/some/rss/feed.xml\n' + ' https://example.com\n' + ' ~/Downloads/firefox_bookmarks_export.html\n' + ' ~/Desktop/sites_list.csv\n' + ) + ) + parser.add_argument( + "--depth", + action="store", + default=0, + choices=[0, 1], + type=int, + help="Recursively archive all linked pages up to this many hops away" + ) + parser.add_argument( + "--overwrite", + default=False, + action="store_true", + help="Re-archive URLs from scratch, overwriting any existing files" + ) + parser.add_argument( + "--init", #'-i', + action='store_true', + help="Init/upgrade the curent data directory before adding", + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + command = parser.parse_args(args or ()) + urls = command.urls + stdin_urls = accept_stdin(stdin) + if (stdin_urls and urls) or (not stdin and not urls): + stderr( + '[X] You must pass URLs/paths to add via stdin or CLI arguments.\n', + color='red', + ) + raise SystemExit(2) + add( + urls=stdin_urls or urls, + depth=command.depth, + update_all=command.update_all, + index_only=command.index_only, + overwrite=command.overwrite, + init=command.init, + extractors=command.extract, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) + + +# TODO: Implement these +# +# parser.add_argument( +# '--mirror', #'-m', +# action='store_true', +# help='Archive an entire site (finding all linked pages below it on the same domain)', +# ) +# parser.add_argument( +# '--crawler', #'-r', +# choices=('depth_first', 'breadth_first'), +# help='Controls which crawler to use in order to find outlinks in a given page', +# default=None, +# ) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_config.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_config.py new file mode 100644 index 0000000..f81286c --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_config.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox config' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import config +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(config.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=config.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--get', #'-g', + action='store_true', + help="Get the value for the given config KEYs", + ) + group.add_argument( + '--set', #'-s', + action='store_true', + help="Set the given KEY=VALUE config values", + ) + group.add_argument( + '--reset', #'-s', + action='store_true', + help="Reset the given KEY config values to their defaults", + ) + parser.add_argument( + 'config_options', + nargs='*', + type=str, + help='KEY or KEY=VALUE formatted config values to get or set', + ) + command = parser.parse_args(args or ()) + config_options_str = accept_stdin(stdin) + + config( + config_options_str=config_options_str, + config_options=command.config_options, + get=command.get, + set=command.set, + reset=command.reset, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_help.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_help.py new file mode 100755 index 0000000..46f17cb --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_help.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox help' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import help +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(help.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=help.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + help(out_dir=pwd or OUTPUT_DIR) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_init.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_init.py new file mode 100755 index 0000000..6255ef2 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_init.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox init' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import init +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(init.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=init.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--force', # '-f', + action='store_true', + help='Ignore unrecognized files in current directory and initialize anyway', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + init( + force=command.force, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_list.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_list.py new file mode 100644 index 0000000..3838cf6 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_list.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox list' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import list_all +from ..util import docstring +from ..config import OUTPUT_DIR +from ..index import ( + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, +) +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(list_all.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=list_all.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--csv', #'-c', + type=str, + help="Print the output in CSV format with the given columns, e.g.: timestamp,url,extension", + default=None, + ) + group.add_argument( + '--json', #'-j', + action='store_true', + help="Print the output in JSON format with all columns included.", + ) + group.add_argument( + '--html', + action='store_true', + help="Print the output in HTML format" + ) + parser.add_argument( + '--with-headers', + action='store_true', + help='Include the headers in the output document' + ) + parser.add_argument( + '--sort', #'-s', + type=str, + help="List the links sorted using the given key, e.g. timestamp or updated.", + default=None, + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="List only links bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="List only links bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--status', + type=str, + choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'), + default='indexed', + help=( + 'List only links or data directories that have the given status\n' + f' indexed {get_indexed_folders.__doc__} (the default)\n' + f' archived {get_archived_folders.__doc__}\n' + f' unarchived {get_unarchived_folders.__doc__}\n' + '\n' + f' present {get_present_folders.__doc__}\n' + f' valid {get_valid_folders.__doc__}\n' + f' invalid {get_invalid_folders.__doc__}\n' + '\n' + f' duplicate {get_duplicate_folders.__doc__}\n' + f' orphaned {get_orphaned_folders.__doc__}\n' + f' corrupted {get_corrupted_folders.__doc__}\n' + f' unrecognized {get_unrecognized_folders.__doc__}\n' + ) + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + default=None, + help='List only URLs matching these filter patterns.' + ) + command = parser.parse_args(args or ()) + filter_patterns_str = accept_stdin(stdin) + + if command.with_headers and not (command.json or command.html or command.csv): + stderr( + '[X] --with-headers can only be used with --json, --html or --csv options.\n', + color='red', + ) + raise SystemExit(2) + + matching_folders = list_all( + filter_patterns_str=filter_patterns_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + status=command.status, + after=command.after, + before=command.before, + sort=command.sort, + csv=command.csv, + json=command.json, + html=command.html, + with_headers=command.with_headers, + out_dir=pwd or OUTPUT_DIR, + ) + raise SystemExit(not matching_folders) + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_manage.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_manage.py new file mode 100644 index 0000000..f05604e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_manage.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox manage' + +import sys + +from typing import Optional, List, IO + +from ..main import manage +from ..util import docstring +from ..config import OUTPUT_DIR + + +@docstring(manage.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + manage( + args=args, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_oneshot.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_oneshot.py new file mode 100644 index 0000000..af68bac --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_oneshot.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox oneshot' + +import sys +import argparse + +from pathlib import Path +from typing import List, Optional, IO + +from ..main import oneshot +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(oneshot.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=oneshot.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + 'url', + type=str, + default=None, + help=( + 'URLs or paths to archive e.g.:\n' + ' https://getpocket.com/users/USERNAME/feed/all\n' + ' https://example.com/some/rss/feed.xml\n' + ' https://example.com\n' + ' ~/Downloads/firefox_bookmarks_export.html\n' + ' ~/Desktop/sites_list.csv\n' + ) + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + parser.add_argument( + '--out-dir', + type=str, + default=OUTPUT_DIR, + help= "Path to save the single archive folder to, e.g. ./example.com_archive" + ) + command = parser.parse_args(args or ()) + url = command.url + stdin_url = accept_stdin(stdin) + if (stdin_url and url) or (not stdin and not url): + stderr( + '[X] You must pass a URL/path to add via stdin or CLI arguments.\n', + color='red', + ) + raise SystemExit(2) + + oneshot( + url=stdin_url or url, + out_dir=Path(command.out_dir).resolve(), + extractors=command.extract, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_remove.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_remove.py new file mode 100644 index 0000000..cb073e9 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_remove.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox remove' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import remove +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(remove.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=remove.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--yes', # '-y', + action='store_true', + help='Remove links instantly without prompting to confirm.', + ) + parser.add_argument( + '--delete', # '-r', + action='store_true', + help=( + "In addition to removing the link from the index, " + "also delete its archived content and metadata folder." + ), + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="List only URLs bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="List only URLs bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex','tag'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + help='URLs matching this filter pattern will be removed from the index.' + ) + command = parser.parse_args(args or ()) + filter_str = accept_stdin(stdin) + + remove( + filter_str=filter_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + before=command.before, + after=command.after, + yes=command.yes, + delete=command.delete, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_schedule.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_schedule.py new file mode 100644 index 0000000..ec5e914 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_schedule.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox schedule' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import schedule +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(schedule.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=schedule.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--quiet', '-q', + action='store_true', + help=("Don't warn about storage space."), + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--add', # '-a', + action='store_true', + help='Add a new scheduled ArchiveBox update job to cron', + ) + parser.add_argument( + '--every', # '-e', + type=str, + default=None, + help='Run ArchiveBox once every [timeperiod] (hour/day/month/year or cron format e.g. "0 0 * * *")', + ) + parser.add_argument( + '--depth', # '-d', + type=int, + default=0, + help='Depth to archive to [0] or 1, see "add" command help for more info.', + ) + group.add_argument( + '--clear', # '-c' + action='store_true', + help=("Stop all ArchiveBox scheduled runs (remove cron jobs)"), + ) + group.add_argument( + '--show', # '-s' + action='store_true', + help=("Print a list of currently active ArchiveBox cron jobs"), + ) + group.add_argument( + '--foreground', '-f', + action='store_true', + help=("Launch ArchiveBox scheduler as a long-running foreground task " + "instead of using cron."), + ) + group.add_argument( + '--run-all', # '-a', + action='store_true', + help=("Run all the scheduled jobs once immediately, independent of " + "their configured schedules, can be used together with --foreground"), + ) + parser.add_argument( + 'import_path', + nargs='?', + type=str, + default=None, + help=("Check this path and import any new links on every run " + "(can be either local file or remote URL)"), + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + schedule( + add=command.add, + show=command.show, + clear=command.clear, + foreground=command.foreground, + run_all=command.run_all, + quiet=command.quiet, + every=command.every, + depth=command.depth, + import_path=command.import_path, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_server.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_server.py new file mode 100644 index 0000000..dbacf7e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_server.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox server' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import server +from ..util import docstring +from ..config import OUTPUT_DIR, BIND_ADDR +from ..logging_util import SmartFormatter, reject_stdin + +@docstring(server.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=server.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + 'runserver_args', + nargs='*', + type=str, + default=[BIND_ADDR], + help='Arguments to pass to Django runserver' + ) + parser.add_argument( + '--reload', + action='store_true', + help='Enable auto-reloading when code or templates change', + ) + parser.add_argument( + '--debug', + action='store_true', + help='Enable DEBUG=True mode with more verbose errors', + ) + parser.add_argument( + '--init', + action='store_true', + help='Run archivebox init before starting the server', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + server( + runserver_args=command.runserver_args, + reload=command.reload, + debug=command.debug, + init=command.init, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_shell.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_shell.py new file mode 100644 index 0000000..bcd5fdd --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_shell.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox shell' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import shell +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(shell.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=shell.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + shell( + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_status.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_status.py new file mode 100644 index 0000000..2bef19c --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_status.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox status' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import status +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(status.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=status.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + status(out_dir=pwd or OUTPUT_DIR) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_update.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_update.py new file mode 100644 index 0000000..6748096 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_update.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox update' + +import sys +import argparse + +from typing import List, Optional, IO + +from ..main import update +from ..util import docstring +from ..config import OUTPUT_DIR +from ..index import ( + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, +) +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(update.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=update.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--only-new', #'-n', + action='store_true', + help="Don't attempt to retry previously skipped/failed links when updating", + ) + parser.add_argument( + '--index-only', #'-o', + action='store_true', + help="Update the main index without archiving any content", + ) + parser.add_argument( + '--resume', #'-r', + type=float, + help='Resume the update process from a given timestamp', + default=None, + ) + parser.add_argument( + '--overwrite', #'-x', + action='store_true', + help='Ignore existing archived content and overwrite with new versions (DANGEROUS)', + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="Update only links bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="Update only links bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--status', + type=str, + choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'), + default='indexed', + help=( + 'Update only links or data directories that have the given status\n' + f' indexed {get_indexed_folders.__doc__} (the default)\n' + f' archived {get_archived_folders.__doc__}\n' + f' unarchived {get_unarchived_folders.__doc__}\n' + '\n' + f' present {get_present_folders.__doc__}\n' + f' valid {get_valid_folders.__doc__}\n' + f' invalid {get_invalid_folders.__doc__}\n' + '\n' + f' duplicate {get_duplicate_folders.__doc__}\n' + f' orphaned {get_orphaned_folders.__doc__}\n' + f' corrupted {get_corrupted_folders.__doc__}\n' + f' unrecognized {get_unrecognized_folders.__doc__}\n' + ) + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + default=None, + help='Update only URLs matching these filter patterns.' + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + command = parser.parse_args(args or ()) + filter_patterns_str = accept_stdin(stdin) + + update( + resume=command.resume, + only_new=command.only_new, + index_only=command.index_only, + overwrite=command.overwrite, + filter_patterns_str=filter_patterns_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + status=command.status, + after=command.after, + before=command.before, + out_dir=pwd or OUTPUT_DIR, + extractors=command.extract, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_version.py b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_version.py new file mode 100755 index 0000000..e7922f3 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/archivebox_version.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox version' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import version +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(version.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=version.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--quiet', '-q', + action='store_true', + help='Only print ArchiveBox version number and nothing else.', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + version( + quiet=command.quiet, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/build/lib/archivebox/cli/tests.py b/archivebox-0.5.3/build/lib/archivebox/cli/tests.py new file mode 100755 index 0000000..4d7016a --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/cli/tests.py @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' + + +import os +import sys +import shutil +import unittest +from pathlib import Path + +from contextlib import contextmanager + +TEST_CONFIG = { + 'USE_COLOR': 'False', + 'SHOW_PROGRESS': 'False', + + 'OUTPUT_DIR': 'data.tests', + + 'SAVE_ARCHIVE_DOT_ORG': 'False', + 'SAVE_TITLE': 'False', + + 'USE_CURL': 'False', + 'USE_WGET': 'False', + 'USE_GIT': 'False', + 'USE_CHROME': 'False', + 'USE_YOUTUBEDL': 'False', +} + +OUTPUT_DIR = 'data.tests' +os.environ.update(TEST_CONFIG) + +from ..main import init +from ..index import load_main_index +from ..config import ( + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, +) + +from . import ( + archivebox_init, + archivebox_add, + archivebox_remove, +) + +HIDE_CLI_OUTPUT = True + +test_urls = ''' +https://example1.com/what/is/happening.html?what=1#how-about-this=1 +https://example2.com/what/is/happening/?what=1#how-about-this=1 +HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f +https://example4.com/what/is/happening.html +https://example5.com/ +https://example6.com + +http://example7.com +[https://example8.com/what/is/this.php?what=1] +[and http://example9.com?what=1&other=3#and-thing=2] +https://example10.com#and-thing=2 " +abcdef +sdflkf[what](https://subb.example12.com/who/what.php?whoami=1#whatami=2)?am=hi +example13.bada +and example14.badb +htt://example15.badc +''' + +stdout = sys.stdout +stderr = sys.stderr + + +@contextmanager +def output_hidden(show_failing=True): + if not HIDE_CLI_OUTPUT: + yield + return + + sys.stdout = open('stdout.txt', 'w+') + sys.stderr = open('stderr.txt', 'w+') + try: + yield + sys.stdout.close() + sys.stderr.close() + sys.stdout = stdout + sys.stderr = stderr + except: + sys.stdout.close() + sys.stderr.close() + sys.stdout = stdout + sys.stderr = stderr + if show_failing: + with open('stdout.txt', 'r') as f: + print(f.read()) + with open('stderr.txt', 'r') as f: + print(f.read()) + raise + finally: + os.remove('stdout.txt') + os.remove('stderr.txt') + + +class TestInit(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + + def tearDown(self): + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + def test_basic_init(self): + with output_hidden(): + archivebox_init.main([]) + + assert (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists() + assert (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists() + assert (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists() + assert len(load_main_index(out_dir=OUTPUT_DIR)) == 0 + + def test_conflicting_init(self): + with open(Path(OUTPUT_DIR) / 'test_conflict.txt', 'w+') as f: + f.write('test') + + try: + with output_hidden(show_failing=False): + archivebox_init.main([]) + assert False, 'Init should have exited with an exception' + except SystemExit: + pass + + assert not (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists() + assert not (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists() + assert not (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists() + try: + load_main_index(out_dir=OUTPUT_DIR) + assert False, 'load_main_index should raise an exception when no index is present' + except: + pass + + def test_no_dirty_state(self): + with output_hidden(): + init() + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + with output_hidden(): + init() + + +class TestAdd(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + with output_hidden(): + init() + + def tearDown(self): + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + def test_add_arg_url(self): + with output_hidden(): + archivebox_add.main(['https://getpocket.com/users/nikisweeting/feed/all']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 30 + + def test_add_arg_file(self): + test_file = Path(OUTPUT_DIR) / 'test.txt' + with open(test_file, 'w+') as f: + f.write(test_urls) + + with output_hidden(): + archivebox_add.main([test_file]) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 12 + os.remove(test_file) + + def test_add_stdin_url(self): + with output_hidden(): + archivebox_add.main([], stdin=test_urls) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 12 + + +class TestRemove(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + with output_hidden(): + init() + archivebox_add.main([], stdin=test_urls) + + # def tearDown(self): + # shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + + def test_remove_exact(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', 'https://example5.com/']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 11 + + def test_remove_regex(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', '--filter-type=regex', r'http(s)?:\/\/(.+\.)?(example\d\.com)']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 4 + + def test_remove_domain(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', '--filter-type=domain', 'example5.com', 'example6.com']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 10 + + def test_remove_none(self): + try: + with output_hidden(show_failing=False): + archivebox_remove.main(['--yes', '--delete', 'https://doesntexist.com']) + assert False, 'Should raise if no URLs match' + except: + pass + + +if __name__ == '__main__': + if '--verbose' in sys.argv or '-v' in sys.argv: + HIDE_CLI_OUTPUT = False + + unittest.main() diff --git a/archivebox-0.5.3/build/lib/archivebox/config.py b/archivebox-0.5.3/build/lib/archivebox/config.py new file mode 100644 index 0000000..9a3f9a7 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/config.py @@ -0,0 +1,1081 @@ +""" +ArchiveBox config definitons (including defaults and dynamic config options). + +Config Usage Example: + + archivebox config --set MEDIA_TIMEOUT=600 + env MEDIA_TIMEOUT=600 USE_COLOR=False ... archivebox [subcommand] ... + +Config Precedence Order: + + 1. cli args (--update-all / --index-only / etc.) + 2. shell environment vars (env USE_COLOR=False archivebox add '...') + 3. config file (echo "SAVE_FAVICON=False" >> ArchiveBox.conf) + 4. defaults (defined below in Python) + +Documentation: + + https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration + +""" + +__package__ = 'archivebox' + +import os +import io +import re +import sys +import json +import getpass +import shutil +import django + +from hashlib import md5 +from pathlib import Path +from typing import Optional, Type, Tuple, Dict, Union, List +from subprocess import run, PIPE, DEVNULL +from configparser import ConfigParser +from collections import defaultdict + +from .config_stubs import ( + SimpleConfigValueDict, + ConfigValue, + ConfigDict, + ConfigDefaultValue, + ConfigDefaultDict, +) + +############################### Config Schema ################################## + +CONFIG_SCHEMA: Dict[str, ConfigDefaultDict] = { + 'SHELL_CONFIG': { + 'IS_TTY': {'type': bool, 'default': lambda _: sys.stdout.isatty()}, + 'USE_COLOR': {'type': bool, 'default': lambda c: c['IS_TTY']}, + 'SHOW_PROGRESS': {'type': bool, 'default': lambda c: c['IS_TTY']}, + 'IN_DOCKER': {'type': bool, 'default': False}, + # TODO: 'SHOW_HINTS': {'type: bool, 'default': True}, + }, + + 'GENERAL_CONFIG': { + 'OUTPUT_DIR': {'type': str, 'default': None}, + 'CONFIG_FILE': {'type': str, 'default': None}, + 'ONLY_NEW': {'type': bool, 'default': True}, + 'TIMEOUT': {'type': int, 'default': 60}, + 'MEDIA_TIMEOUT': {'type': int, 'default': 3600}, + 'OUTPUT_PERMISSIONS': {'type': str, 'default': '755'}, + 'RESTRICT_FILE_NAMES': {'type': str, 'default': 'windows'}, + 'URL_BLACKLIST': {'type': str, 'default': r'\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$'}, # to avoid downloading code assets as their own pages + }, + + 'SERVER_CONFIG': { + 'SECRET_KEY': {'type': str, 'default': None}, + 'BIND_ADDR': {'type': str, 'default': lambda c: ['127.0.0.1:8000', '0.0.0.0:8000'][c['IN_DOCKER']]}, + 'ALLOWED_HOSTS': {'type': str, 'default': '*'}, + 'DEBUG': {'type': bool, 'default': False}, + 'PUBLIC_INDEX': {'type': bool, 'default': True}, + 'PUBLIC_SNAPSHOTS': {'type': bool, 'default': True}, + 'PUBLIC_ADD_VIEW': {'type': bool, 'default': False}, + 'FOOTER_INFO': {'type': str, 'default': 'Content is hosted for personal archiving purposes only. Contact server owner for any takedown requests.'}, + 'ACTIVE_THEME': {'type': str, 'default': 'default'}, + }, + + 'ARCHIVE_METHOD_TOGGLES': { + 'SAVE_TITLE': {'type': bool, 'default': True, 'aliases': ('FETCH_TITLE',)}, + 'SAVE_FAVICON': {'type': bool, 'default': True, 'aliases': ('FETCH_FAVICON',)}, + 'SAVE_WGET': {'type': bool, 'default': True, 'aliases': ('FETCH_WGET',)}, + 'SAVE_WGET_REQUISITES': {'type': bool, 'default': True, 'aliases': ('FETCH_WGET_REQUISITES',)}, + 'SAVE_SINGLEFILE': {'type': bool, 'default': True, 'aliases': ('FETCH_SINGLEFILE',)}, + 'SAVE_READABILITY': {'type': bool, 'default': True, 'aliases': ('FETCH_READABILITY',)}, + 'SAVE_MERCURY': {'type': bool, 'default': True, 'aliases': ('FETCH_MERCURY',)}, + 'SAVE_PDF': {'type': bool, 'default': True, 'aliases': ('FETCH_PDF',)}, + 'SAVE_SCREENSHOT': {'type': bool, 'default': True, 'aliases': ('FETCH_SCREENSHOT',)}, + 'SAVE_DOM': {'type': bool, 'default': True, 'aliases': ('FETCH_DOM',)}, + 'SAVE_HEADERS': {'type': bool, 'default': True, 'aliases': ('FETCH_HEADERS',)}, + 'SAVE_WARC': {'type': bool, 'default': True, 'aliases': ('FETCH_WARC',)}, + 'SAVE_GIT': {'type': bool, 'default': True, 'aliases': ('FETCH_GIT',)}, + 'SAVE_MEDIA': {'type': bool, 'default': True, 'aliases': ('FETCH_MEDIA',)}, + 'SAVE_ARCHIVE_DOT_ORG': {'type': bool, 'default': True, 'aliases': ('SUBMIT_ARCHIVE_DOT_ORG',)}, + }, + + 'ARCHIVE_METHOD_OPTIONS': { + 'RESOLUTION': {'type': str, 'default': '1440,2000', 'aliases': ('SCREENSHOT_RESOLUTION',)}, + 'GIT_DOMAINS': {'type': str, 'default': 'github.com,bitbucket.org,gitlab.com'}, + 'CHECK_SSL_VALIDITY': {'type': bool, 'default': True}, + + 'CURL_USER_AGENT': {'type': str, 'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) curl/{CURL_VERSION}'}, + 'WGET_USER_AGENT': {'type': str, 'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) wget/{WGET_VERSION}'}, + 'CHROME_USER_AGENT': {'type': str, 'default': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36'}, + + 'COOKIES_FILE': {'type': str, 'default': None}, + 'CHROME_USER_DATA_DIR': {'type': str, 'default': None}, + + 'CHROME_HEADLESS': {'type': bool, 'default': True}, + 'CHROME_SANDBOX': {'type': bool, 'default': lambda c: not c['IN_DOCKER']}, + 'YOUTUBEDL_ARGS': {'type': list, 'default': ['--write-description', + '--write-info-json', + '--write-annotations', + '--write-thumbnail', + '--no-call-home', + '--user-agent', + '--all-subs', + '--extract-audio', + '--keep-video', + '--ignore-errors', + '--geo-bypass', + '--audio-format', 'mp3', + '--audio-quality', '320K', + '--embed-thumbnail', + '--add-metadata']}, + + 'WGET_ARGS': {'type': list, 'default': ['--no-verbose', + '--adjust-extension', + '--convert-links', + '--force-directories', + '--backup-converted', + '--span-hosts', + '--no-parent', + '-e', 'robots=off', + ]}, + 'CURL_ARGS': {'type': list, 'default': ['--silent', + '--location', + '--compressed' + ]}, + 'GIT_ARGS': {'type': list, 'default': ['--recursive']}, + }, + + 'SEARCH_BACKEND_CONFIG' : { + 'USE_INDEXING_BACKEND': {'type': bool, 'default': True}, + 'USE_SEARCHING_BACKEND': {'type': bool, 'default': True}, + 'SEARCH_BACKEND_ENGINE': {'type': str, 'default': 'ripgrep'}, + 'SEARCH_BACKEND_HOST_NAME': {'type': str, 'default': 'localhost'}, + 'SEARCH_BACKEND_PORT': {'type': int, 'default': 1491}, + 'SEARCH_BACKEND_PASSWORD': {'type': str, 'default': 'SecretPassword'}, + # SONIC + 'SONIC_COLLECTION': {'type': str, 'default': 'archivebox'}, + 'SONIC_BUCKET': {'type': str, 'default': 'snapshots'}, + }, + + 'DEPENDENCY_CONFIG': { + 'USE_CURL': {'type': bool, 'default': True}, + 'USE_WGET': {'type': bool, 'default': True}, + 'USE_SINGLEFILE': {'type': bool, 'default': True}, + 'USE_READABILITY': {'type': bool, 'default': True}, + 'USE_MERCURY': {'type': bool, 'default': True}, + 'USE_GIT': {'type': bool, 'default': True}, + 'USE_CHROME': {'type': bool, 'default': True}, + 'USE_NODE': {'type': bool, 'default': True}, + 'USE_YOUTUBEDL': {'type': bool, 'default': True}, + 'USE_RIPGREP': {'type': bool, 'default': True}, + + 'CURL_BINARY': {'type': str, 'default': 'curl'}, + 'GIT_BINARY': {'type': str, 'default': 'git'}, + 'WGET_BINARY': {'type': str, 'default': 'wget'}, + 'SINGLEFILE_BINARY': {'type': str, 'default': 'single-file'}, + 'READABILITY_BINARY': {'type': str, 'default': 'readability-extractor'}, + 'MERCURY_BINARY': {'type': str, 'default': 'mercury-parser'}, + 'YOUTUBEDL_BINARY': {'type': str, 'default': 'youtube-dl'}, + 'NODE_BINARY': {'type': str, 'default': 'node'}, + 'RIPGREP_BINARY': {'type': str, 'default': 'rg'}, + 'CHROME_BINARY': {'type': str, 'default': None}, + + 'POCKET_CONSUMER_KEY': {'type': str, 'default': None}, + 'POCKET_ACCESS_TOKENS': {'type': dict, 'default': {}}, + }, +} + + +########################## Backwards-Compatibility ############################# + + +# for backwards compatibility with old config files, check old/deprecated names for each key +CONFIG_ALIASES = { + alias: key + for section in CONFIG_SCHEMA.values() + for key, default in section.items() + for alias in default.get('aliases', ()) +} +USER_CONFIG = {key for section in CONFIG_SCHEMA.values() for key in section.keys()} + +def get_real_name(key: str) -> str: + """get the current canonical name for a given deprecated config key""" + return CONFIG_ALIASES.get(key.upper().strip(), key.upper().strip()) + + + +################################ Constants ##################################### + +PACKAGE_DIR_NAME = 'archivebox' +TEMPLATES_DIR_NAME = 'themes' + +ARCHIVE_DIR_NAME = 'archive' +SOURCES_DIR_NAME = 'sources' +LOGS_DIR_NAME = 'logs' +STATIC_DIR_NAME = 'static' +SQL_INDEX_FILENAME = 'index.sqlite3' +JSON_INDEX_FILENAME = 'index.json' +HTML_INDEX_FILENAME = 'index.html' +ROBOTS_TXT_FILENAME = 'robots.txt' +FAVICON_FILENAME = 'favicon.ico' +CONFIG_FILENAME = 'ArchiveBox.conf' + +DEFAULT_CLI_COLORS = { + 'reset': '\033[00;00m', + 'lightblue': '\033[01;30m', + 'lightyellow': '\033[01;33m', + 'lightred': '\033[01;35m', + 'red': '\033[01;31m', + 'green': '\033[01;32m', + 'blue': '\033[01;34m', + 'white': '\033[01;37m', + 'black': '\033[01;30m', +} +ANSI = {k: '' for k in DEFAULT_CLI_COLORS.keys()} + +COLOR_DICT = defaultdict(lambda: [(0, 0, 0), (0, 0, 0)], { + '00': [(0, 0, 0), (0, 0, 0)], + '30': [(0, 0, 0), (0, 0, 0)], + '31': [(255, 0, 0), (128, 0, 0)], + '32': [(0, 200, 0), (0, 128, 0)], + '33': [(255, 255, 0), (128, 128, 0)], + '34': [(0, 0, 255), (0, 0, 128)], + '35': [(255, 0, 255), (128, 0, 128)], + '36': [(0, 255, 255), (0, 128, 128)], + '37': [(255, 255, 255), (255, 255, 255)], +}) + +STATICFILE_EXTENSIONS = { + # 99.999% of the time, URLs ending in these extensions are static files + # that can be downloaded as-is, not html pages that need to be rendered + 'gif', 'jpeg', 'jpg', 'png', 'tif', 'tiff', 'wbmp', 'ico', 'jng', 'bmp', + 'svg', 'svgz', 'webp', 'ps', 'eps', 'ai', + 'mp3', 'mp4', 'm4a', 'mpeg', 'mpg', 'mkv', 'mov', 'webm', 'm4v', + 'flv', 'wmv', 'avi', 'ogg', 'ts', 'm3u8', + 'pdf', 'txt', 'rtf', 'rtfd', 'doc', 'docx', 'ppt', 'pptx', 'xls', 'xlsx', + 'atom', 'rss', 'css', 'js', 'json', + 'dmg', 'iso', 'img', + 'rar', 'war', 'hqx', 'zip', 'gz', 'bz2', '7z', + + # Less common extensions to consider adding later + # jar, swf, bin, com, exe, dll, deb + # ear, hqx, eot, wmlc, kml, kmz, cco, jardiff, jnlp, run, msi, msp, msm, + # pl pm, prc pdb, rar, rpm, sea, sit, tcl tk, der, pem, crt, xpi, xspf, + # ra, mng, asx, asf, 3gpp, 3gp, mid, midi, kar, jad, wml, htc, mml + + # These are always treated as pages, not as static files, never add them: + # html, htm, shtml, xhtml, xml, aspx, php, cgi +} + + + +############################## Derived Config ################################## + + +DYNAMIC_CONFIG_SCHEMA: ConfigDefaultDict = { + 'TERM_WIDTH': {'default': lambda c: lambda: shutil.get_terminal_size((100, 10)).columns}, + 'USER': {'default': lambda c: getpass.getuser() or os.getlogin()}, + 'ANSI': {'default': lambda c: DEFAULT_CLI_COLORS if c['USE_COLOR'] else {k: '' for k in DEFAULT_CLI_COLORS.keys()}}, + + 'PACKAGE_DIR': {'default': lambda c: Path(__file__).resolve().parent}, + 'TEMPLATES_DIR': {'default': lambda c: c['PACKAGE_DIR'] / TEMPLATES_DIR_NAME}, + + 'OUTPUT_DIR': {'default': lambda c: Path(c['OUTPUT_DIR']).resolve() if c['OUTPUT_DIR'] else Path(os.curdir).resolve()}, + 'ARCHIVE_DIR': {'default': lambda c: c['OUTPUT_DIR'] / ARCHIVE_DIR_NAME}, + 'SOURCES_DIR': {'default': lambda c: c['OUTPUT_DIR'] / SOURCES_DIR_NAME}, + 'LOGS_DIR': {'default': lambda c: c['OUTPUT_DIR'] / LOGS_DIR_NAME}, + 'CONFIG_FILE': {'default': lambda c: Path(c['CONFIG_FILE']).resolve() if c['CONFIG_FILE'] else c['OUTPUT_DIR'] / CONFIG_FILENAME}, + 'COOKIES_FILE': {'default': lambda c: c['COOKIES_FILE'] and Path(c['COOKIES_FILE']).resolve()}, + 'CHROME_USER_DATA_DIR': {'default': lambda c: find_chrome_data_dir() if c['CHROME_USER_DATA_DIR'] is None else (Path(c['CHROME_USER_DATA_DIR']).resolve() if c['CHROME_USER_DATA_DIR'] else None)}, # None means unset, so we autodetect it with find_chrome_Data_dir(), but emptystring '' means user manually set it to '', and we should store it as None + 'URL_BLACKLIST_PTN': {'default': lambda c: c['URL_BLACKLIST'] and re.compile(c['URL_BLACKLIST'] or '', re.IGNORECASE | re.UNICODE | re.MULTILINE)}, + + 'ARCHIVEBOX_BINARY': {'default': lambda c: sys.argv[0]}, + 'VERSION': {'default': lambda c: json.loads((Path(c['PACKAGE_DIR']) / 'package.json').read_text().strip())['version']}, + 'GIT_SHA': {'default': lambda c: c['VERSION'].split('+')[-1] or 'unknown'}, + + 'PYTHON_BINARY': {'default': lambda c: sys.executable}, + 'PYTHON_ENCODING': {'default': lambda c: sys.stdout.encoding.upper()}, + 'PYTHON_VERSION': {'default': lambda c: '{}.{}.{}'.format(*sys.version_info[:3])}, + + 'DJANGO_BINARY': {'default': lambda c: django.__file__.replace('__init__.py', 'bin/django-admin.py')}, + 'DJANGO_VERSION': {'default': lambda c: '{}.{}.{} {} ({})'.format(*django.VERSION)}, + + 'USE_CURL': {'default': lambda c: c['USE_CURL'] and (c['SAVE_FAVICON'] or c['SAVE_TITLE'] or c['SAVE_ARCHIVE_DOT_ORG'])}, + 'CURL_VERSION': {'default': lambda c: bin_version(c['CURL_BINARY']) if c['USE_CURL'] else None}, + 'CURL_USER_AGENT': {'default': lambda c: c['CURL_USER_AGENT'].format(**c)}, + 'CURL_ARGS': {'default': lambda c: c['CURL_ARGS'] or []}, + 'SAVE_FAVICON': {'default': lambda c: c['USE_CURL'] and c['SAVE_FAVICON']}, + 'SAVE_ARCHIVE_DOT_ORG': {'default': lambda c: c['USE_CURL'] and c['SAVE_ARCHIVE_DOT_ORG']}, + + 'USE_WGET': {'default': lambda c: c['USE_WGET'] and (c['SAVE_WGET'] or c['SAVE_WARC'])}, + 'WGET_VERSION': {'default': lambda c: bin_version(c['WGET_BINARY']) if c['USE_WGET'] else None}, + 'WGET_AUTO_COMPRESSION': {'default': lambda c: wget_supports_compression(c) if c['USE_WGET'] else False}, + 'WGET_USER_AGENT': {'default': lambda c: c['WGET_USER_AGENT'].format(**c)}, + 'SAVE_WGET': {'default': lambda c: c['USE_WGET'] and c['SAVE_WGET']}, + 'SAVE_WARC': {'default': lambda c: c['USE_WGET'] and c['SAVE_WARC']}, + 'WGET_ARGS': {'default': lambda c: c['WGET_ARGS'] or []}, + + 'RIPGREP_VERSION': {'default': lambda c: bin_version(c['RIPGREP_BINARY']) if c['USE_RIPGREP'] else None}, + + 'USE_SINGLEFILE': {'default': lambda c: c['USE_SINGLEFILE'] and c['SAVE_SINGLEFILE']}, + 'SINGLEFILE_VERSION': {'default': lambda c: bin_version(c['SINGLEFILE_BINARY']) if c['USE_SINGLEFILE'] else None}, + + 'USE_READABILITY': {'default': lambda c: c['USE_READABILITY'] and c['SAVE_READABILITY']}, + 'READABILITY_VERSION': {'default': lambda c: bin_version(c['READABILITY_BINARY']) if c['USE_READABILITY'] else None}, + + 'USE_MERCURY': {'default': lambda c: c['USE_MERCURY'] and c['SAVE_MERCURY']}, + 'MERCURY_VERSION': {'default': lambda c: '1.0.0' if shutil.which(str(bin_path(c['MERCURY_BINARY']))) else None}, # mercury is unversioned + + 'USE_GIT': {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']}, + 'GIT_VERSION': {'default': lambda c: bin_version(c['GIT_BINARY']) if c['USE_GIT'] else None}, + 'SAVE_GIT': {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']}, + + 'USE_YOUTUBEDL': {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']}, + 'YOUTUBEDL_VERSION': {'default': lambda c: bin_version(c['YOUTUBEDL_BINARY']) if c['USE_YOUTUBEDL'] else None}, + 'SAVE_MEDIA': {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']}, + 'YOUTUBEDL_ARGS': {'default': lambda c: c['YOUTUBEDL_ARGS'] or []}, + + 'USE_CHROME': {'default': lambda c: c['USE_CHROME'] and (c['SAVE_PDF'] or c['SAVE_SCREENSHOT'] or c['SAVE_DOM'] or c['SAVE_SINGLEFILE'])}, + 'CHROME_BINARY': {'default': lambda c: c['CHROME_BINARY'] if c['CHROME_BINARY'] else find_chrome_binary()}, + 'CHROME_VERSION': {'default': lambda c: bin_version(c['CHROME_BINARY']) if c['USE_CHROME'] else None}, + + 'SAVE_PDF': {'default': lambda c: c['USE_CHROME'] and c['SAVE_PDF']}, + 'SAVE_SCREENSHOT': {'default': lambda c: c['USE_CHROME'] and c['SAVE_SCREENSHOT']}, + 'SAVE_DOM': {'default': lambda c: c['USE_CHROME'] and c['SAVE_DOM']}, + 'SAVE_SINGLEFILE': {'default': lambda c: c['USE_CHROME'] and c['SAVE_SINGLEFILE'] and c['USE_NODE']}, + 'SAVE_READABILITY': {'default': lambda c: c['USE_READABILITY'] and c['USE_NODE']}, + 'SAVE_MERCURY': {'default': lambda c: c['USE_MERCURY'] and c['USE_NODE']}, + + 'USE_NODE': {'default': lambda c: c['USE_NODE'] and (c['SAVE_READABILITY'] or c['SAVE_SINGLEFILE'] or c['SAVE_MERCURY'])}, + 'NODE_VERSION': {'default': lambda c: bin_version(c['NODE_BINARY']) if c['USE_NODE'] else None}, + + 'DEPENDENCIES': {'default': lambda c: get_dependency_info(c)}, + 'CODE_LOCATIONS': {'default': lambda c: get_code_locations(c)}, + 'EXTERNAL_LOCATIONS': {'default': lambda c: get_external_locations(c)}, + 'DATA_LOCATIONS': {'default': lambda c: get_data_locations(c)}, + 'CHROME_OPTIONS': {'default': lambda c: get_chrome_info(c)}, +} + + + +################################### Helpers #################################### + + +def load_config_val(key: str, + default: ConfigDefaultValue=None, + type: Optional[Type]=None, + aliases: Optional[Tuple[str, ...]]=None, + config: Optional[ConfigDict]=None, + env_vars: Optional[os._Environ]=None, + config_file_vars: Optional[Dict[str, str]]=None) -> ConfigValue: + """parse bool, int, and str key=value pairs from env""" + + + config_keys_to_check = (key, *(aliases or ())) + for key in config_keys_to_check: + if env_vars: + val = env_vars.get(key) + if val: + break + if config_file_vars: + val = config_file_vars.get(key) + if val: + break + + if type is None or val is None: + if callable(default): + assert isinstance(config, dict) + return default(config) + + return default + + elif type is bool: + if val.lower() in ('true', 'yes', '1'): + return True + elif val.lower() in ('false', 'no', '0'): + return False + else: + raise ValueError(f'Invalid configuration option {key}={val} (expected a boolean: True/False)') + + elif type is str: + if val.lower() in ('true', 'false', 'yes', 'no', '1', '0'): + raise ValueError(f'Invalid configuration option {key}={val} (expected a string)') + return val.strip() + + elif type is int: + if not val.isdigit(): + raise ValueError(f'Invalid configuration option {key}={val} (expected an integer)') + return int(val) + + elif type is list or type is dict: + return json.loads(val) + + raise Exception('Config values can only be str, bool, int or json') + + +def load_config_file(out_dir: str=None) -> Optional[Dict[str, str]]: + """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf""" + + out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve() + config_path = Path(out_dir) / CONFIG_FILENAME + if config_path.exists(): + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(config_path) + # flatten into one namespace + config_file_vars = { + key.upper(): val + for section, options in config_file.items() + for key, val in options.items() + } + # print('[i] Loaded config file', os.path.abspath(config_path)) + # print(config_file_vars) + return config_file_vars + return None + + +def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict: + """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf""" + + from .system import atomic_write + + CONFIG_HEADER = ( + """# This is the config file for your ArchiveBox collection. + # + # You can add options here manually in INI format, or automatically by running: + # archivebox config --set KEY=VALUE + # + # If you modify this file manually, make sure to update your archive after by running: + # archivebox init + # + # A list of all possible config with documentation and examples can be found here: + # https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration + + """) + + out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve() + config_path = Path(out_dir) / CONFIG_FILENAME + + if not config_path.exists(): + atomic_write(config_path, CONFIG_HEADER) + + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(config_path) + + with open(config_path, 'r') as old: + atomic_write(f'{config_path}.bak', old.read()) + + find_section = lambda key: [name for name, opts in CONFIG_SCHEMA.items() if key in opts][0] + + # Set up sections in empty config file + for key, val in config.items(): + section = find_section(key) + if section in config_file: + existing_config = dict(config_file[section]) + else: + existing_config = {} + config_file[section] = {**existing_config, key: val} + + # always make sure there's a SECRET_KEY defined for Django + existing_secret_key = None + if 'SERVER_CONFIG' in config_file and 'SECRET_KEY' in config_file['SERVER_CONFIG']: + existing_secret_key = config_file['SERVER_CONFIG']['SECRET_KEY'] + + if (not existing_secret_key) or ('not a valid secret' in existing_secret_key): + from django.utils.crypto import get_random_string + chars = 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.' + random_secret_key = get_random_string(50, chars) + if 'SERVER_CONFIG' in config_file: + config_file['SERVER_CONFIG']['SECRET_KEY'] = random_secret_key + else: + config_file['SERVER_CONFIG'] = {'SECRET_KEY': random_secret_key} + + with open(config_path, 'w+') as new: + config_file.write(new) + + try: + # validate the config by attempting to re-parse it + CONFIG = load_all_config() + return { + key.upper(): CONFIG.get(key.upper()) + for key in config.keys() + } + except: + # something went horribly wrong, rever to the previous version + with open(f'{config_path}.bak', 'r') as old: + atomic_write(config_path, old.read()) + + if Path(f'{config_path}.bak').exists(): + os.remove(f'{config_path}.bak') + + return {} + + + +def load_config(defaults: ConfigDefaultDict, + config: Optional[ConfigDict]=None, + out_dir: Optional[str]=None, + env_vars: Optional[os._Environ]=None, + config_file_vars: Optional[Dict[str, str]]=None) -> ConfigDict: + + env_vars = env_vars or os.environ + config_file_vars = config_file_vars or load_config_file(out_dir=out_dir) + + extended_config: ConfigDict = config.copy() if config else {} + for key, default in defaults.items(): + try: + extended_config[key] = load_config_val( + key, + default=default['default'], + type=default.get('type'), + aliases=default.get('aliases'), + config=extended_config, + env_vars=env_vars, + config_file_vars=config_file_vars, + ) + except KeyboardInterrupt: + raise SystemExit(0) + except Exception as e: + stderr() + stderr(f'[X] Error while loading configuration value: {key}', color='red', config=extended_config) + stderr(' {}: {}'.format(e.__class__.__name__, e)) + stderr() + stderr(' Check your config for mistakes and try again (your archive data is unaffected).') + stderr() + stderr(' For config documentation and examples see:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration') + stderr() + raise + raise SystemExit(2) + + return extended_config + +# def write_config(config: ConfigDict): + +# with open(os.path.join(config['OUTPUT_DIR'], CONFIG_FILENAME), 'w+') as f: + + +# Logging Helpers +def stdout(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if color: + strs = [ansi[color], ' '.join(str(a) for a in args), ansi['reset'], '\n'] + else: + strs = [' '.join(str(a) for a in args), '\n'] + + sys.stdout.write(prefix + ''.join(strs)) + +def stderr(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if color: + strs = [ansi[color], ' '.join(str(a) for a in args), ansi['reset'], '\n'] + else: + strs = [' '.join(str(a) for a in args), '\n'] + + sys.stderr.write(prefix + ''.join(strs)) + +def hint(text: Union[Tuple[str, ...], List[str], str], prefix=' ', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if isinstance(text, str): + stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text, **ansi)) + else: + stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text[0], **ansi)) + for line in text[1:]: + stderr('{} {}'.format(prefix, line)) + + +# Dependency Metadata Helpers +def bin_version(binary: Optional[str]) -> Optional[str]: + """check the presence and return valid version line of a specified binary""" + + abspath = bin_path(binary) + if not binary or not abspath: + return None + + try: + version_str = run([abspath, "--version"], stdout=PIPE).stdout.strip().decode() + # take first 3 columns of first line of version info + return ' '.join(version_str.split('\n')[0].strip().split()[:3]) + except OSError: + pass + # stderr(f'[X] Unable to find working version of dependency: {binary}', color='red') + # stderr(' Make sure it\'s installed, then confirm it\'s working by running:') + # stderr(f' {binary} --version') + # stderr() + # stderr(' If you don\'t want to install it, you can disable it via config. See here for more info:') + # stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Install') + return None + +def bin_path(binary: Optional[str]) -> Optional[str]: + if binary is None: + return None + + node_modules_bin = Path('.') / 'node_modules' / '.bin' / binary + if node_modules_bin.exists(): + return str(node_modules_bin.resolve()) + + return shutil.which(str(Path(binary).expanduser())) or shutil.which(str(binary)) or binary + +def bin_hash(binary: Optional[str]) -> Optional[str]: + if binary is None: + return None + abs_path = bin_path(binary) + if abs_path is None or not Path(abs_path).exists(): + return None + + file_hash = md5() + with io.open(abs_path, mode='rb') as f: + for chunk in iter(lambda: f.read(io.DEFAULT_BUFFER_SIZE), b''): + file_hash.update(chunk) + + return f'md5:{file_hash.hexdigest()}' + +def find_chrome_binary() -> Optional[str]: + """find any installed chrome binaries in the default locations""" + # Precedence: Chromium, Chrome, Beta, Canary, Unstable, Dev + # make sure data dir finding precedence order always matches binary finding order + default_executable_paths = ( + 'chromium-browser', + 'chromium', + '/Applications/Chromium.app/Contents/MacOS/Chromium', + 'chrome', + 'google-chrome', + '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', + 'google-chrome-stable', + 'google-chrome-beta', + 'google-chrome-canary', + '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary', + 'google-chrome-unstable', + 'google-chrome-dev', + ) + for name in default_executable_paths: + full_path_exists = shutil.which(name) + if full_path_exists: + return name + + return None + +def find_chrome_data_dir() -> Optional[str]: + """find any installed chrome user data directories in the default locations""" + # Precedence: Chromium, Chrome, Beta, Canary, Unstable, Dev + # make sure data dir finding precedence order always matches binary finding order + default_profile_paths = ( + '~/.config/chromium', + '~/Library/Application Support/Chromium', + '~/AppData/Local/Chromium/User Data', + '~/.config/chrome', + '~/.config/google-chrome', + '~/Library/Application Support/Google/Chrome', + '~/AppData/Local/Google/Chrome/User Data', + '~/.config/google-chrome-stable', + '~/.config/google-chrome-beta', + '~/Library/Application Support/Google/Chrome Canary', + '~/AppData/Local/Google/Chrome SxS/User Data', + '~/.config/google-chrome-unstable', + '~/.config/google-chrome-dev', + ) + for path in default_profile_paths: + full_path = Path(path).resolve() + if full_path.exists(): + return full_path + return None + +def wget_supports_compression(config): + try: + cmd = [ + config['WGET_BINARY'], + "--compression=auto", + "--help", + ] + return not run(cmd, stdout=DEVNULL, stderr=DEVNULL).returncode + except (FileNotFoundError, OSError): + return False + +def get_code_locations(config: ConfigDict) -> SimpleConfigValueDict: + return { + 'PACKAGE_DIR': { + 'path': (config['PACKAGE_DIR']).resolve(), + 'enabled': True, + 'is_valid': (config['PACKAGE_DIR'] / '__main__.py').exists(), + }, + 'TEMPLATES_DIR': { + 'path': (config['TEMPLATES_DIR']).resolve(), + 'enabled': True, + 'is_valid': (config['TEMPLATES_DIR'] / config['ACTIVE_THEME'] / 'static').exists(), + }, + # 'NODE_MODULES_DIR': { + # 'path': , + # 'enabled': , + # 'is_valid': (...).exists(), + # }, + } + +def get_external_locations(config: ConfigDict) -> ConfigValue: + abspath = lambda path: None if path is None else Path(path).resolve() + return { + 'CHROME_USER_DATA_DIR': { + 'path': abspath(config['CHROME_USER_DATA_DIR']), + 'enabled': config['USE_CHROME'] and config['CHROME_USER_DATA_DIR'], + 'is_valid': False if config['CHROME_USER_DATA_DIR'] is None else (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(), + }, + 'COOKIES_FILE': { + 'path': abspath(config['COOKIES_FILE']), + 'enabled': config['USE_WGET'] and config['COOKIES_FILE'], + 'is_valid': False if config['COOKIES_FILE'] is None else Path(config['COOKIES_FILE']).exists(), + }, + } + +def get_data_locations(config: ConfigDict) -> ConfigValue: + return { + 'OUTPUT_DIR': { + 'path': config['OUTPUT_DIR'].resolve(), + 'enabled': True, + 'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(), + }, + 'SOURCES_DIR': { + 'path': config['SOURCES_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['SOURCES_DIR'].exists(), + }, + 'LOGS_DIR': { + 'path': config['LOGS_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['LOGS_DIR'].exists(), + }, + 'ARCHIVE_DIR': { + 'path': config['ARCHIVE_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['ARCHIVE_DIR'].exists(), + }, + 'CONFIG_FILE': { + 'path': config['CONFIG_FILE'].resolve(), + 'enabled': True, + 'is_valid': config['CONFIG_FILE'].exists(), + }, + 'SQL_INDEX': { + 'path': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).resolve(), + 'enabled': True, + 'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(), + }, + } + +def get_dependency_info(config: ConfigDict) -> ConfigValue: + return { + 'ARCHIVEBOX_BINARY': { + 'path': bin_path(config['ARCHIVEBOX_BINARY']), + 'version': config['VERSION'], + 'hash': bin_hash(config['ARCHIVEBOX_BINARY']), + 'enabled': True, + 'is_valid': True, + }, + 'PYTHON_BINARY': { + 'path': bin_path(config['PYTHON_BINARY']), + 'version': config['PYTHON_VERSION'], + 'hash': bin_hash(config['PYTHON_BINARY']), + 'enabled': True, + 'is_valid': bool(config['DJANGO_VERSION']), + }, + 'DJANGO_BINARY': { + 'path': bin_path(config['DJANGO_BINARY']), + 'version': config['DJANGO_VERSION'], + 'hash': bin_hash(config['DJANGO_BINARY']), + 'enabled': True, + 'is_valid': bool(config['DJANGO_VERSION']), + }, + 'CURL_BINARY': { + 'path': bin_path(config['CURL_BINARY']), + 'version': config['CURL_VERSION'], + 'hash': bin_hash(config['PYTHON_BINARY']), + 'enabled': config['USE_CURL'], + 'is_valid': bool(config['CURL_VERSION']), + }, + 'WGET_BINARY': { + 'path': bin_path(config['WGET_BINARY']), + 'version': config['WGET_VERSION'], + 'hash': bin_hash(config['WGET_BINARY']), + 'enabled': config['USE_WGET'], + 'is_valid': bool(config['WGET_VERSION']), + }, + 'NODE_BINARY': { + 'path': bin_path(config['NODE_BINARY']), + 'version': config['NODE_VERSION'], + 'hash': bin_hash(config['NODE_BINARY']), + 'enabled': config['USE_NODE'], + 'is_valid': bool(config['SINGLEFILE_VERSION']), + }, + 'SINGLEFILE_BINARY': { + 'path': bin_path(config['SINGLEFILE_BINARY']), + 'version': config['SINGLEFILE_VERSION'], + 'hash': bin_hash(config['SINGLEFILE_BINARY']), + 'enabled': config['USE_SINGLEFILE'], + 'is_valid': bool(config['SINGLEFILE_VERSION']), + }, + 'READABILITY_BINARY': { + 'path': bin_path(config['READABILITY_BINARY']), + 'version': config['READABILITY_VERSION'], + 'hash': bin_hash(config['READABILITY_BINARY']), + 'enabled': config['USE_READABILITY'], + 'is_valid': bool(config['READABILITY_VERSION']), + }, + 'MERCURY_BINARY': { + 'path': bin_path(config['MERCURY_BINARY']), + 'version': config['MERCURY_VERSION'], + 'hash': bin_hash(config['MERCURY_BINARY']), + 'enabled': config['USE_MERCURY'], + 'is_valid': bool(config['MERCURY_VERSION']), + }, + 'GIT_BINARY': { + 'path': bin_path(config['GIT_BINARY']), + 'version': config['GIT_VERSION'], + 'hash': bin_hash(config['GIT_BINARY']), + 'enabled': config['USE_GIT'], + 'is_valid': bool(config['GIT_VERSION']), + }, + 'YOUTUBEDL_BINARY': { + 'path': bin_path(config['YOUTUBEDL_BINARY']), + 'version': config['YOUTUBEDL_VERSION'], + 'hash': bin_hash(config['YOUTUBEDL_BINARY']), + 'enabled': config['USE_YOUTUBEDL'], + 'is_valid': bool(config['YOUTUBEDL_VERSION']), + }, + 'CHROME_BINARY': { + 'path': bin_path(config['CHROME_BINARY']), + 'version': config['CHROME_VERSION'], + 'hash': bin_hash(config['CHROME_BINARY']), + 'enabled': config['USE_CHROME'], + 'is_valid': bool(config['CHROME_VERSION']), + }, + 'RIPGREP_BINARY': { + 'path': bin_path(config['RIPGREP_BINARY']), + 'version': config['RIPGREP_VERSION'], + 'hash': bin_hash(config['RIPGREP_BINARY']), + 'enabled': config['USE_RIPGREP'], + 'is_valid': bool(config['RIPGREP_VERSION']), + }, + # TODO: add an entry for the sonic search backend? + # 'SONIC_BINARY': { + # 'path': bin_path(config['SONIC_BINARY']), + # 'version': config['SONIC_VERSION'], + # 'hash': bin_hash(config['SONIC_BINARY']), + # 'enabled': config['USE_SONIC'], + # 'is_valid': bool(config['SONIC_VERSION']), + # }, + } + +def get_chrome_info(config: ConfigDict) -> ConfigValue: + return { + 'TIMEOUT': config['TIMEOUT'], + 'RESOLUTION': config['RESOLUTION'], + 'CHECK_SSL_VALIDITY': config['CHECK_SSL_VALIDITY'], + 'CHROME_BINARY': config['CHROME_BINARY'], + 'CHROME_HEADLESS': config['CHROME_HEADLESS'], + 'CHROME_SANDBOX': config['CHROME_SANDBOX'], + 'CHROME_USER_AGENT': config['CHROME_USER_AGENT'], + 'CHROME_USER_DATA_DIR': config['CHROME_USER_DATA_DIR'], + } + + +# ****************************************************************************** +# ****************************************************************************** +# ******************************** Load Config ********************************* +# ******* (compile the defaults, configs, and metadata all into CONFIG) ******** +# ****************************************************************************** +# ****************************************************************************** + + +def load_all_config(): + CONFIG: ConfigDict = {} + for section_name, section_config in CONFIG_SCHEMA.items(): + CONFIG = load_config(section_config, CONFIG) + + return load_config(DYNAMIC_CONFIG_SCHEMA, CONFIG) + +# add all final config values in CONFIG to globals in this file +CONFIG = load_all_config() +globals().update(CONFIG) +# this lets us do: from .config import DEBUG, MEDIA_TIMEOUT, ... + + +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** + + + +########################### System Environment Setup ########################### + + +# Set timezone to UTC and umask to OUTPUT_PERMISSIONS +os.environ["TZ"] = 'UTC' +os.umask(0o777 - int(OUTPUT_PERMISSIONS, base=8)) # noqa: F821 + +# add ./node_modules/.bin to $PATH so we can use node scripts in extractors +NODE_BIN_PATH = str((Path(CONFIG["OUTPUT_DIR"]).absolute() / 'node_modules' / '.bin')) +sys.path.append(NODE_BIN_PATH) + + + + +########################### Config Validity Checkers ########################### + + +def check_system_config(config: ConfigDict=CONFIG) -> None: + ### Check system environment + if config['USER'] == 'root': + stderr('[!] ArchiveBox should never be run as root!', color='red') + stderr(' For more information, see the security overview documentation:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root') + raise SystemExit(2) + + ### Check Python environment + if sys.version_info[:3] < (3, 6, 0): + stderr(f'[X] Python version is not new enough: {config["PYTHON_VERSION"]} (>3.6 is required)', color='red') + stderr(' See https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#python for help upgrading your Python installation.') + raise SystemExit(2) + + if config['PYTHON_ENCODING'] not in ('UTF-8', 'UTF8'): + stderr(f'[X] Your system is running python3 scripts with a bad locale setting: {config["PYTHON_ENCODING"]} (it should be UTF-8).', color='red') + stderr(' To fix it, add the line "export PYTHONIOENCODING=UTF-8" to your ~/.bashrc file (without quotes)') + stderr(' Or if you\'re using ubuntu/debian, run "dpkg-reconfigure locales"') + stderr('') + stderr(' Confirm that it\'s fixed by opening a new shell and running:') + stderr(' python3 -c "import sys; print(sys.stdout.encoding)" # should output UTF-8') + raise SystemExit(2) + + # stderr('[i] Using Chrome binary: {}'.format(shutil.which(CHROME_BINARY) or CHROME_BINARY)) + # stderr('[i] Using Chrome data dir: {}'.format(os.path.abspath(CHROME_USER_DATA_DIR))) + if config['CHROME_USER_DATA_DIR'] is not None: + if not (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(): + stderr('[X] Could not find profile "Default" in CHROME_USER_DATA_DIR.', color='red') + stderr(f' {config["CHROME_USER_DATA_DIR"]}') + stderr(' Make sure you set it to a Chrome user data directory containing a Default profile folder.') + stderr(' For more info see:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#CHROME_USER_DATA_DIR') + if '/Default' in str(config['CHROME_USER_DATA_DIR']): + stderr() + stderr(' Try removing /Default from the end e.g.:') + stderr(' CHROME_USER_DATA_DIR="{}"'.format(config['CHROME_USER_DATA_DIR'].split('/Default')[0])) + raise SystemExit(2) + + +def check_dependencies(config: ConfigDict=CONFIG, show_help: bool=True) -> None: + invalid_dependencies = [ + (name, info) for name, info in config['DEPENDENCIES'].items() + if info['enabled'] and not info['is_valid'] + ] + if invalid_dependencies and show_help: + stderr(f'[!] Warning: Missing {len(invalid_dependencies)} recommended dependencies', color='lightyellow') + for dependency, info in invalid_dependencies: + stderr( + ' ! {}: {} ({})'.format( + dependency, + info['path'] or 'unable to find binary', + info['version'] or 'unable to detect version', + ) + ) + if dependency in ('SINGLEFILE_BINARY', 'READABILITY_BINARY', 'MERCURY_BINARY'): + hint(('npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"', + f'or archivebox config --set SAVE_{dependency.rsplit("_", 1)[0]}=False to silence this warning', + ''), prefix=' ') + stderr('') + + if config['TIMEOUT'] < 5: + stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red') + stderr(' You must allow *at least* 5 seconds for indexing and archive methods to run succesfully.') + stderr(' (Setting it to somewhere between 30 and 3000 seconds is recommended)') + stderr() + stderr(' If you want to make ArchiveBox run faster, disable specific archive methods instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles') + stderr() + + elif config['USE_CHROME'] and config['TIMEOUT'] < 15: + stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red') + stderr(' Chrome will fail to archive all sites if set to less than ~15 seconds.') + stderr(' (Setting it to somewhere between 30 and 300 seconds is recommended)') + stderr() + stderr(' If you want to make ArchiveBox run faster, disable specific archive methods instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles') + stderr() + + if config['USE_YOUTUBEDL'] and config['MEDIA_TIMEOUT'] < 20: + stderr(f'[!] Warning: MEDIA_TIMEOUT is set too low! (currently set to MEDIA_TIMEOUT={config["MEDIA_TIMEOUT"]} seconds)', color='red') + stderr(' Youtube-dl will fail to archive all media if set to less than ~20 seconds.') + stderr(' (Setting it somewhere over 60 seconds is recommended)') + stderr() + stderr(' If you want to disable media archiving entirely, set SAVE_MEDIA=False instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_media') + stderr() + +def check_data_folder(out_dir: Union[str, Path, None]=None, config: ConfigDict=CONFIG) -> None: + output_dir = out_dir or config['OUTPUT_DIR'] + assert isinstance(output_dir, (str, Path)) + + sql_index_exists = (Path(output_dir) / SQL_INDEX_FILENAME).exists() + if not sql_index_exists: + stderr('[X] No archivebox index found in the current directory.', color='red') + stderr(f' {output_dir}', color='lightyellow') + stderr() + stderr(' {lightred}Hint{reset}: Are you running archivebox in the right folder?'.format(**config['ANSI'])) + stderr(' cd path/to/your/archive/folder') + stderr(' archivebox [command]') + stderr() + stderr(' {lightred}Hint{reset}: To create a new archive collection or import existing data in this folder, run:'.format(**config['ANSI'])) + stderr(' archivebox init') + raise SystemExit(2) + + from .index.sql import list_migrations + + pending_migrations = [name for status, name in list_migrations() if not status] + + if (not sql_index_exists) or pending_migrations: + if sql_index_exists: + pending_operation = f'apply the {len(pending_migrations)} pending migrations' + else: + pending_operation = 'generate the new SQL main index' + + stderr('[X] This collection was created with an older version of ArchiveBox and must be upgraded first.', color='lightyellow') + stderr(f' {output_dir}') + stderr() + stderr(f' To upgrade it to the latest version and {pending_operation} run:') + stderr(' archivebox init') + raise SystemExit(3) + + sources_dir = Path(output_dir) / SOURCES_DIR_NAME + if not sources_dir.exists(): + sources_dir.mkdir() + + + +def setup_django(out_dir: Path=None, check_db=False, config: ConfigDict=CONFIG, in_memory_db=False) -> None: + check_system_config() + + output_dir = out_dir or Path(config['OUTPUT_DIR']) + + assert isinstance(output_dir, Path) and isinstance(config['PACKAGE_DIR'], Path) + + try: + import django + sys.path.append(str(config['PACKAGE_DIR'])) + os.environ.setdefault('OUTPUT_DIR', str(output_dir)) + assert (config['PACKAGE_DIR'] / 'core' / 'settings.py').exists(), 'settings.py was not found at archivebox/core/settings.py' + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings') + + if in_memory_db: + # Put the db in memory and run migrations in case any command requires it + from django.core.management import call_command + os.environ.setdefault("ARCHIVEBOX_DATABASE_NAME", ":memory:") + django.setup() + call_command("migrate", interactive=False, verbosity=0) + else: + django.setup() + + if check_db: + sql_index_path = Path(output_dir) / SQL_INDEX_FILENAME + assert sql_index_path.exists(), ( + f'No database file {SQL_INDEX_FILENAME} found in OUTPUT_DIR: {config["OUTPUT_DIR"]}') + except KeyboardInterrupt: + raise SystemExit(2) diff --git a/archivebox-0.5.3/build/lib/archivebox/config_stubs.py b/archivebox-0.5.3/build/lib/archivebox/config_stubs.py new file mode 100644 index 0000000..988f58a --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/config_stubs.py @@ -0,0 +1,113 @@ +from pathlib import Path +from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any, List +from mypy_extensions import TypedDict + + + +SimpleConfigValue = Union[str, bool, int, None, Pattern, Dict[str, Any]] +SimpleConfigValueDict = Dict[str, SimpleConfigValue] +SimpleConfigValueGetter = Callable[[], SimpleConfigValue] +ConfigValue = Union[SimpleConfigValue, SimpleConfigValueDict, SimpleConfigValueGetter] + + +class BaseConfig(TypedDict): + pass + +class ConfigDict(BaseConfig, total=False): + """ + # Regenerate by pasting this quine into `archivebox shell` 🥚 + from archivebox.config import ConfigDict, CONFIG_DEFAULTS + print('class ConfigDict(BaseConfig, total=False):') + print(' ' + '"'*3 + ConfigDict.__doc__ + '"'*3) + for section, configs in CONFIG_DEFAULTS.items(): + for key, attrs in configs.items(): + Type, default = attrs['type'], attrs['default'] + if default is None: + print(f' {key}: Optional[{Type.__name__}]') + else: + print(f' {key}: {Type.__name__}') + print() + """ + IS_TTY: bool + USE_COLOR: bool + SHOW_PROGRESS: bool + IN_DOCKER: bool + + PACKAGE_DIR: Path + OUTPUT_DIR: Path + CONFIG_FILE: Path + ONLY_NEW: bool + TIMEOUT: int + MEDIA_TIMEOUT: int + OUTPUT_PERMISSIONS: str + RESTRICT_FILE_NAMES: str + URL_BLACKLIST: str + + SECRET_KEY: Optional[str] + BIND_ADDR: str + ALLOWED_HOSTS: str + DEBUG: bool + PUBLIC_INDEX: bool + PUBLIC_SNAPSHOTS: bool + FOOTER_INFO: str + ACTIVE_THEME: str + + SAVE_TITLE: bool + SAVE_FAVICON: bool + SAVE_WGET: bool + SAVE_WGET_REQUISITES: bool + SAVE_SINGLEFILE: bool + SAVE_READABILITY: bool + SAVE_MERCURY: bool + SAVE_PDF: bool + SAVE_SCREENSHOT: bool + SAVE_DOM: bool + SAVE_WARC: bool + SAVE_GIT: bool + SAVE_MEDIA: bool + SAVE_ARCHIVE_DOT_ORG: bool + + RESOLUTION: str + GIT_DOMAINS: str + CHECK_SSL_VALIDITY: bool + CURL_USER_AGENT: str + WGET_USER_AGENT: str + CHROME_USER_AGENT: str + COOKIES_FILE: Union[str, Path, None] + CHROME_USER_DATA_DIR: Union[str, Path, None] + CHROME_HEADLESS: bool + CHROME_SANDBOX: bool + + USE_CURL: bool + USE_WGET: bool + USE_SINGLEFILE: bool + USE_READABILITY: bool + USE_MERCURY: bool + USE_GIT: bool + USE_CHROME: bool + USE_YOUTUBEDL: bool + CURL_BINARY: str + GIT_BINARY: str + WGET_BINARY: str + SINGLEFILE_BINARY: str + READABILITY_BINARY: str + MERCURY_BINARY: str + YOUTUBEDL_BINARY: str + CHROME_BINARY: Optional[str] + + YOUTUBEDL_ARGS: List[str] + WGET_ARGS: List[str] + CURL_ARGS: List[str] + GIT_ARGS: List[str] + + +ConfigDefaultValueGetter = Callable[[ConfigDict], ConfigValue] +ConfigDefaultValue = Union[ConfigValue, ConfigDefaultValueGetter] + +ConfigDefault = TypedDict('ConfigDefault', { + 'default': ConfigDefaultValue, + 'type': Optional[Type], + 'aliases': Optional[Tuple[str, ...]], +}, total=False) + +ConfigDefaultDict = Dict[str, ConfigDefault] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/__init__.py b/archivebox-0.5.3/build/lib/archivebox/core/__init__.py new file mode 100644 index 0000000..3e1d607 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/__init__.py @@ -0,0 +1 @@ +__package__ = 'archivebox.core' diff --git a/archivebox-0.5.3/build/lib/archivebox/core/admin.py b/archivebox-0.5.3/build/lib/archivebox/core/admin.py new file mode 100644 index 0000000..832bea3 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/admin.py @@ -0,0 +1,257 @@ +__package__ = 'archivebox.core' + +from io import StringIO +from contextlib import redirect_stdout + +from django.contrib import admin +from django.urls import path +from django.utils.html import format_html +from django.utils.safestring import mark_safe +from django.shortcuts import render, redirect +from django.contrib.auth import get_user_model +from django import forms + +from core.models import Snapshot, Tag +from core.forms import AddLinkForm, TagField + +from core.mixins import SearchResultsAdminMixin + +from index.html import snapshot_icons +from util import htmldecode, urldecode, ansi_to_html +from logging_util import printable_filesize +from main import add, remove +from config import OUTPUT_DIR +from extractors import archive_links + +# TODO: https://stackoverflow.com/questions/40760880/add-custom-button-to-django-admin-panel + +def update_snapshots(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], out_dir=OUTPUT_DIR) +update_snapshots.short_description = "Archive" + +def update_titles(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], overwrite=True, methods=('title','favicon'), out_dir=OUTPUT_DIR) +update_titles.short_description = "Pull title" + +def overwrite_snapshots(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], overwrite=True, out_dir=OUTPUT_DIR) +overwrite_snapshots.short_description = "Re-archive (overwrite)" + +def verify_snapshots(modeladmin, request, queryset): + for snapshot in queryset: + print(snapshot.timestamp, snapshot.url, snapshot.is_archived, snapshot.archive_size, len(snapshot.history)) + +verify_snapshots.short_description = "Check" + +def delete_snapshots(modeladmin, request, queryset): + remove(snapshots=queryset, yes=True, delete=True, out_dir=OUTPUT_DIR) + +delete_snapshots.short_description = "Delete" + + +class SnapshotAdminForm(forms.ModelForm): + tags = TagField(required=False) + + class Meta: + model = Snapshot + fields = "__all__" + + def save(self, commit=True): + # Based on: https://stackoverflow.com/a/49933068/3509554 + + # Get the unsave instance + instance = forms.ModelForm.save(self, False) + tags = self.cleaned_data.pop("tags") + + #update save_m2m + def new_save_m2m(): + instance.save_tags(tags) + + # Do we need to save all changes now? + self.save_m2m = new_save_m2m + if commit: + instance.save() + + return instance + + +class SnapshotAdmin(SearchResultsAdminMixin, admin.ModelAdmin): + list_display = ('added', 'title_str', 'url_str', 'files', 'size') + sort_fields = ('title_str', 'url_str', 'added') + readonly_fields = ('id', 'url', 'timestamp', 'num_outputs', 'is_archived', 'url_hash', 'added', 'updated') + search_fields = ['url', 'timestamp', 'title', 'tags__name'] + fields = (*readonly_fields, 'title', 'tags') + list_filter = ('added', 'updated', 'tags') + ordering = ['-added'] + actions = [delete_snapshots, overwrite_snapshots, update_snapshots, update_titles, verify_snapshots] + actions_template = 'admin/actions_as_select.html' + form = SnapshotAdminForm + + def get_urls(self): + urls = super().get_urls() + custom_urls = [ + path('grid/', self.admin_site.admin_view(self.grid_view),name='grid') + ] + return custom_urls + urls + + def get_queryset(self, request): + return super().get_queryset(request).prefetch_related('tags') + + def tag_list(self, obj): + return ', '.join(obj.tags.values_list('name', flat=True)) + + def id_str(self, obj): + return format_html( + '{}', + obj.url_hash[:8], + ) + + def title_str(self, obj): + canon = obj.as_link().canonical_outputs() + tags = ''.join( + format_html('{} ', tag.id, tag) + for tag in obj.tags.all() + if str(tag).strip() + ) + return format_html( + '' + '' + '' + '' + '{}' + '', + obj.archive_path, + obj.archive_path, canon['favicon_path'], + obj.archive_path, + 'fetched' if obj.latest_title or obj.title else 'pending', + urldecode(htmldecode(obj.latest_title or obj.title or ''))[:128] or 'Pending...' + ) + mark_safe(f' {tags}') + + def files(self, obj): + return snapshot_icons(obj) + + def size(self, obj): + archive_size = obj.archive_size + if archive_size: + size_txt = printable_filesize(archive_size) + if archive_size > 52428800: + size_txt = mark_safe(f'{size_txt}') + else: + size_txt = mark_safe('...') + return format_html( + '{}', + obj.archive_path, + size_txt, + ) + + def url_str(self, obj): + return format_html( + '{}', + obj.url, + obj.url.split('://www.', 1)[-1].split('://', 1)[-1][:64], + ) + + def grid_view(self, request): + + # cl = self.get_changelist_instance(request) + + # Save before monkey patching to restore for changelist list view + saved_change_list_template = self.change_list_template + saved_list_per_page = self.list_per_page + saved_list_max_show_all = self.list_max_show_all + + # Monkey patch here plus core_tags.py + self.change_list_template = 'admin/grid_change_list.html' + self.list_per_page = 20 + self.list_max_show_all = self.list_per_page + + # Call monkey patched view + rendered_response = self.changelist_view(request) + + # Restore values + self.change_list_template = saved_change_list_template + self.list_per_page = saved_list_per_page + self.list_max_show_all = saved_list_max_show_all + + return rendered_response + + + id_str.short_description = 'ID' + title_str.short_description = 'Title' + url_str.short_description = 'Original URL' + + id_str.admin_order_field = 'id' + title_str.admin_order_field = 'title' + url_str.admin_order_field = 'url' + +class TagAdmin(admin.ModelAdmin): + list_display = ('slug', 'name', 'id') + sort_fields = ('id', 'name', 'slug') + readonly_fields = ('id',) + search_fields = ('id', 'name', 'slug') + fields = (*readonly_fields, 'name', 'slug') + + +class ArchiveBoxAdmin(admin.AdminSite): + site_header = 'ArchiveBox' + index_title = 'Links' + site_title = 'Index' + + def get_urls(self): + return [ + path('core/snapshot/add/', self.add_view, name='Add'), + ] + super().get_urls() + + def add_view(self, request): + if not request.user.is_authenticated: + return redirect(f'/admin/login/?next={request.path}') + + request.current_app = self.name + context = { + **self.each_context(request), + 'title': 'Add URLs', + } + + if request.method == 'GET': + context['form'] = AddLinkForm() + + elif request.method == 'POST': + form = AddLinkForm(request.POST) + if form.is_valid(): + url = form.cleaned_data["url"] + print(f'[+] Adding URL: {url}') + depth = 0 if form.cleaned_data["depth"] == "0" else 1 + input_kwargs = { + "urls": url, + "depth": depth, + "update_all": False, + "out_dir": OUTPUT_DIR, + } + add_stdout = StringIO() + with redirect_stdout(add_stdout): + add(**input_kwargs) + print(add_stdout.getvalue()) + + context.update({ + "stdout": ansi_to_html(add_stdout.getvalue().strip()), + "form": AddLinkForm() + }) + else: + context["form"] = form + + return render(template_name='add_links.html', request=request, context=context) + +admin.site = ArchiveBoxAdmin() +admin.site.register(get_user_model()) +admin.site.register(Snapshot, SnapshotAdmin) +admin.site.register(Tag, TagAdmin) +admin.site.disable_action('delete_selected') diff --git a/archivebox-0.5.3/build/lib/archivebox/core/apps.py b/archivebox-0.5.3/build/lib/archivebox/core/apps.py new file mode 100644 index 0000000..26f78a8 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class CoreConfig(AppConfig): + name = 'core' diff --git a/archivebox-0.5.3/build/lib/archivebox/core/forms.py b/archivebox-0.5.3/build/lib/archivebox/core/forms.py new file mode 100644 index 0000000..86b29bb --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/forms.py @@ -0,0 +1,67 @@ +__package__ = 'archivebox.core' + +from django import forms + +from ..util import URL_REGEX +from ..vendor.taggit_utils import edit_string_for_tags, parse_tags + +CHOICES = ( + ('0', 'depth = 0 (archive just these URLs)'), + ('1', 'depth = 1 (archive these URLs and all URLs one hop away)'), +) + +from ..extractors import get_default_archive_methods + +ARCHIVE_METHODS = [ + (name, name) + for name, _, _ in get_default_archive_methods() +] + + +class AddLinkForm(forms.Form): + url = forms.RegexField(label="URLs (one per line)", regex=URL_REGEX, min_length='6', strip=True, widget=forms.Textarea, required=True) + depth = forms.ChoiceField(label="Archive depth", choices=CHOICES, widget=forms.RadioSelect, initial='0') + archive_methods = forms.MultipleChoiceField( + required=False, + widget=forms.SelectMultiple, + choices=ARCHIVE_METHODS, + ) +class TagWidgetMixin: + def format_value(self, value): + if value is not None and not isinstance(value, str): + value = edit_string_for_tags(value) + return super().format_value(value) + +class TagWidget(TagWidgetMixin, forms.TextInput): + pass + +class TagField(forms.CharField): + widget = TagWidget + + def clean(self, value): + value = super().clean(value) + try: + return parse_tags(value) + except ValueError: + raise forms.ValidationError( + "Please provide a comma-separated list of tags." + ) + + def has_changed(self, initial_value, data_value): + # Always return False if the field is disabled since self.bound_data + # always uses the initial value in this case. + if self.disabled: + return False + + try: + data_value = self.clean(data_value) + except forms.ValidationError: + pass + + if initial_value is None: + initial_value = [] + + initial_value = [tag.name for tag in initial_value] + initial_value.sort() + + return initial_value != data_value diff --git a/archivebox-0.5.3/build/lib/archivebox/core/management/commands/archivebox.py b/archivebox-0.5.3/build/lib/archivebox/core/management/commands/archivebox.py new file mode 100644 index 0000000..a68b5d9 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/management/commands/archivebox.py @@ -0,0 +1,18 @@ +__package__ = 'archivebox' + +from django.core.management.base import BaseCommand + + +from .cli import run_subcommand + + +class Command(BaseCommand): + help = 'Run an ArchiveBox CLI subcommand (e.g. add, remove, list, etc)' + + def add_arguments(self, parser): + parser.add_argument('subcommand', type=str, help='The subcommand you want to run') + parser.add_argument('command_args', nargs='*', help='Arguments to pass to the subcommand') + + + def handle(self, *args, **kwargs): + run_subcommand(kwargs['subcommand'], args=kwargs['command_args']) diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0001_initial.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0001_initial.py new file mode 100644 index 0000000..73ac78e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0001_initial.py @@ -0,0 +1,27 @@ +# Generated by Django 2.2 on 2019-05-01 03:27 + +from django.db import migrations, models +import uuid + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ] + + operations = [ + migrations.CreateModel( + name='Snapshot', + fields=[ + ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)), + ('url', models.URLField(unique=True)), + ('timestamp', models.CharField(default=None, max_length=32, null=True, unique=True)), + ('title', models.CharField(default=None, max_length=128, null=True)), + ('tags', models.CharField(default=None, max_length=256, null=True)), + ('added', models.DateTimeField(auto_now_add=True)), + ('updated', models.DateTimeField(default=None, null=True)), + ], + ), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0002_auto_20200625_1521.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0002_auto_20200625_1521.py new file mode 100644 index 0000000..4811282 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0002_auto_20200625_1521.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.7 on 2020-06-25 15:21 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0001_initial'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(default=None, max_length=32, null=True), + ), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0003_auto_20200630_1034.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0003_auto_20200630_1034.py new file mode 100644 index 0000000..61fd472 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0003_auto_20200630_1034.py @@ -0,0 +1,38 @@ +# Generated by Django 3.0.7 on 2020-06-30 10:34 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0002_auto_20200625_1521'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='added', + field=models.DateTimeField(auto_now_add=True, db_index=True), + ), + migrations.AlterField( + model_name='snapshot', + name='tags', + field=models.CharField(db_index=True, default=None, max_length=256, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(db_index=True, default=None, max_length=32, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='title', + field=models.CharField(db_index=True, default=None, max_length=128, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='updated', + field=models.DateTimeField(db_index=True, default=None, null=True), + ), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0004_auto_20200713_1552.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0004_auto_20200713_1552.py new file mode 100644 index 0000000..6983662 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0004_auto_20200713_1552.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.7 on 2020-07-13 15:52 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0003_auto_20200630_1034'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(db_index=True, default=None, max_length=32, unique=True), + preserve_default=False, + ), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0005_auto_20200728_0326.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0005_auto_20200728_0326.py new file mode 100644 index 0000000..f367aeb --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0005_auto_20200728_0326.py @@ -0,0 +1,28 @@ +# Generated by Django 3.0.7 on 2020-07-28 03:26 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0004_auto_20200713_1552'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='tags', + field=models.CharField(blank=True, db_index=True, max_length=256, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='title', + field=models.CharField(blank=True, db_index=True, max_length=128, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='updated', + field=models.DateTimeField(blank=True, db_index=True, null=True), + ), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0006_auto_20201012_1520.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0006_auto_20201012_1520.py new file mode 100644 index 0000000..694c990 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0006_auto_20201012_1520.py @@ -0,0 +1,70 @@ +# Generated by Django 3.0.8 on 2020-10-12 15:20 + +from django.db import migrations, models +from django.utils.text import slugify + +def forwards_func(apps, schema_editor): + SnapshotModel = apps.get_model("core", "Snapshot") + TagModel = apps.get_model("core", "Tag") + + db_alias = schema_editor.connection.alias + snapshots = SnapshotModel.objects.all() + for snapshot in snapshots: + tags = snapshot.tags + tag_set = ( + set(tag.strip() for tag in (snapshot.tags_old or '').split(',')) + ) + tag_set.discard("") + + for tag in tag_set: + to_add, _ = TagModel.objects.get_or_create(name=tag, slug=slugify(tag)) + snapshot.tags.add(to_add) + + +def reverse_func(apps, schema_editor): + SnapshotModel = apps.get_model("core", "Snapshot") + TagModel = apps.get_model("core", "Tag") + + db_alias = schema_editor.connection.alias + snapshots = SnapshotModel.objects.all() + for snapshot in snapshots: + tags = snapshot.tags.values_list("name", flat=True) + snapshot.tags_old = ",".join([tag for tag in tags]) + snapshot.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0005_auto_20200728_0326'), + ] + + operations = [ + migrations.RenameField( + model_name='snapshot', + old_name='tags', + new_name='tags_old', + ), + migrations.CreateModel( + name='Tag', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('name', models.CharField(max_length=100, unique=True, verbose_name='name')), + ('slug', models.SlugField(max_length=100, unique=True, verbose_name='slug')), + ], + options={ + 'verbose_name': 'Tag', + 'verbose_name_plural': 'Tags', + }, + ), + migrations.AddField( + model_name='snapshot', + name='tags', + field=models.ManyToManyField(to='core.Tag'), + ), + migrations.RunPython(forwards_func, reverse_func), + migrations.RemoveField( + model_name='snapshot', + name='tags_old', + ), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0007_archiveresult.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0007_archiveresult.py new file mode 100644 index 0000000..a780376 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0007_archiveresult.py @@ -0,0 +1,97 @@ +# Generated by Django 3.0.8 on 2020-11-04 12:25 + +import json +from pathlib import Path + +from django.db import migrations, models +import django.db.models.deletion + +from config import CONFIG +from index.json import to_json + +try: + JSONField = models.JSONField +except AttributeError: + import jsonfield + JSONField = jsonfield.JSONField + + +def forwards_func(apps, schema_editor): + from core.models import EXTRACTORS + + Snapshot = apps.get_model("core", "Snapshot") + ArchiveResult = apps.get_model("core", "ArchiveResult") + + snapshots = Snapshot.objects.all() + for snapshot in snapshots: + out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp + + try: + with open(out_dir / "index.json", "r") as f: + fs_index = json.load(f) + except Exception as e: + continue + + history = fs_index["history"] + + for extractor in history: + for result in history[extractor]: + ArchiveResult.objects.create(extractor=extractor, snapshot=snapshot, cmd=result["cmd"], cmd_version=result["cmd_version"], + start_ts=result["start_ts"], end_ts=result["end_ts"], status=result["status"], pwd=result["pwd"], output=result["output"]) + + +def verify_json_index_integrity(snapshot): + results = snapshot.archiveresult_set.all() + out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp + with open(out_dir / "index.json", "r") as f: + index = json.load(f) + + history = index["history"] + index_results = [result for extractor in history for result in history[extractor]] + flattened_results = [result["start_ts"] for result in index_results] + + missing_results = [result for result in results if result.start_ts.isoformat() not in flattened_results] + + for missing in missing_results: + index["history"][missing.extractor].append({"cmd": missing.cmd, "cmd_version": missing.cmd_version, "end_ts": missing.end_ts.isoformat(), + "start_ts": missing.start_ts.isoformat(), "pwd": missing.pwd, "output": missing.output, + "schema": "ArchiveResult", "status": missing.status}) + + json_index = to_json(index) + with open(out_dir / "index.json", "w") as f: + f.write(json_index) + + +def reverse_func(apps, schema_editor): + Snapshot = apps.get_model("core", "Snapshot") + ArchiveResult = apps.get_model("core", "ArchiveResult") + for snapshot in Snapshot.objects.all(): + verify_json_index_integrity(snapshot) + + ArchiveResult.objects.all().delete() + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0006_auto_20201012_1520'), + ] + + operations = [ + migrations.CreateModel( + name='ArchiveResult', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('cmd', JSONField()), + ('pwd', models.CharField(max_length=256)), + ('cmd_version', models.CharField(max_length=32)), + ('status', models.CharField(choices=[('succeeded', 'succeeded'), ('failed', 'failed'), ('skipped', 'skipped')], max_length=16)), + ('output', models.CharField(max_length=512)), + ('start_ts', models.DateTimeField()), + ('end_ts', models.DateTimeField()), + ('extractor', models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('wget', 'wget'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('headers', 'headers'), ('archive_org', 'archive_org')], max_length=32)), + ('snapshot', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Snapshot')), + ], + ), + migrations.RunPython(forwards_func, reverse_func), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/0008_auto_20210105_1421.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0008_auto_20210105_1421.py new file mode 100644 index 0000000..e5b3387 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/migrations/0008_auto_20210105_1421.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.3 on 2021-01-05 14:21 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0007_archiveresult'), + ] + + operations = [ + migrations.AlterField( + model_name='archiveresult', + name='cmd_version', + field=models.CharField(blank=True, default=None, max_length=32, null=True), + ), + ] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/migrations/__init__.py b/archivebox-0.5.3/build/lib/archivebox/core/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/build/lib/archivebox/core/mixins.py b/archivebox-0.5.3/build/lib/archivebox/core/mixins.py new file mode 100644 index 0000000..538ca1e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/mixins.py @@ -0,0 +1,23 @@ +from django.contrib import messages + +from archivebox.search import query_search_index + +class SearchResultsAdminMixin(object): + def get_search_results(self, request, queryset, search_term): + ''' Enhances the search queryset with results from the search backend. + ''' + qs, use_distinct = \ + super(SearchResultsAdminMixin, self).get_search_results( + request, queryset, search_term) + + search_term = search_term.strip() + if not search_term: + return qs, use_distinct + try: + qsearch = query_search_index(search_term) + except Exception as err: + messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}') + else: + qs = queryset & qsearch + finally: + return qs, use_distinct diff --git a/archivebox-0.5.3/build/lib/archivebox/core/models.py b/archivebox-0.5.3/build/lib/archivebox/core/models.py new file mode 100644 index 0000000..13d75b6 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/models.py @@ -0,0 +1,194 @@ +__package__ = 'archivebox.core' + +import uuid + +from django.db import models, transaction +from django.utils.functional import cached_property +from django.utils.text import slugify +from django.db.models import Case, When, Value, IntegerField + +from ..util import parse_date +from ..index.schema import Link +from ..extractors import get_default_archive_methods, ARCHIVE_METHODS_INDEXING_PRECEDENCE + +EXTRACTORS = [(extractor[0], extractor[0]) for extractor in get_default_archive_methods()] +STATUS_CHOICES = [ + ("succeeded", "succeeded"), + ("failed", "failed"), + ("skipped", "skipped") +] + +try: + JSONField = models.JSONField +except AttributeError: + import jsonfield + JSONField = jsonfield.JSONField + + +class Tag(models.Model): + """ + Based on django-taggit model + """ + name = models.CharField(verbose_name="name", unique=True, blank=False, max_length=100) + slug = models.SlugField(verbose_name="slug", unique=True, max_length=100) + + class Meta: + verbose_name = "Tag" + verbose_name_plural = "Tags" + + def __str__(self): + return self.name + + def slugify(self, tag, i=None): + slug = slugify(tag) + if i is not None: + slug += "_%d" % i + return slug + + def save(self, *args, **kwargs): + if self._state.adding and not self.slug: + self.slug = self.slugify(self.name) + + with transaction.atomic(): + slugs = set( + type(self) + ._default_manager.filter(slug__startswith=self.slug) + .values_list("slug", flat=True) + ) + + i = None + while True: + slug = self.slugify(self.name, i) + if slug not in slugs: + self.slug = slug + return super().save(*args, **kwargs) + i = 1 if i is None else i+1 + else: + return super().save(*args, **kwargs) + + +class Snapshot(models.Model): + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) + + url = models.URLField(unique=True) + timestamp = models.CharField(max_length=32, unique=True, db_index=True) + + title = models.CharField(max_length=128, null=True, blank=True, db_index=True) + + added = models.DateTimeField(auto_now_add=True, db_index=True) + updated = models.DateTimeField(null=True, blank=True, db_index=True) + tags = models.ManyToManyField(Tag) + + keys = ('url', 'timestamp', 'title', 'tags', 'updated') + + def __repr__(self) -> str: + title = self.title or '-' + return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})' + + def __str__(self) -> str: + title = self.title or '-' + return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})' + + @classmethod + def from_json(cls, info: dict): + info = {k: v for k, v in info.items() if k in cls.keys} + return cls(**info) + + def as_json(self, *args) -> dict: + args = args or self.keys + return { + key: getattr(self, key) + if key != 'tags' else self.tags_str() + for key in args + } + + def as_link(self) -> Link: + return Link.from_json(self.as_json()) + + def as_link_with_details(self) -> Link: + from ..index import load_link_details + return load_link_details(self.as_link()) + + def tags_str(self) -> str: + return ','.join(self.tags.order_by('name').values_list('name', flat=True)) + + @cached_property + def bookmarked(self): + return parse_date(self.timestamp) + + @cached_property + def is_archived(self): + return self.as_link().is_archived + + @cached_property + def num_outputs(self): + return self.archiveresult_set.filter(status='succeeded').count() + + @cached_property + def url_hash(self): + return self.as_link().url_hash + + @cached_property + def base_url(self): + return self.as_link().base_url + + @cached_property + def link_dir(self): + return self.as_link().link_dir + + @cached_property + def archive_path(self): + return self.as_link().archive_path + + @cached_property + def archive_size(self): + return self.as_link().archive_size + + @cached_property + def history(self): + # TODO: use ArchiveResult for this instead of json + return self.as_link_with_details().history + + @cached_property + def latest_title(self): + if ('title' in self.history + and self.history['title'] + and (self.history['title'][-1].status == 'succeeded') + and self.history['title'][-1].output.strip()): + return self.history['title'][-1].output.strip() + return None + + def save_tags(self, tags=()): + tags_id = [] + for tag in tags: + tags_id.append(Tag.objects.get_or_create(name=tag)[0].id) + self.tags.clear() + self.tags.add(*tags_id) + + +class ArchiveResultManager(models.Manager): + def indexable(self, sorted: bool = True): + INDEXABLE_METHODS = [ r[0] for r in ARCHIVE_METHODS_INDEXING_PRECEDENCE ] + qs = self.get_queryset().filter(extractor__in=INDEXABLE_METHODS,status='succeeded') + + if sorted: + precedence = [ When(extractor=method, then=Value(precedence)) for method, precedence in ARCHIVE_METHODS_INDEXING_PRECEDENCE ] + qs = qs.annotate(indexing_precedence=Case(*precedence, default=Value(1000),output_field=IntegerField())).order_by('indexing_precedence') + return qs + + +class ArchiveResult(models.Model): + snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE) + cmd = JSONField() + pwd = models.CharField(max_length=256) + cmd_version = models.CharField(max_length=32, default=None, null=True, blank=True) + output = models.CharField(max_length=512) + start_ts = models.DateTimeField() + end_ts = models.DateTimeField() + status = models.CharField(max_length=16, choices=STATUS_CHOICES) + extractor = models.CharField(choices=EXTRACTORS, max_length=32) + + objects = ArchiveResultManager() + + def __str__(self): + return self.extractor diff --git a/archivebox-0.5.3/build/lib/archivebox/core/settings.py b/archivebox-0.5.3/build/lib/archivebox/core/settings.py new file mode 100644 index 0000000..e8ed6b1 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/settings.py @@ -0,0 +1,165 @@ +__package__ = 'archivebox.core' + +import os +import sys + +from pathlib import Path +from django.utils.crypto import get_random_string + +from ..config import ( # noqa: F401 + DEBUG, + SECRET_KEY, + ALLOWED_HOSTS, + PACKAGE_DIR, + ACTIVE_THEME, + TEMPLATES_DIR_NAME, + SQL_INDEX_FILENAME, + OUTPUT_DIR, +) + + +IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3] +IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ +IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3] + +################################################################################ +### Django Core Settings +################################################################################ + +WSGI_APPLICATION = 'core.wsgi.application' +ROOT_URLCONF = 'core.urls' + +LOGIN_URL = '/accounts/login/' +LOGOUT_REDIRECT_URL = '/' +PASSWORD_RESET_URL = '/accounts/password_reset/' +APPEND_SLASH = True + +INSTALLED_APPS = [ + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', + 'django.contrib.admin', + + 'core', + + 'django_extensions', +] + + +MIDDLEWARE = [ + 'django.middleware.security.SecurityMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.common.CommonMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', +] + +AUTHENTICATION_BACKENDS = [ + 'django.contrib.auth.backends.ModelBackend', +] + + +################################################################################ +### Staticfile and Template Settings +################################################################################ + +STATIC_URL = '/static/' + +STATICFILES_DIRS = [ + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / ACTIVE_THEME / 'static'), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'default' / 'static'), +] + +TEMPLATE_DIRS = [ + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / ACTIVE_THEME), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'default'), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME), +] + +TEMPLATES = [ + { + 'BACKEND': 'django.template.backends.django.DjangoTemplates', + 'DIRS': TEMPLATE_DIRS, + 'APP_DIRS': True, + 'OPTIONS': { + 'context_processors': [ + 'django.template.context_processors.debug', + 'django.template.context_processors.request', + 'django.contrib.auth.context_processors.auth', + 'django.contrib.messages.context_processors.messages', + ], + }, + }, +] + + +################################################################################ +### External Service Settings +################################################################################ + +DATABASE_FILE = Path(OUTPUT_DIR) / SQL_INDEX_FILENAME +DATABASE_NAME = os.environ.get("ARCHIVEBOX_DATABASE_NAME", DATABASE_FILE) + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': DATABASE_NAME, + } +} + +EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' + + +################################################################################ +### Security Settings +################################################################################ + +SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.') + +ALLOWED_HOSTS = ALLOWED_HOSTS.split(',') + +SECURE_BROWSER_XSS_FILTER = True +SECURE_CONTENT_TYPE_NOSNIFF = True + +CSRF_COOKIE_SECURE = False +SESSION_COOKIE_SECURE = False +SESSION_COOKIE_DOMAIN = None +SESSION_COOKIE_AGE = 1209600 # 2 weeks +SESSION_EXPIRE_AT_BROWSER_CLOSE = False +SESSION_SAVE_EVERY_REQUEST = True + +AUTH_PASSWORD_VALIDATORS = [ + {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'}, + {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'}, + {'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator'}, + {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'}, +] + + +################################################################################ +### Shell Settings +################################################################################ + +SHELL_PLUS = 'ipython' +SHELL_PLUS_PRINT_SQL = False +IPYTHON_ARGUMENTS = ['--no-confirm-exit', '--no-banner'] +IPYTHON_KERNEL_DISPLAY_NAME = 'ArchiveBox Django Shell' +if IS_SHELL: + os.environ['PYTHONSTARTUP'] = str(Path(PACKAGE_DIR) / 'core' / 'welcome_message.py') + + +################################################################################ +### Internationalization & Localization Settings +################################################################################ + +LANGUAGE_CODE = 'en-us' +TIME_ZONE = 'UTC' +USE_I18N = False +USE_L10N = False +USE_TZ = False + +DATETIME_FORMAT = 'Y-m-d g:iA' +SHORT_DATETIME_FORMAT = 'Y-m-d h:iA' diff --git a/archivebox-0.5.3/build/lib/archivebox/core/templatetags/__init__.py b/archivebox-0.5.3/build/lib/archivebox/core/templatetags/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/build/lib/archivebox/core/templatetags/core_tags.py b/archivebox-0.5.3/build/lib/archivebox/core/templatetags/core_tags.py new file mode 100644 index 0000000..25f0685 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/templatetags/core_tags.py @@ -0,0 +1,47 @@ +from django import template +from django.urls import reverse +from django.contrib.admin.templatetags.base import InclusionAdminNode +from django.templatetags.static import static + + +from typing import Union + +from core.models import ArchiveResult + +register = template.Library() + +@register.simple_tag +def snapshot_image(snapshot): + result = ArchiveResult.objects.filter(snapshot=snapshot, extractor='screenshot', status='succeeded').first() + if result: + return reverse('LinkAssets', args=[f'{str(snapshot.timestamp)}/{result.output}']) + + return static('archive.png') + +@register.filter +def file_size(num_bytes: Union[int, float]) -> str: + for count in ['Bytes','KB','MB','GB']: + if num_bytes > -1024.0 and num_bytes < 1024.0: + return '%3.1f %s' % (num_bytes, count) + num_bytes /= 1024.0 + return '%3.1f %s' % (num_bytes, 'TB') + +def result_list(cl): + """ + Monkey patched result + """ + num_sorted_fields = 0 + return { + 'cl': cl, + 'num_sorted_fields': num_sorted_fields, + 'results': cl.result_list, + } + +@register.tag(name='snapshots_grid') +def result_list_tag(parser, token): + return InclusionAdminNode( + parser, token, + func=result_list, + template_name='snapshots_grid.html', + takes_context=False, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/core/tests.py b/archivebox-0.5.3/build/lib/archivebox/core/tests.py new file mode 100644 index 0000000..4d66077 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/tests.py @@ -0,0 +1,3 @@ +#from django.test import TestCase + +# Create your tests here. diff --git a/archivebox-0.5.3/build/lib/archivebox/core/urls.py b/archivebox-0.5.3/build/lib/archivebox/core/urls.py new file mode 100644 index 0000000..b8e4baf --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/urls.py @@ -0,0 +1,36 @@ +from django.contrib import admin + +from django.urls import path, include +from django.views import static +from django.conf import settings +from django.views.generic.base import RedirectView + +from core.views import MainIndex, LinkDetails, PublicArchiveView, AddView + + +# print('DEBUG', settings.DEBUG) + +urlpatterns = [ + path('robots.txt', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'robots.txt'}), + path('favicon.ico', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'favicon.ico'}), + + path('docs/', RedirectView.as_view(url='https://github.com/ArchiveBox/ArchiveBox/wiki'), name='Docs'), + + path('archive/', RedirectView.as_view(url='/')), + path('archive/', LinkDetails.as_view(), name='LinkAssets'), + + path('admin/core/snapshot/add/', RedirectView.as_view(url='/add/')), + path('add/', AddView.as_view()), + + path('accounts/login/', RedirectView.as_view(url='/admin/login/')), + path('accounts/logout/', RedirectView.as_view(url='/admin/logout/')), + + + path('accounts/', include('django.contrib.auth.urls')), + path('admin/', admin.site.urls), + + path('index.html', RedirectView.as_view(url='/')), + path('index.json', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'index.json'}), + path('', MainIndex.as_view(), name='Home'), + path('public/', PublicArchiveView.as_view(), name='public-index'), +] diff --git a/archivebox-0.5.3/build/lib/archivebox/core/views.py b/archivebox-0.5.3/build/lib/archivebox/core/views.py new file mode 100644 index 0000000..b46e364 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/views.py @@ -0,0 +1,173 @@ +__package__ = 'archivebox.core' + +from io import StringIO +from contextlib import redirect_stdout + +from django.shortcuts import render, redirect + +from django.http import HttpResponse +from django.views import View, static +from django.views.generic.list import ListView +from django.views.generic import FormView +from django.contrib.auth.mixins import UserPassesTestMixin + +from core.models import Snapshot +from core.forms import AddLinkForm + +from ..config import ( + OUTPUT_DIR, + PUBLIC_INDEX, + PUBLIC_SNAPSHOTS, + PUBLIC_ADD_VIEW, + VERSION, + FOOTER_INFO, +) +from main import add +from ..util import base_url, ansi_to_html +from ..index.html import snapshot_icons + + +class MainIndex(View): + template = 'main_index.html' + + def get(self, request): + if request.user.is_authenticated: + return redirect('/admin/core/snapshot/') + + if PUBLIC_INDEX: + return redirect('public-index') + + return redirect(f'/admin/login/?next={request.path}') + + +class LinkDetails(View): + def get(self, request, path): + # missing trailing slash -> redirect to index + if '/' not in path: + return redirect(f'{path}/index.html') + + if not request.user.is_authenticated and not PUBLIC_SNAPSHOTS: + return redirect(f'/admin/login/?next={request.path}') + + try: + slug, archivefile = path.split('/', 1) + except (IndexError, ValueError): + slug, archivefile = path.split('/', 1)[0], 'index.html' + + all_pages = list(Snapshot.objects.all()) + + # slug is a timestamp + by_ts = {page.timestamp: page for page in all_pages} + try: + # print('SERVING STATICFILE', by_ts[slug].link_dir, request.path, path) + response = static.serve(request, archivefile, document_root=by_ts[slug].link_dir, show_indexes=True) + response["Link"] = f'<{by_ts[slug].url}>; rel="canonical"' + return response + except KeyError: + pass + + # slug is a hash + by_hash = {page.url_hash: page for page in all_pages} + try: + timestamp = by_hash[slug].timestamp + return redirect(f'/archive/{timestamp}/{archivefile}') + except KeyError: + pass + + # slug is a URL + by_url = {page.base_url: page for page in all_pages} + try: + # TODO: add multiple snapshot support by showing index of all snapshots + # for given url instead of redirecting to timestamp index + timestamp = by_url[base_url(path)].timestamp + return redirect(f'/archive/{timestamp}/index.html') + except KeyError: + pass + + return HttpResponse( + 'No archived link matches the given timestamp or hash.', + content_type="text/plain", + status=404, + ) + +class PublicArchiveView(ListView): + template = 'snapshot_list.html' + model = Snapshot + paginate_by = 100 + ordering = ['title'] + + def get_context_data(self, **kwargs): + return { + **super().get_context_data(**kwargs), + 'VERSION': VERSION, + 'FOOTER_INFO': FOOTER_INFO, + } + + def get_queryset(self, **kwargs): + qs = super().get_queryset(**kwargs) + query = self.request.GET.get('q') + if query: + qs = qs.filter(title__icontains=query) + for snapshot in qs: + snapshot.icons = snapshot_icons(snapshot) + return qs + + def get(self, *args, **kwargs): + if PUBLIC_INDEX or self.request.user.is_authenticated: + response = super().get(*args, **kwargs) + return response + else: + return redirect(f'/admin/login/?next={self.request.path}') + + +class AddView(UserPassesTestMixin, FormView): + template_name = "add_links.html" + form_class = AddLinkForm + + def get_initial(self): + """Prefill the AddLinkForm with the 'url' GET parameter""" + if self.request.method == 'GET': + url = self.request.GET.get('url', None) + if url: + return {'url': url} + else: + return super().get_initial() + + def test_func(self): + return PUBLIC_ADD_VIEW or self.request.user.is_authenticated + + def get_context_data(self, **kwargs): + return { + **super().get_context_data(**kwargs), + 'title': "Add URLs", + # We can't just call request.build_absolute_uri in the template, because it would include query parameters + 'absolute_add_path': self.request.build_absolute_uri(self.request.path), + 'VERSION': VERSION, + 'FOOTER_INFO': FOOTER_INFO, + } + + def form_valid(self, form): + url = form.cleaned_data["url"] + print(f'[+] Adding URL: {url}') + depth = 0 if form.cleaned_data["depth"] == "0" else 1 + extractors = ','.join(form.cleaned_data["archive_methods"]) + input_kwargs = { + "urls": url, + "depth": depth, + "update_all": False, + "out_dir": OUTPUT_DIR, + } + if extractors: + input_kwargs.update({"extractors": extractors}) + add_stdout = StringIO() + with redirect_stdout(add_stdout): + add(**input_kwargs) + print(add_stdout.getvalue()) + + context = self.get_context_data() + + context.update({ + "stdout": ansi_to_html(add_stdout.getvalue().strip()), + "form": AddLinkForm() + }) + return render(template_name=self.template_name, request=self.request, context=context) diff --git a/archivebox-0.5.3/build/lib/archivebox/core/welcome_message.py b/archivebox-0.5.3/build/lib/archivebox/core/welcome_message.py new file mode 100644 index 0000000..ed5d2d7 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/welcome_message.py @@ -0,0 +1,5 @@ +from archivebox.logging_util import log_shell_welcome_msg + + +if __name__ == '__main__': + log_shell_welcome_msg() diff --git a/archivebox-0.5.3/build/lib/archivebox/core/wsgi.py b/archivebox-0.5.3/build/lib/archivebox/core/wsgi.py new file mode 100644 index 0000000..f933afa --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/core/wsgi.py @@ -0,0 +1,16 @@ +""" +WSGI config for archivebox project. + +It exposes the WSGI callable as a module-level variable named ``application``. + +For more information on this file, see +https://docs.djangoproject.com/en/2.1/howto/deployment/wsgi/ +""" + +import os + +from django.core.wsgi import get_wsgi_application + +os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'archivebox.settings') + +application = get_wsgi_application() diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/__init__.py b/archivebox-0.5.3/build/lib/archivebox/extractors/__init__.py new file mode 100644 index 0000000..a4acef0 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/__init__.py @@ -0,0 +1,182 @@ +__package__ = 'archivebox.extractors' + +import os +from pathlib import Path + +from typing import Optional, List, Iterable, Union +from datetime import datetime +from django.db.models import QuerySet + +from ..index.schema import Link +from ..index.sql import write_link_to_sql_index +from ..index import ( + load_link_details, + write_link_details, +) +from ..util import enforce_types +from ..logging_util import ( + log_archiving_started, + log_archiving_paused, + log_archiving_finished, + log_link_archiving_started, + log_link_archiving_finished, + log_archive_method_started, + log_archive_method_finished, +) +from ..search import write_search_index + +from .title import should_save_title, save_title +from .favicon import should_save_favicon, save_favicon +from .wget import should_save_wget, save_wget +from .singlefile import should_save_singlefile, save_singlefile +from .readability import should_save_readability, save_readability +from .mercury import should_save_mercury, save_mercury +from .pdf import should_save_pdf, save_pdf +from .screenshot import should_save_screenshot, save_screenshot +from .dom import should_save_dom, save_dom +from .git import should_save_git, save_git +from .media import should_save_media, save_media +from .archive_org import should_save_archive_dot_org, save_archive_dot_org +from .headers import should_save_headers, save_headers + + +def get_default_archive_methods(): + return [ + ('title', should_save_title, save_title), + ('favicon', should_save_favicon, save_favicon), + ('wget', should_save_wget, save_wget), + ('singlefile', should_save_singlefile, save_singlefile), + ('pdf', should_save_pdf, save_pdf), + ('screenshot', should_save_screenshot, save_screenshot), + ('dom', should_save_dom, save_dom), + ('readability', should_save_readability, save_readability), #keep readability below wget and singlefile, as it depends on them + ('mercury', should_save_mercury, save_mercury), + ('git', should_save_git, save_git), + ('media', should_save_media, save_media), + ('headers', should_save_headers, save_headers), + ('archive_org', should_save_archive_dot_org, save_archive_dot_org), + ] + +ARCHIVE_METHODS_INDEXING_PRECEDENCE = [('readability', 1), ('singlefile', 2), ('dom', 3), ('wget', 4)] + +@enforce_types +def ignore_methods(to_ignore: List[str]): + ARCHIVE_METHODS = get_default_archive_methods() + methods = filter(lambda x: x[0] not in to_ignore, ARCHIVE_METHODS) + methods = map(lambda x: x[0], methods) + return list(methods) + +@enforce_types +def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> Link: + """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp""" + + # TODO: Remove when the input is changed to be a snapshot. Suboptimal approach. + from core.models import Snapshot, ArchiveResult + try: + snapshot = Snapshot.objects.get(url=link.url) # TODO: This will be unnecessary once everything is a snapshot + except Snapshot.DoesNotExist: + snapshot = write_link_to_sql_index(link) + + ARCHIVE_METHODS = get_default_archive_methods() + + if methods: + ARCHIVE_METHODS = [ + method for method in ARCHIVE_METHODS + if method[0] in methods + ] + + out_dir = out_dir or Path(link.link_dir) + try: + is_new = not Path(out_dir).exists() + if is_new: + os.makedirs(out_dir) + + link = load_link_details(link, out_dir=out_dir) + write_link_details(link, out_dir=out_dir, skip_sql_index=False) + log_link_archiving_started(link, out_dir, is_new) + link = link.overwrite(updated=datetime.now()) + stats = {'skipped': 0, 'succeeded': 0, 'failed': 0} + + for method_name, should_run, method_function in ARCHIVE_METHODS: + try: + if method_name not in link.history: + link.history[method_name] = [] + + if should_run(link, out_dir) or overwrite: + log_archive_method_started(method_name) + + result = method_function(link=link, out_dir=out_dir) + + link.history[method_name].append(result) + + stats[result.status] += 1 + log_archive_method_finished(result) + write_search_index(link=link, texts=result.index_texts) + ArchiveResult.objects.create(snapshot=snapshot, extractor=method_name, cmd=result.cmd, cmd_version=result.cmd_version, + output=result.output, pwd=result.pwd, start_ts=result.start_ts, end_ts=result.end_ts, status=result.status) + + else: + # print('{black} X {}{reset}'.format(method_name, **ANSI)) + stats['skipped'] += 1 + except Exception as e: + raise Exception('Exception in archive_methods.save_{}(Link(url={}))'.format( + method_name, + link.url, + )) from e + + # print(' ', stats) + + try: + latest_title = link.history['title'][-1].output.strip() + if latest_title and len(latest_title) >= len(link.title or ''): + link = link.overwrite(title=latest_title) + except Exception: + pass + + write_link_details(link, out_dir=out_dir, skip_sql_index=False) + + log_link_archiving_finished(link, link.link_dir, is_new, stats) + + except KeyboardInterrupt: + try: + write_link_details(link, out_dir=link.link_dir) + except: + pass + raise + + except Exception as err: + print(' ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err)) + raise + + return link + +@enforce_types +def archive_links(all_links: Union[Iterable[Link], QuerySet], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> List[Link]: + + if type(all_links) is QuerySet: + num_links: int = all_links.count() + get_link = lambda x: x.as_link() + all_links = all_links.iterator() + else: + num_links: int = len(all_links) + get_link = lambda x: x + + if num_links == 0: + return [] + + log_archiving_started(num_links) + idx: int = 0 + try: + for link in all_links: + idx += 1 + to_archive = get_link(link) + archive_link(to_archive, overwrite=overwrite, methods=methods, out_dir=Path(link.link_dir)) + except KeyboardInterrupt: + log_archiving_paused(num_links, idx, link.timestamp) + raise SystemExit(0) + except BaseException: + print() + raise + + log_archiving_finished(num_links) + return all_links diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/archive_org.py b/archivebox-0.5.3/build/lib/archivebox/extractors/archive_org.py new file mode 100644 index 0000000..f5598d6 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/archive_org.py @@ -0,0 +1,112 @@ +__package__ = 'archivebox.extractors' + + +from pathlib import Path +from typing import Optional, List, Dict, Tuple +from collections import defaultdict + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, +) +from ..config import ( + TIMEOUT, + CURL_ARGS, + CHECK_SSL_VALIDITY, + SAVE_ARCHIVE_DOT_ORG, + CURL_BINARY, + CURL_VERSION, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_archive_dot_org(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "archive.org.txt").exists(): + # if open(path, 'r').read().strip() != 'None': + return False + + return SAVE_ARCHIVE_DOT_ORG + +@enforce_types +def save_archive_dot_org(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """submit site to archive.org for archiving via their service, save returned archive url""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'archive.org.txt' + archive_org_url = None + submit_url = 'https://web.archive.org/save/{}'.format(link.url) + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--head', + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + submit_url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + content_location, errors = parse_archive_dot_org_response(result.stdout) + if content_location: + archive_org_url = content_location[0] + elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]: + archive_org_url = None + # raise ArchiveError('Archive.org denied by {}/robots.txt'.format(domain(link.url))) + elif errors: + raise ArchiveError(', '.join(errors)) + else: + raise ArchiveError('Failed to find "content-location" URL header in Archive.org response.') + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + if output and not isinstance(output, Exception): + # instead of writing None when archive.org rejects the url write the + # url to resubmit it to archive.org. This is so when the user visits + # the URL in person, it will attempt to re-archive it, and it'll show the + # nicer error message explaining why the url was rejected if it fails. + archive_org_url = archive_org_url or submit_url + with open(str(out_dir / output), 'w', encoding='utf-8') as f: + f.write(archive_org_url) + chmod_file('archive.org.txt', cwd=str(out_dir)) + output = archive_org_url + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) + +@enforce_types +def parse_archive_dot_org_response(response: bytes) -> Tuple[List[str], List[str]]: + # Parse archive.org response headers + headers: Dict[str, List[str]] = defaultdict(list) + + # lowercase all the header names and store in dict + for header in response.splitlines(): + if b':' not in header or not header.strip(): + continue + name, val = header.decode().split(':', 1) + headers[name.lower().strip()].append(val.strip()) + + # Get successful archive url in "content-location" header or any errors + content_location = headers.get('content-location', headers['location']) + errors = headers['x-archive-wayback-runtime-error'] + return content_location, errors + diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/dom.py b/archivebox-0.5.3/build/lib/archivebox/extractors/dom.py new file mode 100644 index 0000000..babbe71 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/dom.py @@ -0,0 +1,69 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file, atomic_write +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_DOM, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_dom(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / 'output.html').exists(): + return False + + return SAVE_DOM + +@enforce_types +def save_dom(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """print HTML of site to file using chrome --dump-html""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'output.html' + output_path = out_dir / output + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--dump-dom', + link.url + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + atomic_write(output_path, result.stdout) + + if result.returncode: + hints = result.stderr.decode() + raise ArchiveError('Failed to save DOM', hints) + + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/favicon.py b/archivebox-0.5.3/build/lib/archivebox/extractors/favicon.py new file mode 100644 index 0000000..5e7c1fb --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/favicon.py @@ -0,0 +1,64 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput +from ..system import chmod_file, run +from ..util import enforce_types, domain +from ..config import ( + TIMEOUT, + SAVE_FAVICON, + CURL_BINARY, + CURL_ARGS, + CURL_VERSION, + CHECK_SSL_VALIDITY, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_favicon(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if (Path(out_dir) / 'favicon.ico').exists(): + return False + + return SAVE_FAVICON + +@enforce_types +def save_favicon(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download site favicon from google's favicon api""" + + out_dir = out_dir or link.link_dir + output: ArchiveOutput = 'favicon.ico' + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--max-time', str(timeout), + '--output', str(output), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + 'https://www.google.com/s2/favicons?domain={}'.format(domain(link.url)), + ] + status = 'pending' + timer = TimedProgress(timeout, prefix=' ') + try: + run(cmd, cwd=str(out_dir), timeout=timeout) + chmod_file(output, cwd=str(out_dir)) + status = 'succeeded' + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/git.py b/archivebox-0.5.3/build/lib/archivebox/extractors/git.py new file mode 100644 index 0000000..fd20d4b --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/git.py @@ -0,0 +1,90 @@ +__package__ = 'archivebox.extractors' + + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + domain, + extension, + without_query, + without_fragment, +) +from ..config import ( + TIMEOUT, + SAVE_GIT, + GIT_BINARY, + GIT_ARGS, + GIT_VERSION, + GIT_DOMAINS, + CHECK_SSL_VALIDITY +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_git(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + if (out_dir / "git").exists(): + return False + + is_clonable_url = ( + (domain(link.url) in GIT_DOMAINS) + or (extension(link.url) == 'git') + ) + if not is_clonable_url: + return False + + return SAVE_GIT + + +@enforce_types +def save_git(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using git""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'git' + output_path = out_dir / output + output_path.mkdir(exist_ok=True) + cmd = [ + GIT_BINARY, + 'clone', + *GIT_ARGS, + *([] if CHECK_SSL_VALIDITY else ['-c', 'http.sslVerify=false']), + without_query(without_fragment(link.url)), + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(output_path), timeout=timeout + 1) + if result.returncode == 128: + # ignore failed re-download when the folder already exists + pass + elif result.returncode > 0: + hints = 'Got git response code: {}.'.format(result.returncode) + raise ArchiveError('Failed to save git clone', hints) + + chmod_file(output, cwd=str(out_dir)) + + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=GIT_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/headers.py b/archivebox-0.5.3/build/lib/archivebox/extractors/headers.py new file mode 100644 index 0000000..4e69dec --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/headers.py @@ -0,0 +1,69 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput +from ..system import atomic_write +from ..util import ( + enforce_types, + get_headers, +) +from ..config import ( + TIMEOUT, + CURL_BINARY, + CURL_ARGS, + CURL_USER_AGENT, + CURL_VERSION, + CHECK_SSL_VALIDITY, + SAVE_HEADERS +) +from ..logging_util import TimedProgress + +@enforce_types +def should_save_headers(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + + output = Path(out_dir or link.link_dir) / 'headers.json' + return not output.exists() and SAVE_HEADERS + + +@enforce_types +def save_headers(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """Download site headers""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() + output: ArchiveOutput = 'headers.json' + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--head', + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + link.url, + ] + try: + json_headers = get_headers(link.url, timeout=timeout) + output_folder.mkdir(exist_ok=True) + atomic_write(str(output_folder / "headers.json"), json_headers) + except (Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/media.py b/archivebox-0.5.3/build/lib/archivebox/extractors/media.py new file mode 100644 index 0000000..3792fd2 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/media.py @@ -0,0 +1,81 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, +) +from ..config import ( + MEDIA_TIMEOUT, + SAVE_MEDIA, + YOUTUBEDL_ARGS, + YOUTUBEDL_BINARY, + YOUTUBEDL_VERSION, + CHECK_SSL_VALIDITY +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_media(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or link.link_dir + + if is_static_file(link.url): + return False + + if (out_dir / "media").exists(): + return False + + return SAVE_MEDIA + +@enforce_types +def save_media(link: Link, out_dir: Optional[Path]=None, timeout: int=MEDIA_TIMEOUT) -> ArchiveResult: + """Download playlists or individual video, audio, and subtitles using youtube-dl""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'media' + output_path = out_dir / output + output_path.mkdir(exist_ok=True) + cmd = [ + YOUTUBEDL_BINARY, + *YOUTUBEDL_ARGS, + *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate']), + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(output_path), timeout=timeout + 1) + chmod_file(output, cwd=str(out_dir)) + if result.returncode: + if (b'ERROR: Unsupported URL' in result.stderr + or b'HTTP Error 404' in result.stderr + or b'HTTP Error 403' in result.stderr + or b'URL could be a direct video link' in result.stderr + or b'Unable to extract container ID' in result.stderr): + # These happen too frequently on non-media pages to warrant printing to console + pass + else: + hints = ( + 'Got youtube-dl response code: {}.'.format(result.returncode), + *result.stderr.decode().split('\n'), + ) + raise ArchiveError('Failed to save media', hints) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=YOUTUBEDL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/mercury.py b/archivebox-0.5.3/build/lib/archivebox/extractors/mercury.py new file mode 100644 index 0000000..741c329 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/mercury.py @@ -0,0 +1,104 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from subprocess import CompletedProcess +from typing import Optional, List +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, atomic_write +from ..util import ( + enforce_types, + is_static_file, + +) +from ..config import ( + TIMEOUT, + SAVE_MERCURY, + DEPENDENCIES, + MERCURY_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def ShellError(cmd: List[str], result: CompletedProcess, lines: int=20) -> ArchiveError: + # parse out last line of stderr + return ArchiveError( + f'Got {cmd[0]} response code: {result.returncode}).', + *( + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', lines)[-lines:] + if line.strip() + ), + ) + + +@enforce_types +def should_save_mercury(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + output = Path(out_dir or link.link_dir) / 'mercury' + return SAVE_MERCURY and MERCURY_VERSION and (not output.exists()) + + +@enforce_types +def save_mercury(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download reader friendly version using @postlight/mercury-parser""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() / "mercury" + output = str(output_folder) + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + # Get plain text version of article + cmd = [ + DEPENDENCIES['MERCURY_BINARY']['path'], + link.url, + "--format=text" + ] + result = run(cmd, cwd=out_dir, timeout=timeout) + try: + article_text = json.loads(result.stdout) + except json.JSONDecodeError: + raise ShellError(cmd, result) + + # Get HTML version of article + cmd = [ + DEPENDENCIES['MERCURY_BINARY']['path'], + link.url + ] + result = run(cmd, cwd=out_dir, timeout=timeout) + try: + article_json = json.loads(result.stdout) + except json.JSONDecodeError: + raise ShellError(cmd, result) + + output_folder.mkdir(exist_ok=True) + atomic_write(str(output_folder / "content.html"), article_json.pop("content")) + atomic_write(str(output_folder / "content.txt"), article_text["content"]) + atomic_write(str(output_folder / "article.json"), article_json) + + # Check for common failure cases + if (result.returncode > 0): + raise ShellError(cmd, result) + except (ArchiveError, Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=MERCURY_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/pdf.py b/archivebox-0.5.3/build/lib/archivebox/extractors/pdf.py new file mode 100644 index 0000000..1b0201e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/pdf.py @@ -0,0 +1,68 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_PDF, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_pdf(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "output.pdf").exists(): + return False + + return SAVE_PDF + + +@enforce_types +def save_pdf(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """print PDF of site to file using chrome --headless""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'output.pdf' + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--print-to-pdf', + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + if result.returncode: + hints = (result.stderr or result.stdout).decode() + raise ArchiveError('Failed to save PDF', hints) + + chmod_file('output.pdf', cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/readability.py b/archivebox-0.5.3/build/lib/archivebox/extractors/readability.py new file mode 100644 index 0000000..9da620b --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/readability.py @@ -0,0 +1,124 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from tempfile import NamedTemporaryFile + +from typing import Optional +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, atomic_write +from ..util import ( + enforce_types, + download_url, + is_static_file, + +) +from ..config import ( + TIMEOUT, + CURL_BINARY, + SAVE_READABILITY, + DEPENDENCIES, + READABILITY_VERSION, +) +from ..logging_util import TimedProgress + +@enforce_types +def get_html(link: Link, path: Path) -> str: + """ + Try to find wget, singlefile and then dom files. + If none is found, download the url again. + """ + canonical = link.canonical_outputs() + abs_path = path.absolute() + sources = [canonical["singlefile_path"], canonical["wget_path"], canonical["dom_path"]] + document = None + for source in sources: + try: + with open(abs_path / source, "r") as f: + document = f.read() + break + except (FileNotFoundError, TypeError): + continue + if document is None: + return download_url(link.url) + else: + return document + +@enforce_types +def should_save_readability(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + output = Path(out_dir or link.link_dir) / 'readability' + return SAVE_READABILITY and READABILITY_VERSION and (not output.exists()) + + +@enforce_types +def save_readability(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download reader friendly version using @mozilla/readability""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() / "readability" + output = str(output_folder) + + # Readability Docs: https://github.com/mozilla/readability + + status = 'succeeded' + # fake command to show the user so they have something to try debugging if get_html fails + cmd = [ + CURL_BINARY, + link.url + ] + readability_content = None + timer = TimedProgress(timeout, prefix=' ') + try: + document = get_html(link, out_dir) + temp_doc = NamedTemporaryFile(delete=False) + temp_doc.write(document.encode("utf-8")) + temp_doc.close() + + cmd = [ + DEPENDENCIES['READABILITY_BINARY']['path'], + temp_doc.name + ] + + result = run(cmd, cwd=out_dir, timeout=timeout) + result_json = json.loads(result.stdout) + output_folder.mkdir(exist_ok=True) + readability_content = result_json.pop("textContent") + atomic_write(str(output_folder / "content.html"), result_json.pop("content")) + atomic_write(str(output_folder / "content.txt"), readability_content) + atomic_write(str(output_folder / "article.json"), result_json) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + hints = ( + 'Got readability response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0): + raise ArchiveError('Readability was not able to archive the page', hints) + except (Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=READABILITY_VERSION, + output=output, + status=status, + index_texts= [readability_content] if readability_content else [], + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/screenshot.py b/archivebox-0.5.3/build/lib/archivebox/extractors/screenshot.py new file mode 100644 index 0000000..325584e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/screenshot.py @@ -0,0 +1,67 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_SCREENSHOT, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_screenshot(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "screenshot.png").exists(): + return False + + return SAVE_SCREENSHOT + +@enforce_types +def save_screenshot(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """take screenshot of site using chrome --headless""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'screenshot.png' + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--screenshot', + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + if result.returncode: + hints = (result.stderr or result.stdout).decode() + raise ArchiveError('Failed to save screenshot', hints) + + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/singlefile.py b/archivebox-0.5.3/build/lib/archivebox/extractors/singlefile.py new file mode 100644 index 0000000..2e5c389 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/singlefile.py @@ -0,0 +1,90 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_SINGLEFILE, + DEPENDENCIES, + SINGLEFILE_VERSION, + CHROME_BINARY, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_singlefile(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + output = out_dir / 'singlefile.html' + return SAVE_SINGLEFILE and SINGLEFILE_VERSION and (not output.exists()) + + +@enforce_types +def save_singlefile(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using single-file""" + + out_dir = out_dir or Path(link.link_dir) + output = str(out_dir.absolute() / "singlefile.html") + + browser_args = chrome_args(TIMEOUT=0) + + # SingleFile CLI Docs: https://github.com/gildas-lormeau/SingleFile/tree/master/cli + browser_args = '--browser-args={}'.format(json.dumps(browser_args[1:])) + cmd = [ + DEPENDENCIES['SINGLEFILE_BINARY']['path'], + '--browser-executable-path={}'.format(CHROME_BINARY), + browser_args, + link.url, + output + ] + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + hints = ( + 'Got single-file response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0): + raise ArchiveError('SingleFile was not able to archive the page', hints) + chmod_file(output) + except (Exception, OSError) as err: + status = 'failed' + # TODO: Make this prettier. This is necessary to run the command (escape JSON internal quotes). + cmd[2] = browser_args.replace('"', "\\\"") + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=SINGLEFILE_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/title.py b/archivebox-0.5.3/build/lib/archivebox/extractors/title.py new file mode 100644 index 0000000..28cb128 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/title.py @@ -0,0 +1,130 @@ +__package__ = 'archivebox.extractors' + +import re +from html.parser import HTMLParser +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..util import ( + enforce_types, + is_static_file, + download_url, + htmldecode, +) +from ..config import ( + TIMEOUT, + CHECK_SSL_VALIDITY, + SAVE_TITLE, + CURL_BINARY, + CURL_ARGS, + CURL_VERSION, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + + +HTML_TITLE_REGEX = re.compile( + r'' # start matching text after tag + r'(.[^<>]+)', # get everything up to these symbols + re.IGNORECASE | re.MULTILINE | re.DOTALL | re.UNICODE, +) + + +class TitleParser(HTMLParser): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.title_tag = "" + self.title_og = "" + self.inside_title_tag = False + + @property + def title(self): + return self.title_tag or self.title_og or None + + def handle_starttag(self, tag, attrs): + if tag.lower() == "title" and not self.title_tag: + self.inside_title_tag = True + elif tag.lower() == "meta" and not self.title_og: + attrs = dict(attrs) + if attrs.get("property") == "og:title" and attrs.get("content"): + self.title_og = attrs.get("content") + + def handle_data(self, data): + if self.inside_title_tag and data: + self.title_tag += data.strip() + + def handle_endtag(self, tag): + if tag.lower() == "title": + self.inside_title_tag = False + + +@enforce_types +def should_save_title(link: Link, out_dir: Optional[str]=None) -> bool: + # if link already has valid title, skip it + if link.title and not link.title.lower().startswith('http'): + return False + + if is_static_file(link.url): + return False + + return SAVE_TITLE + +def extract_title_with_regex(html): + match = re.search(HTML_TITLE_REGEX, html) + output = htmldecode(match.group(1).strip()) if match else None + return output + +@enforce_types +def save_title(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """try to guess the page's title from its content""" + + from core.models import Snapshot + + output: ArchiveOutput = None + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + html = download_url(link.url, timeout=timeout) + try: + # try using relatively strict html parser first + parser = TitleParser() + parser.feed(html) + output = parser.title + if output is None: + raise + except Exception: + # fallback to regex that can handle broken/malformed html + output = extract_title_with_regex(html) + + # if title is better than the one in the db, update db with new title + if isinstance(output, str) and output: + if not link.title or len(output) >= len(link.title): + Snapshot.objects.filter(url=link.url, + timestamp=link.timestamp)\ + .update(title=output) + else: + raise ArchiveError('Unable to detect page title') + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/extractors/wget.py b/archivebox-0.5.3/build/lib/archivebox/extractors/wget.py new file mode 100644 index 0000000..331f636 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/extractors/wget.py @@ -0,0 +1,184 @@ +__package__ = 'archivebox.extractors' + +import re +from pathlib import Path + +from typing import Optional +from datetime import datetime + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + without_scheme, + without_fragment, + without_query, + path, + domain, + urldecode, +) +from ..config import ( + WGET_ARGS, + TIMEOUT, + SAVE_WGET, + SAVE_WARC, + WGET_BINARY, + WGET_VERSION, + RESTRICT_FILE_NAMES, + CHECK_SSL_VALIDITY, + SAVE_WGET_REQUISITES, + WGET_AUTO_COMPRESSION, + WGET_USER_AGENT, + COOKIES_FILE, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_wget(link: Link, out_dir: Optional[Path]=None) -> bool: + output_path = wget_output_path(link) + out_dir = out_dir or Path(link.link_dir) + if output_path and (out_dir / output_path).exists(): + return False + + return SAVE_WGET + + +@enforce_types +def save_wget(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using wget""" + + out_dir = out_dir or link.link_dir + if SAVE_WARC: + warc_dir = out_dir / "warc" + warc_dir.mkdir(exist_ok=True) + warc_path = warc_dir / str(int(datetime.now().timestamp())) + + # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html + output: ArchiveOutput = None + cmd = [ + WGET_BINARY, + # '--server-response', # print headers for better error parsing + *WGET_ARGS, + '--timeout={}'.format(timeout), + *(['--restrict-file-names={}'.format(RESTRICT_FILE_NAMES)] if RESTRICT_FILE_NAMES else []), + *(['--warc-file={}'.format(str(warc_path))] if SAVE_WARC else []), + *(['--page-requisites'] if SAVE_WGET_REQUISITES else []), + *(['--user-agent={}'.format(WGET_USER_AGENT)] if WGET_USER_AGENT else []), + *(['--load-cookies', COOKIES_FILE] if COOKIES_FILE else []), + *(['--compression=auto'] if WGET_AUTO_COMPRESSION else []), + *([] if SAVE_WARC else ['--timestamping']), + *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate', '--no-hsts']), + link.url, + ] + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + output = wget_output_path(link) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + files_downloaded = ( + int(output_tail[-1].strip().split(' ', 2)[1] or 0) + if 'Downloaded:' in output_tail[-1] + else 0 + ) + hints = ( + 'Got wget response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0 and files_downloaded < 1) or output is None: + if b'403: Forbidden' in result.stderr: + raise ArchiveError('403 Forbidden (try changing WGET_USER_AGENT)', hints) + if b'404: Not Found' in result.stderr: + raise ArchiveError('404 Not Found', hints) + if b'ERROR 500: Internal Server Error' in result.stderr: + raise ArchiveError('500 Internal Server Error', hints) + raise ArchiveError('Wget failed or got an error from the server', hints) + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=WGET_VERSION, + output=output, + status=status, + **timer.stats, + ) + + +@enforce_types +def wget_output_path(link: Link) -> Optional[str]: + """calculate the path to the wgetted .html file, since wget may + adjust some paths to be different than the base_url path. + + See docs on wget --adjust-extension (-E) + """ + if is_static_file(link.url): + return without_scheme(without_fragment(link.url)) + + # Wget downloads can save in a number of different ways depending on the url: + # https://example.com + # > example.com/index.html + # https://example.com?v=zzVa_tX1OiI + # > example.com/index.html?v=zzVa_tX1OiI.html + # https://www.example.com/?v=zzVa_tX1OiI + # > example.com/index.html?v=zzVa_tX1OiI.html + + # https://example.com/abc + # > example.com/abc.html + # https://example.com/abc/ + # > example.com/abc/index.html + # https://example.com/abc?v=zzVa_tX1OiI.html + # > example.com/abc?v=zzVa_tX1OiI.html + # https://example.com/abc/?v=zzVa_tX1OiI.html + # > example.com/abc/index.html?v=zzVa_tX1OiI.html + + # https://example.com/abc/test.html + # > example.com/abc/test.html + # https://example.com/abc/test?v=zzVa_tX1OiI + # > example.com/abc/test?v=zzVa_tX1OiI.html + # https://example.com/abc/test/?v=zzVa_tX1OiI + # > example.com/abc/test/index.html?v=zzVa_tX1OiI.html + + # There's also lots of complexity around how the urlencoding and renaming + # is done for pages with query and hash fragments or extensions like shtml / htm / php / etc + + # Since the wget algorithm for -E (appending .html) is incredibly complex + # and there's no way to get the computed output path from wget + # in order to avoid having to reverse-engineer how they calculate it, + # we just look in the output folder read the filename wget used from the filesystem + full_path = without_fragment(without_query(path(link.url))).strip('/') + search_dir = Path(link.link_dir) / domain(link.url).replace(":", "+") / urldecode(full_path) + for _ in range(4): + if search_dir.exists(): + if search_dir.is_dir(): + html_files = [ + f for f in search_dir.iterdir() + if re.search(".+\\.[Ss]?[Hh][Tt][Mm][Ll]?$", str(f), re.I | re.M) + ] + if html_files: + return str(html_files[0].relative_to(link.link_dir)) + + # Move up one directory level + search_dir = search_dir.parent + + if str(search_dir) == link.link_dir: + break + + return None diff --git a/archivebox-0.5.3/build/lib/archivebox/index/__init__.py b/archivebox-0.5.3/build/lib/archivebox/index/__init__.py new file mode 100644 index 0000000..8eab1d3 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/index/__init__.py @@ -0,0 +1,617 @@ +__package__ = 'archivebox.index' + +import os +import shutil +import json as pyjson +from pathlib import Path + +from itertools import chain +from typing import List, Tuple, Dict, Optional, Iterable +from collections import OrderedDict +from contextlib import contextmanager +from urllib.parse import urlparse +from django.db.models import QuerySet, Q + +from ..util import ( + scheme, + enforce_types, + ExtendedEncoder, +) +from ..config import ( + ARCHIVE_DIR_NAME, + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + OUTPUT_DIR, + TIMEOUT, + URL_BLACKLIST_PTN, + stderr, + OUTPUT_PERMISSIONS +) +from ..logging_util import ( + TimedProgress, + log_indexing_process_started, + log_indexing_process_finished, + log_indexing_started, + log_indexing_finished, + log_parsing_finished, + log_deduping_finished, +) + +from .schema import Link, ArchiveResult +from .html import ( + write_html_link_details, +) +from .json import ( + parse_json_link_details, + write_json_link_details, +) +from .sql import ( + write_sql_main_index, + write_sql_link_details, +) + +from ..search import search_backend_enabled, query_search_index + +### Link filtering and checking + +@enforce_types +def merge_links(a: Link, b: Link) -> Link: + """deterministially merge two links, favoring longer field values over shorter, + and "cleaner" values over worse ones. + """ + assert a.base_url == b.base_url, f'Cannot merge two links with different URLs ({a.base_url} != {b.base_url})' + + # longest url wins (because a fuzzy url will always be shorter) + url = a.url if len(a.url) > len(b.url) else b.url + + # best title based on length and quality + possible_titles = [ + title + for title in (a.title, b.title) + if title and title.strip() and '://' not in title + ] + title = None + if len(possible_titles) == 2: + title = max(possible_titles, key=lambda t: len(t)) + elif len(possible_titles) == 1: + title = possible_titles[0] + + # earliest valid timestamp + timestamp = ( + a.timestamp + if float(a.timestamp or 0) < float(b.timestamp or 0) else + b.timestamp + ) + + # all unique, truthy tags + tags_set = ( + set(tag.strip() for tag in (a.tags or '').split(',')) + | set(tag.strip() for tag in (b.tags or '').split(',')) + ) + tags = ','.join(tags_set) or None + + # all unique source entries + sources = list(set(a.sources + b.sources)) + + # all unique history entries for the combined archive methods + all_methods = set(list(a.history.keys()) + list(a.history.keys())) + history = { + method: (a.history.get(method) or []) + (b.history.get(method) or []) + for method in all_methods + } + for method in all_methods: + deduped_jsons = { + pyjson.dumps(result, sort_keys=True, cls=ExtendedEncoder) + for result in history[method] + } + history[method] = list(reversed(sorted( + (ArchiveResult.from_json(pyjson.loads(result)) for result in deduped_jsons), + key=lambda result: result.start_ts, + ))) + + return Link( + url=url, + timestamp=timestamp, + title=title, + tags=tags, + sources=sources, + history=history, + ) + + +@enforce_types +def validate_links(links: Iterable[Link]) -> List[Link]: + timer = TimedProgress(TIMEOUT * 4) + try: + links = archivable_links(links) # remove chrome://, about:, mailto: etc. + links = sorted_links(links) # deterministically sort the links based on timestamp, url + links = fix_duplicate_links(links) # merge/dedupe duplicate timestamps & urls + finally: + timer.end() + + return list(links) + +@enforce_types +def archivable_links(links: Iterable[Link]) -> Iterable[Link]: + """remove chrome://, about:// or other schemed links that cant be archived""" + for link in links: + try: + urlparse(link.url) + except ValueError: + continue + if scheme(link.url) not in ('http', 'https', 'ftp'): + continue + if URL_BLACKLIST_PTN and URL_BLACKLIST_PTN.search(link.url): + continue + + yield link + + +@enforce_types +def fix_duplicate_links(sorted_links: Iterable[Link]) -> Iterable[Link]: + """ + ensures that all non-duplicate links have monotonically increasing timestamps + """ + # from core.models import Snapshot + + unique_urls: OrderedDict[str, Link] = OrderedDict() + + for link in sorted_links: + if link.url in unique_urls: + # merge with any other links that share the same url + link = merge_links(unique_urls[link.url], link) + unique_urls[link.url] = link + + return unique_urls.values() + + +@enforce_types +def sorted_links(links: Iterable[Link]) -> Iterable[Link]: + sort_func = lambda link: (link.timestamp.split('.', 1)[0], link.url) + return sorted(links, key=sort_func, reverse=True) + + +@enforce_types +def links_after_timestamp(links: Iterable[Link], resume: Optional[float]=None) -> Iterable[Link]: + if not resume: + yield from links + return + + for link in links: + try: + if float(link.timestamp) <= resume: + yield link + except (ValueError, TypeError): + print('Resume value and all timestamp values must be valid numbers.') + + +@enforce_types +def lowest_uniq_timestamp(used_timestamps: OrderedDict, timestamp: str) -> str: + """resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2""" + + timestamp = timestamp.split('.')[0] + nonce = 0 + + # first try 152323423 before 152323423.0 + if timestamp not in used_timestamps: + return timestamp + + new_timestamp = '{}.{}'.format(timestamp, nonce) + while new_timestamp in used_timestamps: + nonce += 1 + new_timestamp = '{}.{}'.format(timestamp, nonce) + + return new_timestamp + + + +### Main Links Index + +@contextmanager +@enforce_types +def timed_index_update(out_path: Path): + log_indexing_started(out_path) + timer = TimedProgress(TIMEOUT * 2, prefix=' ') + try: + yield + finally: + timer.end() + + assert out_path.exists(), f'Failed to write index file: {out_path}' + log_indexing_finished(out_path) + + +@enforce_types +def write_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None: + """Writes links to sqlite3 file for a given list of links""" + + log_indexing_process_started(len(links)) + + try: + with timed_index_update(out_dir / SQL_INDEX_FILENAME): + write_sql_main_index(links, out_dir=out_dir) + os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes + + except (KeyboardInterrupt, SystemExit): + stderr('[!] Warning: Still writing index to disk...', color='lightyellow') + stderr(' Run archivebox init to fix any inconsistencies from an ungraceful exit.') + with timed_index_update(out_dir / SQL_INDEX_FILENAME): + write_sql_main_index(links, out_dir=out_dir) + os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes + raise SystemExit(0) + + log_indexing_process_finished() + +@enforce_types +def load_main_index(out_dir: Path=OUTPUT_DIR, warn: bool=True) -> List[Link]: + """parse and load existing index with any new links from import_path merged in""" + from core.models import Snapshot + try: + return Snapshot.objects.all() + + except (KeyboardInterrupt, SystemExit): + raise SystemExit(0) + +@enforce_types +def load_main_index_meta(out_dir: Path=OUTPUT_DIR) -> Optional[dict]: + index_path = out_dir / JSON_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + meta_dict = pyjson.load(f) + meta_dict.pop('links') + return meta_dict + + return None + + +@enforce_types +def parse_links_from_source(source_path: str, root_url: Optional[str]=None) -> Tuple[List[Link], List[Link]]: + + from ..parsers import parse_links + + new_links: List[Link] = [] + + # parse and validate the import file + raw_links, parser_name = parse_links(source_path, root_url=root_url) + new_links = validate_links(raw_links) + + if parser_name: + num_parsed = len(raw_links) + log_parsing_finished(num_parsed, parser_name) + + return new_links + +@enforce_types +def fix_duplicate_links_in_index(snapshots: QuerySet, links: Iterable[Link]) -> Iterable[Link]: + """ + Given a list of in-memory Links, dedupe and merge them with any conflicting Snapshots in the DB. + """ + unique_urls: OrderedDict[str, Link] = OrderedDict() + + for link in links: + index_link = snapshots.filter(url=link.url) + if index_link: + link = merge_links(index_link[0].as_link(), link) + + unique_urls[link.url] = link + + return unique_urls.values() + +@enforce_types +def dedupe_links(snapshots: QuerySet, + new_links: List[Link]) -> List[Link]: + """ + The validation of links happened at a different stage. This method will + focus on actual deduplication and timestamp fixing. + """ + + # merge existing links in out_dir and new links + dedup_links = fix_duplicate_links_in_index(snapshots, new_links) + + new_links = [ + link for link in new_links + if not snapshots.filter(url=link.url).exists() + ] + + dedup_links_dict = {link.url: link for link in dedup_links} + + # Replace links in new_links with the dedup version + for i in range(len(new_links)): + if new_links[i].url in dedup_links_dict.keys(): + new_links[i] = dedup_links_dict[new_links[i].url] + log_deduping_finished(len(new_links)) + + return new_links + +### Link Details Index + +@enforce_types +def write_link_details(link: Link, out_dir: Optional[str]=None, skip_sql_index: bool=False) -> None: + out_dir = out_dir or link.link_dir + + write_json_link_details(link, out_dir=out_dir) + write_html_link_details(link, out_dir=out_dir) + if not skip_sql_index: + write_sql_link_details(link) + + +@enforce_types +def load_link_details(link: Link, out_dir: Optional[str]=None) -> Link: + """check for an existing link archive in the given directory, + and load+merge it into the given link dict + """ + out_dir = out_dir or link.link_dir + + existing_link = parse_json_link_details(out_dir) + if existing_link: + return merge_links(existing_link, link) + + return link + + + +LINK_FILTERS = { + 'exact': lambda pattern: Q(url=pattern), + 'substring': lambda pattern: Q(url__icontains=pattern), + 'regex': lambda pattern: Q(url__iregex=pattern), + 'domain': lambda pattern: Q(url__istartswith=f"http://{pattern}") | Q(url__istartswith=f"https://{pattern}") | Q(url__istartswith=f"ftp://{pattern}"), + 'tag': lambda pattern: Q(tags__name=pattern), +} + +@enforce_types +def q_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet: + q_filter = Q() + for pattern in filter_patterns: + try: + q_filter = q_filter | LINK_FILTERS[filter_type](pattern) + except KeyError: + stderr() + stderr( + f'[X] Got invalid pattern for --filter-type={filter_type}:', + color='red', + ) + stderr(f' {pattern}') + raise SystemExit(2) + return snapshots.filter(q_filter) + +def search_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='search') -> QuerySet: + if not search_backend_enabled(): + stderr() + stderr( + '[X] The search backend is not enabled, set config.USE_SEARCHING_BACKEND = True', + color='red', + ) + raise SystemExit(2) + from core.models import Snapshot + + qsearch = Snapshot.objects.none() + for pattern in filter_patterns: + try: + qsearch |= query_search_index(pattern) + except: + raise SystemExit(2) + + return snapshots & qsearch + +@enforce_types +def snapshot_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet: + if filter_type != 'search': + return q_filter(snapshots, filter_patterns, filter_type) + else: + return search_filter(snapshots, filter_patterns, filter_type) + + +def get_indexed_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links without checking archive status or data directory validity""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in links + } + +def get_archived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links that are archived with a valid data directory""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_archived, links) + } + +def get_unarchived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links that are unarchived with no data directory or an empty data directory""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_unarchived, links) + } + +def get_present_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that actually exist in the archive/ folder""" + + all_folders = {} + + for entry in (out_dir / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + link = None + try: + link = parse_json_link_details(entry.path) + except Exception: + pass + + all_folders[entry.name] = link + + return all_folders + +def get_valid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs with a valid index matched to the main index and archived content""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_valid, links) + } + +def get_invalid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized""" + duplicate = get_duplicate_folders(snapshots, out_dir=OUTPUT_DIR) + orphaned = get_orphaned_folders(snapshots, out_dir=OUTPUT_DIR) + corrupted = get_corrupted_folders(snapshots, out_dir=OUTPUT_DIR) + unrecognized = get_unrecognized_folders(snapshots, out_dir=OUTPUT_DIR) + return {**duplicate, **orphaned, **corrupted, **unrecognized} + + +def get_duplicate_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that conflict with other directories that have the same link URL or timestamp""" + by_url = {} + by_timestamp = {} + duplicate_folders = {} + + data_folders = ( + str(entry) + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir() + if entry.is_dir() and not snapshots.filter(timestamp=entry.name).exists() + ) + + for path in chain(snapshots.iterator(), data_folders): + link = None + if type(path) is not str: + path = path.as_link().link_dir + + try: + link = parse_json_link_details(path) + except Exception: + pass + + if link: + # link folder has same timestamp as different link folder + by_timestamp[link.timestamp] = by_timestamp.get(link.timestamp, 0) + 1 + if by_timestamp[link.timestamp] > 1: + duplicate_folders[path] = link + + # link folder has same url as different link folder + by_url[link.url] = by_url.get(link.url, 0) + 1 + if by_url[link.url] > 1: + duplicate_folders[path] = link + return duplicate_folders + +def get_orphaned_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that contain a valid index but aren't listed in the main index""" + orphaned_folders = {} + + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + link = None + try: + link = parse_json_link_details(str(entry)) + except Exception: + pass + + if link and not snapshots.filter(timestamp=entry.name).exists(): + # folder is a valid link data dir with index details, but it's not in the main index + orphaned_folders[str(entry)] = link + + return orphaned_folders + +def get_corrupted_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that don't contain a valid index and aren't listed in the main index""" + corrupted = {} + for snapshot in snapshots.iterator(): + link = snapshot.as_link() + if is_corrupt(link): + corrupted[link.link_dir] = link + return corrupted + +def get_unrecognized_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that don't contain recognizable archive data and aren't listed in the main index""" + unrecognized_folders: Dict[str, Optional[Link]] = {} + + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + index_exists = (entry / "index.json").exists() + link = None + try: + link = parse_json_link_details(str(entry)) + except KeyError: + # Try to fix index + if index_exists: + try: + # Last attempt to repair the detail index + link_guessed = parse_json_link_details(str(entry), guess=True) + write_json_link_details(link_guessed, out_dir=str(entry)) + link = parse_json_link_details(str(entry)) + except Exception: + pass + + if index_exists and link is None: + # index exists but it's corrupted or unparseable + unrecognized_folders[str(entry)] = link + + elif not index_exists: + # link details index doesn't exist and the folder isn't in the main index + timestamp = entry.name + if not snapshots.filter(timestamp=timestamp).exists(): + unrecognized_folders[str(entry)] = link + + return unrecognized_folders + + +def is_valid(link: Link) -> bool: + dir_exists = Path(link.link_dir).exists() + index_exists = (Path(link.link_dir) / "index.json").exists() + if not dir_exists: + # unarchived links are not included in the valid list + return False + if dir_exists and not index_exists: + return False + if dir_exists and index_exists: + try: + parsed_link = parse_json_link_details(link.link_dir, guess=True) + return link.url == parsed_link.url + except Exception: + pass + return False + +def is_corrupt(link: Link) -> bool: + if not Path(link.link_dir).exists(): + # unarchived links are not considered corrupt + return False + + if is_valid(link): + return False + + return True + +def is_archived(link: Link) -> bool: + return is_valid(link) and link.is_archived + +def is_unarchived(link: Link) -> bool: + if not Path(link.link_dir).exists(): + return True + return not link.is_archived + + +def fix_invalid_folder_locations(out_dir: Path=OUTPUT_DIR) -> Tuple[List[str], List[str]]: + fixed = [] + cant_fix = [] + for entry in os.scandir(out_dir / ARCHIVE_DIR_NAME): + if entry.is_dir(follow_symlinks=True): + if (Path(entry.path) / 'index.json').exists(): + try: + link = parse_json_link_details(entry.path) + except KeyError: + link = None + if not link: + continue + + if not entry.path.endswith(f'/{link.timestamp}'): + dest = out_dir / ARCHIVE_DIR_NAME / link.timestamp + if dest.exists(): + cant_fix.append(entry.path) + else: + shutil.move(entry.path, dest) + fixed.append(dest) + timestamp = entry.path.rsplit('/', 1)[-1] + assert link.link_dir == entry.path + assert link.timestamp == timestamp + write_json_link_details(link, out_dir=entry.path) + + return fixed, cant_fix diff --git a/archivebox-0.5.3/build/lib/archivebox/index/csv.py b/archivebox-0.5.3/build/lib/archivebox/index/csv.py new file mode 100644 index 0000000..804e646 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/index/csv.py @@ -0,0 +1,37 @@ +__package__ = 'archivebox.index' + +from typing import List, Optional, Any + +from ..util import enforce_types +from .schema import Link + + +@enforce_types +def links_to_csv(links: List[Link], + cols: Optional[List[str]]=None, + header: bool=True, + separator: str=',', + ljust: int=0) -> str: + + cols = cols or ['timestamp', 'is_archived', 'url'] + + header_str = '' + if header: + header_str = separator.join(col.ljust(ljust) for col in cols) + + row_strs = ( + link.to_csv(cols=cols, ljust=ljust, separator=separator) + for link in links + ) + + return '\n'.join((header_str, *row_strs)) + + +@enforce_types +def to_csv(obj: Any, cols: List[str], separator: str=',', ljust: int=0) -> str: + from .json import to_json + + return separator.join( + to_json(getattr(obj, col), indent=None).ljust(ljust) + for col in cols + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/index/html.py b/archivebox-0.5.3/build/lib/archivebox/index/html.py new file mode 100644 index 0000000..a62e2c7 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/index/html.py @@ -0,0 +1,164 @@ +__package__ = 'archivebox.index' + +from datetime import datetime +from typing import List, Optional, Iterator, Mapping +from pathlib import Path + +from django.utils.html import format_html +from collections import defaultdict + +from .schema import Link +from ..system import atomic_write +from ..logging_util import printable_filesize +from ..util import ( + enforce_types, + ts_to_date, + urlencode, + htmlencode, + urldecode, +) +from ..config import ( + OUTPUT_DIR, + VERSION, + GIT_SHA, + FOOTER_INFO, + HTML_INDEX_FILENAME, +) + +MAIN_INDEX_TEMPLATE = 'main_index.html' +MINIMAL_INDEX_TEMPLATE = 'main_index_minimal.html' +LINK_DETAILS_TEMPLATE = 'link_details.html' +TITLE_LOADING_MSG = 'Not yet archived...' + + +### Main Links Index + +@enforce_types +def parse_html_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[str]: + """parse an archive index html file and return the list of urls""" + + index_path = Path(out_dir) / HTML_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + for line in f: + if 'class="link-url"' in line: + yield line.split('"')[1] + return () + +@enforce_types +def generate_index_from_links(links: List[Link], with_headers: bool): + if with_headers: + output = main_index_template(links) + else: + output = main_index_template(links, template=MINIMAL_INDEX_TEMPLATE) + return output + +@enforce_types +def main_index_template(links: List[Link], template: str=MAIN_INDEX_TEMPLATE) -> str: + """render the template for the entire main index""" + + return render_django_template(template, { + 'version': VERSION, + 'git_sha': GIT_SHA, + 'num_links': str(len(links)), + 'date_updated': datetime.now().strftime('%Y-%m-%d'), + 'time_updated': datetime.now().strftime('%Y-%m-%d %H:%M'), + 'links': [link._asdict(extended=True) for link in links], + 'FOOTER_INFO': FOOTER_INFO, + }) + + +### Link Details Index + +@enforce_types +def write_html_link_details(link: Link, out_dir: Optional[str]=None) -> None: + out_dir = out_dir or link.link_dir + + rendered_html = link_details_template(link) + atomic_write(str(Path(out_dir) / HTML_INDEX_FILENAME), rendered_html) + + +@enforce_types +def link_details_template(link: Link) -> str: + + from ..extractors.wget import wget_output_path + + link_info = link._asdict(extended=True) + + return render_django_template(LINK_DETAILS_TEMPLATE, { + **link_info, + **link_info['canonical'], + 'title': htmlencode( + link.title + or (link.base_url if link.is_archived else TITLE_LOADING_MSG) + ), + 'url_str': htmlencode(urldecode(link.base_url)), + 'archive_url': urlencode( + wget_output_path(link) + or (link.domain if link.is_archived else '') + ) or 'about:blank', + 'extension': link.extension or 'html', + 'tags': link.tags or 'untagged', + 'size': printable_filesize(link.archive_size) if link.archive_size else 'pending', + 'status': 'archived' if link.is_archived else 'not yet archived', + 'status_color': 'success' if link.is_archived else 'danger', + 'oldest_archive_date': ts_to_date(link.oldest_archive_date), + }) + +@enforce_types +def render_django_template(template: str, context: Mapping[str, str]) -> str: + """render a given html template string with the given template content""" + from django.template.loader import render_to_string + + return render_to_string(template, context) + + +def snapshot_icons(snapshot) -> str: + from core.models import EXTRACTORS + + archive_results = snapshot.archiveresult_set.filter(status="succeeded") + link = snapshot.as_link() + path = link.archive_path + canon = link.canonical_outputs() + output = "" + output_template = '<a href="/{}/{}" class="exists-{}" title="{}">{} </a>' + icons = { + "singlefile": "❶", + "wget": "🆆", + "dom": "🅷", + "pdf": "📄", + "screenshot": "💻", + "media": "📼", + "git": "🅶", + "archive_org": "🏛", + "readability": "🆁", + "mercury": "🅼", + "warc": "📦" + } + exclude = ["favicon", "title", "headers", "archive_org"] + # Missing specific entry for WARC + + extractor_items = defaultdict(lambda: None) + for extractor, _ in EXTRACTORS: + for result in archive_results: + if result.extractor == extractor: + extractor_items[extractor] = result + + for extractor, _ in EXTRACTORS: + if extractor not in exclude: + exists = extractor_items[extractor] is not None + output += output_template.format(path, canon[f"{extractor}_path"], str(exists), + extractor, icons.get(extractor, "?")) + if extractor == "wget": + # warc isn't technically it's own extractor, so we have to add it after wget + exists = list((Path(path) / canon["warc_path"]).glob("*.warc.gz")) + output += output_template.format(exists[0] if exists else '#', canon["warc_path"], str(bool(exists)), "warc", icons.get("warc", "?")) + + if extractor == "archive_org": + # The check for archive_org is different, so it has to be handled separately + target_path = Path(path) / "archive.org.txt" + exists = target_path.exists() + output += '<a href="{}" class="exists-{}" title="{}">{}</a> '.format(canon["archive_org_path"], str(exists), + "archive_org", icons.get("archive_org", "?")) + + return format_html(f'<span class="files-icons" style="font-size: 1.1em; opacity: 0.8">{output}<span>') diff --git a/archivebox-0.5.3/build/lib/archivebox/index/json.py b/archivebox-0.5.3/build/lib/archivebox/index/json.py new file mode 100644 index 0000000..f24b969 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/index/json.py @@ -0,0 +1,154 @@ +__package__ = 'archivebox.index' + +import os +import sys +import json as pyjson +from pathlib import Path + +from datetime import datetime +from typing import List, Optional, Iterator, Any, Union + +from .schema import Link +from ..system import atomic_write +from ..util import enforce_types +from ..config import ( + VERSION, + OUTPUT_DIR, + FOOTER_INFO, + GIT_SHA, + DEPENDENCIES, + JSON_INDEX_FILENAME, + ARCHIVE_DIR_NAME, + ANSI +) + + +MAIN_INDEX_HEADER = { + 'info': 'This is an index of site data archived by ArchiveBox: The self-hosted web archive.', + 'schema': 'archivebox.index.json', + 'copyright_info': FOOTER_INFO, + 'meta': { + 'project': 'ArchiveBox', + 'version': VERSION, + 'git_sha': GIT_SHA, + 'website': 'https://ArchiveBox.io', + 'docs': 'https://github.com/ArchiveBox/ArchiveBox/wiki', + 'source': 'https://github.com/ArchiveBox/ArchiveBox', + 'issues': 'https://github.com/ArchiveBox/ArchiveBox/issues', + 'dependencies': DEPENDENCIES, + }, +} + +@enforce_types +def generate_json_index_from_links(links: List[Link], with_headers: bool): + if with_headers: + output = { + **MAIN_INDEX_HEADER, + 'num_links': len(links), + 'updated': datetime.now(), + 'last_run_cmd': sys.argv, + 'links': links, + } + else: + output = links + return to_json(output, indent=4, sort_keys=True) + + +@enforce_types +def parse_json_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]: + """parse an archive index json file and return the list of links""" + + index_path = Path(out_dir) / JSON_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + links = pyjson.load(f)['links'] + for link_json in links: + try: + yield Link.from_json(link_json) + except KeyError: + try: + detail_index_path = Path(OUTPUT_DIR) / ARCHIVE_DIR_NAME / link_json['timestamp'] + yield parse_json_link_details(str(detail_index_path)) + except KeyError: + # as a last effort, try to guess the missing values out of existing ones + try: + yield Link.from_json(link_json, guess=True) + except KeyError: + print(" {lightyellow}! Failed to load the index.json from {}".format(detail_index_path, **ANSI)) + continue + return () + +### Link Details Index + +@enforce_types +def write_json_link_details(link: Link, out_dir: Optional[str]=None) -> None: + """write a json file with some info about the link""" + + out_dir = out_dir or link.link_dir + path = Path(out_dir) / JSON_INDEX_FILENAME + atomic_write(str(path), link._asdict(extended=True)) + + +@enforce_types +def parse_json_link_details(out_dir: Union[Path, str], guess: Optional[bool]=False) -> Optional[Link]: + """load the json link index from a given directory""" + existing_index = Path(out_dir) / JSON_INDEX_FILENAME + if existing_index.exists(): + with open(existing_index, 'r', encoding='utf-8') as f: + try: + link_json = pyjson.load(f) + return Link.from_json(link_json, guess) + except pyjson.JSONDecodeError: + pass + return None + + +@enforce_types +def parse_json_links_details(out_dir: Union[Path, str]) -> Iterator[Link]: + """read through all the archive data folders and return the parsed links""" + + for entry in os.scandir(Path(out_dir) / ARCHIVE_DIR_NAME): + if entry.is_dir(follow_symlinks=True): + if (Path(entry.path) / 'index.json').exists(): + try: + link = parse_json_link_details(entry.path) + except KeyError: + link = None + if link: + yield link + + + +### Helpers + +class ExtendedEncoder(pyjson.JSONEncoder): + """ + Extended json serializer that supports serializing several model + fields and objects + """ + + def default(self, obj): + cls_name = obj.__class__.__name__ + + if hasattr(obj, '_asdict'): + return obj._asdict() + + elif isinstance(obj, bytes): + return obj.decode() + + elif isinstance(obj, datetime): + return obj.isoformat() + + elif isinstance(obj, Exception): + return '{}: {}'.format(obj.__class__.__name__, obj) + + elif cls_name in ('dict_items', 'dict_keys', 'dict_values'): + return tuple(obj) + + return pyjson.JSONEncoder.default(self, obj) + + +@enforce_types +def to_json(obj: Any, indent: Optional[int]=4, sort_keys: bool=True, cls=ExtendedEncoder) -> str: + return pyjson.dumps(obj, indent=indent, sort_keys=sort_keys, cls=ExtendedEncoder) + diff --git a/archivebox-0.5.3/build/lib/archivebox/index/schema.py b/archivebox-0.5.3/build/lib/archivebox/index/schema.py new file mode 100644 index 0000000..bc3a25d --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/index/schema.py @@ -0,0 +1,448 @@ +""" + +WARNING: THIS FILE IS ALL LEGACY CODE TO BE REMOVED. + +DO NOT ADD ANY NEW FEATURES TO THIS FILE, NEW CODE GOES HERE: core/models.py + +""" + +__package__ = 'archivebox.index' + +from pathlib import Path + +from datetime import datetime, timedelta + +from typing import List, Dict, Any, Optional, Union + +from dataclasses import dataclass, asdict, field, fields + + +from ..system import get_dir_size + +from ..config import OUTPUT_DIR, ARCHIVE_DIR_NAME + +class ArchiveError(Exception): + def __init__(self, message, hints=None): + super().__init__(message) + self.hints = hints + +LinkDict = Dict[str, Any] + +ArchiveOutput = Union[str, Exception, None] + +@dataclass(frozen=True) +class ArchiveResult: + cmd: List[str] + pwd: Optional[str] + cmd_version: Optional[str] + output: ArchiveOutput + status: str + start_ts: datetime + end_ts: datetime + index_texts: Union[List[str], None] = None + schema: str = 'ArchiveResult' + + def __post_init__(self): + self.typecheck() + + def _asdict(self): + return asdict(self) + + def typecheck(self) -> None: + assert self.schema == self.__class__.__name__ + assert isinstance(self.status, str) and self.status + assert isinstance(self.start_ts, datetime) + assert isinstance(self.end_ts, datetime) + assert isinstance(self.cmd, list) + assert all(isinstance(arg, str) and arg for arg in self.cmd) + assert self.pwd is None or isinstance(self.pwd, str) and self.pwd + assert self.cmd_version is None or isinstance(self.cmd_version, str) and self.cmd_version + assert self.output is None or isinstance(self.output, (str, Exception)) + if isinstance(self.output, str): + assert self.output + + @classmethod + def guess_ts(_cls, dict_info): + from ..util import parse_date + parsed_timestamp = parse_date(dict_info["timestamp"]) + start_ts = parsed_timestamp + end_ts = parsed_timestamp + timedelta(seconds=int(dict_info["duration"])) + return start_ts, end_ts + + @classmethod + def from_json(cls, json_info, guess=False): + from ..util import parse_date + + info = { + key: val + for key, val in json_info.items() + if key in cls.field_names() + } + if guess: + keys = info.keys() + if "start_ts" not in keys: + info["start_ts"], info["end_ts"] = cls.guess_ts(json_info) + else: + info['start_ts'] = parse_date(info['start_ts']) + info['end_ts'] = parse_date(info['end_ts']) + if "pwd" not in keys: + info["pwd"] = str(Path(OUTPUT_DIR) / ARCHIVE_DIR_NAME / json_info["timestamp"]) + if "cmd_version" not in keys: + info["cmd_version"] = "Undefined" + if "cmd" not in keys: + info["cmd"] = [] + else: + info['start_ts'] = parse_date(info['start_ts']) + info['end_ts'] = parse_date(info['end_ts']) + info['cmd_version'] = info.get('cmd_version') + if type(info["cmd"]) is str: + info["cmd"] = [info["cmd"]] + return cls(**info) + + def to_dict(self, *keys) -> dict: + if keys: + return {k: v for k, v in asdict(self).items() if k in keys} + return asdict(self) + + def to_json(self, indent=4, sort_keys=True) -> str: + from .json import to_json + + return to_json(self, indent=indent, sort_keys=sort_keys) + + def to_csv(self, cols: Optional[List[str]]=None, separator: str=',', ljust: int=0) -> str: + from .csv import to_csv + + return to_csv(self, csv_col=cols or self.field_names(), separator=separator, ljust=ljust) + + @classmethod + def field_names(cls): + return [f.name for f in fields(cls)] + + @property + def duration(self) -> int: + return (self.end_ts - self.start_ts).seconds + +@dataclass(frozen=True) +class Link: + timestamp: str + url: str + title: Optional[str] + tags: Optional[str] + sources: List[str] + history: Dict[str, List[ArchiveResult]] = field(default_factory=lambda: {}) + updated: Optional[datetime] = None + schema: str = 'Link' + + + def __str__(self) -> str: + return f'[{self.timestamp}] {self.url} "{self.title}"' + + def __post_init__(self): + self.typecheck() + + def overwrite(self, **kwargs): + """pure functional version of dict.update that returns a new instance""" + return Link(**{**self._asdict(), **kwargs}) + + def __eq__(self, other): + if not isinstance(other, Link): + return NotImplemented + return self.url == other.url + + def __gt__(self, other): + if not isinstance(other, Link): + return NotImplemented + if not self.timestamp or not other.timestamp: + return + return float(self.timestamp) > float(other.timestamp) + + def typecheck(self) -> None: + from ..config import stderr, ANSI + try: + assert self.schema == self.__class__.__name__ + assert isinstance(self.timestamp, str) and self.timestamp + assert self.timestamp.replace('.', '').isdigit() + assert isinstance(self.url, str) and '://' in self.url + assert self.updated is None or isinstance(self.updated, datetime) + assert self.title is None or (isinstance(self.title, str) and self.title) + assert self.tags is None or isinstance(self.tags, str) + assert isinstance(self.sources, list) + assert all(isinstance(source, str) and source for source in self.sources) + assert isinstance(self.history, dict) + for method, results in self.history.items(): + assert isinstance(method, str) and method + assert isinstance(results, list) + assert all(isinstance(result, ArchiveResult) for result in results) + except Exception: + stderr('{red}[X] Error while loading link! [{}] {} "{}"{reset}'.format(self.timestamp, self.url, self.title, **ANSI)) + raise + + def _asdict(self, extended=False): + info = { + 'schema': 'Link', + 'url': self.url, + 'title': self.title or None, + 'timestamp': self.timestamp, + 'updated': self.updated or None, + 'tags': self.tags or None, + 'sources': self.sources or [], + 'history': self.history or {}, + } + if extended: + info.update({ + 'link_dir': self.link_dir, + 'archive_path': self.archive_path, + + 'hash': self.url_hash, + 'base_url': self.base_url, + 'scheme': self.scheme, + 'domain': self.domain, + 'path': self.path, + 'basename': self.basename, + 'extension': self.extension, + 'is_static': self.is_static, + + 'bookmarked_date': self.bookmarked_date, + 'updated_date': self.updated_date, + 'oldest_archive_date': self.oldest_archive_date, + 'newest_archive_date': self.newest_archive_date, + + 'is_archived': self.is_archived, + 'num_outputs': self.num_outputs, + 'num_failures': self.num_failures, + + 'latest': self.latest_outputs(), + 'canonical': self.canonical_outputs(), + }) + return info + + def as_snapshot(self): + from core.models import Snapshot + return Snapshot.objects.get(url=self.url) + + @classmethod + def from_json(cls, json_info, guess=False): + from ..util import parse_date + + info = { + key: val + for key, val in json_info.items() + if key in cls.field_names() + } + info['updated'] = parse_date(info.get('updated')) + info['sources'] = info.get('sources') or [] + + json_history = info.get('history') or {} + cast_history = {} + + for method, method_history in json_history.items(): + cast_history[method] = [] + for json_result in method_history: + assert isinstance(json_result, dict), 'Items in Link["history"][method] must be dicts' + cast_result = ArchiveResult.from_json(json_result, guess) + cast_history[method].append(cast_result) + + info['history'] = cast_history + return cls(**info) + + def to_json(self, indent=4, sort_keys=True) -> str: + from .json import to_json + + return to_json(self, indent=indent, sort_keys=sort_keys) + + def to_csv(self, cols: Optional[List[str]]=None, separator: str=',', ljust: int=0) -> str: + from .csv import to_csv + + return to_csv(self, cols=cols or self.field_names(), separator=separator, ljust=ljust) + + @classmethod + def field_names(cls): + return [f.name for f in fields(cls)] + + @property + def link_dir(self) -> str: + from ..config import CONFIG + return str(Path(CONFIG['ARCHIVE_DIR']) / self.timestamp) + + @property + def archive_path(self) -> str: + from ..config import ARCHIVE_DIR_NAME + return '{}/{}'.format(ARCHIVE_DIR_NAME, self.timestamp) + + @property + def archive_size(self) -> float: + try: + return get_dir_size(self.archive_path)[0] + except Exception: + return 0 + + ### URL Helpers + @property + def url_hash(self): + from ..util import hashurl + + return hashurl(self.url) + + @property + def scheme(self) -> str: + from ..util import scheme + return scheme(self.url) + + @property + def extension(self) -> str: + from ..util import extension + return extension(self.url) + + @property + def domain(self) -> str: + from ..util import domain + return domain(self.url) + + @property + def path(self) -> str: + from ..util import path + return path(self.url) + + @property + def basename(self) -> str: + from ..util import basename + return basename(self.url) + + @property + def base_url(self) -> str: + from ..util import base_url + return base_url(self.url) + + ### Pretty Printing Helpers + @property + def bookmarked_date(self) -> Optional[str]: + from ..util import ts_to_date + + max_ts = (datetime.now() + timedelta(days=30)).timestamp() + + if self.timestamp and self.timestamp.replace('.', '').isdigit(): + if 0 < float(self.timestamp) < max_ts: + return ts_to_date(datetime.fromtimestamp(float(self.timestamp))) + else: + return str(self.timestamp) + return None + + + @property + def updated_date(self) -> Optional[str]: + from ..util import ts_to_date + return ts_to_date(self.updated) if self.updated else None + + @property + def archive_dates(self) -> List[datetime]: + return [ + result.start_ts + for method in self.history.keys() + for result in self.history[method] + ] + + @property + def oldest_archive_date(self) -> Optional[datetime]: + return min(self.archive_dates, default=None) + + @property + def newest_archive_date(self) -> Optional[datetime]: + return max(self.archive_dates, default=None) + + ### Archive Status Helpers + @property + def num_outputs(self) -> int: + return self.as_snapshot().num_outputs + + @property + def num_failures(self) -> int: + return sum(1 + for method in self.history.keys() + for result in self.history[method] + if result.status == 'failed') + + @property + def is_static(self) -> bool: + from ..util import is_static_file + return is_static_file(self.url) + + @property + def is_archived(self) -> bool: + from ..config import ARCHIVE_DIR + from ..util import domain + + output_paths = ( + domain(self.url), + 'output.pdf', + 'screenshot.png', + 'output.html', + 'media', + 'singlefile.html' + ) + + return any( + (Path(ARCHIVE_DIR) / self.timestamp / path).exists() + for path in output_paths + ) + + def latest_outputs(self, status: str=None) -> Dict[str, ArchiveOutput]: + """get the latest output that each archive method produced for link""" + + ARCHIVE_METHODS = ( + 'title', 'favicon', 'wget', 'warc', 'singlefile', 'pdf', + 'screenshot', 'dom', 'git', 'media', 'archive_org', + ) + latest: Dict[str, ArchiveOutput] = {} + for archive_method in ARCHIVE_METHODS: + # get most recent succesful result in history for each archive method + history = self.history.get(archive_method) or [] + history = list(filter(lambda result: result.output, reversed(history))) + if status is not None: + history = list(filter(lambda result: result.status == status, history)) + + history = list(history) + if history: + latest[archive_method] = history[0].output + else: + latest[archive_method] = None + return latest + + + def canonical_outputs(self) -> Dict[str, Optional[str]]: + """predict the expected output paths that should be present after archiving""" + + from ..extractors.wget import wget_output_path + canonical = { + 'index_path': 'index.html', + 'favicon_path': 'favicon.ico', + 'google_favicon_path': 'https://www.google.com/s2/favicons?domain={}'.format(self.domain), + 'wget_path': wget_output_path(self), + 'warc_path': 'warc', + 'singlefile_path': 'singlefile.html', + 'readability_path': 'readability/content.html', + 'mercury_path': 'mercury/content.html', + 'pdf_path': 'output.pdf', + 'screenshot_path': 'screenshot.png', + 'dom_path': 'output.html', + 'archive_org_path': 'https://web.archive.org/web/{}'.format(self.base_url), + 'git_path': 'git', + 'media_path': 'media', + } + if self.is_static: + # static binary files like PDF and images are handled slightly differently. + # they're just downloaded once and aren't archived separately multiple times, + # so the wget, screenshot, & pdf urls should all point to the same file + + static_path = wget_output_path(self) + canonical.update({ + 'title': self.basename, + 'wget_path': static_path, + 'pdf_path': static_path, + 'screenshot_path': static_path, + 'dom_path': static_path, + 'singlefile_path': static_path, + 'readability_path': static_path, + 'mercury_path': static_path, + }) + return canonical + diff --git a/archivebox-0.5.3/build/lib/archivebox/index/sql.py b/archivebox-0.5.3/build/lib/archivebox/index/sql.py new file mode 100644 index 0000000..1e99f67 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/index/sql.py @@ -0,0 +1,106 @@ +__package__ = 'archivebox.index' + +from io import StringIO +from pathlib import Path +from typing import List, Tuple, Iterator +from django.db.models import QuerySet +from django.db import transaction + +from .schema import Link +from ..util import enforce_types +from ..config import OUTPUT_DIR + + +### Main Links Index + +@enforce_types +def parse_sql_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]: + from core.models import Snapshot + + return ( + Link.from_json(page.as_json(*Snapshot.keys)) + for page in Snapshot.objects.all() + ) + +@enforce_types +def remove_from_sql_main_index(snapshots: QuerySet, out_dir: Path=OUTPUT_DIR) -> None: + with transaction.atomic(): + snapshots.delete() + +@enforce_types +def write_link_to_sql_index(link: Link): + from core.models import Snapshot + info = {k: v for k, v in link._asdict().items() if k in Snapshot.keys} + tags = info.pop("tags") + if tags is None: + tags = [] + + try: + info["timestamp"] = Snapshot.objects.get(url=link.url).timestamp + except Snapshot.DoesNotExist: + while Snapshot.objects.filter(timestamp=info["timestamp"]).exists(): + info["timestamp"] = str(float(info["timestamp"]) + 1.0) + + snapshot, _ = Snapshot.objects.update_or_create(url=link.url, defaults=info) + snapshot.save_tags(tags) + return snapshot + + +@enforce_types +def write_sql_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None: + with transaction.atomic(): + for link in links: + write_link_to_sql_index(link) + + +@enforce_types +def write_sql_link_details(link: Link, out_dir: Path=OUTPUT_DIR) -> None: + from core.models import Snapshot + + with transaction.atomic(): + try: + snap = Snapshot.objects.get(url=link.url) + except Snapshot.DoesNotExist: + snap = write_link_to_sql_index(link) + snap.title = link.title + + tag_set = ( + set(tag.strip() for tag in (link.tags or '').split(',')) + ) + tag_list = list(tag_set) or [] + + snap.save() + snap.save_tags(tag_list) + + + +@enforce_types +def list_migrations(out_dir: Path=OUTPUT_DIR) -> List[Tuple[bool, str]]: + from django.core.management import call_command + out = StringIO() + call_command("showmigrations", list=True, stdout=out) + out.seek(0) + migrations = [] + for line in out.readlines(): + if line.strip() and ']' in line: + status_str, name_str = line.strip().split(']', 1) + is_applied = 'X' in status_str + migration_name = name_str.strip() + migrations.append((is_applied, migration_name)) + + return migrations + +@enforce_types +def apply_migrations(out_dir: Path=OUTPUT_DIR) -> List[str]: + from django.core.management import call_command + null, out = StringIO(), StringIO() + call_command("makemigrations", interactive=False, stdout=null) + call_command("migrate", interactive=False, stdout=out) + out.seek(0) + + return [line.strip() for line in out.readlines() if line.strip()] + +@enforce_types +def get_admins(out_dir: Path=OUTPUT_DIR) -> List[str]: + from django.contrib.auth.models import User + return User.objects.filter(is_superuser=True) diff --git a/archivebox-0.5.3/build/lib/archivebox/logging_util.py b/archivebox-0.5.3/build/lib/archivebox/logging_util.py new file mode 100644 index 0000000..f2b8673 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/logging_util.py @@ -0,0 +1,569 @@ +__package__ = 'archivebox' + +import re +import os +import sys +import time +import argparse +from math import log +from multiprocessing import Process +from pathlib import Path + +from datetime import datetime +from dataclasses import dataclass +from typing import Optional, List, Dict, Union, IO, TYPE_CHECKING + +if TYPE_CHECKING: + from .index.schema import Link, ArchiveResult + +from .util import enforce_types +from .config import ( + ConfigDict, + OUTPUT_DIR, + PYTHON_ENCODING, + ANSI, + IS_TTY, + TERM_WIDTH, + SHOW_PROGRESS, + SOURCES_DIR_NAME, + stderr, +) + +@dataclass +class RuntimeStats: + """mutable stats counter for logging archiving timing info to CLI output""" + + skipped: int = 0 + succeeded: int = 0 + failed: int = 0 + + parse_start_ts: Optional[datetime] = None + parse_end_ts: Optional[datetime] = None + + index_start_ts: Optional[datetime] = None + index_end_ts: Optional[datetime] = None + + archiving_start_ts: Optional[datetime] = None + archiving_end_ts: Optional[datetime] = None + +# globals are bad, mmkay +_LAST_RUN_STATS = RuntimeStats() + + + +class SmartFormatter(argparse.HelpFormatter): + """Patched formatter that prints newlines in argparse help strings""" + def _split_lines(self, text, width): + if '\n' in text: + return text.splitlines() + return argparse.HelpFormatter._split_lines(self, text, width) + + +def reject_stdin(caller: str, stdin: Optional[IO]=sys.stdin) -> None: + """Tell the user they passed stdin to a command that doesn't accept it""" + + if stdin and not stdin.isatty(): + stdin_raw_text = stdin.read().strip() + if stdin_raw_text: + stderr(f'[X] The "{caller}" command does not accept stdin.', color='red') + stderr(f' Run archivebox "{caller} --help" to see usage and examples.') + stderr() + raise SystemExit(1) + + +def accept_stdin(stdin: Optional[IO]=sys.stdin) -> Optional[str]: + """accept any standard input and return it as a string or None""" + if not stdin: + return None + elif stdin and not stdin.isatty(): + stdin_str = stdin.read().strip() + return stdin_str or None + return None + + +class TimedProgress: + """Show a progress bar and measure elapsed time until .end() is called""" + + def __init__(self, seconds, prefix=''): + self.SHOW_PROGRESS = SHOW_PROGRESS + if self.SHOW_PROGRESS: + self.p = Process(target=progress_bar, args=(seconds, prefix)) + self.p.start() + + self.stats = {'start_ts': datetime.now(), 'end_ts': None} + + def end(self): + """immediately end progress, clear the progressbar line, and save end_ts""" + + end_ts = datetime.now() + self.stats['end_ts'] = end_ts + + if self.SHOW_PROGRESS: + # terminate if we havent already terminated + try: + # kill the progress bar subprocess + try: + self.p.close() # must be closed *before* its terminnated + except: + pass + self.p.terminate() + self.p.join() + + + # clear whole terminal line + try: + sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset'])) + except (IOError, BrokenPipeError): + # ignore when the parent proc has stopped listening to our stdout + pass + except ValueError: + pass + + +@enforce_types +def progress_bar(seconds: int, prefix: str='') -> None: + """show timer in the form of progress bar, with percentage and seconds remaining""" + chunk = '█' if PYTHON_ENCODING == 'UTF-8' else '#' + last_width = TERM_WIDTH() + chunks = last_width - len(prefix) - 20 # number of progress chunks to show (aka max bar width) + try: + for s in range(seconds * chunks): + max_width = TERM_WIDTH() + if max_width < last_width: + # when the terminal size is shrunk, we have to write a newline + # otherwise the progress bar will keep wrapping incorrectly + sys.stdout.write('\r\n') + sys.stdout.flush() + chunks = max_width - len(prefix) - 20 + pct_complete = s / chunks / seconds * 100 + log_pct = (log(pct_complete or 1, 10) / 2) * 100 # everyone likes faster progress bars ;) + bar_width = round(log_pct/(100/chunks)) + last_width = max_width + + # ████████████████████ 0.9% (1/60sec) + sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format( + prefix, + ANSI['green' if pct_complete < 80 else 'lightyellow'], + (chunk * bar_width).ljust(chunks), + ANSI['reset'], + round(pct_complete, 1), + round(s/chunks), + seconds, + )) + sys.stdout.flush() + time.sleep(1 / chunks) + + # ██████████████████████████████████ 100.0% (60/60sec) + sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format( + prefix, + ANSI['red'], + chunk * chunks, + ANSI['reset'], + 100.0, + seconds, + seconds, + )) + sys.stdout.flush() + # uncomment to have it disappear when it hits 100% instead of staying full red: + # time.sleep(0.5) + # sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset'])) + # sys.stdout.flush() + except (KeyboardInterrupt, BrokenPipeError): + print() + pass + + +def log_cli_command(subcommand: str, subcommand_args: List[str], stdin: Optional[str], pwd: str): + from .config import VERSION, ANSI + cmd = ' '.join(('archivebox', subcommand, *subcommand_args)) + stderr('{black}[i] [{now}] ArchiveBox v{VERSION}: {cmd}{reset}'.format( + now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'), + VERSION=VERSION, + cmd=cmd, + **ANSI, + )) + stderr('{black} > {pwd}{reset}'.format(pwd=pwd, **ANSI)) + stderr() + +### Parsing Stage + + +def log_importing_started(urls: Union[str, List[str]], depth: int, index_only: bool): + _LAST_RUN_STATS.parse_start_ts = datetime.now() + print('{green}[+] [{}] Adding {} links to index (crawl depth={}){}...{reset}'.format( + _LAST_RUN_STATS.parse_start_ts.strftime('%Y-%m-%d %H:%M:%S'), + len(urls) if isinstance(urls, list) else len(urls.split('\n')), + depth, + ' (index only)' if index_only else '', + **ANSI, + )) + +def log_source_saved(source_file: str): + print(' > Saved verbatim input to {}/{}'.format(SOURCES_DIR_NAME, source_file.rsplit('/', 1)[-1])) + +def log_parsing_finished(num_parsed: int, parser_name: str): + _LAST_RUN_STATS.parse_end_ts = datetime.now() + print(' > Parsed {} URLs from input ({})'.format(num_parsed, parser_name)) + +def log_deduping_finished(num_new_links: int): + print(' > Found {} new URLs not already in index'.format(num_new_links)) + + +def log_crawl_started(new_links): + print() + print('{green}[*] Starting crawl of {} sites 1 hop out from starting point{reset}'.format(len(new_links), **ANSI)) + +### Indexing Stage + +def log_indexing_process_started(num_links: int): + start_ts = datetime.now() + _LAST_RUN_STATS.index_start_ts = start_ts + print() + print('{black}[*] [{}] Writing {} links to main index...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + **ANSI, + )) + + +def log_indexing_process_finished(): + end_ts = datetime.now() + _LAST_RUN_STATS.index_end_ts = end_ts + + +def log_indexing_started(out_path: str): + if IS_TTY: + sys.stdout.write(f' > {out_path}') + + +def log_indexing_finished(out_path: str): + print(f'\r √ {out_path}') + + +### Archiving Stage + +def log_archiving_started(num_links: int, resume: Optional[float]=None): + start_ts = datetime.now() + _LAST_RUN_STATS.archiving_start_ts = start_ts + print() + if resume: + print('{green}[▶] [{}] Resuming archive updating for {} pages starting from {}...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + resume, + **ANSI, + )) + else: + print('{green}[▶] [{}] Starting archiving of {} snapshots in index...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + **ANSI, + )) + +def log_archiving_paused(num_links: int, idx: int, timestamp: str): + end_ts = datetime.now() + _LAST_RUN_STATS.archiving_end_ts = end_ts + print() + print('\n{lightyellow}[X] [{now}] Downloading paused on link {timestamp} ({idx}/{total}){reset}'.format( + **ANSI, + now=end_ts.strftime('%Y-%m-%d %H:%M:%S'), + idx=idx+1, + timestamp=timestamp, + total=num_links, + )) + print() + print(' {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI)) + print(' archivebox server # then visit http://127.0.0.1:8000') + print(' Continue archiving where you left off by running:') + print(' archivebox update --resume={}'.format(timestamp)) + +def log_archiving_finished(num_links: int): + end_ts = datetime.now() + _LAST_RUN_STATS.archiving_end_ts = end_ts + assert _LAST_RUN_STATS.archiving_start_ts is not None + seconds = end_ts.timestamp() - _LAST_RUN_STATS.archiving_start_ts.timestamp() + if seconds > 60: + duration = '{0:.2f} min'.format(seconds / 60) + else: + duration = '{0:.2f} sec'.format(seconds) + + print() + print('{}[√] [{}] Update of {} pages complete ({}){}'.format( + ANSI['green'], + end_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + duration, + ANSI['reset'], + )) + print(' - {} links skipped'.format(_LAST_RUN_STATS.skipped)) + print(' - {} links updated'.format(_LAST_RUN_STATS.succeeded + _LAST_RUN_STATS.failed)) + print(' - {} links had errors'.format(_LAST_RUN_STATS.failed)) + print() + print(' {lightred}Hint:{reset} To manage your archive in a Web UI, run:'.format(**ANSI)) + print(' archivebox server 0.0.0.0:8000') + + +def log_link_archiving_started(link: "Link", link_dir: str, is_new: bool): + # [*] [2019-03-22 13:46:45] "Log Structured Merge Trees - ben stopford" + # http://www.benstopford.com/2015/02/14/log-structured-merge-trees/ + # > output/archive/1478739709 + + print('\n[{symbol_color}{symbol}{reset}] [{symbol_color}{now}{reset}] "{title}"'.format( + symbol_color=ANSI['green' if is_new else 'black'], + symbol='+' if is_new else '√', + now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'), + title=link.title or link.base_url, + **ANSI, + )) + print(' {blue}{url}{reset}'.format(url=link.url, **ANSI)) + print(' {} {}'.format( + '>' if is_new else '√', + pretty_path(link_dir), + )) + +def log_link_archiving_finished(link: "Link", link_dir: str, is_new: bool, stats: dict): + total = sum(stats.values()) + + if stats['failed'] > 0 : + _LAST_RUN_STATS.failed += 1 + elif stats['skipped'] == total: + _LAST_RUN_STATS.skipped += 1 + else: + _LAST_RUN_STATS.succeeded += 1 + + +def log_archive_method_started(method: str): + print(' > {}'.format(method)) + + +def log_archive_method_finished(result: "ArchiveResult"): + """quote the argument with whitespace in a command so the user can + copy-paste the outputted string directly to run the cmd + """ + # Prettify CMD string and make it safe to copy-paste by quoting arguments + quoted_cmd = ' '.join( + '"{}"'.format(arg) if ' ' in arg else arg + for arg in result.cmd + ) + + if result.status == 'failed': + if result.output.__class__.__name__ == 'TimeoutExpired': + duration = (result.end_ts - result.start_ts).seconds + hint_header = [ + '{lightyellow}Extractor timed out after {}s.{reset}'.format(duration, **ANSI), + ] + else: + hint_header = [ + '{lightyellow}Extractor failed:{reset}'.format(**ANSI), + ' {reset}{} {red}{}{reset}'.format( + result.output.__class__.__name__.replace('ArchiveError', ''), + result.output, + **ANSI, + ), + ] + + # Prettify error output hints string and limit to five lines + hints = getattr(result.output, 'hints', None) or () + if hints: + hints = hints if isinstance(hints, (list, tuple)) else hints.split('\n') + hints = ( + ' {}{}{}'.format(ANSI['lightyellow'], line.strip(), ANSI['reset']) + for line in hints[:5] if line.strip() + ) + + + # Collect and prefix output lines with indentation + output_lines = [ + *hint_header, + *hints, + '{}Run to see full output:{}'.format(ANSI['lightred'], ANSI['reset']), + *([' cd {};'.format(result.pwd)] if result.pwd else []), + ' {}'.format(quoted_cmd), + ] + print('\n'.join( + ' {}'.format(line) + for line in output_lines + if line + )) + print() + + +def log_list_started(filter_patterns: Optional[List[str]], filter_type: str): + print('{green}[*] Finding links in the archive index matching these {} patterns:{reset}'.format( + filter_type, + **ANSI, + )) + print(' {}'.format(' '.join(filter_patterns or ()))) + +def log_list_finished(links): + from .index.csv import links_to_csv + print() + print('---------------------------------------------------------------------------------------------------') + print(links_to_csv(links, cols=['timestamp', 'is_archived', 'num_outputs', 'url'], header=True, ljust=16, separator=' | ')) + print('---------------------------------------------------------------------------------------------------') + print() + + +def log_removal_started(links: List["Link"], yes: bool, delete: bool): + print('{lightyellow}[i] Found {} matching URLs to remove.{reset}'.format(len(links), **ANSI)) + if delete: + file_counts = [link.num_outputs for link in links if Path(link.link_dir).exists()] + print( + f' {len(links)} Links will be de-listed from the main index, and their archived content folders will be deleted from disk.\n' + f' ({len(file_counts)} data folders with {sum(file_counts)} archived files will be deleted!)' + ) + else: + print( + ' Matching links will be de-listed from the main index, but their archived content folders will remain in place on disk.\n' + ' (Pass --delete if you also want to permanently delete the data folders)' + ) + + if not yes: + print() + print('{lightyellow}[?] Do you want to proceed with removing these {} links?{reset}'.format(len(links), **ANSI)) + try: + assert input(' y/[n]: ').lower() == 'y' + except (KeyboardInterrupt, EOFError, AssertionError): + raise SystemExit(0) + +def log_removal_finished(all_links: int, to_remove: int): + if all_links == 0: + print() + print('{red}[X] No matching links found.{reset}'.format(**ANSI)) + else: + print() + print('{red}[√] Removed {} out of {} links from the archive index.{reset}'.format( + to_remove, + all_links, + **ANSI, + )) + print(' Index now contains {} links.'.format(all_links - to_remove)) + + +def log_shell_welcome_msg(): + from .cli import list_subcommands + + print('{green}# ArchiveBox Imports{reset}'.format(**ANSI)) + print('{green}from core.models import Snapshot, User{reset}'.format(**ANSI)) + print('{green}from archivebox import *\n {}{reset}'.format("\n ".join(list_subcommands().keys()), **ANSI)) + print() + print('[i] Welcome to the ArchiveBox Shell!') + print(' https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Shell-Usage') + print() + print(' {lightred}Hint:{reset} Example use:'.format(**ANSI)) + print(' print(Snapshot.objects.filter(is_archived=True).count())') + print(' Snapshot.objects.get(url="https://example.com").as_json()') + print(' add("https://example.com/some/new/url")') + + + +### Helpers + +@enforce_types +def pretty_path(path: Union[Path, str]) -> str: + """convert paths like .../ArchiveBox/archivebox/../output/abc into output/abc""" + pwd = Path('.').resolve() + # parent = os.path.abspath(os.path.join(pwd, os.path.pardir)) + return str(path).replace(str(pwd) + '/', './') + + +@enforce_types +def printable_filesize(num_bytes: Union[int, float]) -> str: + for count in ['Bytes','KB','MB','GB']: + if num_bytes > -1024.0 and num_bytes < 1024.0: + return '%3.1f %s' % (num_bytes, count) + num_bytes /= 1024.0 + return '%3.1f %s' % (num_bytes, 'TB') + + +@enforce_types +def printable_folders(folders: Dict[str, Optional["Link"]], + with_headers: bool=False) -> str: + return '\n'.join( + f'{folder} {link and link.url} "{link and link.title}"' + for folder, link in folders.items() + ) + + + +@enforce_types +def printable_config(config: ConfigDict, prefix: str='') -> str: + return f'\n{prefix}'.join( + f'{key}={val}' + for key, val in config.items() + if not (isinstance(val, dict) or callable(val)) + ) + + +@enforce_types +def printable_folder_status(name: str, folder: Dict) -> str: + if folder['enabled']: + if folder['is_valid']: + color, symbol, note = 'green', '√', 'valid' + else: + color, symbol, note, num_files = 'red', 'X', 'invalid', '?' + else: + color, symbol, note, num_files = 'lightyellow', '-', 'disabled', '-' + + if folder['path']: + if Path(folder['path']).exists(): + num_files = ( + f'{len(os.listdir(folder["path"]))} files' + if Path(folder['path']).is_dir() else + printable_filesize(Path(folder['path']).stat().st_size) + ) + else: + num_files = 'missing' + + path = str(folder['path']).replace(str(OUTPUT_DIR), '.') if folder['path'] else '' + if path and ' ' in path: + path = f'"{path}"' + + # if path is just a plain dot, replace it back with the full path for clarity + if path == '.': + path = str(OUTPUT_DIR) + + return ' '.join(( + ANSI[color], + symbol, + ANSI['reset'], + name.ljust(21), + num_files.ljust(14), + ANSI[color], + note.ljust(8), + ANSI['reset'], + path.ljust(76), + )) + + +@enforce_types +def printable_dependency_version(name: str, dependency: Dict) -> str: + version = None + if dependency['enabled']: + if dependency['is_valid']: + color, symbol, note, version = 'green', '√', 'valid', '' + + parsed_version_num = re.search(r'[\d\.]+', dependency['version']) + if parsed_version_num: + version = f'v{parsed_version_num[0]}' + + if not version: + color, symbol, note, version = 'red', 'X', 'invalid', '?' + else: + color, symbol, note, version = 'lightyellow', '-', 'disabled', '-' + + path = str(dependency["path"]).replace(str(OUTPUT_DIR), '.') if dependency["path"] else '' + if path and ' ' in path: + path = f'"{path}"' + + return ' '.join(( + ANSI[color], + symbol, + ANSI['reset'], + name.ljust(21), + version.ljust(14), + ANSI[color], + note.ljust(8), + ANSI['reset'], + path.ljust(76), + )) diff --git a/archivebox-0.5.3/build/lib/archivebox/main.py b/archivebox-0.5.3/build/lib/archivebox/main.py new file mode 100644 index 0000000..eb8cd6a --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/main.py @@ -0,0 +1,1131 @@ +__package__ = 'archivebox' + +import os +import sys +import shutil +import platform +from pathlib import Path +from datetime import date + +from typing import Dict, List, Optional, Iterable, IO, Union +from crontab import CronTab, CronSlices +from django.db.models import QuerySet + +from .cli import ( + list_subcommands, + run_subcommand, + display_first, + meta_cmds, + main_cmds, + archive_cmds, +) +from .parsers import ( + save_text_as_source, + save_file_as_source, + parse_links_memory, +) +from .index.schema import Link +from .util import enforce_types # type: ignore +from .system import get_dir_size, dedupe_cron_jobs, CRON_COMMENT +from .index import ( + load_main_index, + parse_links_from_source, + dedupe_links, + write_main_index, + snapshot_filter, + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, + fix_invalid_folder_locations, + write_link_details, +) +from .index.json import ( + parse_json_main_index, + parse_json_links_details, + generate_json_index_from_links, +) +from .index.sql import ( + get_admins, + apply_migrations, + remove_from_sql_main_index, +) +from .index.html import ( + generate_index_from_links, +) +from .index.csv import links_to_csv +from .extractors import archive_links, archive_link, ignore_methods +from .config import ( + stderr, + hint, + ConfigDict, + ANSI, + IS_TTY, + IN_DOCKER, + USER, + ARCHIVEBOX_BINARY, + ONLY_NEW, + OUTPUT_DIR, + SOURCES_DIR, + ARCHIVE_DIR, + LOGS_DIR, + CONFIG_FILE, + ARCHIVE_DIR_NAME, + SOURCES_DIR_NAME, + LOGS_DIR_NAME, + STATIC_DIR_NAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, + SQL_INDEX_FILENAME, + ROBOTS_TXT_FILENAME, + FAVICON_FILENAME, + check_dependencies, + check_data_folder, + write_config_file, + VERSION, + CODE_LOCATIONS, + EXTERNAL_LOCATIONS, + DATA_LOCATIONS, + DEPENDENCIES, + load_all_config, + CONFIG, + USER_CONFIG, + get_real_name, +) +from .logging_util import ( + TERM_WIDTH, + TimedProgress, + log_importing_started, + log_crawl_started, + log_removal_started, + log_removal_finished, + log_list_started, + log_list_finished, + printable_config, + printable_folders, + printable_filesize, + printable_folder_status, + printable_dependency_version, +) + +from .search import flush_search_index, index_links + +ALLOWED_IN_OUTPUT_DIR = { + 'lost+found', + '.DS_Store', + '.venv', + 'venv', + 'virtualenv', + '.virtualenv', + 'node_modules', + 'package-lock.json', + ARCHIVE_DIR_NAME, + SOURCES_DIR_NAME, + LOGS_DIR_NAME, + STATIC_DIR_NAME, + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, + ROBOTS_TXT_FILENAME, + FAVICON_FILENAME, +} + +@enforce_types +def help(out_dir: Path=OUTPUT_DIR) -> None: + """Print the ArchiveBox help message and usage""" + + all_subcommands = list_subcommands() + COMMANDS_HELP_TEXT = '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in meta_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in main_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in archive_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd not in display_first + ) + + + if (Path(out_dir) / SQL_INDEX_FILENAME).exists(): + print('''{green}ArchiveBox v{}: The self-hosted internet archive.{reset} + +{lightred}Active data directory:{reset} + {} + +{lightred}Usage:{reset} + archivebox [command] [--help] [--version] [...args] + +{lightred}Commands:{reset} + {} + +{lightred}Example Use:{reset} + mkdir my-archive; cd my-archive/ + archivebox init + archivebox status + + archivebox add https://example.com/some/page + archivebox add --depth=1 ~/Downloads/bookmarks_export.html + + archivebox list --sort=timestamp --csv=timestamp,url,is_archived + archivebox schedule --every=day https://example.com/some/feed.rss + archivebox update --resume=15109948213.123 + +{lightred}Documentation:{reset} + https://github.com/ArchiveBox/ArchiveBox/wiki +'''.format(VERSION, out_dir, COMMANDS_HELP_TEXT, **ANSI)) + + else: + print('{green}Welcome to ArchiveBox v{}!{reset}'.format(VERSION, **ANSI)) + print() + if IN_DOCKER: + print('When using Docker, you need to mount a volume to use as your data dir:') + print(' docker run -v /some/path:/data archivebox ...') + print() + print('To import an existing archive (from a previous version of ArchiveBox):') + print(' 1. cd into your data dir OUTPUT_DIR (usually ArchiveBox/output) and run:') + print(' 2. archivebox init') + print() + print('To start a new archive:') + print(' 1. Create an empty directory, then cd into it and run:') + print(' 2. archivebox init') + print() + print('For more information, see the documentation here:') + print(' https://github.com/ArchiveBox/ArchiveBox/wiki') + + +@enforce_types +def version(quiet: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Print the ArchiveBox version and dependency information""" + + if quiet: + print(VERSION) + else: + print('ArchiveBox v{}'.format(VERSION)) + p = platform.uname() + print(sys.implementation.name.title(), p.system, platform.platform(), p.machine, '(in Docker)' if IN_DOCKER else '(not in Docker)') + print() + + print('{white}[i] Dependency versions:{reset}'.format(**ANSI)) + for name, dependency in DEPENDENCIES.items(): + print(printable_dependency_version(name, dependency)) + + print() + print('{white}[i] Source-code locations:{reset}'.format(**ANSI)) + for name, folder in CODE_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + + print() + print('{white}[i] Secrets locations:{reset}'.format(**ANSI)) + for name, folder in EXTERNAL_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + + print() + if DATA_LOCATIONS['OUTPUT_DIR']['is_valid']: + print('{white}[i] Data locations:{reset}'.format(**ANSI)) + for name, folder in DATA_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + else: + print() + print('{white}[i] Data locations:{reset}'.format(**ANSI)) + + print() + check_dependencies() + + +@enforce_types +def run(subcommand: str, + subcommand_args: Optional[List[str]], + stdin: Optional[IO]=None, + out_dir: Path=OUTPUT_DIR) -> None: + """Run a given ArchiveBox subcommand with the given list of args""" + run_subcommand( + subcommand=subcommand, + subcommand_args=subcommand_args, + stdin=stdin, + pwd=out_dir, + ) + + +@enforce_types +def init(force: bool=False, out_dir: Path=OUTPUT_DIR) -> None: + """Initialize a new ArchiveBox collection in the current directory""" + from core.models import Snapshot + Path(out_dir).mkdir(exist_ok=True) + is_empty = not len(set(os.listdir(out_dir)) - ALLOWED_IN_OUTPUT_DIR) + + if (Path(out_dir) / JSON_INDEX_FILENAME).exists(): + stderr("[!] This folder contains a JSON index. It is deprecated, and will no longer be kept up to date automatically.", color="lightyellow") + stderr(" You can run `archivebox list --json --with-headers > index.json` to manually generate it.", color="lightyellow") + + existing_index = (Path(out_dir) / SQL_INDEX_FILENAME).exists() + + if is_empty and not existing_index: + print('{green}[+] Initializing a new ArchiveBox collection in this folder...{reset}'.format(**ANSI)) + print(f' {out_dir}') + print('{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + elif existing_index: + print('{green}[*] Updating existing ArchiveBox collection in this folder...{reset}'.format(**ANSI)) + print(f' {out_dir}') + print('{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + else: + if force: + stderr('[!] This folder appears to already have files in it, but no index.sqlite3 is present.', color='lightyellow') + stderr(' Because --force was passed, ArchiveBox will initialize anyway (which may overwrite existing files).') + else: + stderr( + ("{red}[X] This folder appears to already have files in it, but no index.sqlite3 present.{reset}\n\n" + " You must run init in a completely empty directory, or an existing data folder.\n\n" + " {lightred}Hint:{reset} To import an existing data folder make sure to cd into the folder first, \n" + " then run and run 'archivebox init' to pick up where you left off.\n\n" + " (Always make sure your data folder is backed up first before updating ArchiveBox)" + ).format(out_dir, **ANSI) + ) + raise SystemExit(2) + + if existing_index: + print('\n{green}[*] Verifying archive folder structure...{reset}'.format(**ANSI)) + else: + print('\n{green}[+] Building archive folder structure...{reset}'.format(**ANSI)) + + Path(SOURCES_DIR).mkdir(exist_ok=True) + print(f' √ {SOURCES_DIR}') + + Path(ARCHIVE_DIR).mkdir(exist_ok=True) + print(f' √ {ARCHIVE_DIR}') + + Path(LOGS_DIR).mkdir(exist_ok=True) + print(f' √ {LOGS_DIR}') + + write_config_file({}, out_dir=out_dir) + print(f' √ {CONFIG_FILE}') + if (Path(out_dir) / SQL_INDEX_FILENAME).exists(): + print('\n{green}[*] Verifying main SQL index and running migrations...{reset}'.format(**ANSI)) + else: + print('\n{green}[+] Building main SQL index and running migrations...{reset}'.format(**ANSI)) + + DATABASE_FILE = Path(out_dir) / SQL_INDEX_FILENAME + print(f' √ {DATABASE_FILE}') + print() + for migration_line in apply_migrations(out_dir): + print(f' {migration_line}') + + + assert DATABASE_FILE.exists() + + # from django.contrib.auth.models import User + # if IS_TTY and not User.objects.filter(is_superuser=True).exists(): + # print('{green}[+] Creating admin user account...{reset}'.format(**ANSI)) + # call_command("createsuperuser", interactive=True) + + print() + print('{green}[*] Collecting links from any existing indexes and archive folders...{reset}'.format(**ANSI)) + + all_links = Snapshot.objects.none() + pending_links: Dict[str, Link] = {} + + if existing_index: + all_links = load_main_index(out_dir=out_dir, warn=False) + print(' √ Loaded {} links from existing main index.'.format(all_links.count())) + + # Links in data folders that dont match their timestamp + fixed, cant_fix = fix_invalid_folder_locations(out_dir=out_dir) + if fixed: + print(' {lightyellow}√ Fixed {} data directory locations that didn\'t match their link timestamps.{reset}'.format(len(fixed), **ANSI)) + if cant_fix: + print(' {lightyellow}! Could not fix {} data directory locations due to conflicts with existing folders.{reset}'.format(len(cant_fix), **ANSI)) + + # Links in JSON index but not in main index + orphaned_json_links = { + link.url: link + for link in parse_json_main_index(out_dir) + if not all_links.filter(url=link.url).exists() + } + if orphaned_json_links: + pending_links.update(orphaned_json_links) + print(' {lightyellow}√ Added {} orphaned links from existing JSON index...{reset}'.format(len(orphaned_json_links), **ANSI)) + + # Links in data dir indexes but not in main index + orphaned_data_dir_links = { + link.url: link + for link in parse_json_links_details(out_dir) + if not all_links.filter(url=link.url).exists() + } + if orphaned_data_dir_links: + pending_links.update(orphaned_data_dir_links) + print(' {lightyellow}√ Added {} orphaned links from existing archive directories.{reset}'.format(len(orphaned_data_dir_links), **ANSI)) + + # Links in invalid/duplicate data dirs + invalid_folders = { + folder: link + for folder, link in get_invalid_folders(all_links, out_dir=out_dir).items() + } + if invalid_folders: + print(' {lightyellow}! Skipped adding {} invalid link data directories.{reset}'.format(len(invalid_folders), **ANSI)) + print(' X ' + '\n X '.join(f'{folder} {link}' for folder, link in invalid_folders.items())) + print() + print(' {lightred}Hint:{reset} For more information about the link data directories that were skipped, run:'.format(**ANSI)) + print(' archivebox status') + print(' archivebox list --status=invalid') + + + write_main_index(list(pending_links.values()), out_dir=out_dir) + + print('\n{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + if existing_index: + print('{green}[√] Done. Verified and updated the existing ArchiveBox collection.{reset}'.format(**ANSI)) + else: + print('{green}[√] Done. A new ArchiveBox collection was initialized ({} links).{reset}'.format(len(all_links), **ANSI)) + print() + print(' {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI)) + print(' archivebox server # then visit http://127.0.0.1:8000') + print() + print(' To add new links, you can run:') + print(" archivebox add ~/some/path/or/url/to/list_of_links.txt") + print() + print(' For more usage and examples, run:') + print(' archivebox help') + + json_index = Path(out_dir) / JSON_INDEX_FILENAME + html_index = Path(out_dir) / HTML_INDEX_FILENAME + index_name = f"{date.today()}_index_old" + if json_index.exists(): + json_index.rename(f"{index_name}.json") + if html_index.exists(): + html_index.rename(f"{index_name}.html") + + + +@enforce_types +def status(out_dir: Path=OUTPUT_DIR) -> None: + """Print out some info and statistics about the archive collection""" + + check_data_folder(out_dir=out_dir) + + from core.models import Snapshot + from django.contrib.auth import get_user_model + User = get_user_model() + + print('{green}[*] Scanning archive main index...{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {out_dir}/*', ANSI['reset']) + num_bytes, num_dirs, num_files = get_dir_size(out_dir, recursive=False, pattern='index.') + size = printable_filesize(num_bytes) + print(f' Index size: {size} across {num_files} files') + print() + + links = load_main_index(out_dir=out_dir) + num_sql_links = links.count() + num_link_details = sum(1 for link in parse_json_links_details(out_dir=out_dir)) + print(f' > SQL Main Index: {num_sql_links} links'.ljust(36), f'(found in {SQL_INDEX_FILENAME})') + print(f' > JSON Link Details: {num_link_details} links'.ljust(36), f'(found in {ARCHIVE_DIR_NAME}/*/index.json)') + print() + print('{green}[*] Scanning archive data directories...{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {ARCHIVE_DIR}/*', ANSI['reset']) + num_bytes, num_dirs, num_files = get_dir_size(ARCHIVE_DIR) + size = printable_filesize(num_bytes) + print(f' Size: {size} across {num_files} files in {num_dirs} directories') + print(ANSI['black']) + num_indexed = len(get_indexed_folders(links, out_dir=out_dir)) + num_archived = len(get_archived_folders(links, out_dir=out_dir)) + num_unarchived = len(get_unarchived_folders(links, out_dir=out_dir)) + print(f' > indexed: {num_indexed}'.ljust(36), f'({get_indexed_folders.__doc__})') + print(f' > archived: {num_archived}'.ljust(36), f'({get_archived_folders.__doc__})') + print(f' > unarchived: {num_unarchived}'.ljust(36), f'({get_unarchived_folders.__doc__})') + + num_present = len(get_present_folders(links, out_dir=out_dir)) + num_valid = len(get_valid_folders(links, out_dir=out_dir)) + print() + print(f' > present: {num_present}'.ljust(36), f'({get_present_folders.__doc__})') + print(f' > valid: {num_valid}'.ljust(36), f'({get_valid_folders.__doc__})') + + duplicate = get_duplicate_folders(links, out_dir=out_dir) + orphaned = get_orphaned_folders(links, out_dir=out_dir) + corrupted = get_corrupted_folders(links, out_dir=out_dir) + unrecognized = get_unrecognized_folders(links, out_dir=out_dir) + num_invalid = len({**duplicate, **orphaned, **corrupted, **unrecognized}) + print(f' > invalid: {num_invalid}'.ljust(36), f'({get_invalid_folders.__doc__})') + print(f' > duplicate: {len(duplicate)}'.ljust(36), f'({get_duplicate_folders.__doc__})') + print(f' > orphaned: {len(orphaned)}'.ljust(36), f'({get_orphaned_folders.__doc__})') + print(f' > corrupted: {len(corrupted)}'.ljust(36), f'({get_corrupted_folders.__doc__})') + print(f' > unrecognized: {len(unrecognized)}'.ljust(36), f'({get_unrecognized_folders.__doc__})') + + print(ANSI['reset']) + + if num_indexed: + print(' {lightred}Hint:{reset} You can list link data directories by status like so:'.format(**ANSI)) + print(' archivebox list --status=<status> (e.g. indexed, corrupted, archived, etc.)') + + if orphaned: + print(' {lightred}Hint:{reset} To automatically import orphaned data directories into the main index, run:'.format(**ANSI)) + print(' archivebox init') + + if num_invalid: + print(' {lightred}Hint:{reset} You may need to manually remove or fix some invalid data directories, afterwards make sure to run:'.format(**ANSI)) + print(' archivebox init') + + print() + print('{green}[*] Scanning recent archive changes and user logins:{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {LOGS_DIR}/*', ANSI['reset']) + users = get_admins().values_list('username', flat=True) + print(f' UI users {len(users)}: {", ".join(users)}') + last_login = User.objects.order_by('last_login').last() + if last_login: + print(f' Last UI login: {last_login.username} @ {str(last_login.last_login)[:16]}') + last_updated = Snapshot.objects.order_by('updated').last() + if last_updated: + print(f' Last changes: {str(last_updated.updated)[:16]}') + + if not users: + print() + print(' {lightred}Hint:{reset} You can create an admin user by running:'.format(**ANSI)) + print(' archivebox manage createsuperuser') + + print() + for snapshot in links.order_by('-updated')[:10]: + if not snapshot.updated: + continue + print( + ANSI['black'], + ( + f' > {str(snapshot.updated)[:16]} ' + f'[{snapshot.num_outputs} {("X", "√")[snapshot.is_archived]} {printable_filesize(snapshot.archive_size)}] ' + f'"{snapshot.title}": {snapshot.url}' + )[:TERM_WIDTH()], + ANSI['reset'], + ) + print(ANSI['black'], ' ...', ANSI['reset']) + + +@enforce_types +def oneshot(url: str, extractors: str="", out_dir: Path=OUTPUT_DIR): + """ + Create a single URL archive folder with an index.json and index.html, and all the archive method outputs. + You can run this to archive single pages without needing to create a whole collection with archivebox init. + """ + oneshot_link, _ = parse_links_memory([url]) + if len(oneshot_link) > 1: + stderr( + '[X] You should pass a single url to the oneshot command', + color='red' + ) + raise SystemExit(2) + + methods = extractors.split(",") if extractors else ignore_methods(['title']) + archive_link(oneshot_link[0], out_dir=out_dir, methods=methods) + return oneshot_link + +@enforce_types +def add(urls: Union[str, List[str]], + depth: int=0, + update_all: bool=not ONLY_NEW, + index_only: bool=False, + overwrite: bool=False, + init: bool=False, + extractors: str="", + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Add a new URL or list of URLs to your archive""" + + assert depth in (0, 1), 'Depth must be 0 or 1 (depth >1 is not supported yet)' + + extractors = extractors.split(",") if extractors else [] + + if init: + run_subcommand('init', stdin=None, pwd=out_dir) + + # Load list of links from the existing index + check_data_folder(out_dir=out_dir) + check_dependencies() + new_links: List[Link] = [] + all_links = load_main_index(out_dir=out_dir) + + log_importing_started(urls=urls, depth=depth, index_only=index_only) + if isinstance(urls, str): + # save verbatim stdin to sources + write_ahead_log = save_text_as_source(urls, filename='{ts}-import.txt', out_dir=out_dir) + elif isinstance(urls, list): + # save verbatim args to sources + write_ahead_log = save_text_as_source('\n'.join(urls), filename='{ts}-import.txt', out_dir=out_dir) + + new_links += parse_links_from_source(write_ahead_log, root_url=None) + + # If we're going one level deeper, download each link and look for more links + new_links_depth = [] + if new_links and depth == 1: + log_crawl_started(new_links) + for new_link in new_links: + downloaded_file = save_file_as_source(new_link.url, filename=f'{new_link.timestamp}-crawl-{new_link.domain}.txt', out_dir=out_dir) + new_links_depth += parse_links_from_source(downloaded_file, root_url=new_link.url) + + imported_links = list({link.url: link for link in (new_links + new_links_depth)}.values()) + new_links = dedupe_links(all_links, imported_links) + + write_main_index(links=new_links, out_dir=out_dir) + all_links = load_main_index(out_dir=out_dir) + + if index_only: + return all_links + + # Run the archive methods for each link + archive_kwargs = { + "out_dir": out_dir, + } + if extractors: + archive_kwargs["methods"] = extractors + if update_all: + archive_links(all_links, overwrite=overwrite, **archive_kwargs) + elif overwrite: + archive_links(imported_links, overwrite=True, **archive_kwargs) + elif new_links: + archive_links(new_links, overwrite=False, **archive_kwargs) + + return all_links + +@enforce_types +def remove(filter_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + snapshots: Optional[QuerySet]=None, + after: Optional[float]=None, + before: Optional[float]=None, + yes: bool=False, + delete: bool=False, + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Remove the specified URLs from the archive""" + + check_data_folder(out_dir=out_dir) + + if snapshots is None: + if filter_str and filter_patterns: + stderr( + '[X] You should pass either a pattern as an argument, ' + 'or pass a list of patterns via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif not (filter_str or filter_patterns): + stderr( + '[X] You should pass either a pattern as an argument, ' + 'or pass a list of patterns via stdin.', + color='red', + ) + stderr() + hint(('To remove all urls you can run:', + 'archivebox remove --filter-type=regex ".*"')) + stderr() + raise SystemExit(2) + elif filter_str: + filter_patterns = [ptn.strip() for ptn in filter_str.split('\n')] + + list_kwargs = { + "filter_patterns": filter_patterns, + "filter_type": filter_type, + "after": after, + "before": before, + } + if snapshots: + list_kwargs["snapshots"] = snapshots + + log_list_started(filter_patterns, filter_type) + timer = TimedProgress(360, prefix=' ') + try: + snapshots = list_links(**list_kwargs) + finally: + timer.end() + + + if not snapshots.exists(): + log_removal_finished(0, 0) + raise SystemExit(1) + + + log_links = [link.as_link() for link in snapshots] + log_list_finished(log_links) + log_removal_started(log_links, yes=yes, delete=delete) + + timer = TimedProgress(360, prefix=' ') + try: + for snapshot in snapshots: + if delete: + shutil.rmtree(snapshot.as_link().link_dir, ignore_errors=True) + finally: + timer.end() + + to_remove = snapshots.count() + + flush_search_index(snapshots=snapshots) + remove_from_sql_main_index(snapshots=snapshots, out_dir=out_dir) + all_snapshots = load_main_index(out_dir=out_dir) + log_removal_finished(all_snapshots.count(), to_remove) + + return all_snapshots + +@enforce_types +def update(resume: Optional[float]=None, + only_new: bool=ONLY_NEW, + index_only: bool=False, + overwrite: bool=False, + filter_patterns_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: Optional[str]=None, + status: Optional[str]=None, + after: Optional[str]=None, + before: Optional[str]=None, + extractors: str="", + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Import any new links from subscriptions and retry any previously failed/skipped links""" + + check_data_folder(out_dir=out_dir) + check_dependencies() + new_links: List[Link] = [] # TODO: Remove input argument: only_new + + extractors = extractors.split(",") if extractors else [] + + # Step 1: Filter for selected_links + matching_snapshots = list_links( + filter_patterns=filter_patterns, + filter_type=filter_type, + before=before, + after=after, + ) + + matching_folders = list_folders( + links=matching_snapshots, + status=status, + out_dir=out_dir, + ) + all_links = [link for link in matching_folders.values() if link] + + if index_only: + for link in all_links: + write_link_details(link, out_dir=out_dir, skip_sql_index=True) + index_links(all_links, out_dir=out_dir) + return all_links + + # Step 2: Run the archive methods for each link + to_archive = new_links if only_new else all_links + if resume: + to_archive = [ + link for link in to_archive + if link.timestamp >= str(resume) + ] + if not to_archive: + stderr('') + stderr(f'[√] Nothing found to resume after {resume}', color='green') + return all_links + + archive_kwargs = { + "out_dir": out_dir, + } + if extractors: + archive_kwargs["methods"] = extractors + + archive_links(to_archive, overwrite=overwrite, **archive_kwargs) + + # Step 4: Re-write links index with updated titles, icons, and resources + all_links = load_main_index(out_dir=out_dir) + return all_links + +@enforce_types +def list_all(filter_patterns_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + status: Optional[str]=None, + after: Optional[float]=None, + before: Optional[float]=None, + sort: Optional[str]=None, + csv: Optional[str]=None, + json: bool=False, + html: bool=False, + with_headers: bool=False, + out_dir: Path=OUTPUT_DIR) -> Iterable[Link]: + """List, filter, and export information about archive entries""" + + check_data_folder(out_dir=out_dir) + + if filter_patterns and filter_patterns_str: + stderr( + '[X] You should either pass filter patterns as an arguments ' + 'or via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif filter_patterns_str: + filter_patterns = filter_patterns_str.split('\n') + + snapshots = list_links( + filter_patterns=filter_patterns, + filter_type=filter_type, + before=before, + after=after, + ) + + if sort: + snapshots = snapshots.order_by(sort) + + folders = list_folders( + links=snapshots, + status=status, + out_dir=out_dir, + ) + + if json: + output = generate_json_index_from_links(folders.values(), with_headers) + elif html: + output = generate_index_from_links(folders.values(), with_headers) + elif csv: + output = links_to_csv(folders.values(), cols=csv.split(','), header=with_headers) + else: + output = printable_folders(folders, with_headers=with_headers) + print(output) + return folders + + +@enforce_types +def list_links(snapshots: Optional[QuerySet]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + after: Optional[float]=None, + before: Optional[float]=None, + out_dir: Path=OUTPUT_DIR) -> Iterable[Link]: + + check_data_folder(out_dir=out_dir) + + if snapshots: + all_snapshots = snapshots + else: + all_snapshots = load_main_index(out_dir=out_dir) + + if after is not None: + all_snapshots = all_snapshots.filter(timestamp__lt=after) + if before is not None: + all_snapshots = all_snapshots.filter(timestamp__gt=before) + if filter_patterns: + all_snapshots = snapshot_filter(all_snapshots, filter_patterns, filter_type) + return all_snapshots + +@enforce_types +def list_folders(links: List[Link], + status: str, + out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + + check_data_folder(out_dir=out_dir) + + STATUS_FUNCTIONS = { + "indexed": get_indexed_folders, + "archived": get_archived_folders, + "unarchived": get_unarchived_folders, + "present": get_present_folders, + "valid": get_valid_folders, + "invalid": get_invalid_folders, + "duplicate": get_duplicate_folders, + "orphaned": get_orphaned_folders, + "corrupted": get_corrupted_folders, + "unrecognized": get_unrecognized_folders, + } + + try: + return STATUS_FUNCTIONS[status](links, out_dir=out_dir) + except KeyError: + raise ValueError('Status not recognized.') + + +@enforce_types +def config(config_options_str: Optional[str]=None, + config_options: Optional[List[str]]=None, + get: bool=False, + set: bool=False, + reset: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Get and set your ArchiveBox project configuration values""" + + check_data_folder(out_dir=out_dir) + + if config_options and config_options_str: + stderr( + '[X] You should either pass config values as an arguments ' + 'or via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif config_options_str: + config_options = config_options_str.split('\n') + + config_options = config_options or [] + + no_args = not (get or set or reset or config_options) + + matching_config: ConfigDict = {} + if get or no_args: + if config_options: + config_options = [get_real_name(key) for key in config_options] + matching_config = {key: CONFIG[key] for key in config_options if key in CONFIG} + failed_config = [key for key in config_options if key not in CONFIG] + if failed_config: + stderr() + stderr('[X] These options failed to get', color='red') + stderr(' {}'.format('\n '.join(config_options))) + raise SystemExit(1) + else: + matching_config = CONFIG + + print(printable_config(matching_config)) + raise SystemExit(not matching_config) + elif set: + new_config = {} + failed_options = [] + for line in config_options: + if line.startswith('#') or not line.strip(): + continue + if '=' not in line: + stderr('[X] Config KEY=VALUE must have an = sign in it', color='red') + stderr(f' {line}') + raise SystemExit(2) + + raw_key, val = line.split('=', 1) + raw_key = raw_key.upper().strip() + key = get_real_name(raw_key) + if key != raw_key: + stderr(f'[i] Note: The config option {raw_key} has been renamed to {key}, please use the new name going forwards.', color='lightyellow') + + if key in CONFIG: + new_config[key] = val.strip() + else: + failed_options.append(line) + + if new_config: + before = CONFIG + matching_config = write_config_file(new_config, out_dir=OUTPUT_DIR) + after = load_all_config() + print(printable_config(matching_config)) + + side_effect_changes: ConfigDict = {} + for key, val in after.items(): + if key in USER_CONFIG and (before[key] != after[key]) and (key not in matching_config): + side_effect_changes[key] = after[key] + + if side_effect_changes: + stderr() + stderr('[i] Note: This change also affected these other options that depended on it:', color='lightyellow') + print(' {}'.format(printable_config(side_effect_changes, prefix=' '))) + if failed_options: + stderr() + stderr('[X] These options failed to set (check for typos):', color='red') + stderr(' {}'.format('\n '.join(failed_options))) + raise SystemExit(bool(failed_options)) + elif reset: + stderr('[X] This command is not implemented yet.', color='red') + stderr(' Please manually remove the relevant lines from your config file:') + stderr(f' {CONFIG_FILE}') + raise SystemExit(2) + else: + stderr('[X] You must pass either --get or --set, or no arguments to get the whole config.', color='red') + stderr(' archivebox config') + stderr(' archivebox config --get SOME_KEY') + stderr(' archivebox config --set SOME_KEY=SOME_VALUE') + raise SystemExit(2) + + +@enforce_types +def schedule(add: bool=False, + show: bool=False, + clear: bool=False, + foreground: bool=False, + run_all: bool=False, + quiet: bool=False, + every: Optional[str]=None, + depth: int=0, + import_path: Optional[str]=None, + out_dir: Path=OUTPUT_DIR): + """Set ArchiveBox to regularly import URLs at specific times using cron""" + + check_data_folder(out_dir=out_dir) + + (Path(out_dir) / LOGS_DIR_NAME).mkdir(exist_ok=True) + + cron = CronTab(user=True) + cron = dedupe_cron_jobs(cron) + + if clear: + print(cron.remove_all(comment=CRON_COMMENT)) + cron.write() + raise SystemExit(0) + + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + if every or add: + every = every or 'day' + quoted = lambda s: f'"{s}"' if s and ' ' in str(s) else str(s) + cmd = [ + 'cd', + quoted(out_dir), + '&&', + quoted(ARCHIVEBOX_BINARY), + *(['add', f'--depth={depth}', f'"{import_path}"'] if import_path else ['update']), + '>', + quoted(Path(LOGS_DIR) / 'archivebox.log'), + '2>&1', + + ] + new_job = cron.new(command=' '.join(cmd), comment=CRON_COMMENT) + + if every in ('minute', 'hour', 'day', 'month', 'year'): + set_every = getattr(new_job.every(), every) + set_every() + elif CronSlices.is_valid(every): + new_job.setall(every) + else: + stderr('{red}[X] Got invalid timeperiod for cron task.{reset}'.format(**ANSI)) + stderr(' It must be one of minute/hour/day/month') + stderr(' or a quoted cron-format schedule like:') + stderr(' archivebox init --every=day https://example.com/some/rss/feed.xml') + stderr(' archivebox init --every="0/5 * * * *" https://example.com/some/rss/feed.xml') + raise SystemExit(1) + + cron = dedupe_cron_jobs(cron) + cron.write() + + total_runs = sum(j.frequency_per_year() for j in cron) + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + print() + print('{green}[√] Scheduled new ArchiveBox cron job for user: {} ({} jobs are active).{reset}'.format(USER, len(existing_jobs), **ANSI)) + print('\n'.join(f' > {cmd}' if str(cmd) == str(new_job) else f' {cmd}' for cmd in existing_jobs)) + if total_runs > 60 and not quiet: + stderr() + stderr('{lightyellow}[!] With the current cron config, ArchiveBox is estimated to run >{} times per year.{reset}'.format(total_runs, **ANSI)) + stderr(' Congrats on being an enthusiastic internet archiver! 👌') + stderr() + stderr(' Make sure you have enough storage space available to hold all the data.') + stderr(' Using a compressed/deduped filesystem like ZFS is recommended if you plan on archiving a lot.') + stderr('') + elif show: + if existing_jobs: + print('\n'.join(str(cmd) for cmd in existing_jobs)) + else: + stderr('{red}[X] There are no ArchiveBox cron jobs scheduled for your user ({}).{reset}'.format(USER, **ANSI)) + stderr(' To schedule a new job, run:') + stderr(' archivebox schedule --every=[timeperiod] https://example.com/some/rss/feed.xml') + raise SystemExit(0) + + cron = CronTab(user=True) + cron = dedupe_cron_jobs(cron) + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + if foreground or run_all: + if not existing_jobs: + stderr('{red}[X] You must schedule some jobs first before running in foreground mode.{reset}'.format(**ANSI)) + stderr(' archivebox schedule --every=hour https://example.com/some/rss/feed.xml') + raise SystemExit(1) + + print('{green}[*] Running {} ArchiveBox jobs in foreground task scheduler...{reset}'.format(len(existing_jobs), **ANSI)) + if run_all: + try: + for job in existing_jobs: + sys.stdout.write(f' > {job.command.split("/archivebox ")[0].split(" && ")[0]}\n') + sys.stdout.write(f' > {job.command.split("/archivebox ")[-1].split(" > ")[0]}') + sys.stdout.flush() + job.run() + sys.stdout.write(f'\r √ {job.command.split("/archivebox ")[-1]}\n') + except KeyboardInterrupt: + print('\n{green}[√] Stopped.{reset}'.format(**ANSI)) + raise SystemExit(1) + + if foreground: + try: + for job in existing_jobs: + print(f' > {job.command.split("/archivebox ")[-1].split(" > ")[0]}') + for result in cron.run_scheduler(): + print(result) + except KeyboardInterrupt: + print('\n{green}[√] Stopped.{reset}'.format(**ANSI)) + raise SystemExit(1) + + +@enforce_types +def server(runserver_args: Optional[List[str]]=None, + reload: bool=False, + debug: bool=False, + init: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Run the ArchiveBox HTTP server""" + + runserver_args = runserver_args or [] + + if init: + run_subcommand('init', stdin=None, pwd=out_dir) + + # setup config for django runserver + from . import config + config.SHOW_PROGRESS = False + config.DEBUG = config.DEBUG or debug + + check_data_folder(out_dir=out_dir) + + from django.core.management import call_command + from django.contrib.auth.models import User + + admin_user = User.objects.filter(is_superuser=True).order_by('date_joined').only('username').last() + + print('{green}[+] Starting ArchiveBox webserver...{reset}'.format(**ANSI)) + if admin_user: + hint('The admin username is{lightblue} {}{reset}\n'.format(admin_user.username, **ANSI)) + else: + print('{lightyellow}[!] No admin users exist yet, you will not be able to edit links in the UI.{reset}'.format(**ANSI)) + print() + print(' To create an admin user, run:') + print(' archivebox manage createsuperuser') + print() + + # fallback to serving staticfiles insecurely with django when DEBUG=False + if not config.DEBUG: + runserver_args.append('--insecure') # TODO: serve statics w/ nginx instead + + # toggle autoreloading when archivebox code changes (it's on by default) + if not reload: + runserver_args.append('--noreload') + + config.SHOW_PROGRESS = False + config.DEBUG = config.DEBUG or debug + + + call_command("runserver", *runserver_args) + + +@enforce_types +def manage(args: Optional[List[str]]=None, out_dir: Path=OUTPUT_DIR) -> None: + """Run an ArchiveBox Django management command""" + + check_data_folder(out_dir=out_dir) + from django.core.management import execute_from_command_line + + if (args and "createsuperuser" in args) and (IN_DOCKER and not IS_TTY): + stderr('[!] Warning: you need to pass -it to use interactive commands in docker', color='lightyellow') + stderr(' docker run -it archivebox manage {}'.format(' '.join(args or ['...'])), color='lightyellow') + stderr() + + execute_from_command_line([f'{ARCHIVEBOX_BINARY} manage', *(args or ['help'])]) + + +@enforce_types +def shell(out_dir: Path=OUTPUT_DIR) -> None: + """Enter an interactive ArchiveBox Django shell""" + + check_data_folder(out_dir=out_dir) + + from django.core.management import call_command + call_command("shell_plus") + diff --git a/archivebox-0.5.3/build/lib/archivebox/manage.py b/archivebox-0.5.3/build/lib/archivebox/manage.py new file mode 100644 index 0000000..1a9b297 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/manage.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python +import os +import sys + +if __name__ == '__main__': + # if you're a developer working on archivebox, still prefer the archivebox + # versions of ./manage.py commands whenever possible. When that's not possible + # (e.g. makemigrations), you can comment out this check temporarily + + if not ('makemigrations' in sys.argv or 'migrate' in sys.argv): + print("[X] Don't run ./manage.py directly (unless you are a developer running makemigrations):") + print() + print(' Hint: Use these archivebox CLI commands instead of the ./manage.py equivalents:') + print(' archivebox init (migrates the databse to latest version)') + print(' archivebox server (runs the Django web server)') + print(' archivebox shell (opens an iPython Django shell with all models imported)') + print(' archivebox manage [cmd] (any other management commands)') + raise SystemExit(2) + + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings') + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) diff --git a/archivebox-0.5.3/build/lib/archivebox/mypy.ini b/archivebox-0.5.3/build/lib/archivebox/mypy.ini new file mode 100644 index 0000000..b1b4489 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/mypy.ini @@ -0,0 +1,3 @@ +[mypy] +plugins = + mypy_django_plugin.main diff --git a/archivebox-0.5.3/build/lib/archivebox/package.json b/archivebox-0.5.3/build/lib/archivebox/package.json new file mode 100644 index 0000000..7f8bf66 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/package.json @@ -0,0 +1,21 @@ +{ + "name": "archivebox", + "version": "0.5.3", + "description": "ArchiveBox: The self-hosted internet archive", + "author": "Nick Sweeting <archivebox-npm@sweeting.me>", + "license": "MIT", + "scripts": { + "archivebox": "./bin/archive" + }, + "bin": { + "archivebox-node": "./bin/archive", + "single-file": "./node_modules/.bin/single-file", + "readability-extractor": "./node_modules/.bin/readability-extractor", + "mercury-parser": "./node_modules/.bin/mercury-parser" + }, + "dependencies": { + "@postlight/mercury-parser": "^2.2.0", + "readability-extractor": "git+https://github.com/pirate/readability-extractor.git", + "single-file": "git+https://github.com/gildas-lormeau/SingleFile.git" + } +} diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/__init__.py b/archivebox-0.5.3/build/lib/archivebox/parsers/__init__.py new file mode 100644 index 0000000..441c08a --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/__init__.py @@ -0,0 +1,203 @@ +""" +Everything related to parsing links from input sources. + +For a list of supported services, see the README.md. +For examples of supported import formats see tests/. +""" + +__package__ = 'archivebox.parsers' + +import re +from io import StringIO + +from typing import IO, Tuple, List, Optional +from datetime import datetime +from pathlib import Path + +from ..system import atomic_write +from ..config import ( + ANSI, + OUTPUT_DIR, + SOURCES_DIR_NAME, + TIMEOUT, +) +from ..util import ( + basename, + htmldecode, + download_url, + enforce_types, + URL_REGEX, +) +from ..index.schema import Link +from ..logging_util import TimedProgress, log_source_saved + +from .pocket_html import parse_pocket_html_export +from .pocket_api import parse_pocket_api_export +from .pinboard_rss import parse_pinboard_rss_export +from .wallabag_atom import parse_wallabag_atom_export +from .shaarli_rss import parse_shaarli_rss_export +from .medium_rss import parse_medium_rss_export +from .netscape_html import parse_netscape_html_export +from .generic_rss import parse_generic_rss_export +from .generic_json import parse_generic_json_export +from .generic_html import parse_generic_html_export +from .generic_txt import parse_generic_txt_export + +PARSERS = ( + # Specialized parsers + ('Pocket API', parse_pocket_api_export), + ('Wallabag ATOM', parse_wallabag_atom_export), + ('Pocket HTML', parse_pocket_html_export), + ('Pinboard RSS', parse_pinboard_rss_export), + ('Shaarli RSS', parse_shaarli_rss_export), + ('Medium RSS', parse_medium_rss_export), + + # General parsers + ('Netscape HTML', parse_netscape_html_export), + ('Generic RSS', parse_generic_rss_export), + ('Generic JSON', parse_generic_json_export), + ('Generic HTML', parse_generic_html_export), + + # Fallback parser + ('Plain Text', parse_generic_txt_export), +) + + +@enforce_types +def parse_links_memory(urls: List[str], root_url: Optional[str]=None): + """ + parse a list of URLS without touching the filesystem + """ + check_url_parsing_invariants() + + timer = TimedProgress(TIMEOUT * 4) + #urls = list(map(lambda x: x + "\n", urls)) + file = StringIO() + file.writelines(urls) + file.name = "io_string" + links, parser = run_parser_functions(file, timer, root_url=root_url) + timer.end() + + if parser is None: + return [], 'Failed to parse' + return links, parser + + +@enforce_types +def parse_links(source_file: str, root_url: Optional[str]=None) -> Tuple[List[Link], str]: + """parse a list of URLs with their metadata from an + RSS feed, bookmarks export, or text file + """ + + check_url_parsing_invariants() + + timer = TimedProgress(TIMEOUT * 4) + with open(source_file, 'r', encoding='utf-8') as file: + links, parser = run_parser_functions(file, timer, root_url=root_url) + + timer.end() + if parser is None: + return [], 'Failed to parse' + return links, parser + + +def run_parser_functions(to_parse: IO[str], timer, root_url: Optional[str]=None) -> Tuple[List[Link], Optional[str]]: + most_links: List[Link] = [] + best_parser_name = None + + for parser_name, parser_func in PARSERS: + try: + parsed_links = list(parser_func(to_parse, root_url=root_url)) + if not parsed_links: + raise Exception('no links found') + + # print(f'[√] Parser {parser_name} succeeded: {len(parsed_links)} links parsed') + if len(parsed_links) > len(most_links): + most_links = parsed_links + best_parser_name = parser_name + + except Exception as err: # noqa + # Parsers are tried one by one down the list, and the first one + # that succeeds is used. To see why a certain parser was not used + # due to error or format incompatibility, uncomment this line: + + # print('[!] Parser {} failed: {} {}'.format(parser_name, err.__class__.__name__, err)) + # raise + pass + timer.end() + return most_links, best_parser_name + + +@enforce_types +def save_text_as_source(raw_text: str, filename: str='{ts}-stdin.txt', out_dir: Path=OUTPUT_DIR) -> str: + ts = str(datetime.now().timestamp()).split('.', 1)[0] + source_path = str(out_dir / SOURCES_DIR_NAME / filename.format(ts=ts)) + atomic_write(source_path, raw_text) + log_source_saved(source_file=source_path) + return source_path + + +@enforce_types +def save_file_as_source(path: str, timeout: int=TIMEOUT, filename: str='{ts}-{basename}.txt', out_dir: Path=OUTPUT_DIR) -> str: + """download a given url's content into output/sources/domain-<timestamp>.txt""" + ts = str(datetime.now().timestamp()).split('.', 1)[0] + source_path = str(OUTPUT_DIR / SOURCES_DIR_NAME / filename.format(basename=basename(path), ts=ts)) + + if any(path.startswith(s) for s in ('http://', 'https://', 'ftp://')): + # Source is a URL that needs to be downloaded + print(f' > Downloading {path} contents') + timer = TimedProgress(timeout, prefix=' ') + try: + raw_source_text = download_url(path, timeout=timeout) + raw_source_text = htmldecode(raw_source_text) + timer.end() + except Exception as e: + timer.end() + print('{}[!] Failed to download {}{}\n'.format( + ANSI['red'], + path, + ANSI['reset'], + )) + print(' ', e) + raise SystemExit(1) + + else: + # Source is a path to a local file on the filesystem + with open(path, 'r') as f: + raw_source_text = f.read() + + atomic_write(source_path, raw_source_text) + + log_source_saved(source_file=source_path) + + return source_path + + +def check_url_parsing_invariants() -> None: + """Check that plain text regex URL parsing works as expected""" + + # this is last-line-of-defense to make sure the URL_REGEX isn't + # misbehaving, as the consequences could be disastrous and lead to many + # incorrect/badly parsed links being added to the archive + + test_urls = ''' + https://example1.com/what/is/happening.html?what=1#how-about-this=1 + https://example2.com/what/is/happening/?what=1#how-about-this=1 + HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f + https://example4.com/what/is/happening.html + https://example5.com/ + https://example6.com + + <test>http://example7.com</test> + [https://example8.com/what/is/this.php?what=1] + [and http://example9.com?what=1&other=3#and-thing=2] + <what>https://example10.com#and-thing=2 "</about> + abc<this["https://example11.com/what/is#and-thing=2?whoami=23&where=1"]that>def + sdflkf[what](https://example12.com/who/what.php?whoami=1#whatami=2)?am=hi + example13.bada + and example14.badb + <or>htt://example15.badc</that> + ''' + # print('\n'.join(re.findall(URL_REGEX, test_urls))) + assert len(re.findall(URL_REGEX, test_urls)) == 12 + diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/generic_html.py b/archivebox-0.5.3/build/lib/archivebox/parsers/generic_html.py new file mode 100644 index 0000000..74b3d1f --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/generic_html.py @@ -0,0 +1,53 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable, Optional +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + URL_REGEX, +) +from html.parser import HTMLParser +from urllib.parse import urljoin + + +class HrefParser(HTMLParser): + def __init__(self): + super().__init__() + self.urls = [] + + def handle_starttag(self, tag, attrs): + if tag == "a": + for attr, value in attrs: + if attr == "href": + self.urls.append(value) + + +@enforce_types +def parse_generic_html_export(html_file: IO[str], root_url: Optional[str]=None, **_kwargs) -> Iterable[Link]: + """Parse Generic HTML for href tags and use only the url (support for title coming later)""" + + html_file.seek(0) + for line in html_file: + parser = HrefParser() + # example line + # <li><a href="http://example.com/ time_added="1478739709" tags="tag1,tag2">example title</a></li> + parser.feed(line) + for url in parser.urls: + if root_url: + # resolve relative urls /home.html -> https://example.com/home.html + url = urljoin(root_url, url) + + for archivable_url in re.findall(URL_REGEX, url): + yield Link( + url=htmldecode(archivable_url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[html_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/generic_json.py b/archivebox-0.5.3/build/lib/archivebox/parsers/generic_json.py new file mode 100644 index 0000000..e6ed677 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/generic_json.py @@ -0,0 +1,65 @@ +__package__ = 'archivebox.parsers' + +import json + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_generic_json_export(json_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)""" + + json_file.seek(0) + links = json.load(json_file) + json_date = lambda s: datetime.strptime(s, '%Y-%m-%dT%H:%M:%S%z') + + for link in links: + # example line + # {"href":"http:\/\/www.reddit.com\/r\/example","description":"title here","extended":"","meta":"18a973f09c9cc0608c116967b64e0419","hash":"910293f019c2f4bb1a749fb937ba58e3","time":"2014-06-14T15:51:42Z","shared":"no","toread":"no","tags":"reddit android"}] + if link: + # Parse URL + url = link.get('href') or link.get('url') or link.get('URL') + if not url: + raise Exception('JSON must contain URL in each entry [{"url": "http://...", ...}, ...]') + + # Parse the timestamp + ts_str = str(datetime.now().timestamp()) + if link.get('timestamp'): + # chrome/ff histories use a very precise timestamp + ts_str = str(link['timestamp'] / 10000000) + elif link.get('time'): + ts_str = str(json_date(link['time'].split(',', 1)[0]).timestamp()) + elif link.get('created_at'): + ts_str = str(json_date(link['created_at']).timestamp()) + elif link.get('created'): + ts_str = str(json_date(link['created']).timestamp()) + elif link.get('date'): + ts_str = str(json_date(link['date']).timestamp()) + elif link.get('bookmarked'): + ts_str = str(json_date(link['bookmarked']).timestamp()) + elif link.get('saved'): + ts_str = str(json_date(link['saved']).timestamp()) + + # Parse the title + title = None + if link.get('title'): + title = link['title'].strip() + elif link.get('description'): + title = link['description'].replace(' — Readability', '').strip() + elif link.get('name'): + title = link['name'].strip() + + yield Link( + url=htmldecode(url), + timestamp=ts_str, + title=htmldecode(title) or None, + tags=htmldecode(link.get('tags')) or '', + sources=[json_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/generic_rss.py b/archivebox-0.5.3/build/lib/archivebox/parsers/generic_rss.py new file mode 100644 index 0000000..2831844 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/generic_rss.py @@ -0,0 +1,49 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + +@enforce_types +def parse_generic_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse RSS XML-format files into links""" + + rss_file.seek(0) + items = rss_file.read().split('<item>') + items = items[1:] if items else [] + for item in items: + # example item: + # <item> + # <title><![CDATA[How JavaScript works: inside the V8 engine]]> + # Unread + # https://blog.sessionstack.com/how-javascript-works-inside + # https://blog.sessionstack.com/how-javascript-works-inside + # Mon, 21 Aug 2017 14:21:58 -0500 + # + + trailing_removed = item.split('', 1)[0] + leading_removed = trailing_removed.split('', 1)[-1].strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r for r in rows if r.strip().startswith('<{}>'.format(key))][0] + + url = str_between(get_row('link'), '', '') + ts_str = str_between(get_row('pubDate'), '', '') + time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %z") + title = str_between(get_row('title'), ' Iterable[Link]: + """Parse raw links from each line in a text file""" + + text_file.seek(0) + for line in text_file.readlines(): + if not line.strip(): + continue + + # if the line is a local file path that resolves, then we can archive it + try: + if Path(line).exists(): + yield Link( + url=line, + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) + except (OSError, PermissionError): + # nvm, not a valid path... + pass + + # otherwise look for anything that looks like a URL in the line + for url in re.findall(URL_REGEX, line): + yield Link( + url=htmldecode(url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) + + # look inside the URL for any sub-urls, e.g. for archive.org links + # https://web.archive.org/web/20200531203453/https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/ + # -> https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/ + for url in re.findall(URL_REGEX, line[1:]): + yield Link( + url=htmldecode(url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/medium_rss.py b/archivebox-0.5.3/build/lib/archivebox/parsers/medium_rss.py new file mode 100644 index 0000000..8f14f77 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/medium_rss.py @@ -0,0 +1,35 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from xml.etree import ElementTree + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_medium_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Medium RSS feed files into links""" + + rss_file.seek(0) + root = ElementTree.parse(rss_file).getroot() + items = root.find("channel").findall("item") # type: ignore + for item in items: + url = item.find("link").text # type: ignore + title = item.find("title").text.strip() # type: ignore + ts_str = item.find("pubDate").text # type: ignore + time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %Z") # type: ignore + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/netscape_html.py b/archivebox-0.5.3/build/lib/archivebox/parsers/netscape_html.py new file mode 100644 index 0000000..a063023 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/netscape_html.py @@ -0,0 +1,39 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_netscape_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse netscape-format bookmarks export files (produced by all browsers)""" + + html_file.seek(0) + pattern = re.compile("]*>(.+)", re.UNICODE | re.IGNORECASE) + for line in html_file: + # example line + #
    example bookmark title + + match = pattern.search(line) + if match: + url = match.group(1) + time = datetime.fromtimestamp(float(match.group(2))) + title = match.group(3).strip() + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[html_file.name], + ) + diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/pinboard_rss.py b/archivebox-0.5.3/build/lib/archivebox/parsers/pinboard_rss.py new file mode 100644 index 0000000..98ff14a --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/pinboard_rss.py @@ -0,0 +1,47 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from xml.etree import ElementTree + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_pinboard_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Pinboard RSS feed files into links""" + + rss_file.seek(0) + root = ElementTree.parse(rss_file).getroot() + items = root.findall("{http://purl.org/rss/1.0/}item") + for item in items: + find = lambda p: item.find(p).text.strip() if item.find(p) else None # type: ignore + + url = find("{http://purl.org/rss/1.0/}link") + tags = find("{http://purl.org/dc/elements/1.1/}subject") + title = find("{http://purl.org/rss/1.0/}title") + ts_str = find("{http://purl.org/dc/elements/1.1/}date") + + # Pinboard includes a colon in its date stamp timezone offsets, which + # Python can't parse. Remove it: + if ts_str and ts_str[-3:-2] == ":": + ts_str = ts_str[:-3]+ts_str[-2:] + + if ts_str: + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + else: + time = datetime.now() + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=htmldecode(tags) or None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/pocket_api.py b/archivebox-0.5.3/build/lib/archivebox/parsers/pocket_api.py new file mode 100644 index 0000000..bf3a292 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/pocket_api.py @@ -0,0 +1,113 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable, Optional +from configparser import ConfigParser + +from pathlib import Path +from ..vendor.pocket import Pocket + +from ..index.schema import Link +from ..util import enforce_types +from ..system import atomic_write +from ..config import ( + SOURCES_DIR, + POCKET_CONSUMER_KEY, + POCKET_ACCESS_TOKENS, +) + + +COUNT_PER_PAGE = 500 +API_DB_PATH = Path(SOURCES_DIR) / 'pocket_api.db' + +# search for broken protocols that sometimes come from the Pocket API +_BROKEN_PROTOCOL_RE = re.compile('^(http[s]?)(:/(?!/))') + + +def get_pocket_articles(api: Pocket, since=None, page=0): + body, headers = api.get( + state='archive', + sort='oldest', + since=since, + count=COUNT_PER_PAGE, + offset=page * COUNT_PER_PAGE, + ) + + articles = body['list'].values() if isinstance(body['list'], dict) else body['list'] + returned_count = len(articles) + + yield from articles + + if returned_count == COUNT_PER_PAGE: + yield from get_pocket_articles(api, since=since, page=page + 1) + else: + api.last_since = body['since'] + + +def link_from_article(article: dict, sources: list): + url: str = article['resolved_url'] or article['given_url'] + broken_protocol = _BROKEN_PROTOCOL_RE.match(url) + if broken_protocol: + url = url.replace(f'{broken_protocol.group(1)}:/', f'{broken_protocol.group(1)}://') + title = article['resolved_title'] or article['given_title'] or url + + return Link( + url=url, + timestamp=article['time_read'], + title=title, + tags=article.get('tags'), + sources=sources + ) + + +def write_since(username: str, since: str): + if not API_DB_PATH.exists(): + atomic_write(API_DB_PATH, '') + + since_file = ConfigParser() + since_file.optionxform = str + since_file.read(API_DB_PATH) + + since_file[username] = { + 'since': since + } + + with open(API_DB_PATH, 'w+') as new: + since_file.write(new) + + +def read_since(username: str) -> Optional[str]: + if not API_DB_PATH.exists(): + atomic_write(API_DB_PATH, '') + + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(API_DB_PATH) + + return config_file.get(username, 'since', fallback=None) + + +@enforce_types +def should_parse_as_pocket_api(text: str) -> bool: + return text.startswith('pocket://') + + +@enforce_types +def parse_pocket_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Link]: + """Parse bookmarks from the Pocket API""" + + input_buffer.seek(0) + pattern = re.compile(r"^pocket:\/\/(\w+)") + for line in input_buffer: + if should_parse_as_pocket_api(line): + + username = pattern.search(line).group(1) + api = Pocket(POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKENS[username]) + api.last_since = None + + for article in get_pocket_articles(api, since=read_since(username)): + yield link_from_article(article, sources=[line]) + + write_since(username, api.last_since) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/pocket_html.py b/archivebox-0.5.3/build/lib/archivebox/parsers/pocket_html.py new file mode 100644 index 0000000..653f21b --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/pocket_html.py @@ -0,0 +1,38 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_pocket_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)""" + + html_file.seek(0) + pattern = re.compile("^\\s*
  • (.+)
    • ", re.UNICODE) + for line in html_file: + # example line + #
  • example title
    • + match = pattern.search(line) + if match: + url = match.group(1).replace('http://www.readability.com/read?url=', '') # remove old readability prefixes to get original url + time = datetime.fromtimestamp(float(match.group(2))) + tags = match.group(3) + title = match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=tags or '', + sources=[html_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/shaarli_rss.py b/archivebox-0.5.3/build/lib/archivebox/parsers/shaarli_rss.py new file mode 100644 index 0000000..4a925f4 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/shaarli_rss.py @@ -0,0 +1,50 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + + +@enforce_types +def parse_shaarli_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Shaarli-specific RSS XML-format files into links""" + + rss_file.seek(0) + entries = rss_file.read().split('')[1:] + for entry in entries: + # example entry: + # + # Aktuelle Trojaner-Welle: Emotet lauert in gefälschten Rechnungsmails | heise online + # + # https://demo.shaarli.org/?cEV4vw + # 2019-01-30T06:06:01+00:00 + # 2019-01-30T06:06:01+00:00 + #

    — Permalink

    ]]>     + #     + + trailing_removed = entry.split('    ', 1)[0] + leading_removed = trailing_removed.strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0] + + title = str_between(get_row('title'), '', '').strip() + url = str_between(get_row('link'), '') + ts_str = str_between(get_row('published'), '', '') + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/parsers/wallabag_atom.py b/archivebox-0.5.3/build/lib/archivebox/parsers/wallabag_atom.py new file mode 100644 index 0000000..0d77869 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/parsers/wallabag_atom.py @@ -0,0 +1,57 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + + +@enforce_types +def parse_wallabag_atom_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Wallabag Atom files into links""" + + rss_file.seek(0) + entries = rss_file.read().split('')[1:] + for entry in entries: + # example entry: + # + # <![CDATA[Orient Ray vs Mako: Is There Much Difference? - iknowwatches.com]]> + # + # https://iknowwatches.com/orient-ray-vs-mako/ + # wallabag:wallabag.drycat.fr:milosh:entry:14041 + # 2020-10-18T09:14:02+02:00 + # 2020-10-18T09:13:56+02:00 + # + # + # + + trailing_removed = entry.split('', 1)[0] + leading_removed = trailing_removed.strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0] + + title = str_between(get_row('title'), '<![CDATA[', ']]>').strip() + url = str_between(get_row('link rel="via"'), '', '') + ts_str = str_between(get_row('published'), '', '') + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + try: + tags = str_between(get_row('category'), 'label="', '" />') + except: + tags = None + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=tags or '', + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/build/lib/archivebox/search/__init__.py b/archivebox-0.5.3/build/lib/archivebox/search/__init__.py new file mode 100644 index 0000000..6191ede --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/search/__init__.py @@ -0,0 +1,108 @@ +from typing import List, Union +from pathlib import Path +from importlib import import_module + +from django.db.models import QuerySet + +from archivebox.index.schema import Link +from archivebox.util import enforce_types +from archivebox.config import stderr, OUTPUT_DIR, USE_INDEXING_BACKEND, USE_SEARCHING_BACKEND, SEARCH_BACKEND_ENGINE + +from .utils import get_indexable_content, log_index_started + +def indexing_enabled(): + return USE_INDEXING_BACKEND + +def search_backend_enabled(): + return USE_SEARCHING_BACKEND + +def get_backend(): + return f'search.backends.{SEARCH_BACKEND_ENGINE}' + +def import_backend(): + backend_string = get_backend() + try: + backend = import_module(backend_string) + except Exception as err: + raise Exception("Could not load '%s' as a backend: %s" % (backend_string, err)) + return backend + +@enforce_types +def write_search_index(link: Link, texts: Union[List[str], None]=None, out_dir: Path=OUTPUT_DIR, skip_text_index: bool=False) -> None: + if not indexing_enabled(): + return + + if not skip_text_index and texts: + from core.models import Snapshot + + snap = Snapshot.objects.filter(url=link.url).first() + backend = import_backend() + if snap: + try: + backend.index(snapshot_id=str(snap.id), texts=texts) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + +@enforce_types +def query_search_index(query: str, out_dir: Path=OUTPUT_DIR) -> QuerySet: + from core.models import Snapshot + + if search_backend_enabled(): + backend = import_backend() + try: + snapshot_ids = backend.search(query) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + raise + else: + # TODO preserve ordering from backend + qsearch = Snapshot.objects.filter(pk__in=snapshot_ids) + return qsearch + + return Snapshot.objects.none() + +@enforce_types +def flush_search_index(snapshots: QuerySet): + if not indexing_enabled() or not snapshots: + return + backend = import_backend() + snapshot_ids=(str(pk) for pk in snapshots.values_list('pk',flat=True)) + try: + backend.flush(snapshot_ids) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + +@enforce_types +def index_links(links: Union[List[Link],None], out_dir: Path=OUTPUT_DIR): + if not links: + return + + from core.models import Snapshot, ArchiveResult + + for link in links: + snap = Snapshot.objects.filter(url=link.url).first() + if snap: + results = ArchiveResult.objects.indexable().filter(snapshot=snap) + log_index_started(link.url) + try: + texts = get_indexable_content(results) + except Exception as err: + stderr() + stderr( + f'[X] An Exception ocurred reading the indexable content={err}:', + color='red', + ) + else: + write_search_index(link, texts, out_dir=out_dir) diff --git a/archivebox-0.5.3/build/lib/archivebox/search/backends/__init__.py b/archivebox-0.5.3/build/lib/archivebox/search/backends/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/build/lib/archivebox/search/backends/ripgrep.py b/archivebox-0.5.3/build/lib/archivebox/search/backends/ripgrep.py new file mode 100644 index 0000000..840d2d2 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/search/backends/ripgrep.py @@ -0,0 +1,45 @@ +import re +from subprocess import run, PIPE +from typing import List, Generator + +from archivebox.config import ARCHIVE_DIR, RIPGREP_VERSION +from archivebox.util import enforce_types + +RG_IGNORE_EXTENSIONS = ('css','js','orig','svg') + +RG_ADD_TYPE = '--type-add' +RG_IGNORE_ARGUMENTS = f"ignore:*.{{{','.join(RG_IGNORE_EXTENSIONS)}}}" +RG_DEFAULT_ARGUMENTS = "-ilTignore" # Case insensitive(i), matching files results(l) +RG_REGEX_ARGUMENT = '-e' + +TIMESTAMP_REGEX = r'\/([\d]+\.[\d]+)\/' + +ts_regex = re.compile(TIMESTAMP_REGEX) + +@enforce_types +def index(snapshot_id: str, texts: List[str]): + return + +@enforce_types +def flush(snapshot_ids: Generator[str, None, None]): + return + +@enforce_types +def search(text: str) -> List[str]: + if not RIPGREP_VERSION: + raise Exception("ripgrep binary not found, install ripgrep to use this search backend") + + from core.models import Snapshot + + rg_cmd = ['rg', RG_ADD_TYPE, RG_IGNORE_ARGUMENTS, RG_DEFAULT_ARGUMENTS, RG_REGEX_ARGUMENT, text, str(ARCHIVE_DIR)] + rg = run(rg_cmd, stdout=PIPE, stderr=PIPE, timeout=60) + file_paths = [p.decode() for p in rg.stdout.splitlines()] + timestamps = set() + for path in file_paths: + ts = ts_regex.findall(path) + if ts: + timestamps.add(ts[0]) + + snap_ids = [str(id) for id in Snapshot.objects.filter(timestamp__in=timestamps).values_list('pk', flat=True)] + + return snap_ids diff --git a/archivebox-0.5.3/build/lib/archivebox/search/backends/sonic.py b/archivebox-0.5.3/build/lib/archivebox/search/backends/sonic.py new file mode 100644 index 0000000..f0beadd --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/search/backends/sonic.py @@ -0,0 +1,28 @@ +from typing import List, Generator + +from sonic import IngestClient, SearchClient + +from archivebox.util import enforce_types +from archivebox.config import SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD, SONIC_BUCKET, SONIC_COLLECTION + +MAX_SONIC_TEXT_LENGTH = 20000 + +@enforce_types +def index(snapshot_id: str, texts: List[str]): + with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl: + for text in texts: + chunks = [text[i:i+MAX_SONIC_TEXT_LENGTH] for i in range(0, len(text), MAX_SONIC_TEXT_LENGTH)] + for chunk in chunks: + ingestcl.push(SONIC_COLLECTION, SONIC_BUCKET, snapshot_id, str(chunk)) + +@enforce_types +def search(text: str) -> List[str]: + with SearchClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as querycl: + snap_ids = querycl.query(SONIC_COLLECTION, SONIC_BUCKET, text) + return snap_ids + +@enforce_types +def flush(snapshot_ids: Generator[str, None, None]): + with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl: + for id in snapshot_ids: + ingestcl.flush_object(SONIC_COLLECTION, SONIC_BUCKET, str(id)) diff --git a/archivebox-0.5.3/build/lib/archivebox/search/utils.py b/archivebox-0.5.3/build/lib/archivebox/search/utils.py new file mode 100644 index 0000000..55c97e7 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/search/utils.py @@ -0,0 +1,44 @@ +from django.db.models import QuerySet + +from archivebox.util import enforce_types +from archivebox.config import ANSI + +def log_index_started(url): + print('{green}[*] Indexing url: {} in the search index {reset}'.format(url, **ANSI)) + print( ) + +def get_file_result_content(res, extra_path, use_pwd=False): + if use_pwd: + fpath = f'{res.pwd}/{res.output}' + else: + fpath = f'{res.output}' + + if extra_path: + fpath = f'{fpath}/{extra_path}' + + with open(fpath, 'r') as file: + data = file.read() + if data: + return [data] + return [] + + +# This should be abstracted by a plugin interface for extractors +@enforce_types +def get_indexable_content(results: QuerySet): + if not results: + return [] + # Only use the first method available + res, method = results.first(), results.first().extractor + if method not in ('readability', 'singlefile', 'dom', 'wget'): + return [] + # This should come from a plugin interface + + if method == 'readability': + return get_file_result_content(res, 'content.txt') + elif method == 'singlefile': + return get_file_result_content(res, '') + elif method == 'dom': + return get_file_result_content(res,'',use_pwd=True) + elif method == 'wget': + return get_file_result_content(res,'',use_pwd=True) diff --git a/archivebox-0.5.3/build/lib/archivebox/system.py b/archivebox-0.5.3/build/lib/archivebox/system.py new file mode 100644 index 0000000..b27c5e4 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/system.py @@ -0,0 +1,163 @@ +__package__ = 'archivebox' + + +import os +import shutil + +from json import dump +from pathlib import Path +from typing import Optional, Union, Set, Tuple +from subprocess import run as subprocess_run + +from crontab import CronTab +from atomicwrites import atomic_write as lib_atomic_write + +from .util import enforce_types, ExtendedEncoder +from .config import OUTPUT_PERMISSIONS + + + +def run(*args, input=None, capture_output=True, text=False, **kwargs): + """Patched of subprocess.run to fix blocking io making timeout=innefective""" + + if input is not None: + if 'stdin' in kwargs: + raise ValueError('stdin and input arguments may not both be used.') + + if capture_output: + if ('stdout' in kwargs) or ('stderr' in kwargs): + raise ValueError('stdout and stderr arguments may not be used ' + 'with capture_output.') + + return subprocess_run(*args, input=input, capture_output=capture_output, text=text, **kwargs) + + +@enforce_types +def atomic_write(path: Union[Path, str], contents: Union[dict, str, bytes], overwrite: bool=True) -> None: + """Safe atomic write to filesystem by writing to temp file + atomic rename""" + + mode = 'wb+' if isinstance(contents, bytes) else 'w' + + # print('\n> Atomic Write:', mode, path, len(contents), f'overwrite={overwrite}') + try: + with lib_atomic_write(path, mode=mode, overwrite=overwrite) as f: + if isinstance(contents, dict): + dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder) + elif isinstance(contents, (bytes, str)): + f.write(contents) + except OSError as e: + print(f"[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC. ({e})") + print(" For data integrity, ArchiveBox requires a filesystem that supports atomic writes.") + print(" Filesystems and network drives that don't implement FSYNC are incompatible and require workarounds.") + raise SystemExit(1) + os.chmod(path, int(OUTPUT_PERMISSIONS, base=8)) + +@enforce_types +def chmod_file(path: str, cwd: str='.', permissions: str=OUTPUT_PERMISSIONS) -> None: + """chmod -R /""" + + root = Path(cwd) / path + if not root.exists(): + raise Exception('Failed to chmod: {} does not exist (did the previous step fail?)'.format(path)) + + if not root.is_dir(): + os.chmod(root, int(OUTPUT_PERMISSIONS, base=8)) + else: + for subpath in Path(path).glob('**/*'): + os.chmod(subpath, int(OUTPUT_PERMISSIONS, base=8)) + + +@enforce_types +def copy_and_overwrite(from_path: Union[str, Path], to_path: Union[str, Path]): + """copy a given file or directory to a given path, overwriting the destination""" + if Path(from_path).is_dir(): + shutil.rmtree(to_path, ignore_errors=True) + shutil.copytree(from_path, to_path) + else: + with open(from_path, 'rb') as src: + contents = src.read() + atomic_write(to_path, contents) + + +@enforce_types +def get_dir_size(path: Union[str, Path], recursive: bool=True, pattern: Optional[str]=None) -> Tuple[int, int, int]: + """get the total disk size of a given directory, optionally summing up + recursively and limiting to a given filter list + """ + num_bytes, num_dirs, num_files = 0, 0, 0 + for entry in os.scandir(path): + if (pattern is not None) and (pattern not in entry.path): + continue + if entry.is_dir(follow_symlinks=False): + if not recursive: + continue + num_dirs += 1 + bytes_inside, dirs_inside, files_inside = get_dir_size(entry.path) + num_bytes += bytes_inside + num_dirs += dirs_inside + num_files += files_inside + else: + num_bytes += entry.stat(follow_symlinks=False).st_size + num_files += 1 + return num_bytes, num_dirs, num_files + + +CRON_COMMENT = 'archivebox_schedule' + + +@enforce_types +def dedupe_cron_jobs(cron: CronTab) -> CronTab: + deduped: Set[Tuple[str, str]] = set() + + for job in list(cron): + unique_tuple = (str(job.slices), job.command) + if unique_tuple not in deduped: + deduped.add(unique_tuple) + cron.remove(job) + + for schedule, command in deduped: + job = cron.new(command=command, comment=CRON_COMMENT) + job.setall(schedule) + job.enable() + + return cron + + +class suppress_output(object): + ''' + A context manager for doing a "deep suppression" of stdout and stderr in + Python, i.e. will suppress all print, even if the print originates in a + compiled C/Fortran sub-function. + This will not suppress raised exceptions, since exceptions are printed + to stderr just before a script exits, and after the context manager has + exited (at least, I think that is why it lets exceptions through). + + with suppress_stdout_stderr(): + rogue_function() + ''' + def __init__(self, stdout=True, stderr=True): + # Open a pair of null files + # Save the actual stdout (1) and stderr (2) file descriptors. + self.stdout, self.stderr = stdout, stderr + if stdout: + self.null_stdout = os.open(os.devnull, os.O_RDWR) + self.real_stdout = os.dup(1) + if stderr: + self.null_stderr = os.open(os.devnull, os.O_RDWR) + self.real_stderr = os.dup(2) + + def __enter__(self): + # Assign the null pointers to stdout and stderr. + if self.stdout: + os.dup2(self.null_stdout, 1) + if self.stderr: + os.dup2(self.null_stderr, 2) + + def __exit__(self, *_): + # Re-assign the real stdout/stderr back to (1) and (2) + if self.stdout: + os.dup2(self.real_stdout, 1) + os.close(self.null_stdout) + if self.stderr: + os.dup2(self.real_stderr, 2) + os.close(self.null_stderr) diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/admin/actions_as_select.html b/archivebox-0.5.3/build/lib/archivebox/themes/admin/actions_as_select.html new file mode 100644 index 0000000..86a7719 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/admin/actions_as_select.html @@ -0,0 +1 @@ +actions_as_select diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/admin/app_index.html b/archivebox-0.5.3/build/lib/archivebox/themes/admin/app_index.html new file mode 100644 index 0000000..6868b49 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/admin/app_index.html @@ -0,0 +1,18 @@ +{% extends "admin/index.html" %} +{% load i18n %} + +{% block bodyclass %}{{ block.super }} app-{{ app_label }}{% endblock %} + +{% if not is_popup %} +{% block breadcrumbs %} +
    +{% trans 'Home' %} +› +{% for app in app_list %} +{{ app.name }} +{% endfor %} +
    +{% endblock %} +{% endif %} + +{% block sidebar %}{% endblock %} diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/admin/base.html b/archivebox-0.5.3/build/lib/archivebox/themes/admin/base.html new file mode 100644 index 0000000..d8ad8d0 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/admin/base.html @@ -0,0 +1,246 @@ +{% load i18n static %} +{% get_current_language as LANGUAGE_CODE %}{% get_current_language_bidi as LANGUAGE_BIDI %} + + +{% block title %}{% endblock %} | ArchiveBox + +{% block extrastyle %}{% endblock %} +{% if LANGUAGE_BIDI %}{% endif %} +{% block extrahead %}{% endblock %} +{% block responsive %} + + + {% if LANGUAGE_BIDI %}{% endif %} +{% endblock %} +{% block blockbots %}{% endblock %} + + +{% load i18n %} + + + + + + + + + +
    + + {% if not is_popup %} + + + + {% block breadcrumbs %} +
    + {% trans 'Home' %} + {% if title %} › {{ title }}{% endif %} +
    + {% endblock %} + {% endif %} + + {% block messages %} + {% if messages %} +
      {% for message in messages %} + {{ message|capfirst }} + {% endfor %}
    + {% endif %} + {% endblock messages %} + + +
    + {% block pretitle %}{% endblock %} + {% block content_title %}{# {% if title %}

    {{ title }}

    {% endif %} #}{% endblock %} + {% block content %} + {% block object-tools %}{% endblock %} + {{ content }} + {% endblock %} + {% block sidebar %}{% endblock %} +
    +
    + + + {% block footer %}{% endblock %} +
    + + + + + diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/admin/grid_change_list.html b/archivebox-0.5.3/build/lib/archivebox/themes/admin/grid_change_list.html new file mode 100644 index 0000000..6894efd --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/admin/grid_change_list.html @@ -0,0 +1,91 @@ +{% extends "admin/base_site.html" %} +{% load i18n admin_urls static admin_list %} +{% load core_tags %} + +{% block extrastyle %} + {{ block.super }} + + {% if cl.formset %} + + {% endif %} + {% if cl.formset or action_form %} + + {% endif %} + {{ media.css }} + {% if not actions_on_top and not actions_on_bottom %} + + {% endif %} +{% endblock %} + +{% block extrahead %} +{{ block.super }} +{{ media.js }} +{% endblock %} + +{% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-list{% endblock %} + +{% if not is_popup %} +{% block breadcrumbs %} +
    +{% translate 'Home' %} +› {{ cl.opts.app_config.verbose_name }} +› {{ cl.opts.verbose_name_plural|capfirst }} +
    +{% endblock %} +{% endif %} + +{% block coltype %}{% endblock %} + +{% block content %} +
    + {% block object-tools %} +
      + {% block object-tools-items %} + {% change_list_object_tools %} + {% endblock %} +
    + {% endblock %} + {% if cl.formset and cl.formset.errors %} +

    + {% if cl.formset.total_error_count == 1 %}{% translate "Please correct the error below." %}{% else %}{% translate "Please correct the errors below." %}{% endif %} +

    + {{ cl.formset.non_form_errors }} + {% endif %} +
    +
    + {% block search %}{% search_form cl %}{% endblock %} + {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %} + +
    {% csrf_token %} + {% if cl.formset %} +
    {{ cl.formset.management_form }}
    + {% endif %} + + {% block result_list %} + {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %} + {% comment %} + Table grid + {% result_list cl %} + {% endcomment %} + {% snapshots_grid cl %} + {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %} + {% endblock %} + {% block pagination %}{% pagination cl %}{% endblock %} +
    +
    + {% block filters %} + {% if cl.has_filters %} +
    +

    {% translate 'Filter' %}

    + {% if cl.has_active_filters %}

    + ✖ {% translate "Clear all filters" %} +

    {% endif %} + {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %} +
    + {% endif %} + {% endblock %} +
    +
    +{% endblock %} \ No newline at end of file diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/admin/login.html b/archivebox-0.5.3/build/lib/archivebox/themes/admin/login.html new file mode 100644 index 0000000..98283f8 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/admin/login.html @@ -0,0 +1,100 @@ +{% extends "admin/base_site.html" %} +{% load i18n static %} + +{% block extrastyle %}{{ block.super }} +{{ form.media }} +{% endblock %} + +{% block bodyclass %}{{ block.super }} login{% endblock %} + +{% block branding %}

    ArchiveBox Admin

    {% endblock %} + +{% block usertools %} +
    + Back to Main Index +{% endblock %} + +{% block nav-global %}{% endblock %} + +{% block content_title %} +
    + Log in to add, edit, and remove links from your archive. +


    +
    +{% endblock %} + +{% block breadcrumbs %}{% endblock %} + +{% block content %} +{% if form.errors and not form.non_field_errors %} +

    +{% if form.errors.items|length == 1 %}{% trans "Please correct the error below." %}{% else %}{% trans "Please correct the errors below." %}{% endif %} +

    +{% endif %} + +{% if form.non_field_errors %} +{% for error in form.non_field_errors %} +

    + {{ error }} +

    +{% endfor %} +{% endif %} + +
    + +{% if user.is_authenticated %} +

    +{% blocktrans trimmed %} + You are authenticated as {{ username }}, but are not authorized to + access this page. Would you like to login to a different account? +{% endblocktrans %} +

    +{% endif %} + +
    +
    {% csrf_token %} +
    + {{ form.username.errors }} + {{ form.username.label_tag }} {{ form.username }} +
    +
    + {{ form.password.errors }} + {{ form.password.label_tag }} {{ form.password }} + +
    + {% url 'admin_password_reset' as password_reset_url %} + {% if password_reset_url %} +
    + {% trans 'Forgotten your password or username?' %} +
    + {% endif %} +
    + +
    +
    + +
    +

    +
    +
    + If you forgot your password, reset it here or run:
    +
    +archivebox manage changepassword USERNAME
+
    + +

    +
    +
    + To create a new admin user, run the following: +
    +archivebox manage createsuperuser
+
    +
    +
    + + (cd into your archive folder before running commands) +
    + + +
    +{% endblock %} diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/admin/snapshots_grid.html b/archivebox-0.5.3/build/lib/archivebox/themes/admin/snapshots_grid.html new file mode 100644 index 0000000..a7a2d4f --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/admin/snapshots_grid.html @@ -0,0 +1,162 @@ +{% load i18n admin_urls static admin_list %} +{% load core_tags %} + +{% block extrastyle %} + + +{% endblock %} + +{% block content %} +
    + {% for obj in results %} +
    + + + + + +
    + {% if obj.tags_str %} +

    {{obj.tags_str}}

    + {% endif %} + {% if obj.title %} + +

    {{obj.title|truncatechars:55 }}

    +     + {% endif %} + {% comment %}

    TEXT If needed.

    {% endcomment %} +
    +
    +
    + + 🕑 {{obj.added}} + 📖{{obj.num_outputs}} + 🗄️{{ obj.archive_size | file_size }} +
    +
    +
    + {% endfor %} +
    + +{% endblock %} \ No newline at end of file diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/add_links.html b/archivebox-0.5.3/build/lib/archivebox/themes/default/add_links.html new file mode 100644 index 0000000..0b384f5 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/add_links.html @@ -0,0 +1,71 @@ +{% extends "base.html" %} + +{% load static %} +{% load i18n %} + +{% block breadcrumbs %} +
    + {% trans 'Home' %} + {% if title %} › {{ title }}{% endif %} +
    +{% endblock %} + +{% block extra_head %} + +{% endblock %} + +{% block body %} +
    +

    + {% if stdout %} +

    Add new URLs to your archive: results

    + 
    +                {{ stdout | safe }}
+                

    
+
    +
    +
    +   Add more URLs ➕ +
    + {% else %} +
    {% csrf_token %} +

    Add new URLs to your archive

    +
    + {{ form.as_p }} +
    + +
    +
    +


    + + {% if absolute_add_path %} +
    +

    Bookmark this link to quickly add to your archive: + Add to ArchiveBox

    +
    + {% endif %} + + {% endif %} +
    +{% endblock %} + +{% block sidebar %}{% endblock %} diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/base.html b/archivebox-0.5.3/build/lib/archivebox/themes/default/base.html new file mode 100644 index 0000000..a70430e --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/base.html @@ -0,0 +1,284 @@ +{% load static %} + + + + + + Archived Sites + + + + + + {% block extra_head %} + {% endblock %} + + + + + + +
    +
    +
    +
    + + Logo + ArchiveBox: Index + +
    +
    + Add Links   |   + Admin   |   + Docs +
    +
    +
    +
    + {% block body %} + {% endblock %} +
    + + + + diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/core/snapshot_list.html b/archivebox-0.5.3/build/lib/archivebox/themes/default/core/snapshot_list.html new file mode 100644 index 0000000..ce2b2fa --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/core/snapshot_list.html @@ -0,0 +1,51 @@ +{% extends "base.html" %} +{% load static %} + +{% block body %} +
    +
    + + + +
    + + + + + + + + + + + {% for link in object_list %} + {% include 'main_index_row.html' with link=link %} + {% endfor %} + +
    BookmarkedSnapshot ({{object_list|length}})FilesOriginal URL
    +
    + + {% if page_obj.has_previous %} + « first + previous + {% endif %} + + + Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}. + + + {% if page_obj.has_next %} + next + last » + {% endif %} + + + {% if page_obj.has_next %} + next + last » + {% endif %} + +
    +
    +{% endblock %} diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/link_details.html b/archivebox-0.5.3/build/lib/archivebox/themes/default/link_details.html new file mode 100644 index 0000000..b1edcfe --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/link_details.html @@ -0,0 +1,488 @@ + + + + {{title}} + + + + + +
    +
    +
    +
    + + + Archive Icon + ArchiveBox + +
    +
    + Favicon +    + {{title}} +    + ▾ +
    + + + {{url_str}} + + +
    +
    +
    +
    +
    +
    +
    +
    Added
    + {{bookmarked_date}} +
    +
    +
    First Archived
    + {{oldest_archive_date}} +
    +
    +
    Last Checked
    + {{updated_date}} +
    +
    +
    +
    +
    Type
    +
    {{extension}}
    +
    +
    +
    Tags
    +
    {{tags}}
    +
    +
    +
    Status
    +
    {{status}}
    +
    +
    +
    Saved
    + ✅ {{num_outputs}} +
    +
    +
    Errors
    + ❌ {{num_failures}} +
    +
    +
    Size
    + {{size}} +
    +
    +
    +
    +
    🗃 Files
    + JSON | + WARC | + Media | + Git | + Favicon | + See all... +
    +
    +
    +
    +
    +
    + +
    + + + +

    Wget > WARC

     +

    archive/{{domain}}

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > SingleFile

     +

    archive/singlefile.html

    +
    +
    +
    +
    +
    + +
    + + + +

    Archive.Org

     +

    web.archive.org/web/...

    +
    +
    +
    +
    +
    + +
    + + + +

    Original

     +

    {{domain}}

    +
    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > PDF

     +

    archive/output.pdf

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > Screenshot

     +

    archive/screenshot.png

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > HTML

     +

    archive/output.html

    +
    +
    +
    +
    +
    + +
    + + + +

    Readability

     +

    archive/readability/...

    +
    +
    +
    +
    +
    +
    + +
    + + + +

    mercury

     +

    archive/mercury/...

    +
    +
    +
    +
    +
    +
    + + + + + + + + diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index.html b/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index.html new file mode 100644 index 0000000..95af196 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index.html @@ -0,0 +1,255 @@ +{% load static %} + + + + + Archived Sites + + + + + + + + + +
    +
    +
    +
    + + Logo + ArchiveBox: Index + +
    +
    + Add Links   |   + Admin   |   + Docs +
    +
    +
    +
    + + + + + + + + + + + + {% for link in links %} + {% include 'main_index_row.html' with link=link %} + {% endfor %} + +
    BookmarkedSaved Link ({{num_links}})FilesOriginal URL
    + + + diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index_minimal.html b/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index_minimal.html new file mode 100644 index 0000000..dcfaa23 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index_minimal.html @@ -0,0 +1,24 @@ + + + + Archived Sites + + + + + + + + + + + + + + {% for link in links %} + {% include "main_index_row.html" with link=link %} + {% endfor %} + +
    BookmarkedSaved Link ({{num_links}})FilesOriginal URL
    + + \ No newline at end of file diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index_row.html b/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index_row.html new file mode 100644 index 0000000..5e21a8c --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/main_index_row.html @@ -0,0 +1,22 @@ +{% load static %} + + + {% if link.bookmarked_date %} {{ link.bookmarked_date }} {% else %} {{ link.added }} {% endif %} + + {% if link.is_archived %} + + {% else %} + + {% endif %} + + {{link.title|default:'Loading...'}} + {% if link.tags_str != None %} {{link.tags_str|default:''}} {% else %} {{ link.tags|default:'' }} {% endif %} + + + + 📄 + {% if link.icons %} {{link.icons}} {% else %} {{ link.num_outputs}} {% endif %} + + + {{link.url}} + \ No newline at end of file diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/add.css b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/add.css new file mode 100644 index 0000000..b128bf4 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/add.css @@ -0,0 +1,62 @@ +.dashboard #content { + width: 100%; + margin-right: 0px; + margin-left: 0px; +} +#submit { + border: 1px solid rgba(0, 0, 0, 0.2); + padding: 10px; + border-radius: 4px; + background-color: #f5dd5d; + color: #333; + font-size: 18px; + font-weight: 800; +} +#add-form button[role="submit"]:hover { + background-color: #e5cd4d; +} +#add-form label { + display: block; + font-size: 16px; +} +#add-form textarea { + width: 100%; + min-height: 300px; +} +#delay-warning div { + border: 1px solid red; + border-radius: 4px; + margin: 10px; + padding: 10px; + font-size: 15px; + background-color: #f5dd5d; +} +#stdout { + background-color: #ded; + padding: 10px 10px; + border-radius: 4px; + white-space: normal; +} +ul#id_depth { + list-style-type: none; + padding: 0; +} + +@keyframes spin { + 0% { + transform: rotate(0deg); + } + 100% { + transform: rotate(360deg); + } +} + +.loader { + border: 16px solid #f3f3f3; /* Light grey */ + border-top: 16px solid #3498db; /* Blue */ + border-radius: 50%; + width: 30px; + height: 30px; + box-sizing: border-box; + animation: spin 2s linear infinite; +} diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/admin.css b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/admin.css new file mode 100644 index 0000000..181c06d --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/admin.css @@ -0,0 +1,234 @@ +#logo { + height: 30px; + vertical-align: -6px; + padding-right: 5px; +} +#site-name:hover a { + opacity: 0.9; +} +#site-name .loader { + height: 25px; + width: 25px; + display: inline-block; + border-width: 3px; + vertical-align: -3px; + margin-right: 5px; + margin-top: 2px; +} +#branding h1, #branding h1 a:link, #branding h1 a:visited { + color: mintcream; +} +#header { + background: #aa1e55; + padding: 6px 14px; +} +#content { + padding: 8px 8px; +} +#user-tools { + font-size: 13px; + +} + +div.breadcrumbs { + background: #772948; + color: #f5dd5d; + padding: 6px 15px; +} + +body.model-snapshot.change-list div.breadcrumbs, +body.model-snapshot.change-list #content .object-tools { + display: none; +} + +.module h2, .module caption, .inline-group h2 { + background: #772948; +} + +#content .object-tools { + margin-top: -35px; + margin-right: -10px; + float: right; +} + +#content .object-tools a:link, #content .object-tools a:visited { + border-radius: 0px; + background-color: #f5dd5d; + color: #333; + font-size: 12px; + font-weight: 800; +} + +#content .object-tools a.addlink { + background-blend-mode: difference; +} + +#content #changelist #toolbar { + padding: 0px; + background: none; + margin-bottom: 10px; + border-top: 0px; + border-bottom: 0px; +} + +#content #changelist #toolbar form input[type="submit"] { + border-color: #aa1e55; +} + +#content #changelist-filter li.selected a { + color: #aa1e55; +} + + +/*#content #changelist .actions { + position: fixed; + bottom: 0px; + z-index: 800; +}*/ +#content #changelist .actions { + float: right; + margin-top: -34px; + padding: 0px; + background: none; + margin-right: 0px; + width: auto; +} + +#content #changelist .actions .button { + border-radius: 2px; + background-color: #f5dd5d; + color: #333; + font-size: 12px; + font-weight: 800; + margin-right: 4px; + box-shadow: 4px 4px 4px rgba(0,0,0,0.02); + border: 1px solid rgba(0,0,0,0.08); +} +#content #changelist .actions .button:hover { + border: 1px solid rgba(0,0,0,0.2); + opacity: 0.9; +} +#content #changelist .actions .button[name=verify_snapshots], #content #changelist .actions .button[name=update_titles] { + background-color: #dedede; + color: #333; +} +#content #changelist .actions .button[name=update_snapshots] { + background-color:lightseagreen; + color: #333; +} +#content #changelist .actions .button[name=overwrite_snapshots] { + background-color: #ffaa31; + color: #333; +} +#content #changelist .actions .button[name=delete_snapshots] { + background-color: #f91f74; + color: rgb(255 248 252 / 64%); +} + + +#content #changelist-filter h2 { + border-radius: 4px 4px 0px 0px; +} + +@media (min-width: 767px) { + #content #changelist-filter { + top: 35px; + width: 110px; + margin-bottom: 35px; + } + + .change-list .filtered .results, + .change-list .filtered .paginator, + .filtered #toolbar, + .filtered div.xfull { + margin-right: 115px; + } +} + +@media (max-width: 1127px) { + #content #changelist .actions { + position: fixed; + bottom: 6px; + left: 10px; + float: left; + z-index: 1000; + } +} + +#content a img.favicon { + height: 20px; + width: 20px; + vertical-align: -5px; + padding-right: 6px; +} + +#content td, #content th { + vertical-align: middle; + padding: 4px; +} + +#content #changelist table input { + vertical-align: -2px; +} + +#content thead th .text a { + padding: 8px 4px; +} + +#content th.field-added, #content td.field-updated { + word-break: break-word; + min-width: 128px; + white-space: normal; +} + +#content th.field-title_str { + min-width: 300px; +} + +#content td.field-files { + white-space: nowrap; +} +#content td.field-files .exists-True { + opacity: 1; +} +#content td.field-files .exists-False { + opacity: 0.1; + filter: grayscale(100%); +} +#content td.field-size { + white-space: nowrap; +} + +#content td.field-url_str { + word-break: break-all; + min-width: 200px; +} + +#content tr b.status-pending { + font-weight: 200; + opacity: 0.6; +} + +.loader { + border: 16px solid #f3f3f3; /* Light grey */ + border-top: 16px solid #3498db; /* Blue */ + border-radius: 50%; + width: 30px; + height: 30px; + box-sizing: border-box; + animation: spin 2s linear infinite; +} + +@keyframes spin { + 0% { transform: rotate(0deg); } + 100% { transform: rotate(360deg); } +} + +.tags > a > .tag { + float: right; + border-radius: 5px; + background-color: #bfdfff; + padding: 2px 5px; + margin-left: 4px; + margin-top: 1px; +} diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/archive.png b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/archive.png new file mode 100644 index 0000000..307b450 Binary files /dev/null and b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/archive.png differ diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/bootstrap.min.css b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/bootstrap.min.css new file mode 100644 index 0000000..a8da074 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/bootstrap.min.css @@ -0,0 +1,6 @@ +/*! + * Bootstrap v4.0.0-alpha.6 (https://getbootstrap.com) + * Copyright 2011-2017 The Bootstrap Authors + * Copyright 2011-2017 Twitter, Inc. + * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE) + *//*! normalize.css v5.0.0 | MIT License | github.com/necolas/normalize.css */html{font-family:sans-serif;line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}figcaption,figure,main{display:block}figure{margin:1em 40px}hr{-webkit-box-sizing:content-box;box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1em}a{background-color:transparent;-webkit-text-decoration-skip:objects}a:active,a:hover{outline-width:0}abbr[title]{border-bottom:none;text-decoration:underline;text-decoration:underline dotted}b,strong{font-weight:inherit}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1em}dfn{font-style:italic}mark{background-color:#ff0;color:#000}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}audio,video{display:inline-block}audio:not([controls]){display:none;height:0}img{border-style:none}svg:not(:root){overflow:hidden}button,input,optgroup,select,textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}[type=reset],[type=submit],button,html [type=button]{-webkit-appearance:button}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}legend{-webkit-box-sizing:border-box;box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{display:inline-block;vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{-webkit-box-sizing:border-box;box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-cancel-button,[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details,menu{display:block}summary{display:list-item}canvas{display:inline-block}template{display:none}[hidden]{display:none}@media print{*,::after,::before,blockquote::first-letter,blockquote::first-line,div::first-letter,div::first-line,li::first-letter,li::first-line,p::first-letter,p::first-line{text-shadow:none!important;-webkit-box-shadow:none!important;box-shadow:none!important}a,a:visited{text-decoration:underline}abbr[title]::after{content:" (" attr(title) ")"}pre{white-space:pre-wrap!important}blockquote,pre{border:1px solid #999;page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}h2,h3,p{orphans:3;widows:3}h2,h3{page-break-after:avoid}.navbar{display:none}.badge{border:1px solid #000}.table{border-collapse:collapse!important}.table td,.table th{background-color:#fff!important}.table-bordered td,.table-bordered th{border:1px solid #ddd!important}}html{-webkit-box-sizing:border-box;box-sizing:border-box}*,::after,::before{-webkit-box-sizing:inherit;box-sizing:inherit}@-ms-viewport{width:device-width}html{-ms-overflow-style:scrollbar;-webkit-tap-highlight-color:transparent}body{font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-size:1rem;font-weight:400;line-height:1.5;color:#292b2c;background-color:#fff}[tabindex="-1"]:focus{outline:0!important}h1,h2,h3,h4,h5,h6{margin-top:0;margin-bottom:.5rem}p{margin-top:0;margin-bottom:1rem}abbr[data-original-title],abbr[title]{cursor:help}address{margin-bottom:1rem;font-style:normal;line-height:inherit}dl,ol,ul{margin-top:0;margin-bottom:1rem}ol ol,ol ul,ul ol,ul ul{margin-bottom:0}dt{font-weight:700}dd{margin-bottom:.5rem;margin-left:0}blockquote{margin:0 0 1rem}a{color:#0275d8;text-decoration:none}a:focus,a:hover{color:#014c8c;text-decoration:underline}a:not([href]):not([tabindex]){color:inherit;text-decoration:none}a:not([href]):not([tabindex]):focus,a:not([href]):not([tabindex]):hover{color:inherit;text-decoration:none}a:not([href]):not([tabindex]):focus{outline:0}pre{margin-top:0;margin-bottom:1rem;overflow:auto}figure{margin:0 0 1rem}img{vertical-align:middle}[role=button]{cursor:pointer}[role=button],a,area,button,input,label,select,summary,textarea{-ms-touch-action:manipulation;touch-action:manipulation}table{border-collapse:collapse;background-color:transparent}caption{padding-top:.75rem;padding-bottom:.75rem;color:#636c72;text-align:left;caption-side:bottom}th{text-align:left}label{display:inline-block;margin-bottom:.5rem}button:focus{outline:1px dotted;outline:5px auto -webkit-focus-ring-color}button,input,select,textarea{line-height:inherit}input[type=checkbox]:disabled,input[type=radio]:disabled{cursor:not-allowed}input[type=date],input[type=time],input[type=datetime-local],input[type=month]{-webkit-appearance:listbox}textarea{resize:vertical}fieldset{min-width:0;padding:0;margin:0;border:0}legend{display:block;width:100%;padding:0;margin-bottom:.5rem;font-size:1.5rem;line-height:inherit}input[type=search]{-webkit-appearance:none}output{display:inline-block}[hidden]{display:none!important}.h1,.h2,.h3,.h4,.h5,.h6,h1,h2,h3,h4,h5,h6{margin-bottom:.5rem;font-family:inherit;font-weight:500;line-height:1.1;color:inherit}.h1,h1{font-size:2.5rem}.h2,h2{font-size:2rem}.h3,h3{font-size:1.75rem}.h4,h4{font-size:1.5rem}.h5,h5{font-size:1.25rem}.h6,h6{font-size:1rem}.lead{font-size:1.25rem;font-weight:300}.display-1{font-size:6rem;font-weight:300;line-height:1.1}.display-2{font-size:5.5rem;font-weight:300;line-height:1.1}.display-3{font-size:4.5rem;font-weight:300;line-height:1.1}.display-4{font-size:3.5rem;font-weight:300;line-height:1.1}hr{margin-top:1rem;margin-bottom:1rem;border:0;border-top:1px solid rgba(0,0,0,.1)}.small,small{font-size:80%;font-weight:400}.mark,mark{padding:.2em;background-color:#fcf8e3}.list-unstyled{padding-left:0;list-style:none}.list-inline{padding-left:0;list-style:none}.list-inline-item{display:inline-block}.list-inline-item:not(:last-child){margin-right:5px}.initialism{font-size:90%;text-transform:uppercase}.blockquote{padding:.5rem 1rem;margin-bottom:1rem;font-size:1.25rem;border-left:.25rem solid #eceeef}.blockquote-footer{display:block;font-size:80%;color:#636c72}.blockquote-footer::before{content:"\2014 \00A0"}.blockquote-reverse{padding-right:1rem;padding-left:0;text-align:right;border-right:.25rem solid #eceeef;border-left:0}.blockquote-reverse .blockquote-footer::before{content:""}.blockquote-reverse .blockquote-footer::after{content:"\00A0 \2014"}.img-fluid{max-width:100%;height:auto}.img-thumbnail{padding:.25rem;background-color:#fff;border:1px solid #ddd;border-radius:.25rem;-webkit-transition:all .2s ease-in-out;-o-transition:all .2s ease-in-out;transition:all .2s ease-in-out;max-width:100%;height:auto}.figure{display:inline-block}.figure-img{margin-bottom:.5rem;line-height:1}.figure-caption{font-size:90%;color:#636c72}code,kbd,pre,samp{font-family:Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace}code{padding:.2rem .4rem;font-size:90%;color:#bd4147;background-color:#f7f7f9;border-radius:.25rem}a>code{padding:0;color:inherit;background-color:inherit}kbd{padding:.2rem .4rem;font-size:90%;color:#fff;background-color:#292b2c;border-radius:.2rem}kbd kbd{padding:0;font-size:100%;font-weight:700}pre{display:block;margin-top:0;margin-bottom:1rem;font-size:90%;color:#292b2c}pre code{padding:0;font-size:inherit;color:inherit;background-color:transparent;border-radius:0}.pre-scrollable{max-height:340px;overflow-y:scroll}.container{position:relative;margin-left:auto;margin-right:auto;padding-right:15px;padding-left:15px}@media (min-width:576px){.container{padding-right:15px;padding-left:15px}}@media (min-width:768px){.container{padding-right:15px;padding-left:15px}}@media (min-width:992px){.container{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.container{padding-right:15px;padding-left:15px}}@media (min-width:576px){.container{width:540px;max-width:100%}}@media (min-width:768px){.container{width:720px;max-width:100%}}@media (min-width:992px){.container{width:960px;max-width:100%}}@media (min-width:1200px){.container{width:1140px;max-width:100%}}.container-fluid{position:relative;margin-left:auto;margin-right:auto;padding-right:15px;padding-left:15px}@media (min-width:576px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:768px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:992px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.container-fluid{padding-right:15px;padding-left:15px}}.row{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;margin-right:-15px;margin-left:-15px}@media (min-width:576px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:768px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:992px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:1200px){.row{margin-right:-15px;margin-left:-15px}}.no-gutters{margin-right:0;margin-left:0}.no-gutters>.col,.no-gutters>[class*=col-]{padding-right:0;padding-left:0}.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{position:relative;width:100%;min-height:1px;padding-right:15px;padding-left:15px}@media (min-width:576px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:768px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:992px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}.col{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-0{right:auto}.pull-1{right:8.333333%}.pull-2{right:16.666667%}.pull-3{right:25%}.pull-4{right:33.333333%}.pull-5{right:41.666667%}.pull-6{right:50%}.pull-7{right:58.333333%}.pull-8{right:66.666667%}.pull-9{right:75%}.pull-10{right:83.333333%}.pull-11{right:91.666667%}.pull-12{right:100%}.push-0{left:auto}.push-1{left:8.333333%}.push-2{left:16.666667%}.push-3{left:25%}.push-4{left:33.333333%}.push-5{left:41.666667%}.push-6{left:50%}.push-7{left:58.333333%}.push-8{left:66.666667%}.push-9{left:75%}.push-10{left:83.333333%}.push-11{left:91.666667%}.push-12{left:100%}.offset-1{margin-left:8.333333%}.offset-2{margin-left:16.666667%}.offset-3{margin-left:25%}.offset-4{margin-left:33.333333%}.offset-5{margin-left:41.666667%}.offset-6{margin-left:50%}.offset-7{margin-left:58.333333%}.offset-8{margin-left:66.666667%}.offset-9{margin-left:75%}.offset-10{margin-left:83.333333%}.offset-11{margin-left:91.666667%}@media (min-width:576px){.col-sm{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-sm-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-sm-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-sm-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-sm-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-sm-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-sm-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-sm-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-sm-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-sm-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-sm-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-sm-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-sm-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-sm-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-sm-0{right:auto}.pull-sm-1{right:8.333333%}.pull-sm-2{right:16.666667%}.pull-sm-3{right:25%}.pull-sm-4{right:33.333333%}.pull-sm-5{right:41.666667%}.pull-sm-6{right:50%}.pull-sm-7{right:58.333333%}.pull-sm-8{right:66.666667%}.pull-sm-9{right:75%}.pull-sm-10{right:83.333333%}.pull-sm-11{right:91.666667%}.pull-sm-12{right:100%}.push-sm-0{left:auto}.push-sm-1{left:8.333333%}.push-sm-2{left:16.666667%}.push-sm-3{left:25%}.push-sm-4{left:33.333333%}.push-sm-5{left:41.666667%}.push-sm-6{left:50%}.push-sm-7{left:58.333333%}.push-sm-8{left:66.666667%}.push-sm-9{left:75%}.push-sm-10{left:83.333333%}.push-sm-11{left:91.666667%}.push-sm-12{left:100%}.offset-sm-0{margin-left:0}.offset-sm-1{margin-left:8.333333%}.offset-sm-2{margin-left:16.666667%}.offset-sm-3{margin-left:25%}.offset-sm-4{margin-left:33.333333%}.offset-sm-5{margin-left:41.666667%}.offset-sm-6{margin-left:50%}.offset-sm-7{margin-left:58.333333%}.offset-sm-8{margin-left:66.666667%}.offset-sm-9{margin-left:75%}.offset-sm-10{margin-left:83.333333%}.offset-sm-11{margin-left:91.666667%}}@media (min-width:768px){.col-md{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-md-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-md-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-md-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-md-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-md-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-md-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-md-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-md-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-md-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-md-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-md-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-md-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-md-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-md-0{right:auto}.pull-md-1{right:8.333333%}.pull-md-2{right:16.666667%}.pull-md-3{right:25%}.pull-md-4{right:33.333333%}.pull-md-5{right:41.666667%}.pull-md-6{right:50%}.pull-md-7{right:58.333333%}.pull-md-8{right:66.666667%}.pull-md-9{right:75%}.pull-md-10{right:83.333333%}.pull-md-11{right:91.666667%}.pull-md-12{right:100%}.push-md-0{left:auto}.push-md-1{left:8.333333%}.push-md-2{left:16.666667%}.push-md-3{left:25%}.push-md-4{left:33.333333%}.push-md-5{left:41.666667%}.push-md-6{left:50%}.push-md-7{left:58.333333%}.push-md-8{left:66.666667%}.push-md-9{left:75%}.push-md-10{left:83.333333%}.push-md-11{left:91.666667%}.push-md-12{left:100%}.offset-md-0{margin-left:0}.offset-md-1{margin-left:8.333333%}.offset-md-2{margin-left:16.666667%}.offset-md-3{margin-left:25%}.offset-md-4{margin-left:33.333333%}.offset-md-5{margin-left:41.666667%}.offset-md-6{margin-left:50%}.offset-md-7{margin-left:58.333333%}.offset-md-8{margin-left:66.666667%}.offset-md-9{margin-left:75%}.offset-md-10{margin-left:83.333333%}.offset-md-11{margin-left:91.666667%}}@media (min-width:992px){.col-lg{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-lg-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-lg-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-lg-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-lg-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-lg-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-lg-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-lg-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-lg-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-lg-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-lg-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-lg-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-lg-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-lg-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-lg-0{right:auto}.pull-lg-1{right:8.333333%}.pull-lg-2{right:16.666667%}.pull-lg-3{right:25%}.pull-lg-4{right:33.333333%}.pull-lg-5{right:41.666667%}.pull-lg-6{right:50%}.pull-lg-7{right:58.333333%}.pull-lg-8{right:66.666667%}.pull-lg-9{right:75%}.pull-lg-10{right:83.333333%}.pull-lg-11{right:91.666667%}.pull-lg-12{right:100%}.push-lg-0{left:auto}.push-lg-1{left:8.333333%}.push-lg-2{left:16.666667%}.push-lg-3{left:25%}.push-lg-4{left:33.333333%}.push-lg-5{left:41.666667%}.push-lg-6{left:50%}.push-lg-7{left:58.333333%}.push-lg-8{left:66.666667%}.push-lg-9{left:75%}.push-lg-10{left:83.333333%}.push-lg-11{left:91.666667%}.push-lg-12{left:100%}.offset-lg-0{margin-left:0}.offset-lg-1{margin-left:8.333333%}.offset-lg-2{margin-left:16.666667%}.offset-lg-3{margin-left:25%}.offset-lg-4{margin-left:33.333333%}.offset-lg-5{margin-left:41.666667%}.offset-lg-6{margin-left:50%}.offset-lg-7{margin-left:58.333333%}.offset-lg-8{margin-left:66.666667%}.offset-lg-9{margin-left:75%}.offset-lg-10{margin-left:83.333333%}.offset-lg-11{margin-left:91.666667%}}@media (min-width:1200px){.col-xl{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-xl-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-xl-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-xl-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-xl-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-xl-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-xl-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-xl-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-xl-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-xl-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-xl-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-xl-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-xl-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-xl-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-xl-0{right:auto}.pull-xl-1{right:8.333333%}.pull-xl-2{right:16.666667%}.pull-xl-3{right:25%}.pull-xl-4{right:33.333333%}.pull-xl-5{right:41.666667%}.pull-xl-6{right:50%}.pull-xl-7{right:58.333333%}.pull-xl-8{right:66.666667%}.pull-xl-9{right:75%}.pull-xl-10{right:83.333333%}.pull-xl-11{right:91.666667%}.pull-xl-12{right:100%}.push-xl-0{left:auto}.push-xl-1{left:8.333333%}.push-xl-2{left:16.666667%}.push-xl-3{left:25%}.push-xl-4{left:33.333333%}.push-xl-5{left:41.666667%}.push-xl-6{left:50%}.push-xl-7{left:58.333333%}.push-xl-8{left:66.666667%}.push-xl-9{left:75%}.push-xl-10{left:83.333333%}.push-xl-11{left:91.666667%}.push-xl-12{left:100%}.offset-xl-0{margin-left:0}.offset-xl-1{margin-left:8.333333%}.offset-xl-2{margin-left:16.666667%}.offset-xl-3{margin-left:25%}.offset-xl-4{margin-left:33.333333%}.offset-xl-5{margin-left:41.666667%}.offset-xl-6{margin-left:50%}.offset-xl-7{margin-left:58.333333%}.offset-xl-8{margin-left:66.666667%}.offset-xl-9{margin-left:75%}.offset-xl-10{margin-left:83.333333%}.offset-xl-11{margin-left:91.666667%}}.table{width:100%;max-width:100%;margin-bottom:1rem}.table td,.table th{padding:.75rem;vertical-align:top;border-top:1px solid #eceeef}.table thead th{vertical-align:bottom;border-bottom:2px solid #eceeef}.table tbody+tbody{border-top:2px solid #eceeef}.table .table{background-color:#fff}.table-sm td,.table-sm th{padding:.3rem}.table-bordered{border:1px solid #eceeef}.table-bordered td,.table-bordered th{border:1px solid #eceeef}.table-bordered thead td,.table-bordered thead th{border-bottom-width:2px}.table-striped tbody tr:nth-of-type(odd){background-color:rgba(0,0,0,.05)}.table-hover tbody tr:hover{background-color:rgba(0,0,0,.075)}.table-active,.table-active>td,.table-active>th{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover>td,.table-hover .table-active:hover>th{background-color:rgba(0,0,0,.075)}.table-success,.table-success>td,.table-success>th{background-color:#dff0d8}.table-hover .table-success:hover{background-color:#d0e9c6}.table-hover .table-success:hover>td,.table-hover .table-success:hover>th{background-color:#d0e9c6}.table-info,.table-info>td,.table-info>th{background-color:#d9edf7}.table-hover .table-info:hover{background-color:#c4e3f3}.table-hover .table-info:hover>td,.table-hover .table-info:hover>th{background-color:#c4e3f3}.table-warning,.table-warning>td,.table-warning>th{background-color:#fcf8e3}.table-hover .table-warning:hover{background-color:#faf2cc}.table-hover .table-warning:hover>td,.table-hover .table-warning:hover>th{background-color:#faf2cc}.table-danger,.table-danger>td,.table-danger>th{background-color:#f2dede}.table-hover .table-danger:hover{background-color:#ebcccc}.table-hover .table-danger:hover>td,.table-hover .table-danger:hover>th{background-color:#ebcccc}.thead-inverse th{color:#fff;background-color:#292b2c}.thead-default th{color:#464a4c;background-color:#eceeef}.table-inverse{color:#fff;background-color:#292b2c}.table-inverse td,.table-inverse th,.table-inverse thead th{border-color:#fff}.table-inverse.table-bordered{border:0}.table-responsive{display:block;width:100%;overflow-x:auto;-ms-overflow-style:-ms-autohiding-scrollbar}.table-responsive.table-bordered{border:0}.form-control{display:block;width:100%;padding:.5rem .75rem;font-size:1rem;line-height:1.25;color:#464a4c;background-color:#fff;background-image:none;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.15);border-radius:.25rem;-webkit-transition:border-color ease-in-out .15s,-webkit-box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,-webkit-box-shadow ease-in-out .15s;-o-transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s,-webkit-box-shadow ease-in-out .15s}.form-control::-ms-expand{background-color:transparent;border:0}.form-control:focus{color:#464a4c;background-color:#fff;border-color:#5cb3fd;outline:0}.form-control::-webkit-input-placeholder{color:#636c72;opacity:1}.form-control::-moz-placeholder{color:#636c72;opacity:1}.form-control:-ms-input-placeholder{color:#636c72;opacity:1}.form-control::placeholder{color:#636c72;opacity:1}.form-control:disabled,.form-control[readonly]{background-color:#eceeef;opacity:1}.form-control:disabled{cursor:not-allowed}select.form-control:not([size]):not([multiple]){height:calc(2.25rem + 2px)}select.form-control:focus::-ms-value{color:#464a4c;background-color:#fff}.form-control-file,.form-control-range{display:block}.col-form-label{padding-top:calc(.5rem - 1px * 2);padding-bottom:calc(.5rem - 1px * 2);margin-bottom:0}.col-form-label-lg{padding-top:calc(.75rem - 1px * 2);padding-bottom:calc(.75rem - 1px * 2);font-size:1.25rem}.col-form-label-sm{padding-top:calc(.25rem - 1px * 2);padding-bottom:calc(.25rem - 1px * 2);font-size:.875rem}.col-form-legend{padding-top:.5rem;padding-bottom:.5rem;margin-bottom:0;font-size:1rem}.form-control-static{padding-top:.5rem;padding-bottom:.5rem;margin-bottom:0;line-height:1.25;border:solid transparent;border-width:1px 0}.form-control-static.form-control-lg,.form-control-static.form-control-sm,.input-group-lg>.form-control-static.form-control,.input-group-lg>.form-control-static.input-group-addon,.input-group-lg>.input-group-btn>.form-control-static.btn,.input-group-sm>.form-control-static.form-control,.input-group-sm>.form-control-static.input-group-addon,.input-group-sm>.input-group-btn>.form-control-static.btn{padding-right:0;padding-left:0}.form-control-sm,.input-group-sm>.form-control,.input-group-sm>.input-group-addon,.input-group-sm>.input-group-btn>.btn{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.input-group-sm>.input-group-btn>select.btn:not([size]):not([multiple]),.input-group-sm>select.form-control:not([size]):not([multiple]),.input-group-sm>select.input-group-addon:not([size]):not([multiple]),select.form-control-sm:not([size]):not([multiple]){height:1.8125rem}.form-control-lg,.input-group-lg>.form-control,.input-group-lg>.input-group-addon,.input-group-lg>.input-group-btn>.btn{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.input-group-lg>.input-group-btn>select.btn:not([size]):not([multiple]),.input-group-lg>select.form-control:not([size]):not([multiple]),.input-group-lg>select.input-group-addon:not([size]):not([multiple]),select.form-control-lg:not([size]):not([multiple]){height:3.166667rem}.form-group{margin-bottom:1rem}.form-text{display:block;margin-top:.25rem}.form-check{position:relative;display:block;margin-bottom:.5rem}.form-check.disabled .form-check-label{color:#636c72;cursor:not-allowed}.form-check-label{padding-left:1.25rem;margin-bottom:0;cursor:pointer}.form-check-input{position:absolute;margin-top:.25rem;margin-left:-1.25rem}.form-check-input:only-child{position:static}.form-check-inline{display:inline-block}.form-check-inline .form-check-label{vertical-align:middle}.form-check-inline+.form-check-inline{margin-left:.75rem}.form-control-feedback{margin-top:.25rem}.form-control-danger,.form-control-success,.form-control-warning{padding-right:2.25rem;background-repeat:no-repeat;background-position:center right .5625rem;-webkit-background-size:1.125rem 1.125rem;background-size:1.125rem 1.125rem}.has-success .col-form-label,.has-success .custom-control,.has-success .form-check-label,.has-success .form-control-feedback,.has-success .form-control-label{color:#5cb85c}.has-success .form-control{border-color:#5cb85c}.has-success .input-group-addon{color:#5cb85c;border-color:#5cb85c;background-color:#eaf6ea}.has-success .form-control-success{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%235cb85c' d='M2.3 6.73L.6 4.53c-.4-1.04.46-1.4 1.1-.8l1.1 1.4 3.4-3.8c.6-.63 1.6-.27 1.2.7l-4 4.6c-.43.5-.8.4-1.1.1z'/%3E%3C/svg%3E")}.has-warning .col-form-label,.has-warning .custom-control,.has-warning .form-check-label,.has-warning .form-control-feedback,.has-warning .form-control-label{color:#f0ad4e}.has-warning .form-control{border-color:#f0ad4e}.has-warning .input-group-addon{color:#f0ad4e;border-color:#f0ad4e;background-color:#fff}.has-warning .form-control-warning{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%23f0ad4e' d='M4.4 5.324h-.8v-2.46h.8zm0 1.42h-.8V5.89h.8zM3.76.63L.04 7.075c-.115.2.016.425.26.426h7.397c.242 0 .372-.226.258-.426C6.726 4.924 5.47 2.79 4.253.63c-.113-.174-.39-.174-.494 0z'/%3E%3C/svg%3E")}.has-danger .col-form-label,.has-danger .custom-control,.has-danger .form-check-label,.has-danger .form-control-feedback,.has-danger .form-control-label{color:#d9534f}.has-danger .form-control{border-color:#d9534f}.has-danger .input-group-addon{color:#d9534f;border-color:#d9534f;background-color:#fdf7f7}.has-danger .form-control-danger{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23d9534f' viewBox='-2 -2 7 7'%3E%3Cpath stroke='%23d9534f' d='M0 0l3 3m0-3L0 3'/%3E%3Ccircle r='.5'/%3E%3Ccircle cx='3' r='.5'/%3E%3Ccircle cy='3' r='.5'/%3E%3Ccircle cx='3' cy='3' r='.5'/%3E%3C/svg%3E")}.form-inline{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.form-inline .form-check{width:100%}@media (min-width:576px){.form-inline label{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;margin-bottom:0}.form-inline .form-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;margin-bottom:0}.form-inline .form-control{display:inline-block;width:auto;vertical-align:middle}.form-inline .form-control-static{display:inline-block}.form-inline .input-group{width:auto}.form-inline .form-control-label{margin-bottom:0;vertical-align:middle}.form-inline .form-check{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;width:auto;margin-top:0;margin-bottom:0}.form-inline .form-check-label{padding-left:0}.form-inline .form-check-input{position:relative;margin-top:0;margin-right:.25rem;margin-left:0}.form-inline .custom-control{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;padding-left:0}.form-inline .custom-control-indicator{position:static;display:inline-block;margin-right:.25rem;vertical-align:text-bottom}.form-inline .has-feedback .form-control-feedback{top:0}}.btn{display:inline-block;font-weight:400;line-height:1.25;text-align:center;white-space:nowrap;vertical-align:middle;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;border:1px solid transparent;padding:.5rem 1rem;font-size:1rem;border-radius:.25rem;-webkit-transition:all .2s ease-in-out;-o-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.btn:focus,.btn:hover{text-decoration:none}.btn.focus,.btn:focus{outline:0;-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.25);box-shadow:0 0 0 2px rgba(2,117,216,.25)}.btn.disabled,.btn:disabled{cursor:not-allowed;opacity:.65}.btn.active,.btn:active{background-image:none}a.btn.disabled,fieldset[disabled] a.btn{pointer-events:none}.btn-primary{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-primary:hover{color:#fff;background-color:#025aa5;border-color:#01549b}.btn-primary.focus,.btn-primary:focus{-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.5);box-shadow:0 0 0 2px rgba(2,117,216,.5)}.btn-primary.disabled,.btn-primary:disabled{background-color:#0275d8;border-color:#0275d8}.btn-primary.active,.btn-primary:active,.show>.btn-primary.dropdown-toggle{color:#fff;background-color:#025aa5;background-image:none;border-color:#01549b}.btn-secondary{color:#292b2c;background-color:#fff;border-color:#ccc}.btn-secondary:hover{color:#292b2c;background-color:#e6e6e6;border-color:#adadad}.btn-secondary.focus,.btn-secondary:focus{-webkit-box-shadow:0 0 0 2px rgba(204,204,204,.5);box-shadow:0 0 0 2px rgba(204,204,204,.5)}.btn-secondary.disabled,.btn-secondary:disabled{background-color:#fff;border-color:#ccc}.btn-secondary.active,.btn-secondary:active,.show>.btn-secondary.dropdown-toggle{color:#292b2c;background-color:#e6e6e6;background-image:none;border-color:#adadad}.btn-info{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-info:hover{color:#fff;background-color:#31b0d5;border-color:#2aabd2}.btn-info.focus,.btn-info:focus{-webkit-box-shadow:0 0 0 2px rgba(91,192,222,.5);box-shadow:0 0 0 2px rgba(91,192,222,.5)}.btn-info.disabled,.btn-info:disabled{background-color:#5bc0de;border-color:#5bc0de}.btn-info.active,.btn-info:active,.show>.btn-info.dropdown-toggle{color:#fff;background-color:#31b0d5;background-image:none;border-color:#2aabd2}.btn-success{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-success:hover{color:#fff;background-color:#449d44;border-color:#419641}.btn-success.focus,.btn-success:focus{-webkit-box-shadow:0 0 0 2px rgba(92,184,92,.5);box-shadow:0 0 0 2px rgba(92,184,92,.5)}.btn-success.disabled,.btn-success:disabled{background-color:#5cb85c;border-color:#5cb85c}.btn-success.active,.btn-success:active,.show>.btn-success.dropdown-toggle{color:#fff;background-color:#449d44;background-image:none;border-color:#419641}.btn-warning{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-warning:hover{color:#fff;background-color:#ec971f;border-color:#eb9316}.btn-warning.focus,.btn-warning:focus{-webkit-box-shadow:0 0 0 2px rgba(240,173,78,.5);box-shadow:0 0 0 2px rgba(240,173,78,.5)}.btn-warning.disabled,.btn-warning:disabled{background-color:#f0ad4e;border-color:#f0ad4e}.btn-warning.active,.btn-warning:active,.show>.btn-warning.dropdown-toggle{color:#fff;background-color:#ec971f;background-image:none;border-color:#eb9316}.btn-danger{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-danger:hover{color:#fff;background-color:#c9302c;border-color:#c12e2a}.btn-danger.focus,.btn-danger:focus{-webkit-box-shadow:0 0 0 2px rgba(217,83,79,.5);box-shadow:0 0 0 2px rgba(217,83,79,.5)}.btn-danger.disabled,.btn-danger:disabled{background-color:#d9534f;border-color:#d9534f}.btn-danger.active,.btn-danger:active,.show>.btn-danger.dropdown-toggle{color:#fff;background-color:#c9302c;background-image:none;border-color:#c12e2a}.btn-outline-primary{color:#0275d8;background-image:none;background-color:transparent;border-color:#0275d8}.btn-outline-primary:hover{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-outline-primary.focus,.btn-outline-primary:focus{-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.5);box-shadow:0 0 0 2px rgba(2,117,216,.5)}.btn-outline-primary.disabled,.btn-outline-primary:disabled{color:#0275d8;background-color:transparent}.btn-outline-primary.active,.btn-outline-primary:active,.show>.btn-outline-primary.dropdown-toggle{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-outline-secondary{color:#ccc;background-image:none;background-color:transparent;border-color:#ccc}.btn-outline-secondary:hover{color:#fff;background-color:#ccc;border-color:#ccc}.btn-outline-secondary.focus,.btn-outline-secondary:focus{-webkit-box-shadow:0 0 0 2px rgba(204,204,204,.5);box-shadow:0 0 0 2px rgba(204,204,204,.5)}.btn-outline-secondary.disabled,.btn-outline-secondary:disabled{color:#ccc;background-color:transparent}.btn-outline-secondary.active,.btn-outline-secondary:active,.show>.btn-outline-secondary.dropdown-toggle{color:#fff;background-color:#ccc;border-color:#ccc}.btn-outline-info{color:#5bc0de;background-image:none;background-color:transparent;border-color:#5bc0de}.btn-outline-info:hover{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-outline-info.focus,.btn-outline-info:focus{-webkit-box-shadow:0 0 0 2px rgba(91,192,222,.5);box-shadow:0 0 0 2px rgba(91,192,222,.5)}.btn-outline-info.disabled,.btn-outline-info:disabled{color:#5bc0de;background-color:transparent}.btn-outline-info.active,.btn-outline-info:active,.show>.btn-outline-info.dropdown-toggle{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-outline-success{color:#5cb85c;background-image:none;background-color:transparent;border-color:#5cb85c}.btn-outline-success:hover{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-outline-success.focus,.btn-outline-success:focus{-webkit-box-shadow:0 0 0 2px rgba(92,184,92,.5);box-shadow:0 0 0 2px rgba(92,184,92,.5)}.btn-outline-success.disabled,.btn-outline-success:disabled{color:#5cb85c;background-color:transparent}.btn-outline-success.active,.btn-outline-success:active,.show>.btn-outline-success.dropdown-toggle{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-outline-warning{color:#f0ad4e;background-image:none;background-color:transparent;border-color:#f0ad4e}.btn-outline-warning:hover{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-outline-warning.focus,.btn-outline-warning:focus{-webkit-box-shadow:0 0 0 2px rgba(240,173,78,.5);box-shadow:0 0 0 2px rgba(240,173,78,.5)}.btn-outline-warning.disabled,.btn-outline-warning:disabled{color:#f0ad4e;background-color:transparent}.btn-outline-warning.active,.btn-outline-warning:active,.show>.btn-outline-warning.dropdown-toggle{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-outline-danger{color:#d9534f;background-image:none;background-color:transparent;border-color:#d9534f}.btn-outline-danger:hover{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-outline-danger.focus,.btn-outline-danger:focus{-webkit-box-shadow:0 0 0 2px rgba(217,83,79,.5);box-shadow:0 0 0 2px rgba(217,83,79,.5)}.btn-outline-danger.disabled,.btn-outline-danger:disabled{color:#d9534f;background-color:transparent}.btn-outline-danger.active,.btn-outline-danger:active,.show>.btn-outline-danger.dropdown-toggle{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-link{font-weight:400;color:#0275d8;border-radius:0}.btn-link,.btn-link.active,.btn-link:active,.btn-link:disabled{background-color:transparent}.btn-link,.btn-link:active,.btn-link:focus{border-color:transparent}.btn-link:hover{border-color:transparent}.btn-link:focus,.btn-link:hover{color:#014c8c;text-decoration:underline;background-color:transparent}.btn-link:disabled{color:#636c72}.btn-link:disabled:focus,.btn-link:disabled:hover{text-decoration:none}.btn-group-lg>.btn,.btn-lg{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.btn-group-sm>.btn,.btn-sm{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.btn-block{display:block;width:100%}.btn-block+.btn-block{margin-top:.5rem}input[type=button].btn-block,input[type=reset].btn-block,input[type=submit].btn-block{width:100%}.fade{opacity:0;-webkit-transition:opacity .15s linear;-o-transition:opacity .15s linear;transition:opacity .15s linear}.fade.show{opacity:1}.collapse{display:none}.collapse.show{display:block}tr.collapse.show{display:table-row}tbody.collapse.show{display:table-row-group}.collapsing{position:relative;height:0;overflow:hidden;-webkit-transition:height .35s ease;-o-transition:height .35s ease;transition:height .35s ease}.dropdown,.dropup{position:relative}.dropdown-toggle::after{display:inline-block;width:0;height:0;margin-left:.3em;vertical-align:middle;content:"";border-top:.3em solid;border-right:.3em solid transparent;border-left:.3em solid transparent}.dropdown-toggle:focus{outline:0}.dropup .dropdown-toggle::after{border-top:0;border-bottom:.3em solid}.dropdown-menu{position:absolute;top:100%;left:0;z-index:1000;display:none;float:left;min-width:10rem;padding:.5rem 0;margin:.125rem 0 0;font-size:1rem;color:#292b2c;text-align:left;list-style:none;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.dropdown-divider{height:1px;margin:.5rem 0;overflow:hidden;background-color:#eceeef}.dropdown-item{display:block;width:100%;padding:3px 1.5rem;clear:both;font-weight:400;color:#292b2c;text-align:inherit;white-space:nowrap;background:0 0;border:0}.dropdown-item:focus,.dropdown-item:hover{color:#1d1e1f;text-decoration:none;background-color:#f7f7f9}.dropdown-item.active,.dropdown-item:active{color:#fff;text-decoration:none;background-color:#0275d8}.dropdown-item.disabled,.dropdown-item:disabled{color:#636c72;cursor:not-allowed;background-color:transparent}.show>.dropdown-menu{display:block}.show>a{outline:0}.dropdown-menu-right{right:0;left:auto}.dropdown-menu-left{right:auto;left:0}.dropdown-header{display:block;padding:.5rem 1.5rem;margin-bottom:0;font-size:.875rem;color:#636c72;white-space:nowrap}.dropdown-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;z-index:990}.dropup .dropdown-menu{top:auto;bottom:100%;margin-bottom:.125rem}.btn-group,.btn-group-vertical{position:relative;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;vertical-align:middle}.btn-group-vertical>.btn,.btn-group>.btn{position:relative;-webkit-box-flex:0;-webkit-flex:0 1 auto;-ms-flex:0 1 auto;flex:0 1 auto}.btn-group-vertical>.btn:hover,.btn-group>.btn:hover{z-index:2}.btn-group-vertical>.btn.active,.btn-group-vertical>.btn:active,.btn-group-vertical>.btn:focus,.btn-group>.btn.active,.btn-group>.btn:active,.btn-group>.btn:focus{z-index:2}.btn-group .btn+.btn,.btn-group .btn+.btn-group,.btn-group .btn-group+.btn,.btn-group .btn-group+.btn-group,.btn-group-vertical .btn+.btn,.btn-group-vertical .btn+.btn-group,.btn-group-vertical .btn-group+.btn,.btn-group-vertical .btn-group+.btn-group{margin-left:-1px}.btn-toolbar{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-pack:start;-webkit-justify-content:flex-start;-ms-flex-pack:start;justify-content:flex-start}.btn-toolbar .input-group{width:auto}.btn-group>.btn:not(:first-child):not(:last-child):not(.dropdown-toggle){border-radius:0}.btn-group>.btn:first-child{margin-left:0}.btn-group>.btn:first-child:not(:last-child):not(.dropdown-toggle){border-bottom-right-radius:0;border-top-right-radius:0}.btn-group>.btn:last-child:not(:first-child),.btn-group>.dropdown-toggle:not(:first-child){border-bottom-left-radius:0;border-top-left-radius:0}.btn-group>.btn-group{float:left}.btn-group>.btn-group:not(:first-child):not(:last-child)>.btn{border-radius:0}.btn-group>.btn-group:first-child:not(:last-child)>.btn:last-child,.btn-group>.btn-group:first-child:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-top-right-radius:0}.btn-group>.btn-group:last-child:not(:first-child)>.btn:first-child{border-bottom-left-radius:0;border-top-left-radius:0}.btn-group .dropdown-toggle:active,.btn-group.open .dropdown-toggle{outline:0}.btn+.dropdown-toggle-split{padding-right:.75rem;padding-left:.75rem}.btn+.dropdown-toggle-split::after{margin-left:0}.btn-group-sm>.btn+.dropdown-toggle-split,.btn-sm+.dropdown-toggle-split{padding-right:.375rem;padding-left:.375rem}.btn-group-lg>.btn+.dropdown-toggle-split,.btn-lg+.dropdown-toggle-split{padding-right:1.125rem;padding-left:1.125rem}.btn-group-vertical{display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;-webkit-box-align:start;-webkit-align-items:flex-start;-ms-flex-align:start;align-items:flex-start;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center}.btn-group-vertical .btn,.btn-group-vertical .btn-group{width:100%}.btn-group-vertical>.btn+.btn,.btn-group-vertical>.btn+.btn-group,.btn-group-vertical>.btn-group+.btn,.btn-group-vertical>.btn-group+.btn-group{margin-top:-1px;margin-left:0}.btn-group-vertical>.btn:not(:first-child):not(:last-child){border-radius:0}.btn-group-vertical>.btn:first-child:not(:last-child){border-bottom-right-radius:0;border-bottom-left-radius:0}.btn-group-vertical>.btn:last-child:not(:first-child){border-top-right-radius:0;border-top-left-radius:0}.btn-group-vertical>.btn-group:not(:first-child):not(:last-child)>.btn{border-radius:0}.btn-group-vertical>.btn-group:first-child:not(:last-child)>.btn:last-child,.btn-group-vertical>.btn-group:first-child:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-bottom-left-radius:0}.btn-group-vertical>.btn-group:last-child:not(:first-child)>.btn:first-child{border-top-right-radius:0;border-top-left-radius:0}[data-toggle=buttons]>.btn input[type=checkbox],[data-toggle=buttons]>.btn input[type=radio],[data-toggle=buttons]>.btn-group>.btn input[type=checkbox],[data-toggle=buttons]>.btn-group>.btn input[type=radio]{position:absolute;clip:rect(0,0,0,0);pointer-events:none}.input-group{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;width:100%}.input-group .form-control{position:relative;z-index:2;-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;width:1%;margin-bottom:0}.input-group .form-control:active,.input-group .form-control:focus,.input-group .form-control:hover{z-index:3}.input-group .form-control,.input-group-addon,.input-group-btn{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center}.input-group .form-control:not(:first-child):not(:last-child),.input-group-addon:not(:first-child):not(:last-child),.input-group-btn:not(:first-child):not(:last-child){border-radius:0}.input-group-addon,.input-group-btn{white-space:nowrap;vertical-align:middle}.input-group-addon{padding:.5rem .75rem;margin-bottom:0;font-size:1rem;font-weight:400;line-height:1.25;color:#464a4c;text-align:center;background-color:#eceeef;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.input-group-addon.form-control-sm,.input-group-sm>.input-group-addon,.input-group-sm>.input-group-btn>.input-group-addon.btn{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.input-group-addon.form-control-lg,.input-group-lg>.input-group-addon,.input-group-lg>.input-group-btn>.input-group-addon.btn{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.input-group-addon input[type=checkbox],.input-group-addon input[type=radio]{margin-top:0}.input-group .form-control:not(:last-child),.input-group-addon:not(:last-child),.input-group-btn:not(:first-child)>.btn-group:not(:last-child)>.btn,.input-group-btn:not(:first-child)>.btn:not(:last-child):not(.dropdown-toggle),.input-group-btn:not(:last-child)>.btn,.input-group-btn:not(:last-child)>.btn-group>.btn,.input-group-btn:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-top-right-radius:0}.input-group-addon:not(:last-child){border-right:0}.input-group .form-control:not(:first-child),.input-group-addon:not(:first-child),.input-group-btn:not(:first-child)>.btn,.input-group-btn:not(:first-child)>.btn-group>.btn,.input-group-btn:not(:first-child)>.dropdown-toggle,.input-group-btn:not(:last-child)>.btn-group:not(:first-child)>.btn,.input-group-btn:not(:last-child)>.btn:not(:first-child){border-bottom-left-radius:0;border-top-left-radius:0}.form-control+.input-group-addon:not(:first-child){border-left:0}.input-group-btn{position:relative;font-size:0;white-space:nowrap}.input-group-btn>.btn{position:relative;-webkit-box-flex:1;-webkit-flex:1 1 0%;-ms-flex:1 1 0%;flex:1 1 0%}.input-group-btn>.btn+.btn{margin-left:-1px}.input-group-btn>.btn:active,.input-group-btn>.btn:focus,.input-group-btn>.btn:hover{z-index:3}.input-group-btn:not(:last-child)>.btn,.input-group-btn:not(:last-child)>.btn-group{margin-right:-1px}.input-group-btn:not(:first-child)>.btn,.input-group-btn:not(:first-child)>.btn-group{z-index:2;margin-left:-1px}.input-group-btn:not(:first-child)>.btn-group:active,.input-group-btn:not(:first-child)>.btn-group:focus,.input-group-btn:not(:first-child)>.btn-group:hover,.input-group-btn:not(:first-child)>.btn:active,.input-group-btn:not(:first-child)>.btn:focus,.input-group-btn:not(:first-child)>.btn:hover{z-index:3}.custom-control{position:relative;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;min-height:1.5rem;padding-left:1.5rem;margin-right:1rem;cursor:pointer}.custom-control-input{position:absolute;z-index:-1;opacity:0}.custom-control-input:checked~.custom-control-indicator{color:#fff;background-color:#0275d8}.custom-control-input:focus~.custom-control-indicator{-webkit-box-shadow:0 0 0 1px #fff,0 0 0 3px #0275d8;box-shadow:0 0 0 1px #fff,0 0 0 3px #0275d8}.custom-control-input:active~.custom-control-indicator{color:#fff;background-color:#8fcafe}.custom-control-input:disabled~.custom-control-indicator{cursor:not-allowed;background-color:#eceeef}.custom-control-input:disabled~.custom-control-description{color:#636c72;cursor:not-allowed}.custom-control-indicator{position:absolute;top:.25rem;left:0;display:block;width:1rem;height:1rem;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;background-color:#ddd;background-repeat:no-repeat;background-position:center center;-webkit-background-size:50% 50%;background-size:50% 50%}.custom-checkbox .custom-control-indicator{border-radius:.25rem}.custom-checkbox .custom-control-input:checked~.custom-control-indicator{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%23fff' d='M6.564.75l-3.59 3.612-1.538-1.55L0 4.26 2.974 7.25 8 2.193z'/%3E%3C/svg%3E")}.custom-checkbox .custom-control-input:indeterminate~.custom-control-indicator{background-color:#0275d8;background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 4 4'%3E%3Cpath stroke='%23fff' d='M0 2h4'/%3E%3C/svg%3E")}.custom-radio .custom-control-indicator{border-radius:50%}.custom-radio .custom-control-input:checked~.custom-control-indicator{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3E%3Ccircle r='3' fill='%23fff'/%3E%3C/svg%3E")}.custom-controls-stacked{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column}.custom-controls-stacked .custom-control{margin-bottom:.25rem}.custom-controls-stacked .custom-control+.custom-control{margin-left:0}.custom-select{display:inline-block;max-width:100%;height:calc(2.25rem + 2px);padding:.375rem 1.75rem .375rem .75rem;line-height:1.25;color:#464a4c;vertical-align:middle;background:#fff url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 4 5'%3E%3Cpath fill='%23333' d='M2 0L0 2h4zm0 5L0 3h4z'/%3E%3C/svg%3E") no-repeat right .75rem center;-webkit-background-size:8px 10px;background-size:8px 10px;border:1px solid rgba(0,0,0,.15);border-radius:.25rem;-moz-appearance:none;-webkit-appearance:none}.custom-select:focus{border-color:#5cb3fd;outline:0}.custom-select:focus::-ms-value{color:#464a4c;background-color:#fff}.custom-select:disabled{color:#636c72;cursor:not-allowed;background-color:#eceeef}.custom-select::-ms-expand{opacity:0}.custom-select-sm{padding-top:.375rem;padding-bottom:.375rem;font-size:75%}.custom-file{position:relative;display:inline-block;max-width:100%;height:2.5rem;margin-bottom:0;cursor:pointer}.custom-file-input{min-width:14rem;max-width:100%;height:2.5rem;margin:0;filter:alpha(opacity=0);opacity:0}.custom-file-control{position:absolute;top:0;right:0;left:0;z-index:5;height:2.5rem;padding:.5rem 1rem;line-height:1.5;color:#464a4c;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;background-color:#fff;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.custom-file-control:lang(en)::after{content:"Choose file..."}.custom-file-control::before{position:absolute;top:-1px;right:-1px;bottom:-1px;z-index:6;display:block;height:2.5rem;padding:.5rem 1rem;line-height:1.5;color:#464a4c;background-color:#eceeef;border:1px solid rgba(0,0,0,.15);border-radius:0 .25rem .25rem 0}.custom-file-control:lang(en)::before{content:"Browse"}.nav{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;padding-left:0;margin-bottom:0;list-style:none}.nav-link{display:block;padding:.5em 1em}.nav-link:focus,.nav-link:hover{text-decoration:none}.nav-link.disabled{color:#636c72;cursor:not-allowed}.nav-tabs{border-bottom:1px solid #ddd}.nav-tabs .nav-item{margin-bottom:-1px}.nav-tabs .nav-link{border:1px solid transparent;border-top-right-radius:.25rem;border-top-left-radius:.25rem}.nav-tabs .nav-link:focus,.nav-tabs .nav-link:hover{border-color:#eceeef #eceeef #ddd}.nav-tabs .nav-link.disabled{color:#636c72;background-color:transparent;border-color:transparent}.nav-tabs .nav-item.show .nav-link,.nav-tabs .nav-link.active{color:#464a4c;background-color:#fff;border-color:#ddd #ddd #fff}.nav-tabs .dropdown-menu{margin-top:-1px;border-top-right-radius:0;border-top-left-radius:0}.nav-pills .nav-link{border-radius:.25rem}.nav-pills .nav-item.show .nav-link,.nav-pills .nav-link.active{color:#fff;cursor:default;background-color:#0275d8}.nav-fill .nav-item{-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;text-align:center}.nav-justified .nav-item{-webkit-box-flex:1;-webkit-flex:1 1 100%;-ms-flex:1 1 100%;flex:1 1 100%;text-align:center}.tab-content>.tab-pane{display:none}.tab-content>.active{display:block}.navbar{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding:.5rem 1rem}.navbar-brand{display:inline-block;padding-top:.25rem;padding-bottom:.25rem;margin-right:1rem;font-size:1.25rem;line-height:inherit;white-space:nowrap}.navbar-brand:focus,.navbar-brand:hover{text-decoration:none}.navbar-nav{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding-left:0;margin-bottom:0;list-style:none}.navbar-nav .nav-link{padding-right:0;padding-left:0}.navbar-text{display:inline-block;padding-top:.425rem;padding-bottom:.425rem}.navbar-toggler{-webkit-align-self:flex-start;-ms-flex-item-align:start;align-self:flex-start;padding:.25rem .75rem;font-size:1.25rem;line-height:1;background:0 0;border:1px solid transparent;border-radius:.25rem}.navbar-toggler:focus,.navbar-toggler:hover{text-decoration:none}.navbar-toggler-icon{display:inline-block;width:1.5em;height:1.5em;vertical-align:middle;content:"";background:no-repeat center center;-webkit-background-size:100% 100%;background-size:100% 100%}.navbar-toggler-left{position:absolute;left:1rem}.navbar-toggler-right{position:absolute;right:1rem}@media (max-width:575px){.navbar-toggleable .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable>.container{padding-right:0;padding-left:0}}@media (min-width:576px){.navbar-toggleable{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable .navbar-toggler{display:none}}@media (max-width:767px){.navbar-toggleable-sm .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-sm>.container{padding-right:0;padding-left:0}}@media (min-width:768px){.navbar-toggleable-sm{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-sm .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-sm .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-sm>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-sm .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-sm .navbar-toggler{display:none}}@media (max-width:991px){.navbar-toggleable-md .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-md>.container{padding-right:0;padding-left:0}}@media (min-width:992px){.navbar-toggleable-md{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-md .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-md .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-md>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-md .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-md .navbar-toggler{display:none}}@media (max-width:1199px){.navbar-toggleable-lg .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-lg>.container{padding-right:0;padding-left:0}}@media (min-width:1200px){.navbar-toggleable-lg{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-lg .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-lg .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-lg>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-lg .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-lg .navbar-toggler{display:none}}.navbar-toggleable-xl{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-xl .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-xl>.container{padding-right:0;padding-left:0}.navbar-toggleable-xl .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-xl .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-xl>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-xl .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-xl .navbar-toggler{display:none}.navbar-light .navbar-brand,.navbar-light .navbar-toggler{color:rgba(0,0,0,.9)}.navbar-light .navbar-brand:focus,.navbar-light .navbar-brand:hover,.navbar-light .navbar-toggler:focus,.navbar-light .navbar-toggler:hover{color:rgba(0,0,0,.9)}.navbar-light .navbar-nav .nav-link{color:rgba(0,0,0,.5)}.navbar-light .navbar-nav .nav-link:focus,.navbar-light .navbar-nav .nav-link:hover{color:rgba(0,0,0,.7)}.navbar-light .navbar-nav .nav-link.disabled{color:rgba(0,0,0,.3)}.navbar-light .navbar-nav .active>.nav-link,.navbar-light .navbar-nav .nav-link.active,.navbar-light .navbar-nav .nav-link.open,.navbar-light .navbar-nav .open>.nav-link{color:rgba(0,0,0,.9)}.navbar-light .navbar-toggler{border-color:rgba(0,0,0,.1)}.navbar-light .navbar-toggler-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg viewBox='0 0 32 32' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath stroke='rgba(0, 0, 0, 0.5)' stroke-width='2' stroke-linecap='round' stroke-miterlimit='10' d='M4 8h24M4 16h24M4 24h24'/%3E%3C/svg%3E")}.navbar-light .navbar-text{color:rgba(0,0,0,.5)}.navbar-inverse .navbar-brand,.navbar-inverse .navbar-toggler{color:#fff}.navbar-inverse .navbar-brand:focus,.navbar-inverse .navbar-brand:hover,.navbar-inverse .navbar-toggler:focus,.navbar-inverse .navbar-toggler:hover{color:#fff}.navbar-inverse .navbar-nav .nav-link{color:rgba(255,255,255,.5)}.navbar-inverse .navbar-nav .nav-link:focus,.navbar-inverse .navbar-nav .nav-link:hover{color:rgba(255,255,255,.75)}.navbar-inverse .navbar-nav .nav-link.disabled{color:rgba(255,255,255,.25)}.navbar-inverse .navbar-nav .active>.nav-link,.navbar-inverse .navbar-nav .nav-link.active,.navbar-inverse .navbar-nav .nav-link.open,.navbar-inverse .navbar-nav .open>.nav-link{color:#fff}.navbar-inverse .navbar-toggler{border-color:rgba(255,255,255,.1)}.navbar-inverse .navbar-toggler-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg viewBox='0 0 32 32' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath stroke='rgba(255, 255, 255, 0.5)' stroke-width='2' stroke-linecap='round' stroke-miterlimit='10' d='M4 8h24M4 16h24M4 24h24'/%3E%3C/svg%3E")}.navbar-inverse .navbar-text{color:rgba(255,255,255,.5)}.card{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;background-color:#fff;border:1px solid rgba(0,0,0,.125);border-radius:.25rem}.card-block{-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;padding:1.25rem}.card-title{margin-bottom:.75rem}.card-subtitle{margin-top:-.375rem;margin-bottom:0}.card-text:last-child{margin-bottom:0}.card-link:hover{text-decoration:none}.card-link+.card-link{margin-left:1.25rem}.card>.list-group:first-child .list-group-item:first-child{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.card>.list-group:last-child .list-group-item:last-child{border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.card-header{padding:.75rem 1.25rem;margin-bottom:0;background-color:#f7f7f9;border-bottom:1px solid rgba(0,0,0,.125)}.card-header:first-child{border-radius:calc(.25rem - 1px) calc(.25rem - 1px) 0 0}.card-footer{padding:.75rem 1.25rem;background-color:#f7f7f9;border-top:1px solid rgba(0,0,0,.125)}.card-footer:last-child{border-radius:0 0 calc(.25rem - 1px) calc(.25rem - 1px)}.card-header-tabs{margin-right:-.625rem;margin-bottom:-.75rem;margin-left:-.625rem;border-bottom:0}.card-header-pills{margin-right:-.625rem;margin-left:-.625rem}.card-primary{background-color:#0275d8;border-color:#0275d8}.card-primary .card-footer,.card-primary .card-header{background-color:transparent}.card-success{background-color:#5cb85c;border-color:#5cb85c}.card-success .card-footer,.card-success .card-header{background-color:transparent}.card-info{background-color:#5bc0de;border-color:#5bc0de}.card-info .card-footer,.card-info .card-header{background-color:transparent}.card-warning{background-color:#f0ad4e;border-color:#f0ad4e}.card-warning .card-footer,.card-warning .card-header{background-color:transparent}.card-danger{background-color:#d9534f;border-color:#d9534f}.card-danger .card-footer,.card-danger .card-header{background-color:transparent}.card-outline-primary{background-color:transparent;border-color:#0275d8}.card-outline-secondary{background-color:transparent;border-color:#ccc}.card-outline-info{background-color:transparent;border-color:#5bc0de}.card-outline-success{background-color:transparent;border-color:#5cb85c}.card-outline-warning{background-color:transparent;border-color:#f0ad4e}.card-outline-danger{background-color:transparent;border-color:#d9534f}.card-inverse{color:rgba(255,255,255,.65)}.card-inverse .card-footer,.card-inverse .card-header{background-color:transparent;border-color:rgba(255,255,255,.2)}.card-inverse .card-blockquote,.card-inverse .card-footer,.card-inverse .card-header,.card-inverse .card-title{color:#fff}.card-inverse .card-blockquote .blockquote-footer,.card-inverse .card-link,.card-inverse .card-subtitle,.card-inverse .card-text{color:rgba(255,255,255,.65)}.card-inverse .card-link:focus,.card-inverse .card-link:hover{color:#fff}.card-blockquote{padding:0;margin-bottom:0;border-left:0}.card-img{border-radius:calc(.25rem - 1px)}.card-img-overlay{position:absolute;top:0;right:0;bottom:0;left:0;padding:1.25rem}.card-img-top{border-top-right-radius:calc(.25rem - 1px);border-top-left-radius:calc(.25rem - 1px)}.card-img-bottom{border-bottom-right-radius:calc(.25rem - 1px);border-bottom-left-radius:calc(.25rem - 1px)}@media (min-width:576px){.card-deck{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap}.card-deck .card{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-flex:1;-webkit-flex:1 0 0%;-ms-flex:1 0 0%;flex:1 0 0%;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column}.card-deck .card:not(:first-child){margin-left:15px}.card-deck .card:not(:last-child){margin-right:15px}}@media (min-width:576px){.card-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap}.card-group .card{-webkit-box-flex:1;-webkit-flex:1 0 0%;-ms-flex:1 0 0%;flex:1 0 0%}.card-group .card+.card{margin-left:0;border-left:0}.card-group .card:first-child{border-bottom-right-radius:0;border-top-right-radius:0}.card-group .card:first-child .card-img-top{border-top-right-radius:0}.card-group .card:first-child .card-img-bottom{border-bottom-right-radius:0}.card-group .card:last-child{border-bottom-left-radius:0;border-top-left-radius:0}.card-group .card:last-child .card-img-top{border-top-left-radius:0}.card-group .card:last-child .card-img-bottom{border-bottom-left-radius:0}.card-group .card:not(:first-child):not(:last-child){border-radius:0}.card-group .card:not(:first-child):not(:last-child) .card-img-bottom,.card-group .card:not(:first-child):not(:last-child) .card-img-top{border-radius:0}}@media (min-width:576px){.card-columns{-webkit-column-count:3;-moz-column-count:3;column-count:3;-webkit-column-gap:1.25rem;-moz-column-gap:1.25rem;column-gap:1.25rem}.card-columns .card{display:inline-block;width:100%;margin-bottom:.75rem}}.breadcrumb{padding:.75rem 1rem;margin-bottom:1rem;list-style:none;background-color:#eceeef;border-radius:.25rem}.breadcrumb::after{display:block;content:"";clear:both}.breadcrumb-item{float:left}.breadcrumb-item+.breadcrumb-item::before{display:inline-block;padding-right:.5rem;padding-left:.5rem;color:#636c72;content:"/"}.breadcrumb-item+.breadcrumb-item:hover::before{text-decoration:underline}.breadcrumb-item+.breadcrumb-item:hover::before{text-decoration:none}.breadcrumb-item.active{color:#636c72}.pagination{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;padding-left:0;list-style:none;border-radius:.25rem}.page-item:first-child .page-link{margin-left:0;border-bottom-left-radius:.25rem;border-top-left-radius:.25rem}.page-item:last-child .page-link{border-bottom-right-radius:.25rem;border-top-right-radius:.25rem}.page-item.active .page-link{z-index:2;color:#fff;background-color:#0275d8;border-color:#0275d8}.page-item.disabled .page-link{color:#636c72;pointer-events:none;cursor:not-allowed;background-color:#fff;border-color:#ddd}.page-link{position:relative;display:block;padding:.5rem .75rem;margin-left:-1px;line-height:1.25;color:#0275d8;background-color:#fff;border:1px solid #ddd}.page-link:focus,.page-link:hover{color:#014c8c;text-decoration:none;background-color:#eceeef;border-color:#ddd}.pagination-lg .page-link{padding:.75rem 1.5rem;font-size:1.25rem}.pagination-lg .page-item:first-child .page-link{border-bottom-left-radius:.3rem;border-top-left-radius:.3rem}.pagination-lg .page-item:last-child .page-link{border-bottom-right-radius:.3rem;border-top-right-radius:.3rem}.pagination-sm .page-link{padding:.25rem .5rem;font-size:.875rem}.pagination-sm .page-item:first-child .page-link{border-bottom-left-radius:.2rem;border-top-left-radius:.2rem}.pagination-sm .page-item:last-child .page-link{border-bottom-right-radius:.2rem;border-top-right-radius:.2rem}.badge{display:inline-block;padding:.25em .4em;font-size:75%;font-weight:700;line-height:1;color:#fff;text-align:center;white-space:nowrap;vertical-align:baseline;border-radius:.25rem}.badge:empty{display:none}.btn .badge{position:relative;top:-1px}a.badge:focus,a.badge:hover{color:#fff;text-decoration:none;cursor:pointer}.badge-pill{padding-right:.6em;padding-left:.6em;border-radius:10rem}.badge-default{background-color:#636c72}.badge-default[href]:focus,.badge-default[href]:hover{background-color:#4b5257}.badge-primary{background-color:#0275d8}.badge-primary[href]:focus,.badge-primary[href]:hover{background-color:#025aa5}.badge-success{background-color:#5cb85c}.badge-success[href]:focus,.badge-success[href]:hover{background-color:#449d44}.badge-info{background-color:#5bc0de}.badge-info[href]:focus,.badge-info[href]:hover{background-color:#31b0d5}.badge-warning{background-color:#f0ad4e}.badge-warning[href]:focus,.badge-warning[href]:hover{background-color:#ec971f}.badge-danger{background-color:#d9534f}.badge-danger[href]:focus,.badge-danger[href]:hover{background-color:#c9302c}.jumbotron{padding:2rem 1rem;margin-bottom:2rem;background-color:#eceeef;border-radius:.3rem}@media (min-width:576px){.jumbotron{padding:4rem 2rem}}.jumbotron-hr{border-top-color:#d0d5d8}.jumbotron-fluid{padding-right:0;padding-left:0;border-radius:0}.alert{padding:.75rem 1.25rem;margin-bottom:1rem;border:1px solid transparent;border-radius:.25rem}.alert-heading{color:inherit}.alert-link{font-weight:700}.alert-dismissible .close{position:relative;top:-.75rem;right:-1.25rem;padding:.75rem 1.25rem;color:inherit}.alert-success{background-color:#dff0d8;border-color:#d0e9c6;color:#3c763d}.alert-success hr{border-top-color:#c1e2b3}.alert-success .alert-link{color:#2b542c}.alert-info{background-color:#d9edf7;border-color:#bcdff1;color:#31708f}.alert-info hr{border-top-color:#a6d5ec}.alert-info .alert-link{color:#245269}.alert-warning{background-color:#fcf8e3;border-color:#faf2cc;color:#8a6d3b}.alert-warning hr{border-top-color:#f7ecb5}.alert-warning .alert-link{color:#66512c}.alert-danger{background-color:#f2dede;border-color:#ebcccc;color:#a94442}.alert-danger hr{border-top-color:#e4b9b9}.alert-danger .alert-link{color:#843534}@-webkit-keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}@-o-keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}@keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}.progress{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;overflow:hidden;font-size:.75rem;line-height:1rem;text-align:center;background-color:#eceeef;border-radius:.25rem}.progress-bar{height:1rem;color:#fff;background-color:#0275d8}.progress-bar-striped{background-image:-webkit-linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);background-image:-o-linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);background-image:linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);-webkit-background-size:1rem 1rem;background-size:1rem 1rem}.progress-bar-animated{-webkit-animation:progress-bar-stripes 1s linear infinite;-o-animation:progress-bar-stripes 1s linear infinite;animation:progress-bar-stripes 1s linear infinite}.media{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:start;-webkit-align-items:flex-start;-ms-flex-align:start;align-items:flex-start}.media-body{-webkit-box-flex:1;-webkit-flex:1 1 0%;-ms-flex:1 1 0%;flex:1 1 0%}.list-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding-left:0;margin-bottom:0}.list-group-item-action{width:100%;color:#464a4c;text-align:inherit}.list-group-item-action .list-group-item-heading{color:#292b2c}.list-group-item-action:focus,.list-group-item-action:hover{color:#464a4c;text-decoration:none;background-color:#f7f7f9}.list-group-item-action:active{color:#292b2c;background-color:#eceeef}.list-group-item{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;padding:.75rem 1.25rem;margin-bottom:-1px;background-color:#fff;border:1px solid rgba(0,0,0,.125)}.list-group-item:first-child{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.list-group-item:last-child{margin-bottom:0;border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.list-group-item:focus,.list-group-item:hover{text-decoration:none}.list-group-item.disabled,.list-group-item:disabled{color:#636c72;cursor:not-allowed;background-color:#fff}.list-group-item.disabled .list-group-item-heading,.list-group-item:disabled .list-group-item-heading{color:inherit}.list-group-item.disabled .list-group-item-text,.list-group-item:disabled .list-group-item-text{color:#636c72}.list-group-item.active{z-index:2;color:#fff;background-color:#0275d8;border-color:#0275d8}.list-group-item.active .list-group-item-heading,.list-group-item.active .list-group-item-heading>.small,.list-group-item.active .list-group-item-heading>small{color:inherit}.list-group-item.active .list-group-item-text{color:#daeeff}.list-group-flush .list-group-item{border-right:0;border-left:0;border-radius:0}.list-group-flush:first-child .list-group-item:first-child{border-top:0}.list-group-flush:last-child .list-group-item:last-child{border-bottom:0}.list-group-item-success{color:#3c763d;background-color:#dff0d8}a.list-group-item-success,button.list-group-item-success{color:#3c763d}a.list-group-item-success .list-group-item-heading,button.list-group-item-success .list-group-item-heading{color:inherit}a.list-group-item-success:focus,a.list-group-item-success:hover,button.list-group-item-success:focus,button.list-group-item-success:hover{color:#3c763d;background-color:#d0e9c6}a.list-group-item-success.active,button.list-group-item-success.active{color:#fff;background-color:#3c763d;border-color:#3c763d}.list-group-item-info{color:#31708f;background-color:#d9edf7}a.list-group-item-info,button.list-group-item-info{color:#31708f}a.list-group-item-info .list-group-item-heading,button.list-group-item-info .list-group-item-heading{color:inherit}a.list-group-item-info:focus,a.list-group-item-info:hover,button.list-group-item-info:focus,button.list-group-item-info:hover{color:#31708f;background-color:#c4e3f3}a.list-group-item-info.active,button.list-group-item-info.active{color:#fff;background-color:#31708f;border-color:#31708f}.list-group-item-warning{color:#8a6d3b;background-color:#fcf8e3}a.list-group-item-warning,button.list-group-item-warning{color:#8a6d3b}a.list-group-item-warning .list-group-item-heading,button.list-group-item-warning .list-group-item-heading{color:inherit}a.list-group-item-warning:focus,a.list-group-item-warning:hover,button.list-group-item-warning:focus,button.list-group-item-warning:hover{color:#8a6d3b;background-color:#faf2cc}a.list-group-item-warning.active,button.list-group-item-warning.active{color:#fff;background-color:#8a6d3b;border-color:#8a6d3b}.list-group-item-danger{color:#a94442;background-color:#f2dede}a.list-group-item-danger,button.list-group-item-danger{color:#a94442}a.list-group-item-danger .list-group-item-heading,button.list-group-item-danger .list-group-item-heading{color:inherit}a.list-group-item-danger:focus,a.list-group-item-danger:hover,button.list-group-item-danger:focus,button.list-group-item-danger:hover{color:#a94442;background-color:#ebcccc}a.list-group-item-danger.active,button.list-group-item-danger.active{color:#fff;background-color:#a94442;border-color:#a94442}.embed-responsive{position:relative;display:block;width:100%;padding:0;overflow:hidden}.embed-responsive::before{display:block;content:""}.embed-responsive .embed-responsive-item,.embed-responsive embed,.embed-responsive iframe,.embed-responsive object,.embed-responsive video{position:absolute;top:0;bottom:0;left:0;width:100%;height:100%;border:0}.embed-responsive-21by9::before{padding-top:42.857143%}.embed-responsive-16by9::before{padding-top:56.25%}.embed-responsive-4by3::before{padding-top:75%}.embed-responsive-1by1::before{padding-top:100%}.close{float:right;font-size:1.5rem;font-weight:700;line-height:1;color:#000;text-shadow:0 1px 0 #fff;opacity:.5}.close:focus,.close:hover{color:#000;text-decoration:none;cursor:pointer;opacity:.75}button.close{padding:0;cursor:pointer;background:0 0;border:0;-webkit-appearance:none}.modal-open{overflow:hidden}.modal{position:fixed;top:0;right:0;bottom:0;left:0;z-index:1050;display:none;overflow:hidden;outline:0}.modal.fade .modal-dialog{-webkit-transition:-webkit-transform .3s ease-out;transition:-webkit-transform .3s ease-out;-o-transition:-o-transform .3s ease-out;transition:transform .3s ease-out;transition:transform .3s ease-out,-webkit-transform .3s ease-out,-o-transform .3s ease-out;-webkit-transform:translate(0,-25%);-o-transform:translate(0,-25%);transform:translate(0,-25%)}.modal.show .modal-dialog{-webkit-transform:translate(0,0);-o-transform:translate(0,0);transform:translate(0,0)}.modal-open .modal{overflow-x:hidden;overflow-y:auto}.modal-dialog{position:relative;width:auto;margin:10px}.modal-content{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.2);border-radius:.3rem;outline:0}.modal-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;z-index:1040;background-color:#000}.modal-backdrop.fade{opacity:0}.modal-backdrop.show{opacity:.5}.modal-header{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:justify;-webkit-justify-content:space-between;-ms-flex-pack:justify;justify-content:space-between;padding:15px;border-bottom:1px solid #eceeef}.modal-title{margin-bottom:0;line-height:1.5}.modal-body{position:relative;-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;padding:15px}.modal-footer{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:end;-webkit-justify-content:flex-end;-ms-flex-pack:end;justify-content:flex-end;padding:15px;border-top:1px solid #eceeef}.modal-footer>:not(:first-child){margin-left:.25rem}.modal-footer>:not(:last-child){margin-right:.25rem}.modal-scrollbar-measure{position:absolute;top:-9999px;width:50px;height:50px;overflow:scroll}@media (min-width:576px){.modal-dialog{max-width:500px;margin:30px auto}.modal-sm{max-width:300px}}@media (min-width:992px){.modal-lg{max-width:800px}}.tooltip{position:absolute;z-index:1070;display:block;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-style:normal;font-weight:400;letter-spacing:normal;line-break:auto;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;white-space:normal;word-break:normal;word-spacing:normal;font-size:.875rem;word-wrap:break-word;opacity:0}.tooltip.show{opacity:.9}.tooltip.bs-tether-element-attached-bottom,.tooltip.tooltip-top{padding:5px 0;margin-top:-3px}.tooltip.bs-tether-element-attached-bottom .tooltip-inner::before,.tooltip.tooltip-top .tooltip-inner::before{bottom:0;left:50%;margin-left:-5px;content:"";border-width:5px 5px 0;border-top-color:#000}.tooltip.bs-tether-element-attached-left,.tooltip.tooltip-right{padding:0 5px;margin-left:3px}.tooltip.bs-tether-element-attached-left .tooltip-inner::before,.tooltip.tooltip-right .tooltip-inner::before{top:50%;left:0;margin-top:-5px;content:"";border-width:5px 5px 5px 0;border-right-color:#000}.tooltip.bs-tether-element-attached-top,.tooltip.tooltip-bottom{padding:5px 0;margin-top:3px}.tooltip.bs-tether-element-attached-top .tooltip-inner::before,.tooltip.tooltip-bottom .tooltip-inner::before{top:0;left:50%;margin-left:-5px;content:"";border-width:0 5px 5px;border-bottom-color:#000}.tooltip.bs-tether-element-attached-right,.tooltip.tooltip-left{padding:0 5px;margin-left:-3px}.tooltip.bs-tether-element-attached-right .tooltip-inner::before,.tooltip.tooltip-left .tooltip-inner::before{top:50%;right:0;margin-top:-5px;content:"";border-width:5px 0 5px 5px;border-left-color:#000}.tooltip-inner{max-width:200px;padding:3px 8px;color:#fff;text-align:center;background-color:#000;border-radius:.25rem}.tooltip-inner::before{position:absolute;width:0;height:0;border-color:transparent;border-style:solid}.popover{position:absolute;top:0;left:0;z-index:1060;display:block;max-width:276px;padding:1px;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-style:normal;font-weight:400;letter-spacing:normal;line-break:auto;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;white-space:normal;word-break:normal;word-spacing:normal;font-size:.875rem;word-wrap:break-word;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.2);border-radius:.3rem}.popover.bs-tether-element-attached-bottom,.popover.popover-top{margin-top:-10px}.popover.bs-tether-element-attached-bottom::after,.popover.bs-tether-element-attached-bottom::before,.popover.popover-top::after,.popover.popover-top::before{left:50%;border-bottom-width:0}.popover.bs-tether-element-attached-bottom::before,.popover.popover-top::before{bottom:-11px;margin-left:-11px;border-top-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-bottom::after,.popover.popover-top::after{bottom:-10px;margin-left:-10px;border-top-color:#fff}.popover.bs-tether-element-attached-left,.popover.popover-right{margin-left:10px}.popover.bs-tether-element-attached-left::after,.popover.bs-tether-element-attached-left::before,.popover.popover-right::after,.popover.popover-right::before{top:50%;border-left-width:0}.popover.bs-tether-element-attached-left::before,.popover.popover-right::before{left:-11px;margin-top:-11px;border-right-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-left::after,.popover.popover-right::after{left:-10px;margin-top:-10px;border-right-color:#fff}.popover.bs-tether-element-attached-top,.popover.popover-bottom{margin-top:10px}.popover.bs-tether-element-attached-top::after,.popover.bs-tether-element-attached-top::before,.popover.popover-bottom::after,.popover.popover-bottom::before{left:50%;border-top-width:0}.popover.bs-tether-element-attached-top::before,.popover.popover-bottom::before{top:-11px;margin-left:-11px;border-bottom-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-top::after,.popover.popover-bottom::after{top:-10px;margin-left:-10px;border-bottom-color:#f7f7f7}.popover.bs-tether-element-attached-top .popover-title::before,.popover.popover-bottom .popover-title::before{position:absolute;top:0;left:50%;display:block;width:20px;margin-left:-10px;content:"";border-bottom:1px solid #f7f7f7}.popover.bs-tether-element-attached-right,.popover.popover-left{margin-left:-10px}.popover.bs-tether-element-attached-right::after,.popover.bs-tether-element-attached-right::before,.popover.popover-left::after,.popover.popover-left::before{top:50%;border-right-width:0}.popover.bs-tether-element-attached-right::before,.popover.popover-left::before{right:-11px;margin-top:-11px;border-left-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-right::after,.popover.popover-left::after{right:-10px;margin-top:-10px;border-left-color:#fff}.popover-title{padding:8px 14px;margin-bottom:0;font-size:1rem;background-color:#f7f7f7;border-bottom:1px solid #ebebeb;border-top-right-radius:calc(.3rem - 1px);border-top-left-radius:calc(.3rem - 1px)}.popover-title:empty{display:none}.popover-content{padding:9px 14px}.popover::after,.popover::before{position:absolute;display:block;width:0;height:0;border-color:transparent;border-style:solid}.popover::before{content:"";border-width:11px}.popover::after{content:"";border-width:10px}.carousel{position:relative}.carousel-inner{position:relative;width:100%;overflow:hidden}.carousel-item{position:relative;display:none;width:100%}@media (-webkit-transform-3d){.carousel-item{-webkit-transition:-webkit-transform .6s ease-in-out;transition:-webkit-transform .6s ease-in-out;-o-transition:-o-transform .6s ease-in-out;transition:transform .6s ease-in-out;transition:transform .6s ease-in-out,-webkit-transform .6s ease-in-out,-o-transform .6s ease-in-out;-webkit-backface-visibility:hidden;backface-visibility:hidden;-webkit-perspective:1000px;perspective:1000px}}@supports ((-webkit-transform:translate3d(0,0,0)) or (transform:translate3d(0,0,0))){.carousel-item{-webkit-transition:-webkit-transform .6s ease-in-out;transition:-webkit-transform .6s ease-in-out;-o-transition:-o-transform .6s ease-in-out;transition:transform .6s ease-in-out;transition:transform .6s ease-in-out,-webkit-transform .6s ease-in-out,-o-transform .6s ease-in-out;-webkit-backface-visibility:hidden;backface-visibility:hidden;-webkit-perspective:1000px;perspective:1000px}}.carousel-item-next,.carousel-item-prev,.carousel-item.active{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex}.carousel-item-next,.carousel-item-prev{position:absolute;top:0}@media (-webkit-transform-3d){.carousel-item-next.carousel-item-left,.carousel-item-prev.carousel-item-right{-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0)}.active.carousel-item-right,.carousel-item-next{-webkit-transform:translate3d(100%,0,0);transform:translate3d(100%,0,0)}.active.carousel-item-left,.carousel-item-prev{-webkit-transform:translate3d(-100%,0,0);transform:translate3d(-100%,0,0)}}@supports ((-webkit-transform:translate3d(0,0,0)) or (transform:translate3d(0,0,0))){.carousel-item-next.carousel-item-left,.carousel-item-prev.carousel-item-right{-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0)}.active.carousel-item-right,.carousel-item-next{-webkit-transform:translate3d(100%,0,0);transform:translate3d(100%,0,0)}.active.carousel-item-left,.carousel-item-prev{-webkit-transform:translate3d(-100%,0,0);transform:translate3d(-100%,0,0)}}.carousel-control-next,.carousel-control-prev{position:absolute;top:0;bottom:0;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;width:15%;color:#fff;text-align:center;opacity:.5}.carousel-control-next:focus,.carousel-control-next:hover,.carousel-control-prev:focus,.carousel-control-prev:hover{color:#fff;text-decoration:none;outline:0;opacity:.9}.carousel-control-prev{left:0}.carousel-control-next{right:0}.carousel-control-next-icon,.carousel-control-prev-icon{display:inline-block;width:20px;height:20px;background:transparent no-repeat center center;-webkit-background-size:100% 100%;background-size:100% 100%}.carousel-control-prev-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23fff' viewBox='0 0 8 8'%3E%3Cpath d='M4 0l-4 4 4 4 1.5-1.5-2.5-2.5 2.5-2.5-1.5-1.5z'/%3E%3C/svg%3E")}.carousel-control-next-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23fff' viewBox='0 0 8 8'%3E%3Cpath d='M1.5 0l-1.5 1.5 2.5 2.5-2.5 2.5 1.5 1.5 4-4-4-4z'/%3E%3C/svg%3E")}.carousel-indicators{position:absolute;right:0;bottom:10px;left:0;z-index:15;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;padding-left:0;margin-right:15%;margin-left:15%;list-style:none}.carousel-indicators li{position:relative;-webkit-box-flex:1;-webkit-flex:1 0 auto;-ms-flex:1 0 auto;flex:1 0 auto;max-width:30px;height:3px;margin-right:3px;margin-left:3px;text-indent:-999px;cursor:pointer;background-color:rgba(255,255,255,.5)}.carousel-indicators li::before{position:absolute;top:-10px;left:0;display:inline-block;width:100%;height:10px;content:""}.carousel-indicators li::after{position:absolute;bottom:-10px;left:0;display:inline-block;width:100%;height:10px;content:""}.carousel-indicators .active{background-color:#fff}.carousel-caption{position:absolute;right:15%;bottom:20px;left:15%;z-index:10;padding-top:20px;padding-bottom:20px;color:#fff;text-align:center}.align-baseline{vertical-align:baseline!important}.align-top{vertical-align:top!important}.align-middle{vertical-align:middle!important}.align-bottom{vertical-align:bottom!important}.align-text-bottom{vertical-align:text-bottom!important}.align-text-top{vertical-align:text-top!important}.bg-faded{background-color:#f7f7f7}.bg-primary{background-color:#0275d8!important}a.bg-primary:focus,a.bg-primary:hover{background-color:#025aa5!important}.bg-success{background-color:#5cb85c!important}a.bg-success:focus,a.bg-success:hover{background-color:#449d44!important}.bg-info{background-color:#5bc0de!important}a.bg-info:focus,a.bg-info:hover{background-color:#31b0d5!important}.bg-warning{background-color:#f0ad4e!important}a.bg-warning:focus,a.bg-warning:hover{background-color:#ec971f!important}.bg-danger{background-color:#d9534f!important}a.bg-danger:focus,a.bg-danger:hover{background-color:#c9302c!important}.bg-inverse{background-color:#292b2c!important}a.bg-inverse:focus,a.bg-inverse:hover{background-color:#101112!important}.border-0{border:0!important}.border-top-0{border-top:0!important}.border-right-0{border-right:0!important}.border-bottom-0{border-bottom:0!important}.border-left-0{border-left:0!important}.rounded{border-radius:.25rem}.rounded-top{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.rounded-right{border-bottom-right-radius:.25rem;border-top-right-radius:.25rem}.rounded-bottom{border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.rounded-left{border-bottom-left-radius:.25rem;border-top-left-radius:.25rem}.rounded-circle{border-radius:50%}.rounded-0{border-radius:0}.clearfix::after{display:block;content:"";clear:both}.d-none{display:none!important}.d-inline{display:inline!important}.d-inline-block{display:inline-block!important}.d-block{display:block!important}.d-table{display:table!important}.d-table-cell{display:table-cell!important}.d-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}@media (min-width:576px){.d-sm-none{display:none!important}.d-sm-inline{display:inline!important}.d-sm-inline-block{display:inline-block!important}.d-sm-block{display:block!important}.d-sm-table{display:table!important}.d-sm-table-cell{display:table-cell!important}.d-sm-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-sm-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:768px){.d-md-none{display:none!important}.d-md-inline{display:inline!important}.d-md-inline-block{display:inline-block!important}.d-md-block{display:block!important}.d-md-table{display:table!important}.d-md-table-cell{display:table-cell!important}.d-md-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-md-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:992px){.d-lg-none{display:none!important}.d-lg-inline{display:inline!important}.d-lg-inline-block{display:inline-block!important}.d-lg-block{display:block!important}.d-lg-table{display:table!important}.d-lg-table-cell{display:table-cell!important}.d-lg-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-lg-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:1200px){.d-xl-none{display:none!important}.d-xl-inline{display:inline!important}.d-xl-inline-block{display:inline-block!important}.d-xl-block{display:block!important}.d-xl-table{display:table!important}.d-xl-table-cell{display:table-cell!important}.d-xl-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-xl-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}.flex-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}@media (min-width:576px){.flex-sm-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-sm-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-sm-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-sm-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-sm-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-sm-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-sm-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-sm-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-sm-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-sm-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-sm-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-sm-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-sm-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-sm-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-sm-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-sm-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-sm-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-sm-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-sm-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-sm-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-sm-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-sm-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-sm-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-sm-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-sm-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-sm-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-sm-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-sm-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-sm-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-sm-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-sm-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-sm-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:768px){.flex-md-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-md-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-md-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-md-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-md-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-md-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-md-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-md-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-md-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-md-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-md-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-md-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-md-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-md-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-md-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-md-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-md-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-md-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-md-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-md-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-md-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-md-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-md-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-md-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-md-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-md-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-md-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-md-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-md-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-md-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-md-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-md-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:992px){.flex-lg-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-lg-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-lg-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-lg-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-lg-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-lg-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-lg-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-lg-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-lg-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-lg-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-lg-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-lg-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-lg-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-lg-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-lg-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-lg-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-lg-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-lg-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-lg-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-lg-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-lg-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-lg-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-lg-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-lg-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-lg-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-lg-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-lg-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-lg-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-lg-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-lg-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-lg-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-lg-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:1200px){.flex-xl-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-xl-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-xl-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-xl-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-xl-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-xl-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-xl-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-xl-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-xl-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-xl-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-xl-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-xl-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-xl-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-xl-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-xl-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-xl-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-xl-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-xl-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-xl-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-xl-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-xl-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-xl-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-xl-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-xl-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-xl-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-xl-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-xl-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-xl-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-xl-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-xl-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-xl-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-xl-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}.float-left{float:left!important}.float-right{float:right!important}.float-none{float:none!important}@media (min-width:576px){.float-sm-left{float:left!important}.float-sm-right{float:right!important}.float-sm-none{float:none!important}}@media (min-width:768px){.float-md-left{float:left!important}.float-md-right{float:right!important}.float-md-none{float:none!important}}@media (min-width:992px){.float-lg-left{float:left!important}.float-lg-right{float:right!important}.float-lg-none{float:none!important}}@media (min-width:1200px){.float-xl-left{float:left!important}.float-xl-right{float:right!important}.float-xl-none{float:none!important}}.fixed-top{position:fixed;top:0;right:0;left:0;z-index:1030}.fixed-bottom{position:fixed;right:0;bottom:0;left:0;z-index:1030}.sticky-top{position:-webkit-sticky;position:sticky;top:0;z-index:1030}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.w-25{width:25%!important}.w-50{width:50%!important}.w-75{width:75%!important}.w-100{width:100%!important}.h-25{height:25%!important}.h-50{height:50%!important}.h-75{height:75%!important}.h-100{height:100%!important}.mw-100{max-width:100%!important}.mh-100{max-height:100%!important}.m-0{margin:0 0!important}.mt-0{margin-top:0!important}.mr-0{margin-right:0!important}.mb-0{margin-bottom:0!important}.ml-0{margin-left:0!important}.mx-0{margin-right:0!important;margin-left:0!important}.my-0{margin-top:0!important;margin-bottom:0!important}.m-1{margin:.25rem .25rem!important}.mt-1{margin-top:.25rem!important}.mr-1{margin-right:.25rem!important}.mb-1{margin-bottom:.25rem!important}.ml-1{margin-left:.25rem!important}.mx-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-2{margin:.5rem .5rem!important}.mt-2{margin-top:.5rem!important}.mr-2{margin-right:.5rem!important}.mb-2{margin-bottom:.5rem!important}.ml-2{margin-left:.5rem!important}.mx-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-3{margin:1rem 1rem!important}.mt-3{margin-top:1rem!important}.mr-3{margin-right:1rem!important}.mb-3{margin-bottom:1rem!important}.ml-3{margin-left:1rem!important}.mx-3{margin-right:1rem!important;margin-left:1rem!important}.my-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-4{margin:1.5rem 1.5rem!important}.mt-4{margin-top:1.5rem!important}.mr-4{margin-right:1.5rem!important}.mb-4{margin-bottom:1.5rem!important}.ml-4{margin-left:1.5rem!important}.mx-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-5{margin:3rem 3rem!important}.mt-5{margin-top:3rem!important}.mr-5{margin-right:3rem!important}.mb-5{margin-bottom:3rem!important}.ml-5{margin-left:3rem!important}.mx-5{margin-right:3rem!important;margin-left:3rem!important}.my-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-0{padding:0 0!important}.pt-0{padding-top:0!important}.pr-0{padding-right:0!important}.pb-0{padding-bottom:0!important}.pl-0{padding-left:0!important}.px-0{padding-right:0!important;padding-left:0!important}.py-0{padding-top:0!important;padding-bottom:0!important}.p-1{padding:.25rem .25rem!important}.pt-1{padding-top:.25rem!important}.pr-1{padding-right:.25rem!important}.pb-1{padding-bottom:.25rem!important}.pl-1{padding-left:.25rem!important}.px-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-2{padding:.5rem .5rem!important}.pt-2{padding-top:.5rem!important}.pr-2{padding-right:.5rem!important}.pb-2{padding-bottom:.5rem!important}.pl-2{padding-left:.5rem!important}.px-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-3{padding:1rem 1rem!important}.pt-3{padding-top:1rem!important}.pr-3{padding-right:1rem!important}.pb-3{padding-bottom:1rem!important}.pl-3{padding-left:1rem!important}.px-3{padding-right:1rem!important;padding-left:1rem!important}.py-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-4{padding:1.5rem 1.5rem!important}.pt-4{padding-top:1.5rem!important}.pr-4{padding-right:1.5rem!important}.pb-4{padding-bottom:1.5rem!important}.pl-4{padding-left:1.5rem!important}.px-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-5{padding:3rem 3rem!important}.pt-5{padding-top:3rem!important}.pr-5{padding-right:3rem!important}.pb-5{padding-bottom:3rem!important}.pl-5{padding-left:3rem!important}.px-5{padding-right:3rem!important;padding-left:3rem!important}.py-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-auto{margin:auto!important}.mt-auto{margin-top:auto!important}.mr-auto{margin-right:auto!important}.mb-auto{margin-bottom:auto!important}.ml-auto{margin-left:auto!important}.mx-auto{margin-right:auto!important;margin-left:auto!important}.my-auto{margin-top:auto!important;margin-bottom:auto!important}@media (min-width:576px){.m-sm-0{margin:0 0!important}.mt-sm-0{margin-top:0!important}.mr-sm-0{margin-right:0!important}.mb-sm-0{margin-bottom:0!important}.ml-sm-0{margin-left:0!important}.mx-sm-0{margin-right:0!important;margin-left:0!important}.my-sm-0{margin-top:0!important;margin-bottom:0!important}.m-sm-1{margin:.25rem .25rem!important}.mt-sm-1{margin-top:.25rem!important}.mr-sm-1{margin-right:.25rem!important}.mb-sm-1{margin-bottom:.25rem!important}.ml-sm-1{margin-left:.25rem!important}.mx-sm-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-sm-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-sm-2{margin:.5rem .5rem!important}.mt-sm-2{margin-top:.5rem!important}.mr-sm-2{margin-right:.5rem!important}.mb-sm-2{margin-bottom:.5rem!important}.ml-sm-2{margin-left:.5rem!important}.mx-sm-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-sm-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-sm-3{margin:1rem 1rem!important}.mt-sm-3{margin-top:1rem!important}.mr-sm-3{margin-right:1rem!important}.mb-sm-3{margin-bottom:1rem!important}.ml-sm-3{margin-left:1rem!important}.mx-sm-3{margin-right:1rem!important;margin-left:1rem!important}.my-sm-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-sm-4{margin:1.5rem 1.5rem!important}.mt-sm-4{margin-top:1.5rem!important}.mr-sm-4{margin-right:1.5rem!important}.mb-sm-4{margin-bottom:1.5rem!important}.ml-sm-4{margin-left:1.5rem!important}.mx-sm-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-sm-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-sm-5{margin:3rem 3rem!important}.mt-sm-5{margin-top:3rem!important}.mr-sm-5{margin-right:3rem!important}.mb-sm-5{margin-bottom:3rem!important}.ml-sm-5{margin-left:3rem!important}.mx-sm-5{margin-right:3rem!important;margin-left:3rem!important}.my-sm-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-sm-0{padding:0 0!important}.pt-sm-0{padding-top:0!important}.pr-sm-0{padding-right:0!important}.pb-sm-0{padding-bottom:0!important}.pl-sm-0{padding-left:0!important}.px-sm-0{padding-right:0!important;padding-left:0!important}.py-sm-0{padding-top:0!important;padding-bottom:0!important}.p-sm-1{padding:.25rem .25rem!important}.pt-sm-1{padding-top:.25rem!important}.pr-sm-1{padding-right:.25rem!important}.pb-sm-1{padding-bottom:.25rem!important}.pl-sm-1{padding-left:.25rem!important}.px-sm-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-sm-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-sm-2{padding:.5rem .5rem!important}.pt-sm-2{padding-top:.5rem!important}.pr-sm-2{padding-right:.5rem!important}.pb-sm-2{padding-bottom:.5rem!important}.pl-sm-2{padding-left:.5rem!important}.px-sm-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-sm-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-sm-3{padding:1rem 1rem!important}.pt-sm-3{padding-top:1rem!important}.pr-sm-3{padding-right:1rem!important}.pb-sm-3{padding-bottom:1rem!important}.pl-sm-3{padding-left:1rem!important}.px-sm-3{padding-right:1rem!important;padding-left:1rem!important}.py-sm-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-sm-4{padding:1.5rem 1.5rem!important}.pt-sm-4{padding-top:1.5rem!important}.pr-sm-4{padding-right:1.5rem!important}.pb-sm-4{padding-bottom:1.5rem!important}.pl-sm-4{padding-left:1.5rem!important}.px-sm-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-sm-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-sm-5{padding:3rem 3rem!important}.pt-sm-5{padding-top:3rem!important}.pr-sm-5{padding-right:3rem!important}.pb-sm-5{padding-bottom:3rem!important}.pl-sm-5{padding-left:3rem!important}.px-sm-5{padding-right:3rem!important;padding-left:3rem!important}.py-sm-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-sm-auto{margin:auto!important}.mt-sm-auto{margin-top:auto!important}.mr-sm-auto{margin-right:auto!important}.mb-sm-auto{margin-bottom:auto!important}.ml-sm-auto{margin-left:auto!important}.mx-sm-auto{margin-right:auto!important;margin-left:auto!important}.my-sm-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:768px){.m-md-0{margin:0 0!important}.mt-md-0{margin-top:0!important}.mr-md-0{margin-right:0!important}.mb-md-0{margin-bottom:0!important}.ml-md-0{margin-left:0!important}.mx-md-0{margin-right:0!important;margin-left:0!important}.my-md-0{margin-top:0!important;margin-bottom:0!important}.m-md-1{margin:.25rem .25rem!important}.mt-md-1{margin-top:.25rem!important}.mr-md-1{margin-right:.25rem!important}.mb-md-1{margin-bottom:.25rem!important}.ml-md-1{margin-left:.25rem!important}.mx-md-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-md-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-md-2{margin:.5rem .5rem!important}.mt-md-2{margin-top:.5rem!important}.mr-md-2{margin-right:.5rem!important}.mb-md-2{margin-bottom:.5rem!important}.ml-md-2{margin-left:.5rem!important}.mx-md-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-md-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-md-3{margin:1rem 1rem!important}.mt-md-3{margin-top:1rem!important}.mr-md-3{margin-right:1rem!important}.mb-md-3{margin-bottom:1rem!important}.ml-md-3{margin-left:1rem!important}.mx-md-3{margin-right:1rem!important;margin-left:1rem!important}.my-md-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-md-4{margin:1.5rem 1.5rem!important}.mt-md-4{margin-top:1.5rem!important}.mr-md-4{margin-right:1.5rem!important}.mb-md-4{margin-bottom:1.5rem!important}.ml-md-4{margin-left:1.5rem!important}.mx-md-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-md-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-md-5{margin:3rem 3rem!important}.mt-md-5{margin-top:3rem!important}.mr-md-5{margin-right:3rem!important}.mb-md-5{margin-bottom:3rem!important}.ml-md-5{margin-left:3rem!important}.mx-md-5{margin-right:3rem!important;margin-left:3rem!important}.my-md-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-md-0{padding:0 0!important}.pt-md-0{padding-top:0!important}.pr-md-0{padding-right:0!important}.pb-md-0{padding-bottom:0!important}.pl-md-0{padding-left:0!important}.px-md-0{padding-right:0!important;padding-left:0!important}.py-md-0{padding-top:0!important;padding-bottom:0!important}.p-md-1{padding:.25rem .25rem!important}.pt-md-1{padding-top:.25rem!important}.pr-md-1{padding-right:.25rem!important}.pb-md-1{padding-bottom:.25rem!important}.pl-md-1{padding-left:.25rem!important}.px-md-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-md-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-md-2{padding:.5rem .5rem!important}.pt-md-2{padding-top:.5rem!important}.pr-md-2{padding-right:.5rem!important}.pb-md-2{padding-bottom:.5rem!important}.pl-md-2{padding-left:.5rem!important}.px-md-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-md-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-md-3{padding:1rem 1rem!important}.pt-md-3{padding-top:1rem!important}.pr-md-3{padding-right:1rem!important}.pb-md-3{padding-bottom:1rem!important}.pl-md-3{padding-left:1rem!important}.px-md-3{padding-right:1rem!important;padding-left:1rem!important}.py-md-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-md-4{padding:1.5rem 1.5rem!important}.pt-md-4{padding-top:1.5rem!important}.pr-md-4{padding-right:1.5rem!important}.pb-md-4{padding-bottom:1.5rem!important}.pl-md-4{padding-left:1.5rem!important}.px-md-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-md-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-md-5{padding:3rem 3rem!important}.pt-md-5{padding-top:3rem!important}.pr-md-5{padding-right:3rem!important}.pb-md-5{padding-bottom:3rem!important}.pl-md-5{padding-left:3rem!important}.px-md-5{padding-right:3rem!important;padding-left:3rem!important}.py-md-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-md-auto{margin:auto!important}.mt-md-auto{margin-top:auto!important}.mr-md-auto{margin-right:auto!important}.mb-md-auto{margin-bottom:auto!important}.ml-md-auto{margin-left:auto!important}.mx-md-auto{margin-right:auto!important;margin-left:auto!important}.my-md-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:992px){.m-lg-0{margin:0 0!important}.mt-lg-0{margin-top:0!important}.mr-lg-0{margin-right:0!important}.mb-lg-0{margin-bottom:0!important}.ml-lg-0{margin-left:0!important}.mx-lg-0{margin-right:0!important;margin-left:0!important}.my-lg-0{margin-top:0!important;margin-bottom:0!important}.m-lg-1{margin:.25rem .25rem!important}.mt-lg-1{margin-top:.25rem!important}.mr-lg-1{margin-right:.25rem!important}.mb-lg-1{margin-bottom:.25rem!important}.ml-lg-1{margin-left:.25rem!important}.mx-lg-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-lg-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-lg-2{margin:.5rem .5rem!important}.mt-lg-2{margin-top:.5rem!important}.mr-lg-2{margin-right:.5rem!important}.mb-lg-2{margin-bottom:.5rem!important}.ml-lg-2{margin-left:.5rem!important}.mx-lg-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-lg-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-lg-3{margin:1rem 1rem!important}.mt-lg-3{margin-top:1rem!important}.mr-lg-3{margin-right:1rem!important}.mb-lg-3{margin-bottom:1rem!important}.ml-lg-3{margin-left:1rem!important}.mx-lg-3{margin-right:1rem!important;margin-left:1rem!important}.my-lg-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-lg-4{margin:1.5rem 1.5rem!important}.mt-lg-4{margin-top:1.5rem!important}.mr-lg-4{margin-right:1.5rem!important}.mb-lg-4{margin-bottom:1.5rem!important}.ml-lg-4{margin-left:1.5rem!important}.mx-lg-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-lg-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-lg-5{margin:3rem 3rem!important}.mt-lg-5{margin-top:3rem!important}.mr-lg-5{margin-right:3rem!important}.mb-lg-5{margin-bottom:3rem!important}.ml-lg-5{margin-left:3rem!important}.mx-lg-5{margin-right:3rem!important;margin-left:3rem!important}.my-lg-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-lg-0{padding:0 0!important}.pt-lg-0{padding-top:0!important}.pr-lg-0{padding-right:0!important}.pb-lg-0{padding-bottom:0!important}.pl-lg-0{padding-left:0!important}.px-lg-0{padding-right:0!important;padding-left:0!important}.py-lg-0{padding-top:0!important;padding-bottom:0!important}.p-lg-1{padding:.25rem .25rem!important}.pt-lg-1{padding-top:.25rem!important}.pr-lg-1{padding-right:.25rem!important}.pb-lg-1{padding-bottom:.25rem!important}.pl-lg-1{padding-left:.25rem!important}.px-lg-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-lg-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-lg-2{padding:.5rem .5rem!important}.pt-lg-2{padding-top:.5rem!important}.pr-lg-2{padding-right:.5rem!important}.pb-lg-2{padding-bottom:.5rem!important}.pl-lg-2{padding-left:.5rem!important}.px-lg-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-lg-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-lg-3{padding:1rem 1rem!important}.pt-lg-3{padding-top:1rem!important}.pr-lg-3{padding-right:1rem!important}.pb-lg-3{padding-bottom:1rem!important}.pl-lg-3{padding-left:1rem!important}.px-lg-3{padding-right:1rem!important;padding-left:1rem!important}.py-lg-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-lg-4{padding:1.5rem 1.5rem!important}.pt-lg-4{padding-top:1.5rem!important}.pr-lg-4{padding-right:1.5rem!important}.pb-lg-4{padding-bottom:1.5rem!important}.pl-lg-4{padding-left:1.5rem!important}.px-lg-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-lg-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-lg-5{padding:3rem 3rem!important}.pt-lg-5{padding-top:3rem!important}.pr-lg-5{padding-right:3rem!important}.pb-lg-5{padding-bottom:3rem!important}.pl-lg-5{padding-left:3rem!important}.px-lg-5{padding-right:3rem!important;padding-left:3rem!important}.py-lg-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-lg-auto{margin:auto!important}.mt-lg-auto{margin-top:auto!important}.mr-lg-auto{margin-right:auto!important}.mb-lg-auto{margin-bottom:auto!important}.ml-lg-auto{margin-left:auto!important}.mx-lg-auto{margin-right:auto!important;margin-left:auto!important}.my-lg-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:1200px){.m-xl-0{margin:0 0!important}.mt-xl-0{margin-top:0!important}.mr-xl-0{margin-right:0!important}.mb-xl-0{margin-bottom:0!important}.ml-xl-0{margin-left:0!important}.mx-xl-0{margin-right:0!important;margin-left:0!important}.my-xl-0{margin-top:0!important;margin-bottom:0!important}.m-xl-1{margin:.25rem .25rem!important}.mt-xl-1{margin-top:.25rem!important}.mr-xl-1{margin-right:.25rem!important}.mb-xl-1{margin-bottom:.25rem!important}.ml-xl-1{margin-left:.25rem!important}.mx-xl-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-xl-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-xl-2{margin:.5rem .5rem!important}.mt-xl-2{margin-top:.5rem!important}.mr-xl-2{margin-right:.5rem!important}.mb-xl-2{margin-bottom:.5rem!important}.ml-xl-2{margin-left:.5rem!important}.mx-xl-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-xl-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-xl-3{margin:1rem 1rem!important}.mt-xl-3{margin-top:1rem!important}.mr-xl-3{margin-right:1rem!important}.mb-xl-3{margin-bottom:1rem!important}.ml-xl-3{margin-left:1rem!important}.mx-xl-3{margin-right:1rem!important;margin-left:1rem!important}.my-xl-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-xl-4{margin:1.5rem 1.5rem!important}.mt-xl-4{margin-top:1.5rem!important}.mr-xl-4{margin-right:1.5rem!important}.mb-xl-4{margin-bottom:1.5rem!important}.ml-xl-4{margin-left:1.5rem!important}.mx-xl-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-xl-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-xl-5{margin:3rem 3rem!important}.mt-xl-5{margin-top:3rem!important}.mr-xl-5{margin-right:3rem!important}.mb-xl-5{margin-bottom:3rem!important}.ml-xl-5{margin-left:3rem!important}.mx-xl-5{margin-right:3rem!important;margin-left:3rem!important}.my-xl-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-xl-0{padding:0 0!important}.pt-xl-0{padding-top:0!important}.pr-xl-0{padding-right:0!important}.pb-xl-0{padding-bottom:0!important}.pl-xl-0{padding-left:0!important}.px-xl-0{padding-right:0!important;padding-left:0!important}.py-xl-0{padding-top:0!important;padding-bottom:0!important}.p-xl-1{padding:.25rem .25rem!important}.pt-xl-1{padding-top:.25rem!important}.pr-xl-1{padding-right:.25rem!important}.pb-xl-1{padding-bottom:.25rem!important}.pl-xl-1{padding-left:.25rem!important}.px-xl-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-xl-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-xl-2{padding:.5rem .5rem!important}.pt-xl-2{padding-top:.5rem!important}.pr-xl-2{padding-right:.5rem!important}.pb-xl-2{padding-bottom:.5rem!important}.pl-xl-2{padding-left:.5rem!important}.px-xl-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-xl-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-xl-3{padding:1rem 1rem!important}.pt-xl-3{padding-top:1rem!important}.pr-xl-3{padding-right:1rem!important}.pb-xl-3{padding-bottom:1rem!important}.pl-xl-3{padding-left:1rem!important}.px-xl-3{padding-right:1rem!important;padding-left:1rem!important}.py-xl-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-xl-4{padding:1.5rem 1.5rem!important}.pt-xl-4{padding-top:1.5rem!important}.pr-xl-4{padding-right:1.5rem!important}.pb-xl-4{padding-bottom:1.5rem!important}.pl-xl-4{padding-left:1.5rem!important}.px-xl-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-xl-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-xl-5{padding:3rem 3rem!important}.pt-xl-5{padding-top:3rem!important}.pr-xl-5{padding-right:3rem!important}.pb-xl-5{padding-bottom:3rem!important}.pl-xl-5{padding-left:3rem!important}.px-xl-5{padding-right:3rem!important;padding-left:3rem!important}.py-xl-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-xl-auto{margin:auto!important}.mt-xl-auto{margin-top:auto!important}.mr-xl-auto{margin-right:auto!important}.mb-xl-auto{margin-bottom:auto!important}.ml-xl-auto{margin-left:auto!important}.mx-xl-auto{margin-right:auto!important;margin-left:auto!important}.my-xl-auto{margin-top:auto!important;margin-bottom:auto!important}}.text-justify{text-align:justify!important}.text-nowrap{white-space:nowrap!important}.text-truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.text-left{text-align:left!important}.text-right{text-align:right!important}.text-center{text-align:center!important}@media (min-width:576px){.text-sm-left{text-align:left!important}.text-sm-right{text-align:right!important}.text-sm-center{text-align:center!important}}@media (min-width:768px){.text-md-left{text-align:left!important}.text-md-right{text-align:right!important}.text-md-center{text-align:center!important}}@media (min-width:992px){.text-lg-left{text-align:left!important}.text-lg-right{text-align:right!important}.text-lg-center{text-align:center!important}}@media (min-width:1200px){.text-xl-left{text-align:left!important}.text-xl-right{text-align:right!important}.text-xl-center{text-align:center!important}}.text-lowercase{text-transform:lowercase!important}.text-uppercase{text-transform:uppercase!important}.text-capitalize{text-transform:capitalize!important}.font-weight-normal{font-weight:400}.font-weight-bold{font-weight:700}.font-italic{font-style:italic}.text-white{color:#fff!important}.text-muted{color:#636c72!important}a.text-muted:focus,a.text-muted:hover{color:#4b5257!important}.text-primary{color:#0275d8!important}a.text-primary:focus,a.text-primary:hover{color:#025aa5!important}.text-success{color:#5cb85c!important}a.text-success:focus,a.text-success:hover{color:#449d44!important}.text-info{color:#5bc0de!important}a.text-info:focus,a.text-info:hover{color:#31b0d5!important}.text-warning{color:#f0ad4e!important}a.text-warning:focus,a.text-warning:hover{color:#ec971f!important}.text-danger{color:#d9534f!important}a.text-danger:focus,a.text-danger:hover{color:#c9302c!important}.text-gray-dark{color:#292b2c!important}a.text-gray-dark:focus,a.text-gray-dark:hover{color:#101112!important}.text-hide{font:0/0 a;color:transparent;text-shadow:none;background-color:transparent;border:0}.invisible{visibility:hidden!important}.hidden-xs-up{display:none!important}@media (max-width:575px){.hidden-xs-down{display:none!important}}@media (min-width:576px){.hidden-sm-up{display:none!important}}@media (max-width:767px){.hidden-sm-down{display:none!important}}@media (min-width:768px){.hidden-md-up{display:none!important}}@media (max-width:991px){.hidden-md-down{display:none!important}}@media (min-width:992px){.hidden-lg-up{display:none!important}}@media (max-width:1199px){.hidden-lg-down{display:none!important}}@media (min-width:1200px){.hidden-xl-up{display:none!important}}.hidden-xl-down{display:none!important}.visible-print-block{display:none!important}@media print{.visible-print-block{display:block!important}}.visible-print-inline{display:none!important}@media print{.visible-print-inline{display:inline!important}}.visible-print-inline-block{display:none!important}@media print{.visible-print-inline-block{display:inline-block!important}}@media print{.hidden-print{display:none!important}}/*# sourceMappingURL=bootstrap.min.css.map */ \ No newline at end of file diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/external.png b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/external.png new file mode 100755 index 0000000..7e1a5f0 Binary files /dev/null and b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/external.png differ diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.dataTables.min.css b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.dataTables.min.css new file mode 100644 index 0000000..4303138 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.dataTables.min.css @@ -0,0 +1 @@ +table.dataTable{width:100%;margin:0 auto;clear:both;border-collapse:separate;border-spacing:0}table.dataTable thead th,table.dataTable tfoot th{font-weight:bold}table.dataTable thead th,table.dataTable thead td{padding:10px 18px;border-bottom:1px solid #111}table.dataTable thead th:active,table.dataTable thead td:active{outline:none}table.dataTable tfoot th,table.dataTable tfoot td{padding:10px 18px 6px 18px;border-top:1px solid #111}table.dataTable thead .sorting,table.dataTable thead .sorting_asc,table.dataTable thead .sorting_desc,table.dataTable thead .sorting_asc_disabled,table.dataTable thead .sorting_desc_disabled{cursor:pointer;*cursor:hand;background-repeat:no-repeat;background-position:center right}table.dataTable thead .sorting{background-image:url("sort_both.png")}table.dataTable thead .sorting_asc{background-image:url("sort_asc.png")}table.dataTable thead .sorting_desc{background-image:url("sort_desc.png")}table.dataTable thead .sorting_asc_disabled{background-image:url("sort_asc_disabled.png")}table.dataTable thead .sorting_desc_disabled{background-image:url("sort_desc_disabled.png")}table.dataTable tbody tr{background-color:#ffffff}table.dataTable tbody tr.selected{background-color:#B0BED9}table.dataTable tbody th,table.dataTable tbody td{padding:8px 10px}table.dataTable.row-border tbody th,table.dataTable.row-border tbody td,table.dataTable.display tbody th,table.dataTable.display tbody td{border-top:1px solid #ddd}table.dataTable.row-border tbody tr:first-child th,table.dataTable.row-border tbody tr:first-child td,table.dataTable.display tbody tr:first-child th,table.dataTable.display tbody tr:first-child td{border-top:none}table.dataTable.cell-border tbody th,table.dataTable.cell-border tbody td{border-top:1px solid #ddd;border-right:1px solid #ddd}table.dataTable.cell-border tbody tr th:first-child,table.dataTable.cell-border tbody tr td:first-child{border-left:1px solid #ddd}table.dataTable.cell-border tbody tr:first-child th,table.dataTable.cell-border tbody tr:first-child td{border-top:none}table.dataTable.stripe tbody tr.odd,table.dataTable.display tbody tr.odd{background-color:#f9f9f9}table.dataTable.stripe tbody tr.odd.selected,table.dataTable.display tbody tr.odd.selected{background-color:#acbad4}table.dataTable.hover tbody tr:hover,table.dataTable.display tbody tr:hover{background-color:#f6f6f6}table.dataTable.hover tbody tr:hover.selected,table.dataTable.display tbody tr:hover.selected{background-color:#aab7d1}table.dataTable.order-column tbody tr>.sorting_1,table.dataTable.order-column tbody tr>.sorting_2,table.dataTable.order-column tbody tr>.sorting_3,table.dataTable.display tbody tr>.sorting_1,table.dataTable.display tbody tr>.sorting_2,table.dataTable.display tbody tr>.sorting_3{background-color:#fafafa}table.dataTable.order-column tbody tr.selected>.sorting_1,table.dataTable.order-column tbody tr.selected>.sorting_2,table.dataTable.order-column tbody tr.selected>.sorting_3,table.dataTable.display tbody tr.selected>.sorting_1,table.dataTable.display tbody tr.selected>.sorting_2,table.dataTable.display tbody tr.selected>.sorting_3{background-color:#acbad5}table.dataTable.display tbody tr.odd>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd>.sorting_1{background-color:#f1f1f1}table.dataTable.display tbody tr.odd>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd>.sorting_2{background-color:#f3f3f3}table.dataTable.display tbody tr.odd>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd>.sorting_3{background-color:whitesmoke}table.dataTable.display tbody tr.odd.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_1{background-color:#a6b4cd}table.dataTable.display tbody tr.odd.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_2{background-color:#a8b5cf}table.dataTable.display tbody tr.odd.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_3{background-color:#a9b7d1}table.dataTable.display tbody tr.even>.sorting_1,table.dataTable.order-column.stripe tbody tr.even>.sorting_1{background-color:#fafafa}table.dataTable.display tbody tr.even>.sorting_2,table.dataTable.order-column.stripe tbody tr.even>.sorting_2{background-color:#fcfcfc}table.dataTable.display tbody tr.even>.sorting_3,table.dataTable.order-column.stripe tbody tr.even>.sorting_3{background-color:#fefefe}table.dataTable.display tbody tr.even.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_1{background-color:#acbad5}table.dataTable.display tbody tr.even.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_2{background-color:#aebcd6}table.dataTable.display tbody tr.even.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_3{background-color:#afbdd8}table.dataTable.display tbody tr:hover>.sorting_1,table.dataTable.order-column.hover tbody tr:hover>.sorting_1{background-color:#eaeaea}table.dataTable.display tbody tr:hover>.sorting_2,table.dataTable.order-column.hover tbody tr:hover>.sorting_2{background-color:#ececec}table.dataTable.display tbody tr:hover>.sorting_3,table.dataTable.order-column.hover tbody tr:hover>.sorting_3{background-color:#efefef}table.dataTable.display tbody tr:hover.selected>.sorting_1,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_1{background-color:#a2aec7}table.dataTable.display tbody tr:hover.selected>.sorting_2,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_2{background-color:#a3b0c9}table.dataTable.display tbody tr:hover.selected>.sorting_3,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_3{background-color:#a5b2cb}table.dataTable.no-footer{border-bottom:1px solid #111}table.dataTable.nowrap th,table.dataTable.nowrap td{white-space:nowrap}table.dataTable.compact thead th,table.dataTable.compact thead td{padding:4px 17px 4px 4px}table.dataTable.compact tfoot th,table.dataTable.compact tfoot td{padding:4px}table.dataTable.compact tbody th,table.dataTable.compact tbody td{padding:4px}table.dataTable th.dt-left,table.dataTable td.dt-left{text-align:left}table.dataTable th.dt-center,table.dataTable td.dt-center,table.dataTable td.dataTables_empty{text-align:center}table.dataTable th.dt-right,table.dataTable td.dt-right{text-align:right}table.dataTable th.dt-justify,table.dataTable td.dt-justify{text-align:justify}table.dataTable th.dt-nowrap,table.dataTable td.dt-nowrap{white-space:nowrap}table.dataTable thead th.dt-head-left,table.dataTable thead td.dt-head-left,table.dataTable tfoot th.dt-head-left,table.dataTable tfoot td.dt-head-left{text-align:left}table.dataTable thead th.dt-head-center,table.dataTable thead td.dt-head-center,table.dataTable tfoot th.dt-head-center,table.dataTable tfoot td.dt-head-center{text-align:center}table.dataTable thead th.dt-head-right,table.dataTable thead td.dt-head-right,table.dataTable tfoot th.dt-head-right,table.dataTable tfoot td.dt-head-right{text-align:right}table.dataTable thead th.dt-head-justify,table.dataTable thead td.dt-head-justify,table.dataTable tfoot th.dt-head-justify,table.dataTable tfoot td.dt-head-justify{text-align:justify}table.dataTable thead th.dt-head-nowrap,table.dataTable thead td.dt-head-nowrap,table.dataTable tfoot th.dt-head-nowrap,table.dataTable tfoot td.dt-head-nowrap{white-space:nowrap}table.dataTable tbody th.dt-body-left,table.dataTable tbody td.dt-body-left{text-align:left}table.dataTable tbody th.dt-body-center,table.dataTable tbody td.dt-body-center{text-align:center}table.dataTable tbody th.dt-body-right,table.dataTable tbody td.dt-body-right{text-align:right}table.dataTable tbody th.dt-body-justify,table.dataTable tbody td.dt-body-justify{text-align:justify}table.dataTable tbody th.dt-body-nowrap,table.dataTable tbody td.dt-body-nowrap{white-space:nowrap}table.dataTable,table.dataTable th,table.dataTable td{box-sizing:content-box}.dataTables_wrapper{position:relative;clear:both;*zoom:1;zoom:1}.dataTables_wrapper .dataTables_length{float:left}.dataTables_wrapper .dataTables_filter{float:right;text-align:right}.dataTables_wrapper .dataTables_filter input{margin-left:0.5em}.dataTables_wrapper .dataTables_info{clear:both;float:left;padding-top:0.755em}.dataTables_wrapper .dataTables_paginate{float:right;text-align:right;padding-top:0.25em}.dataTables_wrapper .dataTables_paginate .paginate_button{box-sizing:border-box;display:inline-block;min-width:1.5em;padding:0.5em 1em;margin-left:2px;text-align:center;text-decoration:none !important;cursor:pointer;*cursor:hand;color:#333 !important;border:1px solid transparent;border-radius:2px}.dataTables_wrapper .dataTables_paginate .paginate_button.current,.dataTables_wrapper .dataTables_paginate .paginate_button.current:hover{color:#333 !important;border:1px solid #979797;background-color:white;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #fff), color-stop(100%, #dcdcdc));background:-webkit-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-moz-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-ms-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-o-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:linear-gradient(to bottom, #fff 0%, #dcdcdc 100%)}.dataTables_wrapper .dataTables_paginate .paginate_button.disabled,.dataTables_wrapper .dataTables_paginate .paginate_button.disabled:hover,.dataTables_wrapper .dataTables_paginate .paginate_button.disabled:active{cursor:default;color:#666 !important;border:1px solid transparent;background:transparent;box-shadow:none}.dataTables_wrapper .dataTables_paginate .paginate_button:hover{color:white !important;border:1px solid #111;background-color:#585858;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #585858), color-stop(100%, #111));background:-webkit-linear-gradient(top, #585858 0%, #111 100%);background:-moz-linear-gradient(top, #585858 0%, #111 100%);background:-ms-linear-gradient(top, #585858 0%, #111 100%);background:-o-linear-gradient(top, #585858 0%, #111 100%);background:linear-gradient(to bottom, #585858 0%, #111 100%)}.dataTables_wrapper .dataTables_paginate .paginate_button:active{outline:none;background-color:#2b2b2b;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #2b2b2b), color-stop(100%, #0c0c0c));background:-webkit-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-moz-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-ms-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-o-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:linear-gradient(to bottom, #2b2b2b 0%, #0c0c0c 100%);box-shadow:inset 0 0 3px #111}.dataTables_wrapper .dataTables_paginate .ellipsis{padding:0 1em}.dataTables_wrapper .dataTables_processing{position:absolute;top:50%;left:50%;width:100%;height:40px;margin-left:-50%;margin-top:-25px;padding-top:20px;text-align:center;font-size:1.2em;background-color:white;background:-webkit-gradient(linear, left top, right top, color-stop(0%, rgba(255,255,255,0)), color-stop(25%, rgba(255,255,255,0.9)), color-stop(75%, rgba(255,255,255,0.9)), color-stop(100%, rgba(255,255,255,0)));background:-webkit-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-moz-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-ms-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-o-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:linear-gradient(to right, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%)}.dataTables_wrapper .dataTables_length,.dataTables_wrapper .dataTables_filter,.dataTables_wrapper .dataTables_info,.dataTables_wrapper .dataTables_processing,.dataTables_wrapper .dataTables_paginate{color:#333}.dataTables_wrapper .dataTables_scroll{clear:both}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody{*margin-top:-1px;-webkit-overflow-scrolling:touch}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>th,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>td,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>th,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>td{vertical-align:middle}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>th>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>td>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>th>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>td>div.dataTables_sizing{height:0;overflow:hidden;margin:0 !important;padding:0 !important}.dataTables_wrapper.no-footer .dataTables_scrollBody{border-bottom:1px solid #111}.dataTables_wrapper.no-footer div.dataTables_scrollHead table.dataTable,.dataTables_wrapper.no-footer div.dataTables_scrollBody>table{border-bottom:none}.dataTables_wrapper:after{visibility:hidden;display:block;content:"";clear:both;height:0}@media screen and (max-width: 767px){.dataTables_wrapper .dataTables_info,.dataTables_wrapper .dataTables_paginate{float:none;text-align:center}.dataTables_wrapper .dataTables_paginate{margin-top:0.5em}}@media screen and (max-width: 640px){.dataTables_wrapper .dataTables_length,.dataTables_wrapper .dataTables_filter{float:none;text-align:center}.dataTables_wrapper .dataTables_filter{margin-top:0.5em}} diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.dataTables.min.js b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.dataTables.min.js new file mode 100644 index 0000000..07af1c3 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.dataTables.min.js @@ -0,0 +1,166 @@ +/*! + DataTables 1.10.19 + ©2008-2018 SpryMedia Ltd - datatables.net/license +*/ +(function(h){"function"===typeof define&&define.amd?define(["jquery"],function(E){return h(E,window,document)}):"object"===typeof exports?module.exports=function(E,H){E||(E=window);H||(H="undefined"!==typeof window?require("jquery"):require("jquery")(E));return h(H,E,E.document)}:h(jQuery,window,document)})(function(h,E,H,k){function Z(a){var b,c,d={};h.each(a,function(e){if((b=e.match(/^([^A-Z]+?)([A-Z])/))&&-1!=="a aa ai ao as b fn i m o s ".indexOf(b[1]+" "))c=e.replace(b[0],b[2].toLowerCase()), +d[c]=e,"o"===b[1]&&Z(a[e])});a._hungarianMap=d}function J(a,b,c){a._hungarianMap||Z(a);var d;h.each(b,function(e){d=a._hungarianMap[e];if(d!==k&&(c||b[d]===k))"o"===d.charAt(0)?(b[d]||(b[d]={}),h.extend(!0,b[d],b[e]),J(a[d],b[d],c)):b[d]=b[e]})}function Ca(a){var b=n.defaults.oLanguage,c=b.sDecimal;c&&Da(c);if(a){var d=a.sZeroRecords;!a.sEmptyTable&&(d&&"No data available in table"===b.sEmptyTable)&&F(a,a,"sZeroRecords","sEmptyTable");!a.sLoadingRecords&&(d&&"Loading..."===b.sLoadingRecords)&&F(a, +a,"sZeroRecords","sLoadingRecords");a.sInfoThousands&&(a.sThousands=a.sInfoThousands);(a=a.sDecimal)&&c!==a&&Da(a)}}function fb(a){A(a,"ordering","bSort");A(a,"orderMulti","bSortMulti");A(a,"orderClasses","bSortClasses");A(a,"orderCellsTop","bSortCellsTop");A(a,"order","aaSorting");A(a,"orderFixed","aaSortingFixed");A(a,"paging","bPaginate");A(a,"pagingType","sPaginationType");A(a,"pageLength","iDisplayLength");A(a,"searching","bFilter");"boolean"===typeof a.sScrollX&&(a.sScrollX=a.sScrollX?"100%": +"");"boolean"===typeof a.scrollX&&(a.scrollX=a.scrollX?"100%":"");if(a=a.aoSearchCols)for(var b=0,c=a.length;b").css({position:"fixed",top:0,left:-1*h(E).scrollLeft(),height:1,width:1, +overflow:"hidden"}).append(h("
    ").css({position:"absolute",top:1,left:1,width:100,overflow:"scroll"}).append(h("
    ").css({width:"100%",height:10}))).appendTo("body"),d=c.children(),e=d.children();b.barWidth=d[0].offsetWidth-d[0].clientWidth;b.bScrollOversize=100===e[0].offsetWidth&&100!==d[0].clientWidth;b.bScrollbarLeft=1!==Math.round(e.offset().left);b.bBounding=c[0].getBoundingClientRect().width?!0:!1;c.remove()}h.extend(a.oBrowser,n.__browser);a.oScroll.iBarWidth=n.__browser.barWidth} +function ib(a,b,c,d,e,f){var g,j=!1;c!==k&&(g=c,j=!0);for(;d!==e;)a.hasOwnProperty(d)&&(g=j?b(g,a[d],d,a):a[d],j=!0,d+=f);return g}function Ea(a,b){var c=n.defaults.column,d=a.aoColumns.length,c=h.extend({},n.models.oColumn,c,{nTh:b?b:H.createElement("th"),sTitle:c.sTitle?c.sTitle:b?b.innerHTML:"",aDataSort:c.aDataSort?c.aDataSort:[d],mData:c.mData?c.mData:d,idx:d});a.aoColumns.push(c);c=a.aoPreSearchCols;c[d]=h.extend({},n.models.oSearch,c[d]);ka(a,d,h(b).data())}function ka(a,b,c){var b=a.aoColumns[b], +d=a.oClasses,e=h(b.nTh);if(!b.sWidthOrig){b.sWidthOrig=e.attr("width")||null;var f=(e.attr("style")||"").match(/width:\s*(\d+[pxem%]+)/);f&&(b.sWidthOrig=f[1])}c!==k&&null!==c&&(gb(c),J(n.defaults.column,c),c.mDataProp!==k&&!c.mData&&(c.mData=c.mDataProp),c.sType&&(b._sManualType=c.sType),c.className&&!c.sClass&&(c.sClass=c.className),c.sClass&&e.addClass(c.sClass),h.extend(b,c),F(b,c,"sWidth","sWidthOrig"),c.iDataSort!==k&&(b.aDataSort=[c.iDataSort]),F(b,c,"aDataSort"));var g=b.mData,j=S(g),i=b.mRender? +S(b.mRender):null,c=function(a){return"string"===typeof a&&-1!==a.indexOf("@")};b._bAttrSrc=h.isPlainObject(g)&&(c(g.sort)||c(g.type)||c(g.filter));b._setter=null;b.fnGetData=function(a,b,c){var d=j(a,b,k,c);return i&&b?i(d,b,a,c):d};b.fnSetData=function(a,b,c){return N(g)(a,b,c)};"number"!==typeof g&&(a._rowReadObject=!0);a.oFeatures.bSort||(b.bSortable=!1,e.addClass(d.sSortableNone));a=-1!==h.inArray("asc",b.asSorting);c=-1!==h.inArray("desc",b.asSorting);!b.bSortable||!a&&!c?(b.sSortingClass=d.sSortableNone, +b.sSortingClassJUI=""):a&&!c?(b.sSortingClass=d.sSortableAsc,b.sSortingClassJUI=d.sSortJUIAscAllowed):!a&&c?(b.sSortingClass=d.sSortableDesc,b.sSortingClassJUI=d.sSortJUIDescAllowed):(b.sSortingClass=d.sSortable,b.sSortingClassJUI=d.sSortJUI)}function $(a){if(!1!==a.oFeatures.bAutoWidth){var b=a.aoColumns;Fa(a);for(var c=0,d=b.length;cq[f])d(l.length+q[f],m);else if("string"=== +typeof q[f]){j=0;for(i=l.length;jb&&a[e]--; -1!=d&&c===k&&a.splice(d, +1)}function da(a,b,c,d){var e=a.aoData[b],f,g=function(c,d){for(;c.childNodes.length;)c.removeChild(c.firstChild);c.innerHTML=B(a,b,d,"display")};if("dom"===c||(!c||"auto"===c)&&"dom"===e.src)e._aData=Ia(a,e,d,d===k?k:e._aData).data;else{var j=e.anCells;if(j)if(d!==k)g(j[d],d);else{c=0;for(f=j.length;c").appendTo(g));b=0;for(c=l.length;btr").attr("role","row");h(g).find(">tr>th, >tr>td").addClass(m.sHeaderTH);h(j).find(">tr>th, >tr>td").addClass(m.sFooterTH);if(null!==j){a=a.aoFooter[0];b=0;for(c=a.length;b=a.fnRecordsDisplay()?0:g,a.iInitDisplayStart=-1);var g=a._iDisplayStart,m=a.fnDisplayEnd();if(a.bDeferLoading)a.bDeferLoading=!1,a.iDraw++,C(a,!1);else if(j){if(!a.bDestroying&&!mb(a))return}else a.iDraw++;if(0!==i.length){f=j?a.aoData.length:m;for(j=j?0:g;j",{"class":e?d[0]:""}).append(h("",{valign:"top",colSpan:V(a),"class":a.oClasses.sRowEmpty}).html(c))[0];r(a,"aoHeaderCallback","header",[h(a.nTHead).children("tr")[0],Ka(a),g,m,i]);r(a,"aoFooterCallback","footer",[h(a.nTFoot).children("tr")[0],Ka(a),g,m,i]);d=h(a.nTBody);d.children().detach(); +d.append(h(b));r(a,"aoDrawCallback","draw",[a]);a.bSorted=!1;a.bFiltered=!1;a.bDrawing=!1}}function T(a,b){var c=a.oFeatures,d=c.bFilter;c.bSort&&nb(a);d?ga(a,a.oPreviousSearch):a.aiDisplay=a.aiDisplayMaster.slice();!0!==b&&(a._iDisplayStart=0);a._drawHold=b;P(a);a._drawHold=!1}function ob(a){var b=a.oClasses,c=h(a.nTable),c=h("
    ").insertBefore(c),d=a.oFeatures,e=h("
    ",{id:a.sTableId+"_wrapper","class":b.sWrapper+(a.nTFoot?"":" "+b.sNoFooter)});a.nHolding=c[0];a.nTableWrapper=e[0];a.nTableReinsertBefore= +a.nTable.nextSibling;for(var f=a.sDom.split(""),g,j,i,m,l,q,k=0;k")[0];m=f[k+1];if("'"==m||'"'==m){l="";for(q=2;f[k+q]!=m;)l+=f[k+q],q++;"H"==l?l=b.sJUIHeader:"F"==l&&(l=b.sJUIFooter);-1!=l.indexOf(".")?(m=l.split("."),i.id=m[0].substr(1,m[0].length-1),i.className=m[1]):"#"==l.charAt(0)?i.id=l.substr(1,l.length-1):i.className=l;k+=q}e.append(i);e=h(i)}else if(">"==j)e=e.parent();else if("l"==j&&d.bPaginate&&d.bLengthChange)g=pb(a);else if("f"==j&& +d.bFilter)g=qb(a);else if("r"==j&&d.bProcessing)g=rb(a);else if("t"==j)g=sb(a);else if("i"==j&&d.bInfo)g=tb(a);else if("p"==j&&d.bPaginate)g=ub(a);else if(0!==n.ext.feature.length){i=n.ext.feature;q=0;for(m=i.length;q',j=d.sSearch,j=j.match(/_INPUT_/)?j.replace("_INPUT_", +g):j+g,b=h("
    ",{id:!f.f?c+"_filter":null,"class":b.sFilter}).append(h("
    ").addClass(b.sLength);a.aanFeatures.l||(i[0].id=c+"_length");i.children().append(a.oLanguage.sLengthMenu.replace("_MENU_",e[0].outerHTML));h("select",i).val(a._iDisplayLength).on("change.DT",function(){Ra(a,h(this).val());P(a)});h(a.nTable).on("length.dt.DT",function(b,c,d){a=== +c&&h("select",i).val(d)});return i[0]}function ub(a){var b=a.sPaginationType,c=n.ext.pager[b],d="function"===typeof c,e=function(a){P(a)},b=h("
    ").addClass(a.oClasses.sPaging+b)[0],f=a.aanFeatures;d||c.fnInit(a,b,e);f.p||(b.id=a.sTableId+"_paginate",a.aoDrawCallback.push({fn:function(a){if(d){var b=a._iDisplayStart,i=a._iDisplayLength,h=a.fnRecordsDisplay(),l=-1===i,b=l?0:Math.ceil(b/i),i=l?1:Math.ceil(h/i),h=c(b,i),k,l=0;for(k=f.p.length;lf&&(d=0)):"first"==b?d=0:"previous"==b?(d=0<=e?d-e:0,0>d&&(d=0)):"next"==b?d+e",{id:!a.aanFeatures.r?a.sTableId+"_processing":null,"class":a.oClasses.sProcessing}).html(a.oLanguage.sProcessing).insertBefore(a.nTable)[0]} +function C(a,b){a.oFeatures.bProcessing&&h(a.aanFeatures.r).css("display",b?"block":"none");r(a,null,"processing",[a,b])}function sb(a){var b=h(a.nTable);b.attr("role","grid");var c=a.oScroll;if(""===c.sX&&""===c.sY)return a.nTable;var d=c.sX,e=c.sY,f=a.oClasses,g=b.children("caption"),j=g.length?g[0]._captionSide:null,i=h(b[0].cloneNode(!1)),m=h(b[0].cloneNode(!1)),l=b.children("tfoot");l.length||(l=null);i=h("
    ",{"class":f.sScrollWrapper}).append(h("
    ",{"class":f.sScrollHead}).css({overflow:"hidden", +position:"relative",border:0,width:d?!d?null:v(d):"100%"}).append(h("
    ",{"class":f.sScrollHeadInner}).css({"box-sizing":"content-box",width:c.sXInner||"100%"}).append(i.removeAttr("id").css("margin-left",0).append("top"===j?g:null).append(b.children("thead"))))).append(h("
    ",{"class":f.sScrollBody}).css({position:"relative",overflow:"auto",width:!d?null:v(d)}).append(b));l&&i.append(h("
    ",{"class":f.sScrollFoot}).css({overflow:"hidden",border:0,width:d?!d?null:v(d):"100%"}).append(h("
    ", +{"class":f.sScrollFootInner}).append(m.removeAttr("id").css("margin-left",0).append("bottom"===j?g:null).append(b.children("tfoot")))));var b=i.children(),k=b[0],f=b[1],t=l?b[2]:null;if(d)h(f).on("scroll.DT",function(){var a=this.scrollLeft;k.scrollLeft=a;l&&(t.scrollLeft=a)});h(f).css(e&&c.bCollapse?"max-height":"height",e);a.nScrollHead=k;a.nScrollBody=f;a.nScrollFoot=t;a.aoDrawCallback.push({fn:la,sName:"scrolling"});return i[0]}function la(a){var b=a.oScroll,c=b.sX,d=b.sXInner,e=b.sY,b=b.iBarWidth, +f=h(a.nScrollHead),g=f[0].style,j=f.children("div"),i=j[0].style,m=j.children("table"),j=a.nScrollBody,l=h(j),q=j.style,t=h(a.nScrollFoot).children("div"),n=t.children("table"),o=h(a.nTHead),p=h(a.nTable),s=p[0],r=s.style,u=a.nTFoot?h(a.nTFoot):null,x=a.oBrowser,U=x.bScrollOversize,Xb=D(a.aoColumns,"nTh"),Q,L,R,w,Ua=[],y=[],z=[],A=[],B,C=function(a){a=a.style;a.paddingTop="0";a.paddingBottom="0";a.borderTopWidth="0";a.borderBottomWidth="0";a.height=0};L=j.scrollHeight>j.clientHeight;if(a.scrollBarVis!== +L&&a.scrollBarVis!==k)a.scrollBarVis=L,$(a);else{a.scrollBarVis=L;p.children("thead, tfoot").remove();u&&(R=u.clone().prependTo(p),Q=u.find("tr"),R=R.find("tr"));w=o.clone().prependTo(p);o=o.find("tr");L=w.find("tr");w.find("th, td").removeAttr("tabindex");c||(q.width="100%",f[0].style.width="100%");h.each(ra(a,w),function(b,c){B=aa(a,b);c.style.width=a.aoColumns[B].sWidth});u&&I(function(a){a.style.width=""},R);f=p.outerWidth();if(""===c){r.width="100%";if(U&&(p.find("tbody").height()>j.offsetHeight|| +"scroll"==l.css("overflow-y")))r.width=v(p.outerWidth()-b);f=p.outerWidth()}else""!==d&&(r.width=v(d),f=p.outerWidth());I(C,L);I(function(a){z.push(a.innerHTML);Ua.push(v(h(a).css("width")))},L);I(function(a,b){if(h.inArray(a,Xb)!==-1)a.style.width=Ua[b]},o);h(L).height(0);u&&(I(C,R),I(function(a){A.push(a.innerHTML);y.push(v(h(a).css("width")))},R),I(function(a,b){a.style.width=y[b]},Q),h(R).height(0));I(function(a,b){a.innerHTML='
    '+z[b]+"
    ";a.childNodes[0].style.height= +"0";a.childNodes[0].style.overflow="hidden";a.style.width=Ua[b]},L);u&&I(function(a,b){a.innerHTML='
    '+A[b]+"
    ";a.childNodes[0].style.height="0";a.childNodes[0].style.overflow="hidden";a.style.width=y[b]},R);if(p.outerWidth()j.offsetHeight||"scroll"==l.css("overflow-y")?f+b:f;if(U&&(j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=v(Q-b);(""===c||""!==d)&&K(a,1,"Possible column misalignment",6)}else Q="100%";q.width=v(Q); +g.width=v(Q);u&&(a.nScrollFoot.style.width=v(Q));!e&&U&&(q.height=v(s.offsetHeight+b));c=p.outerWidth();m[0].style.width=v(c);i.width=v(c);d=p.height()>j.clientHeight||"scroll"==l.css("overflow-y");e="padding"+(x.bScrollbarLeft?"Left":"Right");i[e]=d?b+"px":"0px";u&&(n[0].style.width=v(c),t[0].style.width=v(c),t[0].style[e]=d?b+"px":"0px");p.children("colgroup").insertBefore(p.children("thead"));l.scroll();if((a.bSorted||a.bFiltered)&&!a._drawHold)j.scrollTop=0}}function I(a,b,c){for(var d=0,e=0, +f=b.length,g,j;e").appendTo(j.find("tbody"));j.find("thead, tfoot").remove();j.append(h(a.nTHead).clone()).append(h(a.nTFoot).clone());j.find("tfoot th, tfoot td").css("width","");m=ra(a,j.find("thead")[0]);for(n=0;n").css({width:o.sWidthOrig,margin:0,padding:0,border:0,height:1}));if(a.aoData.length)for(n=0;n").css(f||e?{position:"absolute",top:0,left:0,height:1,right:0,overflow:"hidden"}:{}).append(j).appendTo(k);f&&g?j.width(g):f?(j.css("width","auto"),j.removeAttr("width"),j.width()").css("width",v(a)).appendTo(b||H.body),d=c[0].offsetWidth;c.remove();return d}function Gb(a, +b){var c=Hb(a,b);if(0>c)return null;var d=a.aoData[c];return!d.nTr?h("").html(B(a,c,b,"display"))[0]:d.anCells[b]}function Hb(a,b){for(var c,d=-1,e=-1,f=0,g=a.aoData.length;fd&&(d=c.length,e=f);return e}function v(a){return null===a?"0px":"number"==typeof a?0>a?"0px":a+"px":a.match(/\d$/)?a+"px":a}function X(a){var b,c,d=[],e=a.aoColumns,f,g,j,i;b=a.aaSortingFixed;c=h.isPlainObject(b);var m=[];f=function(a){a.length&& +!h.isArray(a[0])?m.push(a):h.merge(m,a)};h.isArray(b)&&f(b);c&&b.pre&&f(b.pre);f(a.aaSorting);c&&b.post&&f(b.post);for(a=0;ae?1:0,0!==c)return"asc"===j.dir?c:-c;c=d[a];e=d[b];return ce?1:0}):i.sort(function(a,b){var c,g,j,i,k=h.length,n=f[a]._aSortData,o=f[b]._aSortData;for(j=0;jg?1:0})}a.bSorted=!0}function Jb(a){for(var b,c,d=a.aoColumns,e=X(a),a=a.oLanguage.oAria,f=0,g=d.length;f/g,"");var i=c.nTh;i.removeAttribute("aria-sort");c.bSortable&&(0e?e+1:3));e=0;for(f=d.length;ee?e+1:3))}a.aLastSort=d}function Ib(a,b){var c=a.aoColumns[b],d=n.ext.order[c.sSortDataType],e;d&&(e=d.call(a.oInstance,a,b,ba(a,b)));for(var f,g=n.ext.type.order[c.sType+"-pre"],j=0,i=a.aoData.length;j=f.length?[0,c[1]]:c)}));b.search!==k&&h.extend(a.oPreviousSearch,Cb(b.search));if(b.columns){d=0;for(e=b.columns.length;d=c&&(b=c-d);b-=b%d;if(-1===d||0>b)b=0;a._iDisplayStart=b}function Na(a,b){var c=a.renderer,d=n.ext.renderer[b];return h.isPlainObject(c)&&c[b]?d[c[b]]||d._:"string"=== +typeof c?d[c]||d._:d._}function y(a){return a.oFeatures.bServerSide?"ssp":a.ajax||a.sAjaxSource?"ajax":"dom"}function ia(a,b){var c=[],c=Lb.numbers_length,d=Math.floor(c/2);b<=c?c=Y(0,b):a<=d?(c=Y(0,c-2),c.push("ellipsis"),c.push(b-1)):(a>=b-1-d?c=Y(b-(c-2),b):(c=Y(a-d+2,a+d-1),c.push("ellipsis"),c.push(b-1)),c.splice(0,0,"ellipsis"),c.splice(0,0,0));c.DT_el="span";return c}function Da(a){h.each({num:function(b){return za(b,a)},"num-fmt":function(b){return za(b,a,Ya)},"html-num":function(b){return za(b, +a,Aa)},"html-num-fmt":function(b){return za(b,a,Aa,Ya)}},function(b,c){x.type.order[b+a+"-pre"]=c;b.match(/^html\-/)&&(x.type.search[b+a]=x.type.search.html)})}function Mb(a){return function(){var b=[ya(this[n.ext.iApiIndex])].concat(Array.prototype.slice.call(arguments));return n.ext.internal[a].apply(this,b)}}var n=function(a){this.$=function(a,b){return this.api(!0).$(a,b)};this._=function(a,b){return this.api(!0).rows(a,b).data()};this.api=function(a){return a?new s(ya(this[x.iApiIndex])):new s(this)}; +this.fnAddData=function(a,b){var c=this.api(!0),d=h.isArray(a)&&(h.isArray(a[0])||h.isPlainObject(a[0]))?c.rows.add(a):c.row.add(a);(b===k||b)&&c.draw();return d.flatten().toArray()};this.fnAdjustColumnSizing=function(a){var b=this.api(!0).columns.adjust(),c=b.settings()[0],d=c.oScroll;a===k||a?b.draw(!1):(""!==d.sX||""!==d.sY)&&la(c)};this.fnClearTable=function(a){var b=this.api(!0).clear();(a===k||a)&&b.draw()};this.fnClose=function(a){this.api(!0).row(a).child.hide()};this.fnDeleteRow=function(a, +b,c){var d=this.api(!0),a=d.rows(a),e=a.settings()[0],h=e.aoData[a[0][0]];a.remove();b&&b.call(this,e,h);(c===k||c)&&d.draw();return h};this.fnDestroy=function(a){this.api(!0).destroy(a)};this.fnDraw=function(a){this.api(!0).draw(a)};this.fnFilter=function(a,b,c,d,e,h){e=this.api(!0);null===b||b===k?e.search(a,c,d,h):e.column(b).search(a,c,d,h);e.draw()};this.fnGetData=function(a,b){var c=this.api(!0);if(a!==k){var d=a.nodeName?a.nodeName.toLowerCase():"";return b!==k||"td"==d||"th"==d?c.cell(a,b).data(): +c.row(a).data()||null}return c.data().toArray()};this.fnGetNodes=function(a){var b=this.api(!0);return a!==k?b.row(a).node():b.rows().nodes().flatten().toArray()};this.fnGetPosition=function(a){var b=this.api(!0),c=a.nodeName.toUpperCase();return"TR"==c?b.row(a).index():"TD"==c||"TH"==c?(a=b.cell(a).index(),[a.row,a.columnVisible,a.column]):null};this.fnIsOpen=function(a){return this.api(!0).row(a).child.isShown()};this.fnOpen=function(a,b,c){return this.api(!0).row(a).child(b,c).show().child()[0]}; +this.fnPageChange=function(a,b){var c=this.api(!0).page(a);(b===k||b)&&c.draw(!1)};this.fnSetColumnVis=function(a,b,c){a=this.api(!0).column(a).visible(b);(c===k||c)&&a.columns.adjust().draw()};this.fnSettings=function(){return ya(this[x.iApiIndex])};this.fnSort=function(a){this.api(!0).order(a).draw()};this.fnSortListener=function(a,b,c){this.api(!0).order.listener(a,b,c)};this.fnUpdate=function(a,b,c,d,e){var h=this.api(!0);c===k||null===c?h.row(b).data(a):h.cell(b,c).data(a);(e===k||e)&&h.columns.adjust(); +(d===k||d)&&h.draw();return 0};this.fnVersionCheck=x.fnVersionCheck;var b=this,c=a===k,d=this.length;c&&(a={});this.oApi=this.internal=x.internal;for(var e in n.ext.internal)e&&(this[e]=Mb(e));this.each(function(){var e={},g=1").appendTo(q)); +p.nTHead=b[0];b=q.children("tbody");b.length===0&&(b=h("").appendTo(q));p.nTBody=b[0];b=q.children("tfoot");if(b.length===0&&a.length>0&&(p.oScroll.sX!==""||p.oScroll.sY!==""))b=h("").appendTo(q);if(b.length===0||b.children().length===0)q.addClass(u.sNoFooter);else if(b.length>0){p.nTFoot=b[0];ea(p.aoFooter,p.nTFoot)}if(g.aaData)for(j=0;j/g,Zb=/^\d{2,4}[\.\/\-]\d{1,2}[\.\/\-]\d{1,2}([T ]{1}\d{1,2}[:\.]\d{2}([\.:]\d{2})?)?$/,$b=RegExp("(\\/|\\.|\\*|\\+|\\?|\\||\\(|\\)|\\[|\\]|\\{|\\}|\\\\|\\$|\\^|\\-)","g"),Ya=/[',$£€¥%\u2009\u202F\u20BD\u20a9\u20BArfkɃΞ]/gi,M=function(a){return!a||!0===a||"-"===a?!0:!1},Ob=function(a){var b=parseInt(a,10);return!isNaN(b)&& +isFinite(a)?b:null},Pb=function(a,b){Za[b]||(Za[b]=RegExp(Qa(b),"g"));return"string"===typeof a&&"."!==b?a.replace(/\./g,"").replace(Za[b],"."):a},$a=function(a,b,c){var d="string"===typeof a;if(M(a))return!0;b&&d&&(a=Pb(a,b));c&&d&&(a=a.replace(Ya,""));return!isNaN(parseFloat(a))&&isFinite(a)},Qb=function(a,b,c){return M(a)?!0:!(M(a)||"string"===typeof a)?null:$a(a.replace(Aa,""),b,c)?!0:null},D=function(a,b,c){var d=[],e=0,f=a.length;if(c!==k)for(;ea.length)){b=a.slice().sort();for(var c=b[0],d=1,e=b.length;d")[0],Wb=va.textContent!==k,Yb= +/<.*?>/g,Oa=n.util.throttle,Sb=[],w=Array.prototype,ac=function(a){var b,c,d=n.settings,e=h.map(d,function(a){return a.nTable});if(a){if(a.nTable&&a.oApi)return[a];if(a.nodeName&&"table"===a.nodeName.toLowerCase())return b=h.inArray(a,e),-1!==b?[d[b]]:null;if(a&&"function"===typeof a.settings)return a.settings().toArray();"string"===typeof a?c=h(a):a instanceof h&&(c=a)}else return[];if(c)return c.map(function(){b=h.inArray(this,e);return-1!==b?d[b]:null}).toArray()};s=function(a,b){if(!(this instanceof +s))return new s(a,b);var c=[],d=function(a){(a=ac(a))&&(c=c.concat(a))};if(h.isArray(a))for(var e=0,f=a.length;ea?new s(b[a],this[a]):null},filter:function(a){var b=[];if(w.filter)b=w.filter.call(this,a,this);else for(var c=0,d=this.length;c").addClass(b),h("td",c).addClass(b).html(a)[0].colSpan=V(d),e.push(c[0]))};f(a,b);c._details&&c._details.detach();c._details=h(e); +c._detailsShow&&c._details.insertAfter(c.nTr)}return this});o(["row().child.show()","row().child().show()"],function(){Ub(this,!0);return this});o(["row().child.hide()","row().child().hide()"],function(){Ub(this,!1);return this});o(["row().child.remove()","row().child().remove()"],function(){db(this);return this});o("row().child.isShown()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]]._detailsShow||!1:!1});var bc=/^([^:]+):(name|visIdx|visible)$/,Vb=function(a,b, +c,d,e){for(var c=[],d=0,f=e.length;d=0?b:g.length+b];if(typeof a==="function"){var e=Ba(c,f);return h.map(g,function(b,f){return a(f,Vb(c,f,0,0,e),i[f])?f:null})}var k=typeof a==="string"?a.match(bc): +"";if(k)switch(k[2]){case "visIdx":case "visible":b=parseInt(k[1],10);if(b<0){var n=h.map(g,function(a,b){return a.bVisible?b:null});return[n[n.length+b]]}return[aa(c,b)];case "name":return h.map(j,function(a,b){return a===k[1]?b:null});default:return[]}if(a.nodeName&&a._DT_CellIndex)return[a._DT_CellIndex.column];b=h(i).filter(a).map(function(){return h.inArray(this,i)}).toArray();if(b.length||!a.nodeName)return b;b=h(a).closest("*[data-dt-column]");return b.length?[b.data("dt-column")]:[]},c,f)}, +1);c.selector.cols=a;c.selector.opts=b;return c});u("columns().header()","column().header()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTh},1)});u("columns().footer()","column().footer()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTf},1)});u("columns().data()","column().data()",function(){return this.iterator("column-rows",Vb,1)});u("columns().dataSrc()","column().dataSrc()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].mData}, +1)});u("columns().cache()","column().cache()",function(a){return this.iterator("column-rows",function(b,c,d,e,f){return ja(b.aoData,f,"search"===a?"_aFilterData":"_aSortData",c)},1)});u("columns().nodes()","column().nodes()",function(){return this.iterator("column-rows",function(a,b,c,d,e){return ja(a.aoData,e,"anCells",b)},1)});u("columns().visible()","column().visible()",function(a,b){var c=this.iterator("column",function(b,c){if(a===k)return b.aoColumns[c].bVisible;var f=b.aoColumns,g=f[c],j=b.aoData, +i,m,l;if(a!==k&&g.bVisible!==a){if(a){var n=h.inArray(!0,D(f,"bVisible"),c+1);i=0;for(m=j.length;id;return!0};n.isDataTable= +n.fnIsDataTable=function(a){var b=h(a).get(0),c=!1;if(a instanceof n.Api)return!0;h.each(n.settings,function(a,e){var f=e.nScrollHead?h("table",e.nScrollHead)[0]:null,g=e.nScrollFoot?h("table",e.nScrollFoot)[0]:null;if(e.nTable===b||f===b||g===b)c=!0});return c};n.tables=n.fnTables=function(a){var b=!1;h.isPlainObject(a)&&(b=a.api,a=a.visible);var c=h.map(n.settings,function(b){if(!a||a&&h(b.nTable).is(":visible"))return b.nTable});return b?new s(c):c};n.camelToHungarian=J;o("$()",function(a,b){var c= +this.rows(b).nodes(),c=h(c);return h([].concat(c.filter(a).toArray(),c.find(a).toArray()))});h.each(["on","one","off"],function(a,b){o(b+"()",function(){var a=Array.prototype.slice.call(arguments);a[0]=h.map(a[0].split(/\s/),function(a){return!a.match(/\.dt\b/)?a+".dt":a}).join(" ");var d=h(this.tables().nodes());d[b].apply(d,a);return this})});o("clear()",function(){return this.iterator("table",function(a){oa(a)})});o("settings()",function(){return new s(this.context,this.context)});o("init()",function(){var a= +this.context;return a.length?a[0].oInit:null});o("data()",function(){return this.iterator("table",function(a){return D(a.aoData,"_aData")}).flatten()});o("destroy()",function(a){a=a||!1;return this.iterator("table",function(b){var c=b.nTableWrapper.parentNode,d=b.oClasses,e=b.nTable,f=b.nTBody,g=b.nTHead,j=b.nTFoot,i=h(e),f=h(f),k=h(b.nTableWrapper),l=h.map(b.aoData,function(a){return a.nTr}),o;b.bDestroying=!0;r(b,"aoDestroyCallback","destroy",[b]);a||(new s(b)).columns().visible(!0);k.off(".DT").find(":not(tbody *)").off(".DT"); +h(E).off(".DT-"+b.sInstance);e!=g.parentNode&&(i.children("thead").detach(),i.append(g));j&&e!=j.parentNode&&(i.children("tfoot").detach(),i.append(j));b.aaSorting=[];b.aaSortingFixed=[];wa(b);h(l).removeClass(b.asStripeClasses.join(" "));h("th, td",g).removeClass(d.sSortable+" "+d.sSortableAsc+" "+d.sSortableDesc+" "+d.sSortableNone);f.children().detach();f.append(l);g=a?"remove":"detach";i[g]();k[g]();!a&&c&&(c.insertBefore(e,b.nTableReinsertBefore),i.css("width",b.sDestroyWidth).removeClass(d.sTable), +(o=b.asDestroyStripes.length)&&f.children().each(function(a){h(this).addClass(b.asDestroyStripes[a%o])}));c=h.inArray(b,n.settings);-1!==c&&n.settings.splice(c,1)})});h.each(["column","row","cell"],function(a,b){o(b+"s().every()",function(a){var d=this.selector.opts,e=this;return this.iterator(b,function(f,g,h,i,m){a.call(e[b](g,"cell"===b?h:d,"cell"===b?d:k),g,h,i,m)})})});o("i18n()",function(a,b,c){var d=this.context[0],a=S(a)(d.oLanguage);a===k&&(a=b);c!==k&&h.isPlainObject(a)&&(a=a[c]!==k?a[c]: +a._);return a.replace("%d",c)});n.version="1.10.19";n.settings=[];n.models={};n.models.oSearch={bCaseInsensitive:!0,sSearch:"",bRegex:!1,bSmart:!0};n.models.oRow={nTr:null,anCells:null,_aData:[],_aSortData:null,_aFilterData:null,_sFilterRow:null,_sRowStripe:"",src:null,idx:-1};n.models.oColumn={idx:null,aDataSort:null,asSorting:null,bSearchable:null,bSortable:null,bVisible:null,_sManualType:null,_bAttrSrc:!1,fnCreatedCell:null,fnGetData:null,fnSetData:null,mData:null,mRender:null,nTh:null,nTf:null, +sClass:null,sContentPadding:null,sDefaultContent:null,sName:null,sSortDataType:"std",sSortingClass:null,sSortingClassJUI:null,sTitle:null,sType:null,sWidth:null,sWidthOrig:null};n.defaults={aaData:null,aaSorting:[[0,"asc"]],aaSortingFixed:[],ajax:null,aLengthMenu:[10,25,50,100],aoColumns:null,aoColumnDefs:null,aoSearchCols:[],asStripeClasses:null,bAutoWidth:!0,bDeferRender:!1,bDestroy:!1,bFilter:!0,bInfo:!0,bLengthChange:!0,bPaginate:!0,bProcessing:!1,bRetrieve:!1,bScrollCollapse:!1,bServerSide:!1, +bSort:!0,bSortMulti:!0,bSortCellsTop:!1,bSortClasses:!0,bStateSave:!1,fnCreatedRow:null,fnDrawCallback:null,fnFooterCallback:null,fnFormatNumber:function(a){return a.toString().replace(/\B(?=(\d{3})+(?!\d))/g,this.oLanguage.sThousands)},fnHeaderCallback:null,fnInfoCallback:null,fnInitComplete:null,fnPreDrawCallback:null,fnRowCallback:null,fnServerData:null,fnServerParams:null,fnStateLoadCallback:function(a){try{return JSON.parse((-1===a.iStateDuration?sessionStorage:localStorage).getItem("DataTables_"+ +a.sInstance+"_"+location.pathname))}catch(b){}},fnStateLoadParams:null,fnStateLoaded:null,fnStateSaveCallback:function(a,b){try{(-1===a.iStateDuration?sessionStorage:localStorage).setItem("DataTables_"+a.sInstance+"_"+location.pathname,JSON.stringify(b))}catch(c){}},fnStateSaveParams:null,iStateDuration:7200,iDeferLoading:null,iDisplayLength:10,iDisplayStart:0,iTabIndex:0,oClasses:{},oLanguage:{oAria:{sSortAscending:": activate to sort column ascending",sSortDescending:": activate to sort column descending"}, +oPaginate:{sFirst:"First",sLast:"Last",sNext:"Next",sPrevious:"Previous"},sEmptyTable:"No data available in table",sInfo:"Showing _START_ to _END_ of _TOTAL_ entries",sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({}, +n.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",renderer:null,rowId:"DT_RowId"};Z(n.defaults);n.defaults.column={aDataSort:null,iDataSort:-1,asSorting:["asc","desc"],bSearchable:!0,bSortable:!0,bVisible:!0,fnCreatedCell:null,mData:null,mRender:null,sCellType:"td",sClass:"",sContentPadding:"",sDefaultContent:null,sName:"",sSortDataType:"std",sTitle:null,sType:null,sWidth:null}; +Z(n.defaults.column);n.models.oSettings={oFeatures:{bAutoWidth:null,bDeferRender:null,bFilter:null,bInfo:null,bLengthChange:null,bPaginate:null,bProcessing:null,bServerSide:null,bSort:null,bSortMulti:null,bSortClasses:null,bStateSave:null},oScroll:{bCollapse:null,iBarWidth:0,sX:null,sXInner:null,sY:null},oLanguage:{fnInfoCallback:null},oBrowser:{bScrollOversize:!1,bScrollbarLeft:!1,bBounding:!1,barWidth:0},ajax:null,aanFeatures:[],aoData:[],aiDisplay:[],aiDisplayMaster:[],aIds:{},aoColumns:[],aoHeader:[], +aoFooter:[],oPreviousSearch:{},aoPreSearchCols:[],aaSorting:null,aaSortingFixed:[],asStripeClasses:null,asDestroyStripes:[],sDestroyWidth:0,aoRowCallback:[],aoHeaderCallback:[],aoFooterCallback:[],aoDrawCallback:[],aoRowCreatedCallback:[],aoPreDrawCallback:[],aoInitComplete:[],aoStateSaveParams:[],aoStateLoadParams:[],aoStateLoaded:[],sTableId:"",nTable:null,nTHead:null,nTFoot:null,nTBody:null,nTableWrapper:null,bDeferLoading:!1,bInitialised:!1,aoOpenRows:[],sDom:null,searchDelay:null,sPaginationType:"two_button", +iStateDuration:0,aoStateSave:[],aoStateLoad:[],oSavedState:null,oLoadedState:null,sAjaxSource:null,sAjaxDataProp:null,bAjaxDataGet:!0,jqXHR:null,json:k,oAjaxData:k,fnServerData:null,aoServerParams:[],sServerMethod:null,fnFormatNumber:null,aLengthMenu:null,iDraw:0,bDrawing:!1,iDrawError:-1,_iDisplayLength:10,_iDisplayStart:0,_iRecordsTotal:0,_iRecordsDisplay:0,oClasses:{},bFiltered:!1,bSorted:!1,bSortCellsTop:null,oInit:null,aoDestroyCallback:[],fnRecordsTotal:function(){return"ssp"==y(this)?1*this._iRecordsTotal: +this.aiDisplayMaster.length},fnRecordsDisplay:function(){return"ssp"==y(this)?1*this._iRecordsDisplay:this.aiDisplay.length},fnDisplayEnd:function(){var a=this._iDisplayLength,b=this._iDisplayStart,c=b+a,d=this.aiDisplay.length,e=this.oFeatures,f=e.bPaginate;return e.bServerSide?!1===f||-1===a?b+d:Math.min(b+a,this._iRecordsDisplay):!f||c>d||-1===a?d:c},oInstance:null,sInstance:null,iTabIndex:0,nScrollHead:null,nScrollFoot:null,aLastSort:[],oPlugins:{},rowIdFn:null,rowId:null};n.ext=x={buttons:{}, +classes:{},builder:"-source-",errMode:"alert",feature:[],search:[],selector:{cell:[],column:[],row:[]},internal:{},legacy:{ajax:null},pager:{},renderer:{pageButton:{},header:{}},order:{},type:{detect:[],search:{},order:{}},_unique:0,fnVersionCheck:n.fnVersionCheck,iApiIndex:0,oJUIClasses:{},sVersion:n.version};h.extend(x,{afnFiltering:x.search,aTypes:x.type.detect,ofnSearch:x.type.search,oSort:x.type.order,afnSortData:x.order,aoFeatures:x.feature,oApi:x.internal,oStdClasses:x.classes,oPagination:x.pager}); +h.extend(n.ext.classes,{sTable:"dataTable",sNoFooter:"no-footer",sPageButton:"paginate_button",sPageButtonActive:"current",sPageButtonDisabled:"disabled",sStripeOdd:"odd",sStripeEven:"even",sRowEmpty:"dataTables_empty",sWrapper:"dataTables_wrapper",sFilter:"dataTables_filter",sInfo:"dataTables_info",sPaging:"dataTables_paginate paging_",sLength:"dataTables_length",sProcessing:"dataTables_processing",sSortAsc:"sorting_asc",sSortDesc:"sorting_desc",sSortable:"sorting",sSortableAsc:"sorting_asc_disabled", +sSortableDesc:"sorting_desc_disabled",sSortableNone:"sorting_disabled",sSortColumn:"sorting_",sFilterInput:"",sLengthSelect:"",sScrollWrapper:"dataTables_scroll",sScrollHead:"dataTables_scrollHead",sScrollHeadInner:"dataTables_scrollHeadInner",sScrollBody:"dataTables_scrollBody",sScrollFoot:"dataTables_scrollFoot",sScrollFootInner:"dataTables_scrollFootInner",sHeaderTH:"",sFooterTH:"",sSortJUIAsc:"",sSortJUIDesc:"",sSortJUI:"",sSortJUIAscAllowed:"",sSortJUIDescAllowed:"",sSortJUIWrapper:"",sSortIcon:"", +sJUIHeader:"",sJUIFooter:""});var Lb=n.ext.pager;h.extend(Lb,{simple:function(){return["previous","next"]},full:function(){return["first","previous","next","last"]},numbers:function(a,b){return[ia(a,b)]},simple_numbers:function(a,b){return["previous",ia(a,b),"next"]},full_numbers:function(a,b){return["first","previous",ia(a,b),"next","last"]},first_last_numbers:function(a,b){return["first",ia(a,b),"last"]},_numbers:ia,numbers_length:7});h.extend(!0,n.ext.renderer,{pageButton:{_:function(a,b,c,d,e, +f){var g=a.oClasses,j=a.oLanguage.oPaginate,i=a.oLanguage.oAria.paginate||{},m,l,n=0,o=function(b,d){var k,s,u,r,v=function(b){Ta(a,b.data.action,true)};k=0;for(s=d.length;k").appendTo(b);o(u,r)}else{m=null;l="";switch(r){case "ellipsis":b.append('…');break;case "first":m=j.sFirst;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "previous":m=j.sPrevious;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "next":m= +j.sNext;l=r+(e",{"class":g.sPageButton+" "+l,"aria-controls":a.sTableId,"aria-label":i[r],"data-dt-idx":n,tabindex:a.iTabIndex,id:c===0&&typeof r==="string"?a.sTableId+"_"+r:null}).html(m).appendTo(b);Wa(u,{action:r},v);n++}}}},s;try{s=h(b).find(H.activeElement).data("dt-idx")}catch(u){}o(h(b).empty(),d);s!==k&&h(b).find("[data-dt-idx="+ +s+"]").focus()}}});h.extend(n.ext.type.detect,[function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c)?"num"+c:null},function(a){if(a&&!(a instanceof Date)&&!Zb.test(a))return null;var b=Date.parse(a);return null!==b&&!isNaN(b)||M(a)?"date":null},function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c,!0)?"num-fmt"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c)?"html-num"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c,!0)?"html-num-fmt"+c:null},function(a){return M(a)|| +"string"===typeof a&&-1!==a.indexOf("<")?"html":null}]);h.extend(n.ext.type.search,{html:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," ").replace(Aa,""):""},string:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," "):a}});var za=function(a,b,c,d){if(0!==a&&(!a||"-"===a))return-Infinity;b&&(a=Pb(a,b));a.replace&&(c&&(a=a.replace(c,"")),d&&(a=a.replace(d,"")));return 1*a};h.extend(x.type.order,{"date-pre":function(a){a=Date.parse(a);return isNaN(a)?-Infinity:a},"html-pre":function(a){return M(a)? +"":a.replace?a.replace(/<.*?>/g,"").toLowerCase():a+""},"string-pre":function(a){return M(a)?"":"string"===typeof a?a.toLowerCase():!a.toString?"":a.toString()},"string-asc":function(a,b){return ab?1:0},"string-desc":function(a,b){return ab?-1:0}});Da("");h.extend(!0,n.ext.renderer,{header:{_:function(a,b,c,d){h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(c.sSortingClass+" "+d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc: +c.sSortingClass)}})},jqueryui:function(a,b,c,d){h("
    ").addClass(d.sSortJUIWrapper).append(b.contents()).append(h("").addClass(d.sSortIcon+" "+c.sSortingClassJUI)).appendTo(b);h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass);b.find("span."+d.sSortIcon).removeClass(d.sSortJUIAsc+" "+d.sSortJUIDesc+" "+d.sSortJUI+" "+d.sSortJUIAscAllowed+" "+d.sSortJUIDescAllowed).addClass(h[e]== +"asc"?d.sSortJUIAsc:h[e]=="desc"?d.sSortJUIDesc:c.sSortingClassJUI)}})}}});var eb=function(a){return"string"===typeof a?a.replace(//g,">").replace(/"/g,"""):a};n.render={number:function(a,b,c,d,e){return{display:function(f){if("number"!==typeof f&&"string"!==typeof f)return f;var g=0>f?"-":"",h=parseFloat(f);if(isNaN(h))return eb(f);h=h.toFixed(c);f=Math.abs(h);h=parseInt(f,10);f=c?b+(f-h).toFixed(c).substring(2):"";return g+(d||"")+h.toString().replace(/\B(?=(\d{3})+(?!\d))/g, +a)+f+(e||"")}}},text:function(){return{display:eb,filter:eb}}};h.extend(n.ext.internal,{_fnExternApiFunc:Mb,_fnBuildAjax:sa,_fnAjaxUpdate:mb,_fnAjaxParameters:vb,_fnAjaxUpdateDraw:wb,_fnAjaxDataSrc:ta,_fnAddColumn:Ea,_fnColumnOptions:ka,_fnAdjustColumnSizing:$,_fnVisibleToColumnIndex:aa,_fnColumnIndexToVisible:ba,_fnVisbleColumns:V,_fnGetColumns:ma,_fnColumnTypes:Ga,_fnApplyColumnDefs:jb,_fnHungarianMap:Z,_fnCamelToHungarian:J,_fnLanguageCompat:Ca,_fnBrowserDetect:hb,_fnAddData:O,_fnAddTr:na,_fnNodeToDataIndex:function(a, +b){return b._DT_RowIndex!==k?b._DT_RowIndex:null},_fnNodeToColumnIndex:function(a,b,c){return h.inArray(c,a.aoData[b].anCells)},_fnGetCellData:B,_fnSetCellData:kb,_fnSplitObjNotation:Ja,_fnGetObjectDataFn:S,_fnSetObjectDataFn:N,_fnGetDataMaster:Ka,_fnClearTable:oa,_fnDeleteIndex:pa,_fnInvalidate:da,_fnGetRowElements:Ia,_fnCreateTr:Ha,_fnBuildHead:lb,_fnDrawHead:fa,_fnDraw:P,_fnReDraw:T,_fnAddOptionsHtml:ob,_fnDetectHeader:ea,_fnGetUniqueThs:ra,_fnFeatureHtmlFilter:qb,_fnFilterComplete:ga,_fnFilterCustom:zb, +_fnFilterColumn:yb,_fnFilter:xb,_fnFilterCreateSearch:Pa,_fnEscapeRegex:Qa,_fnFilterData:Ab,_fnFeatureHtmlInfo:tb,_fnUpdateInfo:Db,_fnInfoMacros:Eb,_fnInitialise:ha,_fnInitComplete:ua,_fnLengthChange:Ra,_fnFeatureHtmlLength:pb,_fnFeatureHtmlPaginate:ub,_fnPageChange:Ta,_fnFeatureHtmlProcessing:rb,_fnProcessingDisplay:C,_fnFeatureHtmlTable:sb,_fnScrollDraw:la,_fnApplyToChildren:I,_fnCalculateColumnWidths:Fa,_fnThrottle:Oa,_fnConvertToWidth:Fb,_fnGetWidestNode:Gb,_fnGetMaxLenString:Hb,_fnStringToCss:v, +_fnSortFlatten:X,_fnSort:nb,_fnSortAria:Jb,_fnSortListener:Va,_fnSortAttachListener:Ma,_fnSortingClasses:wa,_fnSortData:Ib,_fnSaveState:xa,_fnLoadState:Kb,_fnSettingsFromNode:ya,_fnLog:K,_fnMap:F,_fnBindAction:Wa,_fnCallbackReg:z,_fnCallbackFire:r,_fnLengthOverflow:Sa,_fnRenderer:Na,_fnDataSource:y,_fnRowAttributes:La,_fnExtend:Xa,_fnCalculateEnd:function(){}});h.fn.dataTable=n;n.$=h;h.fn.dataTableSettings=n.settings;h.fn.dataTableExt=n.ext;h.fn.DataTable=function(a){return h(this).dataTable(a).api()}; +h.each(n,function(a,b){h.fn.DataTable[a]=b});return h.fn.dataTable}); diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.min.js b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.min.js new file mode 100644 index 0000000..4d9b3a2 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/default/static/jquery.min.js @@ -0,0 +1,2 @@ +/*! jQuery v3.3.1 | (c) JS Foundation and other contributors | jquery.org/license */ +!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(e,t){"use strict";var n=[],r=e.document,i=Object.getPrototypeOf,o=n.slice,a=n.concat,s=n.push,u=n.indexOf,l={},c=l.toString,f=l.hasOwnProperty,p=f.toString,d=p.call(Object),h={},g=function e(t){return"function"==typeof t&&"number"!=typeof t.nodeType},y=function e(t){return null!=t&&t===t.window},v={type:!0,src:!0,noModule:!0};function m(e,t,n){var i,o=(t=t||r).createElement("script");if(o.text=e,n)for(i in v)n[i]&&(o[i]=n[i]);t.head.appendChild(o).parentNode.removeChild(o)}function x(e){return null==e?e+"":"object"==typeof e||"function"==typeof e?l[c.call(e)]||"object":typeof e}var b="3.3.1",w=function(e,t){return new w.fn.init(e,t)},T=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g;w.fn=w.prototype={jquery:"3.3.1",constructor:w,length:0,toArray:function(){return o.call(this)},get:function(e){return null==e?o.call(this):e<0?this[e+this.length]:this[e]},pushStack:function(e){var t=w.merge(this.constructor(),e);return t.prevObject=this,t},each:function(e){return w.each(this,e)},map:function(e){return this.pushStack(w.map(this,function(t,n){return e.call(t,n,t)}))},slice:function(){return this.pushStack(o.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(e){var t=this.length,n=+e+(e<0?t:0);return this.pushStack(n>=0&&n0&&t-1 in e)}var E=function(e){var t,n,r,i,o,a,s,u,l,c,f,p,d,h,g,y,v,m,x,b="sizzle"+1*new Date,w=e.document,T=0,C=0,E=ae(),k=ae(),S=ae(),D=function(e,t){return e===t&&(f=!0),0},N={}.hasOwnProperty,A=[],j=A.pop,q=A.push,L=A.push,H=A.slice,O=function(e,t){for(var n=0,r=e.length;n+~]|"+M+")"+M+"*"),z=new RegExp("="+M+"*([^\\]'\"]*?)"+M+"*\\]","g"),X=new RegExp(W),U=new RegExp("^"+R+"$"),V={ID:new RegExp("^#("+R+")"),CLASS:new RegExp("^\\.("+R+")"),TAG:new RegExp("^("+R+"|[*])"),ATTR:new RegExp("^"+I),PSEUDO:new RegExp("^"+W),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+M+"*(even|odd|(([+-]|)(\\d*)n|)"+M+"*(?:([+-]|)"+M+"*(\\d+)|))"+M+"*\\)|)","i"),bool:new RegExp("^(?:"+P+")$","i"),needsContext:new RegExp("^"+M+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+M+"*((?:-\\d)?\\d*)"+M+"*\\)|)(?=[^-]|$)","i")},G=/^(?:input|select|textarea|button)$/i,Y=/^h\d$/i,Q=/^[^{]+\{\s*\[native \w/,J=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,K=/[+~]/,Z=new RegExp("\\\\([\\da-f]{1,6}"+M+"?|("+M+")|.)","ig"),ee=function(e,t,n){var r="0x"+t-65536;return r!==r||n?t:r<0?String.fromCharCode(r+65536):String.fromCharCode(r>>10|55296,1023&r|56320)},te=/([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g,ne=function(e,t){return t?"\0"===e?"\ufffd":e.slice(0,-1)+"\\"+e.charCodeAt(e.length-1).toString(16)+" ":"\\"+e},re=function(){p()},ie=me(function(e){return!0===e.disabled&&("form"in e||"label"in e)},{dir:"parentNode",next:"legend"});try{L.apply(A=H.call(w.childNodes),w.childNodes),A[w.childNodes.length].nodeType}catch(e){L={apply:A.length?function(e,t){q.apply(e,H.call(t))}:function(e,t){var n=e.length,r=0;while(e[n++]=t[r++]);e.length=n-1}}}function oe(e,t,r,i){var o,s,l,c,f,h,v,m=t&&t.ownerDocument,T=t?t.nodeType:9;if(r=r||[],"string"!=typeof e||!e||1!==T&&9!==T&&11!==T)return r;if(!i&&((t?t.ownerDocument||t:w)!==d&&p(t),t=t||d,g)){if(11!==T&&(f=J.exec(e)))if(o=f[1]){if(9===T){if(!(l=t.getElementById(o)))return r;if(l.id===o)return r.push(l),r}else if(m&&(l=m.getElementById(o))&&x(t,l)&&l.id===o)return r.push(l),r}else{if(f[2])return L.apply(r,t.getElementsByTagName(e)),r;if((o=f[3])&&n.getElementsByClassName&&t.getElementsByClassName)return L.apply(r,t.getElementsByClassName(o)),r}if(n.qsa&&!S[e+" "]&&(!y||!y.test(e))){if(1!==T)m=t,v=e;else if("object"!==t.nodeName.toLowerCase()){(c=t.getAttribute("id"))?c=c.replace(te,ne):t.setAttribute("id",c=b),s=(h=a(e)).length;while(s--)h[s]="#"+c+" "+ve(h[s]);v=h.join(","),m=K.test(e)&&ge(t.parentNode)||t}if(v)try{return L.apply(r,m.querySelectorAll(v)),r}catch(e){}finally{c===b&&t.removeAttribute("id")}}}return u(e.replace(B,"$1"),t,r,i)}function ae(){var e=[];function t(n,i){return e.push(n+" ")>r.cacheLength&&delete t[e.shift()],t[n+" "]=i}return t}function se(e){return e[b]=!0,e}function ue(e){var t=d.createElement("fieldset");try{return!!e(t)}catch(e){return!1}finally{t.parentNode&&t.parentNode.removeChild(t),t=null}}function le(e,t){var n=e.split("|"),i=n.length;while(i--)r.attrHandle[n[i]]=t}function ce(e,t){var n=t&&e,r=n&&1===e.nodeType&&1===t.nodeType&&e.sourceIndex-t.sourceIndex;if(r)return r;if(n)while(n=n.nextSibling)if(n===t)return-1;return e?1:-1}function fe(e){return function(t){return"input"===t.nodeName.toLowerCase()&&t.type===e}}function pe(e){return function(t){var n=t.nodeName.toLowerCase();return("input"===n||"button"===n)&&t.type===e}}function de(e){return function(t){return"form"in t?t.parentNode&&!1===t.disabled?"label"in t?"label"in t.parentNode?t.parentNode.disabled===e:t.disabled===e:t.isDisabled===e||t.isDisabled!==!e&&ie(t)===e:t.disabled===e:"label"in t&&t.disabled===e}}function he(e){return se(function(t){return t=+t,se(function(n,r){var i,o=e([],n.length,t),a=o.length;while(a--)n[i=o[a]]&&(n[i]=!(r[i]=n[i]))})})}function ge(e){return e&&"undefined"!=typeof e.getElementsByTagName&&e}n=oe.support={},o=oe.isXML=function(e){var t=e&&(e.ownerDocument||e).documentElement;return!!t&&"HTML"!==t.nodeName},p=oe.setDocument=function(e){var t,i,a=e?e.ownerDocument||e:w;return a!==d&&9===a.nodeType&&a.documentElement?(d=a,h=d.documentElement,g=!o(d),w!==d&&(i=d.defaultView)&&i.top!==i&&(i.addEventListener?i.addEventListener("unload",re,!1):i.attachEvent&&i.attachEvent("onunload",re)),n.attributes=ue(function(e){return e.className="i",!e.getAttribute("className")}),n.getElementsByTagName=ue(function(e){return e.appendChild(d.createComment("")),!e.getElementsByTagName("*").length}),n.getElementsByClassName=Q.test(d.getElementsByClassName),n.getById=ue(function(e){return h.appendChild(e).id=b,!d.getElementsByName||!d.getElementsByName(b).length}),n.getById?(r.filter.ID=function(e){var t=e.replace(Z,ee);return function(e){return e.getAttribute("id")===t}},r.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&g){var n=t.getElementById(e);return n?[n]:[]}}):(r.filter.ID=function(e){var t=e.replace(Z,ee);return function(e){var n="undefined"!=typeof e.getAttributeNode&&e.getAttributeNode("id");return n&&n.value===t}},r.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&g){var n,r,i,o=t.getElementById(e);if(o){if((n=o.getAttributeNode("id"))&&n.value===e)return[o];i=t.getElementsByName(e),r=0;while(o=i[r++])if((n=o.getAttributeNode("id"))&&n.value===e)return[o]}return[]}}),r.find.TAG=n.getElementsByTagName?function(e,t){return"undefined"!=typeof t.getElementsByTagName?t.getElementsByTagName(e):n.qsa?t.querySelectorAll(e):void 0}:function(e,t){var n,r=[],i=0,o=t.getElementsByTagName(e);if("*"===e){while(n=o[i++])1===n.nodeType&&r.push(n);return r}return o},r.find.CLASS=n.getElementsByClassName&&function(e,t){if("undefined"!=typeof t.getElementsByClassName&&g)return t.getElementsByClassName(e)},v=[],y=[],(n.qsa=Q.test(d.querySelectorAll))&&(ue(function(e){h.appendChild(e).innerHTML="",e.querySelectorAll("[msallowcapture^='']").length&&y.push("[*^$]="+M+"*(?:''|\"\")"),e.querySelectorAll("[selected]").length||y.push("\\["+M+"*(?:value|"+P+")"),e.querySelectorAll("[id~="+b+"-]").length||y.push("~="),e.querySelectorAll(":checked").length||y.push(":checked"),e.querySelectorAll("a#"+b+"+*").length||y.push(".#.+[+~]")}),ue(function(e){e.innerHTML="";var t=d.createElement("input");t.setAttribute("type","hidden"),e.appendChild(t).setAttribute("name","D"),e.querySelectorAll("[name=d]").length&&y.push("name"+M+"*[*^$|!~]?="),2!==e.querySelectorAll(":enabled").length&&y.push(":enabled",":disabled"),h.appendChild(e).disabled=!0,2!==e.querySelectorAll(":disabled").length&&y.push(":enabled",":disabled"),e.querySelectorAll("*,:x"),y.push(",.*:")})),(n.matchesSelector=Q.test(m=h.matches||h.webkitMatchesSelector||h.mozMatchesSelector||h.oMatchesSelector||h.msMatchesSelector))&&ue(function(e){n.disconnectedMatch=m.call(e,"*"),m.call(e,"[s!='']:x"),v.push("!=",W)}),y=y.length&&new RegExp(y.join("|")),v=v.length&&new RegExp(v.join("|")),t=Q.test(h.compareDocumentPosition),x=t||Q.test(h.contains)?function(e,t){var n=9===e.nodeType?e.documentElement:e,r=t&&t.parentNode;return e===r||!(!r||1!==r.nodeType||!(n.contains?n.contains(r):e.compareDocumentPosition&&16&e.compareDocumentPosition(r)))}:function(e,t){if(t)while(t=t.parentNode)if(t===e)return!0;return!1},D=t?function(e,t){if(e===t)return f=!0,0;var r=!e.compareDocumentPosition-!t.compareDocumentPosition;return r||(1&(r=(e.ownerDocument||e)===(t.ownerDocument||t)?e.compareDocumentPosition(t):1)||!n.sortDetached&&t.compareDocumentPosition(e)===r?e===d||e.ownerDocument===w&&x(w,e)?-1:t===d||t.ownerDocument===w&&x(w,t)?1:c?O(c,e)-O(c,t):0:4&r?-1:1)}:function(e,t){if(e===t)return f=!0,0;var n,r=0,i=e.parentNode,o=t.parentNode,a=[e],s=[t];if(!i||!o)return e===d?-1:t===d?1:i?-1:o?1:c?O(c,e)-O(c,t):0;if(i===o)return ce(e,t);n=e;while(n=n.parentNode)a.unshift(n);n=t;while(n=n.parentNode)s.unshift(n);while(a[r]===s[r])r++;return r?ce(a[r],s[r]):a[r]===w?-1:s[r]===w?1:0},d):d},oe.matches=function(e,t){return oe(e,null,null,t)},oe.matchesSelector=function(e,t){if((e.ownerDocument||e)!==d&&p(e),t=t.replace(z,"='$1']"),n.matchesSelector&&g&&!S[t+" "]&&(!v||!v.test(t))&&(!y||!y.test(t)))try{var r=m.call(e,t);if(r||n.disconnectedMatch||e.document&&11!==e.document.nodeType)return r}catch(e){}return oe(t,d,null,[e]).length>0},oe.contains=function(e,t){return(e.ownerDocument||e)!==d&&p(e),x(e,t)},oe.attr=function(e,t){(e.ownerDocument||e)!==d&&p(e);var i=r.attrHandle[t.toLowerCase()],o=i&&N.call(r.attrHandle,t.toLowerCase())?i(e,t,!g):void 0;return void 0!==o?o:n.attributes||!g?e.getAttribute(t):(o=e.getAttributeNode(t))&&o.specified?o.value:null},oe.escape=function(e){return(e+"").replace(te,ne)},oe.error=function(e){throw new Error("Syntax error, unrecognized expression: "+e)},oe.uniqueSort=function(e){var t,r=[],i=0,o=0;if(f=!n.detectDuplicates,c=!n.sortStable&&e.slice(0),e.sort(D),f){while(t=e[o++])t===e[o]&&(i=r.push(o));while(i--)e.splice(r[i],1)}return c=null,e},i=oe.getText=function(e){var t,n="",r=0,o=e.nodeType;if(o){if(1===o||9===o||11===o){if("string"==typeof e.textContent)return e.textContent;for(e=e.firstChild;e;e=e.nextSibling)n+=i(e)}else if(3===o||4===o)return e.nodeValue}else while(t=e[r++])n+=i(t);return n},(r=oe.selectors={cacheLength:50,createPseudo:se,match:V,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(e){return e[1]=e[1].replace(Z,ee),e[3]=(e[3]||e[4]||e[5]||"").replace(Z,ee),"~="===e[2]&&(e[3]=" "+e[3]+" "),e.slice(0,4)},CHILD:function(e){return e[1]=e[1].toLowerCase(),"nth"===e[1].slice(0,3)?(e[3]||oe.error(e[0]),e[4]=+(e[4]?e[5]+(e[6]||1):2*("even"===e[3]||"odd"===e[3])),e[5]=+(e[7]+e[8]||"odd"===e[3])):e[3]&&oe.error(e[0]),e},PSEUDO:function(e){var t,n=!e[6]&&e[2];return V.CHILD.test(e[0])?null:(e[3]?e[2]=e[4]||e[5]||"":n&&X.test(n)&&(t=a(n,!0))&&(t=n.indexOf(")",n.length-t)-n.length)&&(e[0]=e[0].slice(0,t),e[2]=n.slice(0,t)),e.slice(0,3))}},filter:{TAG:function(e){var t=e.replace(Z,ee).toLowerCase();return"*"===e?function(){return!0}:function(e){return e.nodeName&&e.nodeName.toLowerCase()===t}},CLASS:function(e){var t=E[e+" "];return t||(t=new RegExp("(^|"+M+")"+e+"("+M+"|$)"))&&E(e,function(e){return t.test("string"==typeof e.className&&e.className||"undefined"!=typeof e.getAttribute&&e.getAttribute("class")||"")})},ATTR:function(e,t,n){return function(r){var i=oe.attr(r,e);return null==i?"!="===t:!t||(i+="","="===t?i===n:"!="===t?i!==n:"^="===t?n&&0===i.indexOf(n):"*="===t?n&&i.indexOf(n)>-1:"$="===t?n&&i.slice(-n.length)===n:"~="===t?(" "+i.replace($," ")+" ").indexOf(n)>-1:"|="===t&&(i===n||i.slice(0,n.length+1)===n+"-"))}},CHILD:function(e,t,n,r,i){var o="nth"!==e.slice(0,3),a="last"!==e.slice(-4),s="of-type"===t;return 1===r&&0===i?function(e){return!!e.parentNode}:function(t,n,u){var l,c,f,p,d,h,g=o!==a?"nextSibling":"previousSibling",y=t.parentNode,v=s&&t.nodeName.toLowerCase(),m=!u&&!s,x=!1;if(y){if(o){while(g){p=t;while(p=p[g])if(s?p.nodeName.toLowerCase()===v:1===p.nodeType)return!1;h=g="only"===e&&!h&&"nextSibling"}return!0}if(h=[a?y.firstChild:y.lastChild],a&&m){x=(d=(l=(c=(f=(p=y)[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]||[])[0]===T&&l[1])&&l[2],p=d&&y.childNodes[d];while(p=++d&&p&&p[g]||(x=d=0)||h.pop())if(1===p.nodeType&&++x&&p===t){c[e]=[T,d,x];break}}else if(m&&(x=d=(l=(c=(f=(p=t)[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]||[])[0]===T&&l[1]),!1===x)while(p=++d&&p&&p[g]||(x=d=0)||h.pop())if((s?p.nodeName.toLowerCase()===v:1===p.nodeType)&&++x&&(m&&((c=(f=p[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]=[T,x]),p===t))break;return(x-=i)===r||x%r==0&&x/r>=0}}},PSEUDO:function(e,t){var n,i=r.pseudos[e]||r.setFilters[e.toLowerCase()]||oe.error("unsupported pseudo: "+e);return i[b]?i(t):i.length>1?(n=[e,e,"",t],r.setFilters.hasOwnProperty(e.toLowerCase())?se(function(e,n){var r,o=i(e,t),a=o.length;while(a--)e[r=O(e,o[a])]=!(n[r]=o[a])}):function(e){return i(e,0,n)}):i}},pseudos:{not:se(function(e){var t=[],n=[],r=s(e.replace(B,"$1"));return r[b]?se(function(e,t,n,i){var o,a=r(e,null,i,[]),s=e.length;while(s--)(o=a[s])&&(e[s]=!(t[s]=o))}):function(e,i,o){return t[0]=e,r(t,null,o,n),t[0]=null,!n.pop()}}),has:se(function(e){return function(t){return oe(e,t).length>0}}),contains:se(function(e){return e=e.replace(Z,ee),function(t){return(t.textContent||t.innerText||i(t)).indexOf(e)>-1}}),lang:se(function(e){return U.test(e||"")||oe.error("unsupported lang: "+e),e=e.replace(Z,ee).toLowerCase(),function(t){var n;do{if(n=g?t.lang:t.getAttribute("xml:lang")||t.getAttribute("lang"))return(n=n.toLowerCase())===e||0===n.indexOf(e+"-")}while((t=t.parentNode)&&1===t.nodeType);return!1}}),target:function(t){var n=e.location&&e.location.hash;return n&&n.slice(1)===t.id},root:function(e){return e===h},focus:function(e){return e===d.activeElement&&(!d.hasFocus||d.hasFocus())&&!!(e.type||e.href||~e.tabIndex)},enabled:de(!1),disabled:de(!0),checked:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&!!e.checked||"option"===t&&!!e.selected},selected:function(e){return e.parentNode&&e.parentNode.selectedIndex,!0===e.selected},empty:function(e){for(e=e.firstChild;e;e=e.nextSibling)if(e.nodeType<6)return!1;return!0},parent:function(e){return!r.pseudos.empty(e)},header:function(e){return Y.test(e.nodeName)},input:function(e){return G.test(e.nodeName)},button:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&"button"===e.type||"button"===t},text:function(e){var t;return"input"===e.nodeName.toLowerCase()&&"text"===e.type&&(null==(t=e.getAttribute("type"))||"text"===t.toLowerCase())},first:he(function(){return[0]}),last:he(function(e,t){return[t-1]}),eq:he(function(e,t,n){return[n<0?n+t:n]}),even:he(function(e,t){for(var n=0;n=0;)e.push(r);return e}),gt:he(function(e,t,n){for(var r=n<0?n+t:n;++r1?function(t,n,r){var i=e.length;while(i--)if(!e[i](t,n,r))return!1;return!0}:e[0]}function be(e,t,n){for(var r=0,i=t.length;r-1&&(o[l]=!(a[l]=f))}}else v=we(v===a?v.splice(h,v.length):v),i?i(null,a,v,u):L.apply(a,v)})}function Ce(e){for(var t,n,i,o=e.length,a=r.relative[e[0].type],s=a||r.relative[" "],u=a?1:0,c=me(function(e){return e===t},s,!0),f=me(function(e){return O(t,e)>-1},s,!0),p=[function(e,n,r){var i=!a&&(r||n!==l)||((t=n).nodeType?c(e,n,r):f(e,n,r));return t=null,i}];u1&&xe(p),u>1&&ve(e.slice(0,u-1).concat({value:" "===e[u-2].type?"*":""})).replace(B,"$1"),n,u0,i=e.length>0,o=function(o,a,s,u,c){var f,h,y,v=0,m="0",x=o&&[],b=[],w=l,C=o||i&&r.find.TAG("*",c),E=T+=null==w?1:Math.random()||.1,k=C.length;for(c&&(l=a===d||a||c);m!==k&&null!=(f=C[m]);m++){if(i&&f){h=0,a||f.ownerDocument===d||(p(f),s=!g);while(y=e[h++])if(y(f,a||d,s)){u.push(f);break}c&&(T=E)}n&&((f=!y&&f)&&v--,o&&x.push(f))}if(v+=m,n&&m!==v){h=0;while(y=t[h++])y(x,b,a,s);if(o){if(v>0)while(m--)x[m]||b[m]||(b[m]=j.call(u));b=we(b)}L.apply(u,b),c&&!o&&b.length>0&&v+t.length>1&&oe.uniqueSort(u)}return c&&(T=E,l=w),x};return n?se(o):o}return s=oe.compile=function(e,t){var n,r=[],i=[],o=S[e+" "];if(!o){t||(t=a(e)),n=t.length;while(n--)(o=Ce(t[n]))[b]?r.push(o):i.push(o);(o=S(e,Ee(i,r))).selector=e}return o},u=oe.select=function(e,t,n,i){var o,u,l,c,f,p="function"==typeof e&&e,d=!i&&a(e=p.selector||e);if(n=n||[],1===d.length){if((u=d[0]=d[0].slice(0)).length>2&&"ID"===(l=u[0]).type&&9===t.nodeType&&g&&r.relative[u[1].type]){if(!(t=(r.find.ID(l.matches[0].replace(Z,ee),t)||[])[0]))return n;p&&(t=t.parentNode),e=e.slice(u.shift().value.length)}o=V.needsContext.test(e)?0:u.length;while(o--){if(l=u[o],r.relative[c=l.type])break;if((f=r.find[c])&&(i=f(l.matches[0].replace(Z,ee),K.test(u[0].type)&&ge(t.parentNode)||t))){if(u.splice(o,1),!(e=i.length&&ve(u)))return L.apply(n,i),n;break}}}return(p||s(e,d))(i,t,!g,n,!t||K.test(e)&&ge(t.parentNode)||t),n},n.sortStable=b.split("").sort(D).join("")===b,n.detectDuplicates=!!f,p(),n.sortDetached=ue(function(e){return 1&e.compareDocumentPosition(d.createElement("fieldset"))}),ue(function(e){return e.innerHTML="","#"===e.firstChild.getAttribute("href")})||le("type|href|height|width",function(e,t,n){if(!n)return e.getAttribute(t,"type"===t.toLowerCase()?1:2)}),n.attributes&&ue(function(e){return e.innerHTML="",e.firstChild.setAttribute("value",""),""===e.firstChild.getAttribute("value")})||le("value",function(e,t,n){if(!n&&"input"===e.nodeName.toLowerCase())return e.defaultValue}),ue(function(e){return null==e.getAttribute("disabled")})||le(P,function(e,t,n){var r;if(!n)return!0===e[t]?t.toLowerCase():(r=e.getAttributeNode(t))&&r.specified?r.value:null}),oe}(e);w.find=E,w.expr=E.selectors,w.expr[":"]=w.expr.pseudos,w.uniqueSort=w.unique=E.uniqueSort,w.text=E.getText,w.isXMLDoc=E.isXML,w.contains=E.contains,w.escapeSelector=E.escape;var k=function(e,t,n){var r=[],i=void 0!==n;while((e=e[t])&&9!==e.nodeType)if(1===e.nodeType){if(i&&w(e).is(n))break;r.push(e)}return r},S=function(e,t){for(var n=[];e;e=e.nextSibling)1===e.nodeType&&e!==t&&n.push(e);return n},D=w.expr.match.needsContext;function N(e,t){return e.nodeName&&e.nodeName.toLowerCase()===t.toLowerCase()}var A=/^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i;function j(e,t,n){return g(t)?w.grep(e,function(e,r){return!!t.call(e,r,e)!==n}):t.nodeType?w.grep(e,function(e){return e===t!==n}):"string"!=typeof t?w.grep(e,function(e){return u.call(t,e)>-1!==n}):w.filter(t,e,n)}w.filter=function(e,t,n){var r=t[0];return n&&(e=":not("+e+")"),1===t.length&&1===r.nodeType?w.find.matchesSelector(r,e)?[r]:[]:w.find.matches(e,w.grep(t,function(e){return 1===e.nodeType}))},w.fn.extend({find:function(e){var t,n,r=this.length,i=this;if("string"!=typeof e)return this.pushStack(w(e).filter(function(){for(t=0;t1?w.uniqueSort(n):n},filter:function(e){return this.pushStack(j(this,e||[],!1))},not:function(e){return this.pushStack(j(this,e||[],!0))},is:function(e){return!!j(this,"string"==typeof e&&D.test(e)?w(e):e||[],!1).length}});var q,L=/^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/;(w.fn.init=function(e,t,n){var i,o;if(!e)return this;if(n=n||q,"string"==typeof e){if(!(i="<"===e[0]&&">"===e[e.length-1]&&e.length>=3?[null,e,null]:L.exec(e))||!i[1]&&t)return!t||t.jquery?(t||n).find(e):this.constructor(t).find(e);if(i[1]){if(t=t instanceof w?t[0]:t,w.merge(this,w.parseHTML(i[1],t&&t.nodeType?t.ownerDocument||t:r,!0)),A.test(i[1])&&w.isPlainObject(t))for(i in t)g(this[i])?this[i](t[i]):this.attr(i,t[i]);return this}return(o=r.getElementById(i[2]))&&(this[0]=o,this.length=1),this}return e.nodeType?(this[0]=e,this.length=1,this):g(e)?void 0!==n.ready?n.ready(e):e(w):w.makeArray(e,this)}).prototype=w.fn,q=w(r);var H=/^(?:parents|prev(?:Until|All))/,O={children:!0,contents:!0,next:!0,prev:!0};w.fn.extend({has:function(e){var t=w(e,this),n=t.length;return this.filter(function(){for(var e=0;e-1:1===n.nodeType&&w.find.matchesSelector(n,e))){o.push(n);break}return this.pushStack(o.length>1?w.uniqueSort(o):o)},index:function(e){return e?"string"==typeof e?u.call(w(e),this[0]):u.call(this,e.jquery?e[0]:e):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(e,t){return this.pushStack(w.uniqueSort(w.merge(this.get(),w(e,t))))},addBack:function(e){return this.add(null==e?this.prevObject:this.prevObject.filter(e))}});function P(e,t){while((e=e[t])&&1!==e.nodeType);return e}w.each({parent:function(e){var t=e.parentNode;return t&&11!==t.nodeType?t:null},parents:function(e){return k(e,"parentNode")},parentsUntil:function(e,t,n){return k(e,"parentNode",n)},next:function(e){return P(e,"nextSibling")},prev:function(e){return P(e,"previousSibling")},nextAll:function(e){return k(e,"nextSibling")},prevAll:function(e){return k(e,"previousSibling")},nextUntil:function(e,t,n){return k(e,"nextSibling",n)},prevUntil:function(e,t,n){return k(e,"previousSibling",n)},siblings:function(e){return S((e.parentNode||{}).firstChild,e)},children:function(e){return S(e.firstChild)},contents:function(e){return N(e,"iframe")?e.contentDocument:(N(e,"template")&&(e=e.content||e),w.merge([],e.childNodes))}},function(e,t){w.fn[e]=function(n,r){var i=w.map(this,t,n);return"Until"!==e.slice(-5)&&(r=n),r&&"string"==typeof r&&(i=w.filter(r,i)),this.length>1&&(O[e]||w.uniqueSort(i),H.test(e)&&i.reverse()),this.pushStack(i)}});var M=/[^\x20\t\r\n\f]+/g;function R(e){var t={};return w.each(e.match(M)||[],function(e,n){t[n]=!0}),t}w.Callbacks=function(e){e="string"==typeof e?R(e):w.extend({},e);var t,n,r,i,o=[],a=[],s=-1,u=function(){for(i=i||e.once,r=t=!0;a.length;s=-1){n=a.shift();while(++s-1)o.splice(n,1),n<=s&&s--}),this},has:function(e){return e?w.inArray(e,o)>-1:o.length>0},empty:function(){return o&&(o=[]),this},disable:function(){return i=a=[],o=n="",this},disabled:function(){return!o},lock:function(){return i=a=[],n||t||(o=n=""),this},locked:function(){return!!i},fireWith:function(e,n){return i||(n=[e,(n=n||[]).slice?n.slice():n],a.push(n),t||u()),this},fire:function(){return l.fireWith(this,arguments),this},fired:function(){return!!r}};return l};function I(e){return e}function W(e){throw e}function $(e,t,n,r){var i;try{e&&g(i=e.promise)?i.call(e).done(t).fail(n):e&&g(i=e.then)?i.call(e,t,n):t.apply(void 0,[e].slice(r))}catch(e){n.apply(void 0,[e])}}w.extend({Deferred:function(t){var n=[["notify","progress",w.Callbacks("memory"),w.Callbacks("memory"),2],["resolve","done",w.Callbacks("once memory"),w.Callbacks("once memory"),0,"resolved"],["reject","fail",w.Callbacks("once memory"),w.Callbacks("once memory"),1,"rejected"]],r="pending",i={state:function(){return r},always:function(){return o.done(arguments).fail(arguments),this},"catch":function(e){return i.then(null,e)},pipe:function(){var e=arguments;return w.Deferred(function(t){w.each(n,function(n,r){var i=g(e[r[4]])&&e[r[4]];o[r[1]](function(){var e=i&&i.apply(this,arguments);e&&g(e.promise)?e.promise().progress(t.notify).done(t.resolve).fail(t.reject):t[r[0]+"With"](this,i?[e]:arguments)})}),e=null}).promise()},then:function(t,r,i){var o=0;function a(t,n,r,i){return function(){var s=this,u=arguments,l=function(){var e,l;if(!(t=o&&(r!==W&&(s=void 0,u=[e]),n.rejectWith(s,u))}};t?c():(w.Deferred.getStackHook&&(c.stackTrace=w.Deferred.getStackHook()),e.setTimeout(c))}}return w.Deferred(function(e){n[0][3].add(a(0,e,g(i)?i:I,e.notifyWith)),n[1][3].add(a(0,e,g(t)?t:I)),n[2][3].add(a(0,e,g(r)?r:W))}).promise()},promise:function(e){return null!=e?w.extend(e,i):i}},o={};return w.each(n,function(e,t){var a=t[2],s=t[5];i[t[1]]=a.add,s&&a.add(function(){r=s},n[3-e][2].disable,n[3-e][3].disable,n[0][2].lock,n[0][3].lock),a.add(t[3].fire),o[t[0]]=function(){return o[t[0]+"With"](this===o?void 0:this,arguments),this},o[t[0]+"With"]=a.fireWith}),i.promise(o),t&&t.call(o,o),o},when:function(e){var t=arguments.length,n=t,r=Array(n),i=o.call(arguments),a=w.Deferred(),s=function(e){return function(n){r[e]=this,i[e]=arguments.length>1?o.call(arguments):n,--t||a.resolveWith(r,i)}};if(t<=1&&($(e,a.done(s(n)).resolve,a.reject,!t),"pending"===a.state()||g(i[n]&&i[n].then)))return a.then();while(n--)$(i[n],s(n),a.reject);return a.promise()}});var B=/^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/;w.Deferred.exceptionHook=function(t,n){e.console&&e.console.warn&&t&&B.test(t.name)&&e.console.warn("jQuery.Deferred exception: "+t.message,t.stack,n)},w.readyException=function(t){e.setTimeout(function(){throw t})};var F=w.Deferred();w.fn.ready=function(e){return F.then(e)["catch"](function(e){w.readyException(e)}),this},w.extend({isReady:!1,readyWait:1,ready:function(e){(!0===e?--w.readyWait:w.isReady)||(w.isReady=!0,!0!==e&&--w.readyWait>0||F.resolveWith(r,[w]))}}),w.ready.then=F.then;function _(){r.removeEventListener("DOMContentLoaded",_),e.removeEventListener("load",_),w.ready()}"complete"===r.readyState||"loading"!==r.readyState&&!r.documentElement.doScroll?e.setTimeout(w.ready):(r.addEventListener("DOMContentLoaded",_),e.addEventListener("load",_));var z=function(e,t,n,r,i,o,a){var s=0,u=e.length,l=null==n;if("object"===x(n)){i=!0;for(s in n)z(e,t,s,n[s],!0,o,a)}else if(void 0!==r&&(i=!0,g(r)||(a=!0),l&&(a?(t.call(e,r),t=null):(l=t,t=function(e,t,n){return l.call(w(e),n)})),t))for(;s1,null,!0)},removeData:function(e){return this.each(function(){K.remove(this,e)})}}),w.extend({queue:function(e,t,n){var r;if(e)return t=(t||"fx")+"queue",r=J.get(e,t),n&&(!r||Array.isArray(n)?r=J.access(e,t,w.makeArray(n)):r.push(n)),r||[]},dequeue:function(e,t){t=t||"fx";var n=w.queue(e,t),r=n.length,i=n.shift(),o=w._queueHooks(e,t),a=function(){w.dequeue(e,t)};"inprogress"===i&&(i=n.shift(),r--),i&&("fx"===t&&n.unshift("inprogress"),delete o.stop,i.call(e,a,o)),!r&&o&&o.empty.fire()},_queueHooks:function(e,t){var n=t+"queueHooks";return J.get(e,n)||J.access(e,n,{empty:w.Callbacks("once memory").add(function(){J.remove(e,[t+"queue",n])})})}}),w.fn.extend({queue:function(e,t){var n=2;return"string"!=typeof e&&(t=e,e="fx",n--),arguments.length\x20\t\r\n\f]+)/i,he=/^$|^module$|\/(?:java|ecma)script/i,ge={option:[1,""],thead:[1,"","
    "],col:[2,"","
    "],tr:[2,"","
    "],td:[3,"","
    "],_default:[0,"",""]};ge.optgroup=ge.option,ge.tbody=ge.tfoot=ge.colgroup=ge.caption=ge.thead,ge.th=ge.td;function ye(e,t){var n;return n="undefined"!=typeof e.getElementsByTagName?e.getElementsByTagName(t||"*"):"undefined"!=typeof e.querySelectorAll?e.querySelectorAll(t||"*"):[],void 0===t||t&&N(e,t)?w.merge([e],n):n}function ve(e,t){for(var n=0,r=e.length;n-1)i&&i.push(o);else if(l=w.contains(o.ownerDocument,o),a=ye(f.appendChild(o),"script"),l&&ve(a),n){c=0;while(o=a[c++])he.test(o.type||"")&&n.push(o)}return f}!function(){var e=r.createDocumentFragment().appendChild(r.createElement("div")),t=r.createElement("input");t.setAttribute("type","radio"),t.setAttribute("checked","checked"),t.setAttribute("name","t"),e.appendChild(t),h.checkClone=e.cloneNode(!0).cloneNode(!0).lastChild.checked,e.innerHTML="",h.noCloneChecked=!!e.cloneNode(!0).lastChild.defaultValue}();var be=r.documentElement,we=/^key/,Te=/^(?:mouse|pointer|contextmenu|drag|drop)|click/,Ce=/^([^.]*)(?:\.(.+)|)/;function Ee(){return!0}function ke(){return!1}function Se(){try{return r.activeElement}catch(e){}}function De(e,t,n,r,i,o){var a,s;if("object"==typeof t){"string"!=typeof n&&(r=r||n,n=void 0);for(s in t)De(e,s,n,r,t[s],o);return e}if(null==r&&null==i?(i=n,r=n=void 0):null==i&&("string"==typeof n?(i=r,r=void 0):(i=r,r=n,n=void 0)),!1===i)i=ke;else if(!i)return e;return 1===o&&(a=i,(i=function(e){return w().off(e),a.apply(this,arguments)}).guid=a.guid||(a.guid=w.guid++)),e.each(function(){w.event.add(this,t,i,r,n)})}w.event={global:{},add:function(e,t,n,r,i){var o,a,s,u,l,c,f,p,d,h,g,y=J.get(e);if(y){n.handler&&(n=(o=n).handler,i=o.selector),i&&w.find.matchesSelector(be,i),n.guid||(n.guid=w.guid++),(u=y.events)||(u=y.events={}),(a=y.handle)||(a=y.handle=function(t){return"undefined"!=typeof w&&w.event.triggered!==t.type?w.event.dispatch.apply(e,arguments):void 0}),l=(t=(t||"").match(M)||[""]).length;while(l--)d=g=(s=Ce.exec(t[l])||[])[1],h=(s[2]||"").split(".").sort(),d&&(f=w.event.special[d]||{},d=(i?f.delegateType:f.bindType)||d,f=w.event.special[d]||{},c=w.extend({type:d,origType:g,data:r,handler:n,guid:n.guid,selector:i,needsContext:i&&w.expr.match.needsContext.test(i),namespace:h.join(".")},o),(p=u[d])||((p=u[d]=[]).delegateCount=0,f.setup&&!1!==f.setup.call(e,r,h,a)||e.addEventListener&&e.addEventListener(d,a)),f.add&&(f.add.call(e,c),c.handler.guid||(c.handler.guid=n.guid)),i?p.splice(p.delegateCount++,0,c):p.push(c),w.event.global[d]=!0)}},remove:function(e,t,n,r,i){var o,a,s,u,l,c,f,p,d,h,g,y=J.hasData(e)&&J.get(e);if(y&&(u=y.events)){l=(t=(t||"").match(M)||[""]).length;while(l--)if(s=Ce.exec(t[l])||[],d=g=s[1],h=(s[2]||"").split(".").sort(),d){f=w.event.special[d]||{},p=u[d=(r?f.delegateType:f.bindType)||d]||[],s=s[2]&&new RegExp("(^|\\.)"+h.join("\\.(?:.*\\.|)")+"(\\.|$)"),a=o=p.length;while(o--)c=p[o],!i&&g!==c.origType||n&&n.guid!==c.guid||s&&!s.test(c.namespace)||r&&r!==c.selector&&("**"!==r||!c.selector)||(p.splice(o,1),c.selector&&p.delegateCount--,f.remove&&f.remove.call(e,c));a&&!p.length&&(f.teardown&&!1!==f.teardown.call(e,h,y.handle)||w.removeEvent(e,d,y.handle),delete u[d])}else for(d in u)w.event.remove(e,d+t[l],n,r,!0);w.isEmptyObject(u)&&J.remove(e,"handle events")}},dispatch:function(e){var t=w.event.fix(e),n,r,i,o,a,s,u=new Array(arguments.length),l=(J.get(this,"events")||{})[t.type]||[],c=w.event.special[t.type]||{};for(u[0]=t,n=1;n=1))for(;l!==this;l=l.parentNode||this)if(1===l.nodeType&&("click"!==e.type||!0!==l.disabled)){for(o=[],a={},n=0;n-1:w.find(i,this,null,[l]).length),a[i]&&o.push(r);o.length&&s.push({elem:l,handlers:o})}return l=this,u\x20\t\r\n\f]*)[^>]*)\/>/gi,Ae=/\s*$/g;function Le(e,t){return N(e,"table")&&N(11!==t.nodeType?t:t.firstChild,"tr")?w(e).children("tbody")[0]||e:e}function He(e){return e.type=(null!==e.getAttribute("type"))+"/"+e.type,e}function Oe(e){return"true/"===(e.type||"").slice(0,5)?e.type=e.type.slice(5):e.removeAttribute("type"),e}function Pe(e,t){var n,r,i,o,a,s,u,l;if(1===t.nodeType){if(J.hasData(e)&&(o=J.access(e),a=J.set(t,o),l=o.events)){delete a.handle,a.events={};for(i in l)for(n=0,r=l[i].length;n1&&"string"==typeof y&&!h.checkClone&&je.test(y))return e.each(function(i){var o=e.eq(i);v&&(t[0]=y.call(this,i,o.html())),Re(o,t,n,r)});if(p&&(i=xe(t,e[0].ownerDocument,!1,e,r),o=i.firstChild,1===i.childNodes.length&&(i=o),o||r)){for(u=(s=w.map(ye(i,"script"),He)).length;f")},clone:function(e,t,n){var r,i,o,a,s=e.cloneNode(!0),u=w.contains(e.ownerDocument,e);if(!(h.noCloneChecked||1!==e.nodeType&&11!==e.nodeType||w.isXMLDoc(e)))for(a=ye(s),r=0,i=(o=ye(e)).length;r0&&ve(a,!u&&ye(e,"script")),s},cleanData:function(e){for(var t,n,r,i=w.event.special,o=0;void 0!==(n=e[o]);o++)if(Y(n)){if(t=n[J.expando]){if(t.events)for(r in t.events)i[r]?w.event.remove(n,r):w.removeEvent(n,r,t.handle);n[J.expando]=void 0}n[K.expando]&&(n[K.expando]=void 0)}}}),w.fn.extend({detach:function(e){return Ie(this,e,!0)},remove:function(e){return Ie(this,e)},text:function(e){return z(this,function(e){return void 0===e?w.text(this):this.empty().each(function(){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||(this.textContent=e)})},null,e,arguments.length)},append:function(){return Re(this,arguments,function(e){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||Le(this,e).appendChild(e)})},prepend:function(){return Re(this,arguments,function(e){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var t=Le(this,e);t.insertBefore(e,t.firstChild)}})},before:function(){return Re(this,arguments,function(e){this.parentNode&&this.parentNode.insertBefore(e,this)})},after:function(){return Re(this,arguments,function(e){this.parentNode&&this.parentNode.insertBefore(e,this.nextSibling)})},empty:function(){for(var e,t=0;null!=(e=this[t]);t++)1===e.nodeType&&(w.cleanData(ye(e,!1)),e.textContent="");return this},clone:function(e,t){return e=null!=e&&e,t=null==t?e:t,this.map(function(){return w.clone(this,e,t)})},html:function(e){return z(this,function(e){var t=this[0]||{},n=0,r=this.length;if(void 0===e&&1===t.nodeType)return t.innerHTML;if("string"==typeof e&&!Ae.test(e)&&!ge[(de.exec(e)||["",""])[1].toLowerCase()]){e=w.htmlPrefilter(e);try{for(;n=0&&(u+=Math.max(0,Math.ceil(e["offset"+t[0].toUpperCase()+t.slice(1)]-o-u-s-.5))),u}function et(e,t,n){var r=$e(e),i=Fe(e,t,r),o="border-box"===w.css(e,"boxSizing",!1,r),a=o;if(We.test(i)){if(!n)return i;i="auto"}return a=a&&(h.boxSizingReliable()||i===e.style[t]),("auto"===i||!parseFloat(i)&&"inline"===w.css(e,"display",!1,r))&&(i=e["offset"+t[0].toUpperCase()+t.slice(1)],a=!0),(i=parseFloat(i)||0)+Ze(e,t,n||(o?"border":"content"),a,r,i)+"px"}w.extend({cssHooks:{opacity:{get:function(e,t){if(t){var n=Fe(e,"opacity");return""===n?"1":n}}}},cssNumber:{animationIterationCount:!0,columnCount:!0,fillOpacity:!0,flexGrow:!0,flexShrink:!0,fontWeight:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{},style:function(e,t,n,r){if(e&&3!==e.nodeType&&8!==e.nodeType&&e.style){var i,o,a,s=G(t),u=Xe.test(t),l=e.style;if(u||(t=Je(s)),a=w.cssHooks[t]||w.cssHooks[s],void 0===n)return a&&"get"in a&&void 0!==(i=a.get(e,!1,r))?i:l[t];"string"==(o=typeof n)&&(i=ie.exec(n))&&i[1]&&(n=ue(e,t,i),o="number"),null!=n&&n===n&&("number"===o&&(n+=i&&i[3]||(w.cssNumber[s]?"":"px")),h.clearCloneStyle||""!==n||0!==t.indexOf("background")||(l[t]="inherit"),a&&"set"in a&&void 0===(n=a.set(e,n,r))||(u?l.setProperty(t,n):l[t]=n))}},css:function(e,t,n,r){var i,o,a,s=G(t);return Xe.test(t)||(t=Je(s)),(a=w.cssHooks[t]||w.cssHooks[s])&&"get"in a&&(i=a.get(e,!0,n)),void 0===i&&(i=Fe(e,t,r)),"normal"===i&&t in Ve&&(i=Ve[t]),""===n||n?(o=parseFloat(i),!0===n||isFinite(o)?o||0:i):i}}),w.each(["height","width"],function(e,t){w.cssHooks[t]={get:function(e,n,r){if(n)return!ze.test(w.css(e,"display"))||e.getClientRects().length&&e.getBoundingClientRect().width?et(e,t,r):se(e,Ue,function(){return et(e,t,r)})},set:function(e,n,r){var i,o=$e(e),a="border-box"===w.css(e,"boxSizing",!1,o),s=r&&Ze(e,t,r,a,o);return a&&h.scrollboxSize()===o.position&&(s-=Math.ceil(e["offset"+t[0].toUpperCase()+t.slice(1)]-parseFloat(o[t])-Ze(e,t,"border",!1,o)-.5)),s&&(i=ie.exec(n))&&"px"!==(i[3]||"px")&&(e.style[t]=n,n=w.css(e,t)),Ke(e,n,s)}}}),w.cssHooks.marginLeft=_e(h.reliableMarginLeft,function(e,t){if(t)return(parseFloat(Fe(e,"marginLeft"))||e.getBoundingClientRect().left-se(e,{marginLeft:0},function(){return e.getBoundingClientRect().left}))+"px"}),w.each({margin:"",padding:"",border:"Width"},function(e,t){w.cssHooks[e+t]={expand:function(n){for(var r=0,i={},o="string"==typeof n?n.split(" "):[n];r<4;r++)i[e+oe[r]+t]=o[r]||o[r-2]||o[0];return i}},"margin"!==e&&(w.cssHooks[e+t].set=Ke)}),w.fn.extend({css:function(e,t){return z(this,function(e,t,n){var r,i,o={},a=0;if(Array.isArray(t)){for(r=$e(e),i=t.length;a1)}});function tt(e,t,n,r,i){return new tt.prototype.init(e,t,n,r,i)}w.Tween=tt,tt.prototype={constructor:tt,init:function(e,t,n,r,i,o){this.elem=e,this.prop=n,this.easing=i||w.easing._default,this.options=t,this.start=this.now=this.cur(),this.end=r,this.unit=o||(w.cssNumber[n]?"":"px")},cur:function(){var e=tt.propHooks[this.prop];return e&&e.get?e.get(this):tt.propHooks._default.get(this)},run:function(e){var t,n=tt.propHooks[this.prop];return this.options.duration?this.pos=t=w.easing[this.easing](e,this.options.duration*e,0,1,this.options.duration):this.pos=t=e,this.now=(this.end-this.start)*t+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),n&&n.set?n.set(this):tt.propHooks._default.set(this),this}},tt.prototype.init.prototype=tt.prototype,tt.propHooks={_default:{get:function(e){var t;return 1!==e.elem.nodeType||null!=e.elem[e.prop]&&null==e.elem.style[e.prop]?e.elem[e.prop]:(t=w.css(e.elem,e.prop,""))&&"auto"!==t?t:0},set:function(e){w.fx.step[e.prop]?w.fx.step[e.prop](e):1!==e.elem.nodeType||null==e.elem.style[w.cssProps[e.prop]]&&!w.cssHooks[e.prop]?e.elem[e.prop]=e.now:w.style(e.elem,e.prop,e.now+e.unit)}}},tt.propHooks.scrollTop=tt.propHooks.scrollLeft={set:function(e){e.elem.nodeType&&e.elem.parentNode&&(e.elem[e.prop]=e.now)}},w.easing={linear:function(e){return e},swing:function(e){return.5-Math.cos(e*Math.PI)/2},_default:"swing"},w.fx=tt.prototype.init,w.fx.step={};var nt,rt,it=/^(?:toggle|show|hide)$/,ot=/queueHooks$/;function at(){rt&&(!1===r.hidden&&e.requestAnimationFrame?e.requestAnimationFrame(at):e.setTimeout(at,w.fx.interval),w.fx.tick())}function st(){return e.setTimeout(function(){nt=void 0}),nt=Date.now()}function ut(e,t){var n,r=0,i={height:e};for(t=t?1:0;r<4;r+=2-t)i["margin"+(n=oe[r])]=i["padding"+n]=e;return t&&(i.opacity=i.width=e),i}function lt(e,t,n){for(var r,i=(pt.tweeners[t]||[]).concat(pt.tweeners["*"]),o=0,a=i.length;o1)},removeAttr:function(e){return this.each(function(){w.removeAttr(this,e)})}}),w.extend({attr:function(e,t,n){var r,i,o=e.nodeType;if(3!==o&&8!==o&&2!==o)return"undefined"==typeof e.getAttribute?w.prop(e,t,n):(1===o&&w.isXMLDoc(e)||(i=w.attrHooks[t.toLowerCase()]||(w.expr.match.bool.test(t)?dt:void 0)),void 0!==n?null===n?void w.removeAttr(e,t):i&&"set"in i&&void 0!==(r=i.set(e,n,t))?r:(e.setAttribute(t,n+""),n):i&&"get"in i&&null!==(r=i.get(e,t))?r:null==(r=w.find.attr(e,t))?void 0:r)},attrHooks:{type:{set:function(e,t){if(!h.radioValue&&"radio"===t&&N(e,"input")){var n=e.value;return e.setAttribute("type",t),n&&(e.value=n),t}}}},removeAttr:function(e,t){var n,r=0,i=t&&t.match(M);if(i&&1===e.nodeType)while(n=i[r++])e.removeAttribute(n)}}),dt={set:function(e,t,n){return!1===t?w.removeAttr(e,n):e.setAttribute(n,n),n}},w.each(w.expr.match.bool.source.match(/\w+/g),function(e,t){var n=ht[t]||w.find.attr;ht[t]=function(e,t,r){var i,o,a=t.toLowerCase();return r||(o=ht[a],ht[a]=i,i=null!=n(e,t,r)?a:null,ht[a]=o),i}});var gt=/^(?:input|select|textarea|button)$/i,yt=/^(?:a|area)$/i;w.fn.extend({prop:function(e,t){return z(this,w.prop,e,t,arguments.length>1)},removeProp:function(e){return this.each(function(){delete this[w.propFix[e]||e]})}}),w.extend({prop:function(e,t,n){var r,i,o=e.nodeType;if(3!==o&&8!==o&&2!==o)return 1===o&&w.isXMLDoc(e)||(t=w.propFix[t]||t,i=w.propHooks[t]),void 0!==n?i&&"set"in i&&void 0!==(r=i.set(e,n,t))?r:e[t]=n:i&&"get"in i&&null!==(r=i.get(e,t))?r:e[t]},propHooks:{tabIndex:{get:function(e){var t=w.find.attr(e,"tabindex");return t?parseInt(t,10):gt.test(e.nodeName)||yt.test(e.nodeName)&&e.href?0:-1}}},propFix:{"for":"htmlFor","class":"className"}}),h.optSelected||(w.propHooks.selected={get:function(e){var t=e.parentNode;return t&&t.parentNode&&t.parentNode.selectedIndex,null},set:function(e){var t=e.parentNode;t&&(t.selectedIndex,t.parentNode&&t.parentNode.selectedIndex)}}),w.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){w.propFix[this.toLowerCase()]=this});function vt(e){return(e.match(M)||[]).join(" ")}function mt(e){return e.getAttribute&&e.getAttribute("class")||""}function xt(e){return Array.isArray(e)?e:"string"==typeof e?e.match(M)||[]:[]}w.fn.extend({addClass:function(e){var t,n,r,i,o,a,s,u=0;if(g(e))return this.each(function(t){w(this).addClass(e.call(this,t,mt(this)))});if((t=xt(e)).length)while(n=this[u++])if(i=mt(n),r=1===n.nodeType&&" "+vt(i)+" "){a=0;while(o=t[a++])r.indexOf(" "+o+" ")<0&&(r+=o+" ");i!==(s=vt(r))&&n.setAttribute("class",s)}return this},removeClass:function(e){var t,n,r,i,o,a,s,u=0;if(g(e))return this.each(function(t){w(this).removeClass(e.call(this,t,mt(this)))});if(!arguments.length)return this.attr("class","");if((t=xt(e)).length)while(n=this[u++])if(i=mt(n),r=1===n.nodeType&&" "+vt(i)+" "){a=0;while(o=t[a++])while(r.indexOf(" "+o+" ")>-1)r=r.replace(" "+o+" "," ");i!==(s=vt(r))&&n.setAttribute("class",s)}return this},toggleClass:function(e,t){var n=typeof e,r="string"===n||Array.isArray(e);return"boolean"==typeof t&&r?t?this.addClass(e):this.removeClass(e):g(e)?this.each(function(n){w(this).toggleClass(e.call(this,n,mt(this),t),t)}):this.each(function(){var t,i,o,a;if(r){i=0,o=w(this),a=xt(e);while(t=a[i++])o.hasClass(t)?o.removeClass(t):o.addClass(t)}else void 0!==e&&"boolean"!==n||((t=mt(this))&&J.set(this,"__className__",t),this.setAttribute&&this.setAttribute("class",t||!1===e?"":J.get(this,"__className__")||""))})},hasClass:function(e){var t,n,r=0;t=" "+e+" ";while(n=this[r++])if(1===n.nodeType&&(" "+vt(mt(n))+" ").indexOf(t)>-1)return!0;return!1}});var bt=/\r/g;w.fn.extend({val:function(e){var t,n,r,i=this[0];{if(arguments.length)return r=g(e),this.each(function(n){var i;1===this.nodeType&&(null==(i=r?e.call(this,n,w(this).val()):e)?i="":"number"==typeof i?i+="":Array.isArray(i)&&(i=w.map(i,function(e){return null==e?"":e+""})),(t=w.valHooks[this.type]||w.valHooks[this.nodeName.toLowerCase()])&&"set"in t&&void 0!==t.set(this,i,"value")||(this.value=i))});if(i)return(t=w.valHooks[i.type]||w.valHooks[i.nodeName.toLowerCase()])&&"get"in t&&void 0!==(n=t.get(i,"value"))?n:"string"==typeof(n=i.value)?n.replace(bt,""):null==n?"":n}}}),w.extend({valHooks:{option:{get:function(e){var t=w.find.attr(e,"value");return null!=t?t:vt(w.text(e))}},select:{get:function(e){var t,n,r,i=e.options,o=e.selectedIndex,a="select-one"===e.type,s=a?null:[],u=a?o+1:i.length;for(r=o<0?u:a?o:0;r-1)&&(n=!0);return n||(e.selectedIndex=-1),o}}}}),w.each(["radio","checkbox"],function(){w.valHooks[this]={set:function(e,t){if(Array.isArray(t))return e.checked=w.inArray(w(e).val(),t)>-1}},h.checkOn||(w.valHooks[this].get=function(e){return null===e.getAttribute("value")?"on":e.value})}),h.focusin="onfocusin"in e;var wt=/^(?:focusinfocus|focusoutblur)$/,Tt=function(e){e.stopPropagation()};w.extend(w.event,{trigger:function(t,n,i,o){var a,s,u,l,c,p,d,h,v=[i||r],m=f.call(t,"type")?t.type:t,x=f.call(t,"namespace")?t.namespace.split("."):[];if(s=h=u=i=i||r,3!==i.nodeType&&8!==i.nodeType&&!wt.test(m+w.event.triggered)&&(m.indexOf(".")>-1&&(m=(x=m.split(".")).shift(),x.sort()),c=m.indexOf(":")<0&&"on"+m,t=t[w.expando]?t:new w.Event(m,"object"==typeof t&&t),t.isTrigger=o?2:3,t.namespace=x.join("."),t.rnamespace=t.namespace?new RegExp("(^|\\.)"+x.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,t.result=void 0,t.target||(t.target=i),n=null==n?[t]:w.makeArray(n,[t]),d=w.event.special[m]||{},o||!d.trigger||!1!==d.trigger.apply(i,n))){if(!o&&!d.noBubble&&!y(i)){for(l=d.delegateType||m,wt.test(l+m)||(s=s.parentNode);s;s=s.parentNode)v.push(s),u=s;u===(i.ownerDocument||r)&&v.push(u.defaultView||u.parentWindow||e)}a=0;while((s=v[a++])&&!t.isPropagationStopped())h=s,t.type=a>1?l:d.bindType||m,(p=(J.get(s,"events")||{})[t.type]&&J.get(s,"handle"))&&p.apply(s,n),(p=c&&s[c])&&p.apply&&Y(s)&&(t.result=p.apply(s,n),!1===t.result&&t.preventDefault());return t.type=m,o||t.isDefaultPrevented()||d._default&&!1!==d._default.apply(v.pop(),n)||!Y(i)||c&&g(i[m])&&!y(i)&&((u=i[c])&&(i[c]=null),w.event.triggered=m,t.isPropagationStopped()&&h.addEventListener(m,Tt),i[m](),t.isPropagationStopped()&&h.removeEventListener(m,Tt),w.event.triggered=void 0,u&&(i[c]=u)),t.result}},simulate:function(e,t,n){var r=w.extend(new w.Event,n,{type:e,isSimulated:!0});w.event.trigger(r,null,t)}}),w.fn.extend({trigger:function(e,t){return this.each(function(){w.event.trigger(e,t,this)})},triggerHandler:function(e,t){var n=this[0];if(n)return w.event.trigger(e,t,n,!0)}}),h.focusin||w.each({focus:"focusin",blur:"focusout"},function(e,t){var n=function(e){w.event.simulate(t,e.target,w.event.fix(e))};w.event.special[t]={setup:function(){var r=this.ownerDocument||this,i=J.access(r,t);i||r.addEventListener(e,n,!0),J.access(r,t,(i||0)+1)},teardown:function(){var r=this.ownerDocument||this,i=J.access(r,t)-1;i?J.access(r,t,i):(r.removeEventListener(e,n,!0),J.remove(r,t))}}});var Ct=e.location,Et=Date.now(),kt=/\?/;w.parseXML=function(t){var n;if(!t||"string"!=typeof t)return null;try{n=(new e.DOMParser).parseFromString(t,"text/xml")}catch(e){n=void 0}return n&&!n.getElementsByTagName("parsererror").length||w.error("Invalid XML: "+t),n};var St=/\[\]$/,Dt=/\r?\n/g,Nt=/^(?:submit|button|image|reset|file)$/i,At=/^(?:input|select|textarea|keygen)/i;function jt(e,t,n,r){var i;if(Array.isArray(t))w.each(t,function(t,i){n||St.test(e)?r(e,i):jt(e+"["+("object"==typeof i&&null!=i?t:"")+"]",i,n,r)});else if(n||"object"!==x(t))r(e,t);else for(i in t)jt(e+"["+i+"]",t[i],n,r)}w.param=function(e,t){var n,r=[],i=function(e,t){var n=g(t)?t():t;r[r.length]=encodeURIComponent(e)+"="+encodeURIComponent(null==n?"":n)};if(Array.isArray(e)||e.jquery&&!w.isPlainObject(e))w.each(e,function(){i(this.name,this.value)});else for(n in e)jt(n,e[n],t,i);return r.join("&")},w.fn.extend({serialize:function(){return w.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var e=w.prop(this,"elements");return e?w.makeArray(e):this}).filter(function(){var e=this.type;return this.name&&!w(this).is(":disabled")&&At.test(this.nodeName)&&!Nt.test(e)&&(this.checked||!pe.test(e))}).map(function(e,t){var n=w(this).val();return null==n?null:Array.isArray(n)?w.map(n,function(e){return{name:t.name,value:e.replace(Dt,"\r\n")}}):{name:t.name,value:n.replace(Dt,"\r\n")}}).get()}});var qt=/%20/g,Lt=/#.*$/,Ht=/([?&])_=[^&]*/,Ot=/^(.*?):[ \t]*([^\r\n]*)$/gm,Pt=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Mt=/^(?:GET|HEAD)$/,Rt=/^\/\//,It={},Wt={},$t="*/".concat("*"),Bt=r.createElement("a");Bt.href=Ct.href;function Ft(e){return function(t,n){"string"!=typeof t&&(n=t,t="*");var r,i=0,o=t.toLowerCase().match(M)||[];if(g(n))while(r=o[i++])"+"===r[0]?(r=r.slice(1)||"*",(e[r]=e[r]||[]).unshift(n)):(e[r]=e[r]||[]).push(n)}}function _t(e,t,n,r){var i={},o=e===Wt;function a(s){var u;return i[s]=!0,w.each(e[s]||[],function(e,s){var l=s(t,n,r);return"string"!=typeof l||o||i[l]?o?!(u=l):void 0:(t.dataTypes.unshift(l),a(l),!1)}),u}return a(t.dataTypes[0])||!i["*"]&&a("*")}function zt(e,t){var n,r,i=w.ajaxSettings.flatOptions||{};for(n in t)void 0!==t[n]&&((i[n]?e:r||(r={}))[n]=t[n]);return r&&w.extend(!0,e,r),e}function Xt(e,t,n){var r,i,o,a,s=e.contents,u=e.dataTypes;while("*"===u[0])u.shift(),void 0===r&&(r=e.mimeType||t.getResponseHeader("Content-Type"));if(r)for(i in s)if(s[i]&&s[i].test(r)){u.unshift(i);break}if(u[0]in n)o=u[0];else{for(i in n){if(!u[0]||e.converters[i+" "+u[0]]){o=i;break}a||(a=i)}o=o||a}if(o)return o!==u[0]&&u.unshift(o),n[o]}function Ut(e,t,n,r){var i,o,a,s,u,l={},c=e.dataTypes.slice();if(c[1])for(a in e.converters)l[a.toLowerCase()]=e.converters[a];o=c.shift();while(o)if(e.responseFields[o]&&(n[e.responseFields[o]]=t),!u&&r&&e.dataFilter&&(t=e.dataFilter(t,e.dataType)),u=o,o=c.shift())if("*"===o)o=u;else if("*"!==u&&u!==o){if(!(a=l[u+" "+o]||l["* "+o]))for(i in l)if((s=i.split(" "))[1]===o&&(a=l[u+" "+s[0]]||l["* "+s[0]])){!0===a?a=l[i]:!0!==l[i]&&(o=s[0],c.unshift(s[1]));break}if(!0!==a)if(a&&e["throws"])t=a(t);else try{t=a(t)}catch(e){return{state:"parsererror",error:a?e:"No conversion from "+u+" to "+o}}}return{state:"success",data:t}}w.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:Ct.href,type:"GET",isLocal:Pt.test(Ct.protocol),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":$t,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/\bxml\b/,html:/\bhtml/,json:/\bjson\b/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":JSON.parse,"text xml":w.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(e,t){return t?zt(zt(e,w.ajaxSettings),t):zt(w.ajaxSettings,e)},ajaxPrefilter:Ft(It),ajaxTransport:Ft(Wt),ajax:function(t,n){"object"==typeof t&&(n=t,t=void 0),n=n||{};var i,o,a,s,u,l,c,f,p,d,h=w.ajaxSetup({},n),g=h.context||h,y=h.context&&(g.nodeType||g.jquery)?w(g):w.event,v=w.Deferred(),m=w.Callbacks("once memory"),x=h.statusCode||{},b={},T={},C="canceled",E={readyState:0,getResponseHeader:function(e){var t;if(c){if(!s){s={};while(t=Ot.exec(a))s[t[1].toLowerCase()]=t[2]}t=s[e.toLowerCase()]}return null==t?null:t},getAllResponseHeaders:function(){return c?a:null},setRequestHeader:function(e,t){return null==c&&(e=T[e.toLowerCase()]=T[e.toLowerCase()]||e,b[e]=t),this},overrideMimeType:function(e){return null==c&&(h.mimeType=e),this},statusCode:function(e){var t;if(e)if(c)E.always(e[E.status]);else for(t in e)x[t]=[x[t],e[t]];return this},abort:function(e){var t=e||C;return i&&i.abort(t),k(0,t),this}};if(v.promise(E),h.url=((t||h.url||Ct.href)+"").replace(Rt,Ct.protocol+"//"),h.type=n.method||n.type||h.method||h.type,h.dataTypes=(h.dataType||"*").toLowerCase().match(M)||[""],null==h.crossDomain){l=r.createElement("a");try{l.href=h.url,l.href=l.href,h.crossDomain=Bt.protocol+"//"+Bt.host!=l.protocol+"//"+l.host}catch(e){h.crossDomain=!0}}if(h.data&&h.processData&&"string"!=typeof h.data&&(h.data=w.param(h.data,h.traditional)),_t(It,h,n,E),c)return E;(f=w.event&&h.global)&&0==w.active++&&w.event.trigger("ajaxStart"),h.type=h.type.toUpperCase(),h.hasContent=!Mt.test(h.type),o=h.url.replace(Lt,""),h.hasContent?h.data&&h.processData&&0===(h.contentType||"").indexOf("application/x-www-form-urlencoded")&&(h.data=h.data.replace(qt,"+")):(d=h.url.slice(o.length),h.data&&(h.processData||"string"==typeof h.data)&&(o+=(kt.test(o)?"&":"?")+h.data,delete h.data),!1===h.cache&&(o=o.replace(Ht,"$1"),d=(kt.test(o)?"&":"?")+"_="+Et+++d),h.url=o+d),h.ifModified&&(w.lastModified[o]&&E.setRequestHeader("If-Modified-Since",w.lastModified[o]),w.etag[o]&&E.setRequestHeader("If-None-Match",w.etag[o])),(h.data&&h.hasContent&&!1!==h.contentType||n.contentType)&&E.setRequestHeader("Content-Type",h.contentType),E.setRequestHeader("Accept",h.dataTypes[0]&&h.accepts[h.dataTypes[0]]?h.accepts[h.dataTypes[0]]+("*"!==h.dataTypes[0]?", "+$t+"; q=0.01":""):h.accepts["*"]);for(p in h.headers)E.setRequestHeader(p,h.headers[p]);if(h.beforeSend&&(!1===h.beforeSend.call(g,E,h)||c))return E.abort();if(C="abort",m.add(h.complete),E.done(h.success),E.fail(h.error),i=_t(Wt,h,n,E)){if(E.readyState=1,f&&y.trigger("ajaxSend",[E,h]),c)return E;h.async&&h.timeout>0&&(u=e.setTimeout(function(){E.abort("timeout")},h.timeout));try{c=!1,i.send(b,k)}catch(e){if(c)throw e;k(-1,e)}}else k(-1,"No Transport");function k(t,n,r,s){var l,p,d,b,T,C=n;c||(c=!0,u&&e.clearTimeout(u),i=void 0,a=s||"",E.readyState=t>0?4:0,l=t>=200&&t<300||304===t,r&&(b=Xt(h,E,r)),b=Ut(h,b,E,l),l?(h.ifModified&&((T=E.getResponseHeader("Last-Modified"))&&(w.lastModified[o]=T),(T=E.getResponseHeader("etag"))&&(w.etag[o]=T)),204===t||"HEAD"===h.type?C="nocontent":304===t?C="notmodified":(C=b.state,p=b.data,l=!(d=b.error))):(d=C,!t&&C||(C="error",t<0&&(t=0))),E.status=t,E.statusText=(n||C)+"",l?v.resolveWith(g,[p,C,E]):v.rejectWith(g,[E,C,d]),E.statusCode(x),x=void 0,f&&y.trigger(l?"ajaxSuccess":"ajaxError",[E,h,l?p:d]),m.fireWith(g,[E,C]),f&&(y.trigger("ajaxComplete",[E,h]),--w.active||w.event.trigger("ajaxStop")))}return E},getJSON:function(e,t,n){return w.get(e,t,n,"json")},getScript:function(e,t){return w.get(e,void 0,t,"script")}}),w.each(["get","post"],function(e,t){w[t]=function(e,n,r,i){return g(n)&&(i=i||r,r=n,n=void 0),w.ajax(w.extend({url:e,type:t,dataType:i,data:n,success:r},w.isPlainObject(e)&&e))}}),w._evalUrl=function(e){return w.ajax({url:e,type:"GET",dataType:"script",cache:!0,async:!1,global:!1,"throws":!0})},w.fn.extend({wrapAll:function(e){var t;return this[0]&&(g(e)&&(e=e.call(this[0])),t=w(e,this[0].ownerDocument).eq(0).clone(!0),this[0].parentNode&&t.insertBefore(this[0]),t.map(function(){var e=this;while(e.firstElementChild)e=e.firstElementChild;return e}).append(this)),this},wrapInner:function(e){return g(e)?this.each(function(t){w(this).wrapInner(e.call(this,t))}):this.each(function(){var t=w(this),n=t.contents();n.length?n.wrapAll(e):t.append(e)})},wrap:function(e){var t=g(e);return this.each(function(n){w(this).wrapAll(t?e.call(this,n):e)})},unwrap:function(e){return this.parent(e).not("body").each(function(){w(this).replaceWith(this.childNodes)}),this}}),w.expr.pseudos.hidden=function(e){return!w.expr.pseudos.visible(e)},w.expr.pseudos.visible=function(e){return!!(e.offsetWidth||e.offsetHeight||e.getClientRects().length)},w.ajaxSettings.xhr=function(){try{return new e.XMLHttpRequest}catch(e){}};var Vt={0:200,1223:204},Gt=w.ajaxSettings.xhr();h.cors=!!Gt&&"withCredentials"in Gt,h.ajax=Gt=!!Gt,w.ajaxTransport(function(t){var n,r;if(h.cors||Gt&&!t.crossDomain)return{send:function(i,o){var a,s=t.xhr();if(s.open(t.type,t.url,t.async,t.username,t.password),t.xhrFields)for(a in t.xhrFields)s[a]=t.xhrFields[a];t.mimeType&&s.overrideMimeType&&s.overrideMimeType(t.mimeType),t.crossDomain||i["X-Requested-With"]||(i["X-Requested-With"]="XMLHttpRequest");for(a in i)s.setRequestHeader(a,i[a]);n=function(e){return function(){n&&(n=r=s.onload=s.onerror=s.onabort=s.ontimeout=s.onreadystatechange=null,"abort"===e?s.abort():"error"===e?"number"!=typeof s.status?o(0,"error"):o(s.status,s.statusText):o(Vt[s.status]||s.status,s.statusText,"text"!==(s.responseType||"text")||"string"!=typeof s.responseText?{binary:s.response}:{text:s.responseText},s.getAllResponseHeaders()))}},s.onload=n(),r=s.onerror=s.ontimeout=n("error"),void 0!==s.onabort?s.onabort=r:s.onreadystatechange=function(){4===s.readyState&&e.setTimeout(function(){n&&r()})},n=n("abort");try{s.send(t.hasContent&&t.data||null)}catch(e){if(n)throw e}},abort:function(){n&&n()}}}),w.ajaxPrefilter(function(e){e.crossDomain&&(e.contents.script=!1)}),w.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/\b(?:java|ecma)script\b/},converters:{"text script":function(e){return w.globalEval(e),e}}}),w.ajaxPrefilter("script",function(e){void 0===e.cache&&(e.cache=!1),e.crossDomain&&(e.type="GET")}),w.ajaxTransport("script",function(e){if(e.crossDomain){var t,n;return{send:function(i,o){t=w(" + + + + +
    +
    +
    +
    + + Logo + ArchiveBox + +
    +
    + Documentation   |   + Source   |   + Website +
    +
    +
    +
    + + + + + + + + + + $rows +
    BookmarkedSnapshot ($num_links)FilesOriginal URL
    + + + diff --git a/archivebox-0.5.3/build/lib/archivebox/themes/legacy/main_index_row.html b/archivebox-0.5.3/build/lib/archivebox/themes/legacy/main_index_row.html new file mode 100644 index 0000000..9112eac --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/themes/legacy/main_index_row.html @@ -0,0 +1,16 @@ + + $bookmarked_date + + + + $title + $tags + + + + 📄 + $num_outputs + + + $url + diff --git a/archivebox-0.5.3/build/lib/archivebox/util.py b/archivebox-0.5.3/build/lib/archivebox/util.py new file mode 100644 index 0000000..5530ab4 --- /dev/null +++ b/archivebox-0.5.3/build/lib/archivebox/util.py @@ -0,0 +1,318 @@ +__package__ = 'archivebox' + +import re +import requests +import json as pyjson + +from typing import List, Optional, Any +from pathlib import Path +from inspect import signature +from functools import wraps +from hashlib import sha256 +from urllib.parse import urlparse, quote, unquote +from html import escape, unescape +from datetime import datetime +from dateparser import parse as dateparser +from requests.exceptions import RequestException, ReadTimeout + +from .vendor.base32_crockford import encode as base32_encode # type: ignore +from w3lib.encoding import html_body_declared_encoding, http_content_type_encoding + +try: + import chardet + detect_encoding = lambda rawdata: chardet.detect(rawdata)["encoding"] +except ImportError: + detect_encoding = lambda rawdata: "utf-8" + +### Parsing Helpers + +# All of these are (str) -> str +# shortcuts to: https://docs.python.org/3/library/urllib.parse.html#url-parsing +scheme = lambda url: urlparse(url).scheme.lower() +without_scheme = lambda url: urlparse(url)._replace(scheme='').geturl().strip('//') +without_query = lambda url: urlparse(url)._replace(query='').geturl().strip('//') +without_fragment = lambda url: urlparse(url)._replace(fragment='').geturl().strip('//') +without_path = lambda url: urlparse(url)._replace(path='', fragment='', query='').geturl().strip('//') +path = lambda url: urlparse(url).path +basename = lambda url: urlparse(url).path.rsplit('/', 1)[-1] +domain = lambda url: urlparse(url).netloc +query = lambda url: urlparse(url).query +fragment = lambda url: urlparse(url).fragment +extension = lambda url: basename(url).rsplit('.', 1)[-1].lower() if '.' in basename(url) else '' +base_url = lambda url: without_scheme(url) # uniq base url used to dedupe links + +without_www = lambda url: url.replace('://www.', '://', 1) +without_trailing_slash = lambda url: url[:-1] if url[-1] == '/' else url.replace('/?', '?') +hashurl = lambda url: base32_encode(int(sha256(base_url(url).encode('utf-8')).hexdigest(), 16))[:20] + +urlencode = lambda s: s and quote(s, encoding='utf-8', errors='replace') +urldecode = lambda s: s and unquote(s) +htmlencode = lambda s: s and escape(s, quote=True) +htmldecode = lambda s: s and unescape(s) + +short_ts = lambda ts: str(parse_date(ts).timestamp()).split('.')[0] +ts_to_date = lambda ts: ts and parse_date(ts).strftime('%Y-%m-%d %H:%M') +ts_to_iso = lambda ts: ts and parse_date(ts).isoformat() + + +URL_REGEX = re.compile( + r'http[s]?://' # start matching from allowed schemes + r'(?:[a-zA-Z]|[0-9]' # followed by allowed alphanum characters + r'|[$-_@.&+]|[!*\(\),]' # or allowed symbols + r'|(?:%[0-9a-fA-F][0-9a-fA-F]))' # or allowed unicode bytes + r'[^\]\[\(\)<>"\'\s]+', # stop parsing at these symbols + re.IGNORECASE, +) + +COLOR_REGEX = re.compile(r'\[(?P\d+)(;(?P\d+)(;(?P\d+))?)?m') + +def is_static_file(url: str): + # TODO: the proper way is with MIME type detection + ext, not only extension + from .config import STATICFILE_EXTENSIONS + return extension(url).lower() in STATICFILE_EXTENSIONS + + +def enforce_types(func): + """ + Enforce function arg and kwarg types at runtime using its python3 type hints + """ + # TODO: check return type as well + + @wraps(func) + def typechecked_function(*args, **kwargs): + sig = signature(func) + + def check_argument_type(arg_key, arg_val): + try: + annotation = sig.parameters[arg_key].annotation + except KeyError: + annotation = None + + if annotation is not None and annotation.__class__ is type: + if not isinstance(arg_val, annotation): + raise TypeError( + '{}(..., {}: {}) got unexpected {} argument {}={}'.format( + func.__name__, + arg_key, + annotation.__name__, + type(arg_val).__name__, + arg_key, + str(arg_val)[:64], + ) + ) + + # check args + for arg_val, arg_key in zip(args, sig.parameters): + check_argument_type(arg_key, arg_val) + + # check kwargs + for arg_key, arg_val in kwargs.items(): + check_argument_type(arg_key, arg_val) + + return func(*args, **kwargs) + + return typechecked_function + + +def docstring(text: Optional[str]): + """attach the given docstring to the decorated function""" + def decorator(func): + if text: + func.__doc__ = text + return func + return decorator + + +@enforce_types +def str_between(string: str, start: str, end: str=None) -> str: + """(12345, , ) -> 12345""" + + content = string.split(start, 1)[-1] + if end is not None: + content = content.rsplit(end, 1)[0] + + return content + + +@enforce_types +def parse_date(date: Any) -> Optional[datetime]: + """Parse unix timestamps, iso format, and human-readable strings""" + + if date is None: + return None + + if isinstance(date, datetime): + return date + + if isinstance(date, (float, int)): + date = str(date) + + if isinstance(date, str): + return dateparser(date) + + raise ValueError('Tried to parse invalid date! {}'.format(date)) + + +@enforce_types +def download_url(url: str, timeout: int=None) -> str: + """Download the contents of a remote url and return the text""" + from .config import TIMEOUT, CHECK_SSL_VALIDITY, WGET_USER_AGENT + timeout = timeout or TIMEOUT + response = requests.get( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + ) + + content_type = response.headers.get('Content-Type', '') + encoding = http_content_type_encoding(content_type) or html_body_declared_encoding(response.text) + + if encoding is not None: + response.encoding = encoding + + return response.text + +@enforce_types +def get_headers(url: str, timeout: int=None) -> str: + """Download the contents of a remote url and return the headers""" + from .config import TIMEOUT, CHECK_SSL_VALIDITY, WGET_USER_AGENT + timeout = timeout or TIMEOUT + + try: + response = requests.head( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + allow_redirects=True, + ) + if response.status_code >= 400: + raise RequestException + except ReadTimeout: + raise + except RequestException: + response = requests.get( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + stream=True + ) + + return pyjson.dumps(dict(response.headers), indent=4) + + +@enforce_types +def chrome_args(**options) -> List[str]: + """helper to build up a chrome shell command with arguments""" + + from .config import CHROME_OPTIONS + + options = {**CHROME_OPTIONS, **options} + + cmd_args = [options['CHROME_BINARY']] + + if options['CHROME_HEADLESS']: + cmd_args += ('--headless',) + + if not options['CHROME_SANDBOX']: + # assume this means we are running inside a docker container + # in docker, GPU support is limited, sandboxing is unecessary, + # and SHM is limited to 64MB by default (which is too low to be usable). + cmd_args += ( + '--no-sandbox', + '--disable-gpu', + '--disable-dev-shm-usage', + '--disable-software-rasterizer', + ) + + + if not options['CHECK_SSL_VALIDITY']: + cmd_args += ('--disable-web-security', '--ignore-certificate-errors') + + if options['CHROME_USER_AGENT']: + cmd_args += ('--user-agent={}'.format(options['CHROME_USER_AGENT']),) + + if options['RESOLUTION']: + cmd_args += ('--window-size={}'.format(options['RESOLUTION']),) + + if options['TIMEOUT']: + cmd_args += ('--timeout={}'.format((options['TIMEOUT']) * 1000),) + + if options['CHROME_USER_DATA_DIR']: + cmd_args.append('--user-data-dir={}'.format(options['CHROME_USER_DATA_DIR'])) + + return cmd_args + + +def ansi_to_html(text): + """ + Based on: https://stackoverflow.com/questions/19212665/python-converting-ansi-color-codes-to-html + """ + from .config import COLOR_DICT + + TEMPLATE = '
    ' + text = text.replace('[m', '    ') + + def single_sub(match): + argsdict = match.groupdict() + if argsdict['arg_3'] is None: + if argsdict['arg_2'] is None: + _, color = 0, argsdict['arg_1'] + else: + _, color = argsdict['arg_1'], argsdict['arg_2'] + else: + _, color = argsdict['arg_3'], argsdict['arg_2'] + + return TEMPLATE.format(COLOR_DICT[color][0]) + + return COLOR_REGEX.sub(single_sub, text) + + +class AttributeDict(dict): + """Helper to allow accessing dict values via Example.key or Example['key']""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # Recursively convert nested dicts to AttributeDicts (optional): + # for key, val in self.items(): + # if isinstance(val, dict) and type(val) is not AttributeDict: + # self[key] = AttributeDict(val) + + def __getattr__(self, attr: str) -> Any: + return dict.__getitem__(self, attr) + + def __setattr__(self, attr: str, value: Any) -> None: + return dict.__setitem__(self, attr, value) + + +class ExtendedEncoder(pyjson.JSONEncoder): + """ + Extended json serializer that supports serializing several model + fields and objects + """ + + def default(self, obj): + cls_name = obj.__class__.__name__ + + if hasattr(obj, '_asdict'): + return obj._asdict() + + elif isinstance(obj, bytes): + return obj.decode() + + elif isinstance(obj, datetime): + return obj.isoformat() + + elif isinstance(obj, Exception): + return '{}: {}'.format(obj.__class__.__name__, obj) + + elif isinstance(obj, Path): + return str(obj) + + elif cls_name in ('dict_items', 'dict_keys', 'dict_values'): + return tuple(obj) + + return pyjson.JSONEncoder.default(self, obj) + diff --git a/archivebox-0.5.3/build/lib/archivebox/vendor/__init__.py b/archivebox-0.5.3/build/lib/archivebox/vendor/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/debian/.debhelper/generated/archivebox/installed-by-dh_installdocs b/archivebox-0.5.3/debian/.debhelper/generated/archivebox/installed-by-dh_installdocs new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/debian/archivebox.debhelper.log b/archivebox-0.5.3/debian/archivebox.debhelper.log new file mode 100644 index 0000000..de33f70 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox.debhelper.log @@ -0,0 +1,19 @@ +dh_update_autotools_config +dh_auto_configure +dh_auto_build +dh_auto_test +dh_prep +dh_auto_install +dh_installdocs +dh_installchangelogs +dh_installinit +dh_perl +dh_link +dh_strip_nondeterminism +dh_compress +dh_fixperms +dh_missing +dh_installdeb +dh_gencontrol +dh_md5sums +dh_builddeb diff --git a/archivebox-0.5.3/debian/archivebox.postinst.debhelper b/archivebox-0.5.3/debian/archivebox.postinst.debhelper new file mode 100644 index 0000000..2c9b172 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox.postinst.debhelper @@ -0,0 +1,10 @@ + +# Automatically added by dh_python3: +if which py3compile >/dev/null 2>&1; then + py3compile -p archivebox +fi +if which pypy3compile >/dev/null 2>&1; then + pypy3compile -p archivebox || true +fi + +# End automatically added section diff --git a/archivebox-0.5.3/debian/archivebox.prerm.debhelper b/archivebox-0.5.3/debian/archivebox.prerm.debhelper new file mode 100644 index 0000000..282f72d --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox.prerm.debhelper @@ -0,0 +1,10 @@ + +# Automatically added by dh_python3: +if which py3clean >/dev/null 2>&1; then + py3clean -p archivebox +else + dpkg -L archivebox | perl -ne 's,/([^/]*)\.py$,/__pycache__/\1.*, or next; unlink $_ or die $! foreach glob($_)' + find /usr/lib/python3/dist-packages/ -type d -name __pycache__ -empty -print0 | xargs --null --no-run-if-empty rmdir +fi + +# End automatically added section diff --git a/archivebox-0.5.3/debian/archivebox.substvars b/archivebox-0.5.3/debian/archivebox.substvars new file mode 100644 index 0000000..914fa79 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox.substvars @@ -0,0 +1,3 @@ +python3:Depends=python3-atomicwrites, python3-croniter, python3-crontab, python3-dateparser, python3-django, python3-django-extensions, python3-ipython, python3-mypy-extensions, python3-requests, python3-w3lib, python3:any, youtube-dl +misc:Depends= +misc:Pre-Depends= diff --git a/archivebox-0.5.3/debian/archivebox/DEBIAN/control b/archivebox-0.5.3/debian/archivebox/DEBIAN/control new file mode 100644 index 0000000..a03f45b --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/DEBIAN/control @@ -0,0 +1,30 @@ +Package: archivebox +Version: 0.5.3-1 +Architecture: all +Maintainer: Nick Sweeting +Installed-Size: 938 +Depends: python3-atomicwrites, python3-croniter, python3-crontab, python3-dateparser, python3-django, python3-django-extensions, python3-ipython, python3-mypy-extensions, python3-requests, python3-w3lib, python3:any, youtube-dl, nodejs, chromium-browser, wget, curl, git, ffmpeg, python3-django-jsonfield, ripgrep +Section: python +Priority: optional +Homepage: https://github.com/ArchiveBox/ArchiveBox +Description: The self-hosted internet archive. +
    + +

    ArchiveBox
    The open-source self-hosted web archive.

    + . + ▶️ Quickstart | + Demo | + Github | + Documentation | + Info & Motivation | + Community | + Roadmap + . + 
    + "Your own personal internet archive" (网站存档 / 爬虫)
+
    + . + + . + + diff --git a/archivebox-0.5.3/debian/archivebox/DEBIAN/md5sums b/archivebox-0.5.3/debian/archivebox/DEBIAN/md5sums new file mode 100644 index 0000000..2f1db73 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/DEBIAN/md5sums @@ -0,0 +1,126 @@ +20e0eff07a6f27bb640067c750a3b328 usr/bin/archivebox +b5ad0d3290ff6a43ce15685c2bdeb420 usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/PKG-INFO +68b329da9893e34099c7d8ad5cb9c940 usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/dependency_links.txt +5b15ee07b1a58d2f77efc56c47c66c01 usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/entry_points.txt +10c6a7ca4e7ce3913366a1d3a5ec7635 usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/requires.txt +e65a9b928b38091ffd6a9142efc76d7f usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/top_level.txt +a2f11497252da68a947de7436621ebb4 usr/lib/python3/dist-packages/archivebox/.flake8 +688591e55f2399c2dccca18dc133adc7 usr/lib/python3/dist-packages/archivebox/LICENSE +ca7d59f8588de5f91eed226dd5269833 usr/lib/python3/dist-packages/archivebox/README.md +05cc08c4f7a16eb7b3034371e81ab34e usr/lib/python3/dist-packages/archivebox/__init__.py +2044702060327d07cf4b394f1ee8f004 usr/lib/python3/dist-packages/archivebox/__main__.py +215ae089a12c854f4c5cf54c3537c0dc usr/lib/python3/dist-packages/archivebox/cli/__init__.py +a0235679ec347b17a4da2b7de46058b5 usr/lib/python3/dist-packages/archivebox/cli/archivebox_add.py +4c7a0dbbe6427e3dfe72c71b1f06e105 usr/lib/python3/dist-packages/archivebox/cli/archivebox_config.py +dea57a0d33a8f589017f384b57caf915 usr/lib/python3/dist-packages/archivebox/cli/archivebox_help.py +542e7fd0b458d2d0ebecb9e0054e2476 usr/lib/python3/dist-packages/archivebox/cli/archivebox_init.py +140bffcb8c334f9926df3700fc3a7b3c usr/lib/python3/dist-packages/archivebox/cli/archivebox_list.py +c0dae677c828d80ef6055df7590eb7b1 usr/lib/python3/dist-packages/archivebox/cli/archivebox_manage.py +a76b9bb7a4db20dd1f06cd713ed19882 usr/lib/python3/dist-packages/archivebox/cli/archivebox_oneshot.py +26b9498e1438087df04ddf8769d3ecdb usr/lib/python3/dist-packages/archivebox/cli/archivebox_remove.py +81b171b3cbc2b4b519854610bd4eda28 usr/lib/python3/dist-packages/archivebox/cli/archivebox_schedule.py +217edd5037476e0e435d586f30cdd87f usr/lib/python3/dist-packages/archivebox/cli/archivebox_server.py +75e28b47f4a314a4ad3b45e3b2d2372f usr/lib/python3/dist-packages/archivebox/cli/archivebox_shell.py +abe650a38c3ca70f361b60cac752100a usr/lib/python3/dist-packages/archivebox/cli/archivebox_status.py +ef3c2139225b3e1721384ba8378711d0 usr/lib/python3/dist-packages/archivebox/cli/archivebox_update.py +70baac1da5d511ad1168d85024460cfd usr/lib/python3/dist-packages/archivebox/cli/archivebox_version.py +60a8430087b9ab81e9c575d79869b5bf usr/lib/python3/dist-packages/archivebox/cli/tests.py +025f92c65e10ba509dc8895ca314f67e usr/lib/python3/dist-packages/archivebox/config.py +c374043c2f120c86275ec8c3374bdd7d usr/lib/python3/dist-packages/archivebox/config_stubs.py +7eda7ac9e0c4c4451faf49cf133fd49d usr/lib/python3/dist-packages/archivebox/core/__init__.py +c3c7c8f948eddd54e529d4f816c51659 usr/lib/python3/dist-packages/archivebox/core/admin.py +3416954d5516f91f182e1b0424f0e42e usr/lib/python3/dist-packages/archivebox/core/apps.py +cf16db9fb86e9cda3fcc837f425df114 usr/lib/python3/dist-packages/archivebox/core/forms.py +06b6ed7e1e2a6534697da15278638bc9 usr/lib/python3/dist-packages/archivebox/core/management/commands/archivebox.py +11863b1f400e6f3355fe6fc510758fb1 usr/lib/python3/dist-packages/archivebox/core/migrations/0001_initial.py +4725a5d030235d794cf6376b745e1b91 usr/lib/python3/dist-packages/archivebox/core/migrations/0002_auto_20200625_1521.py +d1d0c9fb198fa8cb65b9139af1287463 usr/lib/python3/dist-packages/archivebox/core/migrations/0003_auto_20200630_1034.py +9b0a3b9a6d15cdd8570be9d3279b3980 usr/lib/python3/dist-packages/archivebox/core/migrations/0004_auto_20200713_1552.py +0c7dbc9ff489bcb7d30e6e75472db1ff usr/lib/python3/dist-packages/archivebox/core/migrations/0005_auto_20200728_0326.py +5c61293713417e23dced052c7c1179dd usr/lib/python3/dist-packages/archivebox/core/migrations/0006_auto_20201012_1520.py +2515235c69e97ef56a2cf841eda2e4d5 usr/lib/python3/dist-packages/archivebox/core/migrations/0007_archiveresult.py +527ec8a2a550cd46b2665459c5135fd3 usr/lib/python3/dist-packages/archivebox/core/migrations/0008_auto_20210105_1421.py +d41d8cd98f00b204e9800998ecf8427e usr/lib/python3/dist-packages/archivebox/core/migrations/__init__.py +fd0a90ed401f1a88cef4b0a17d2a61da usr/lib/python3/dist-packages/archivebox/core/mixins.py +b5206bcea85425e9061ebb38e2775a99 usr/lib/python3/dist-packages/archivebox/core/models.py +ef513d4b93017ff1e2b8123494596134 usr/lib/python3/dist-packages/archivebox/core/settings.py +d41d8cd98f00b204e9800998ecf8427e usr/lib/python3/dist-packages/archivebox/core/templatetags/__init__.py +96c3038e9d1b691917163c51cb8278b5 usr/lib/python3/dist-packages/archivebox/core/templatetags/core_tags.py +7921097fb98105f70ed684323aedfaa2 usr/lib/python3/dist-packages/archivebox/core/tests.py +09145de4c13f82f60cc1b3d85f3093b0 usr/lib/python3/dist-packages/archivebox/core/urls.py +7fb664948dc7792a44209fe146631b24 usr/lib/python3/dist-packages/archivebox/core/views.py +42c5c88dde758dc2b236480e0ad72538 usr/lib/python3/dist-packages/archivebox/core/welcome_message.py +f0c203511cfb622d117ead1efdfb8453 usr/lib/python3/dist-packages/archivebox/core/wsgi.py +084793aecfc2e145c566dafca9a5caf6 usr/lib/python3/dist-packages/archivebox/extractors/__init__.py +83c409be44edf3266bdb34f31f51177c usr/lib/python3/dist-packages/archivebox/extractors/archive_org.py +7d6f2948c16817f000f232047981b2ba usr/lib/python3/dist-packages/archivebox/extractors/dom.py +b3a7de648a59fe2bf23b99666ab81da3 usr/lib/python3/dist-packages/archivebox/extractors/favicon.py +55bb4fa43d994f9bb5127e532be42b2b usr/lib/python3/dist-packages/archivebox/extractors/git.py +e3938e2aed4a7661b1d91091b019af77 usr/lib/python3/dist-packages/archivebox/extractors/headers.py +70137ea70959bd68278c912618d69cd7 usr/lib/python3/dist-packages/archivebox/extractors/media.py +4901e04ffbb39b93c9a184ef8cbafb69 usr/lib/python3/dist-packages/archivebox/extractors/mercury.py +444e061240cba77435ebadb4b504e576 usr/lib/python3/dist-packages/archivebox/extractors/pdf.py +0761c38461ef6c1a4fe47d5b0b9f1584 usr/lib/python3/dist-packages/archivebox/extractors/readability.py +45cb0ccad39c46fea47a6161dbad349e usr/lib/python3/dist-packages/archivebox/extractors/screenshot.py +88f0dcb2a274affaceacd00f54b2f142 usr/lib/python3/dist-packages/archivebox/extractors/singlefile.py +e5287f7969e4e3e27b661002a14c1be6 usr/lib/python3/dist-packages/archivebox/extractors/title.py +2d333caa3aa0ed9042d74517a5653446 usr/lib/python3/dist-packages/archivebox/extractors/wget.py +36be113ed8d578727cd233bbd7a7cf22 usr/lib/python3/dist-packages/archivebox/index/__init__.py +c0ed6a7ef5e8e047ec9ac1a3f976adac usr/lib/python3/dist-packages/archivebox/index/csv.py +20f44840be4044870b9b636c7a677840 usr/lib/python3/dist-packages/archivebox/index/html.py +100604e5708884f50fdc900fd9052462 usr/lib/python3/dist-packages/archivebox/index/json.py +9cb33b270d2c5eac1426219cad522322 usr/lib/python3/dist-packages/archivebox/index/schema.py +fda86ceb8bbb688f54e4fb7e93542ab5 usr/lib/python3/dist-packages/archivebox/index/sql.py +b4f1a7b7a6432d6b321f6709da1befea usr/lib/python3/dist-packages/archivebox/logging_util.py +23d467a209c2382e9da7fcdba9200add usr/lib/python3/dist-packages/archivebox/main.py +9a36bcd681908fe369426f723caae82a usr/lib/python3/dist-packages/archivebox/manage.py +7f29e21daf2f7670c7c39b328d1b54a3 usr/lib/python3/dist-packages/archivebox/mypy.ini +525ea6ae9b6e2017d6965eb95f096cbf usr/lib/python3/dist-packages/archivebox/package.json +45a1cc9fdc8ab57be7d8f757fa56cf71 usr/lib/python3/dist-packages/archivebox/parsers/__init__.py +29e35b556cec2b5655e4f73acf270559 usr/lib/python3/dist-packages/archivebox/parsers/generic_html.py +de65c73fa7fe6c41ac274beb7698212d usr/lib/python3/dist-packages/archivebox/parsers/generic_json.py +c530a159067c31d915dac2a42d78446a usr/lib/python3/dist-packages/archivebox/parsers/generic_rss.py +f94362cf08c9b74defaf61fb5e5d2ee8 usr/lib/python3/dist-packages/archivebox/parsers/generic_txt.py +adb1220adb7c13da1f35798f01ce228a usr/lib/python3/dist-packages/archivebox/parsers/medium_rss.py +7122155a71ef7ef9bea823093226b278 usr/lib/python3/dist-packages/archivebox/parsers/netscape_html.py +6a6acb970fdec2d7183a531deb89d722 usr/lib/python3/dist-packages/archivebox/parsers/pinboard_rss.py +68df240cd37791a1bd6d667c7cf220df usr/lib/python3/dist-packages/archivebox/parsers/pocket_api.py +5e923efe79b067a923d5b6411818e01f usr/lib/python3/dist-packages/archivebox/parsers/pocket_html.py +477be61f390a0d748550fabcac0cc69f usr/lib/python3/dist-packages/archivebox/parsers/shaarli_rss.py +2f298d7f55581a053a1ad4d8c475af3b usr/lib/python3/dist-packages/archivebox/parsers/wallabag_atom.py +7c9d41d94ce0a9154c05aefbdf9571a5 usr/lib/python3/dist-packages/archivebox/search/__init__.py +d41d8cd98f00b204e9800998ecf8427e usr/lib/python3/dist-packages/archivebox/search/backends/__init__.py +4db03e517d549405522a62eee08b8b4e usr/lib/python3/dist-packages/archivebox/search/backends/ripgrep.py +8c8afad7576dd4b95d5338536035b3e1 usr/lib/python3/dist-packages/archivebox/search/backends/sonic.py +1ea8f355301d159b7ea495c6e9ccc109 usr/lib/python3/dist-packages/archivebox/search/utils.py +2b3355691067f09a330c278ebacc1a1d usr/lib/python3/dist-packages/archivebox/system.py +ded98b773616e1d57d2bb406888e932f usr/lib/python3/dist-packages/archivebox/themes/admin/actions_as_select.html +9eefc8fb5d749b8289fa3d92a41a9a92 usr/lib/python3/dist-packages/archivebox/themes/admin/app_index.html +5b15de2865d2c0fed6324c44926db706 usr/lib/python3/dist-packages/archivebox/themes/admin/base.html +e4cad9c3534c9af3b515e77c939cdd44 usr/lib/python3/dist-packages/archivebox/themes/admin/grid_change_list.html +03db44427a828c825d8f8946d34f0ebb usr/lib/python3/dist-packages/archivebox/themes/admin/login.html +3efdef6e81478d298d2ac6e85eb7924f usr/lib/python3/dist-packages/archivebox/themes/admin/snapshots_grid.html +49792143a5c2a661dca41a59f1cd38f5 usr/lib/python3/dist-packages/archivebox/themes/default/add_links.html +a244220de67e4352540792b13d0c4c8c usr/lib/python3/dist-packages/archivebox/themes/default/base.html +7cc13d785bd8578482eb215ecd93f1eb usr/lib/python3/dist-packages/archivebox/themes/default/core/snapshot_list.html +da7c05931cc57d74f2ef783d59ce8a91 usr/lib/python3/dist-packages/archivebox/themes/default/link_details.html +8dcc7de9cd396dfcdf81f0063d693d1d usr/lib/python3/dist-packages/archivebox/themes/default/main_index.html +4a08c4e5893a6b1f6fd04e361daaca34 usr/lib/python3/dist-packages/archivebox/themes/default/main_index_minimal.html +756501b93ead9d6c8d1d2ba1418dd634 usr/lib/python3/dist-packages/archivebox/themes/default/main_index_row.html +6afe60e785b4031a7b9be5592333a7e0 usr/lib/python3/dist-packages/archivebox/themes/default/static/add.css +d2a4769f3d03c6d08ed6c91e282d4477 usr/lib/python3/dist-packages/archivebox/themes/default/static/admin.css +34ad7059ab0170cb762daba9ac253412 usr/lib/python3/dist-packages/archivebox/themes/default/static/archive.png +7e923ad223e9f33e54d22e50cf2bcce5 usr/lib/python3/dist-packages/archivebox/themes/default/static/bootstrap.min.css +49edc22ce56c60c9655b77b8319a7676 usr/lib/python3/dist-packages/archivebox/themes/default/static/external.png +4150ca19a74adc6d9e1d3a5417f2ad6d usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.css +97fd6a774fc6211e7619aca9a61ca804 usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.js +a09e13ee94d51c524b7e2a728c7d4039 usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.min.js +a8dcf333411ca3c52a00e878cbb42e86 usr/lib/python3/dist-packages/archivebox/themes/default/static/sort_asc.png +9a6486086d09bb38cf66a57cc559ade3 usr/lib/python3/dist-packages/archivebox/themes/default/static/sort_both.png +47cd34ea64b71baf3a3e6dee987fc8a9 usr/lib/python3/dist-packages/archivebox/themes/default/static/sort_desc.png +fd31a4f9bc95400457135161abd1760f usr/lib/python3/dist-packages/archivebox/themes/default/static/spinner.gif +d24ab23be1f33c036cb955ea1485bdd1 usr/lib/python3/dist-packages/archivebox/themes/legacy/main_index.html +05eb8d3747632c1f952113afcd4af0b6 usr/lib/python3/dist-packages/archivebox/themes/legacy/main_index_row.html +8ce539505f0c16b82c1d3f09785af98d usr/lib/python3/dist-packages/archivebox/util.py +d41d8cd98f00b204e9800998ecf8427e usr/lib/python3/dist-packages/archivebox/vendor/__init__.py +157fd3564470d9676e2dce3187864c4b usr/share/doc/archivebox/changelog.Debian.gz diff --git a/archivebox-0.5.3/debian/archivebox/DEBIAN/postinst b/archivebox-0.5.3/debian/archivebox/DEBIAN/postinst new file mode 100755 index 0000000..875e3ad --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/DEBIAN/postinst @@ -0,0 +1,12 @@ +#!/bin/sh +set -e + +# Automatically added by dh_python3: +if which py3compile >/dev/null 2>&1; then + py3compile -p archivebox +fi +if which pypy3compile >/dev/null 2>&1; then + pypy3compile -p archivebox || true +fi + +# End automatically added section diff --git a/archivebox-0.5.3/debian/archivebox/DEBIAN/prerm b/archivebox-0.5.3/debian/archivebox/DEBIAN/prerm new file mode 100755 index 0000000..7ab7bae --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/DEBIAN/prerm @@ -0,0 +1,12 @@ +#!/bin/sh +set -e + +# Automatically added by dh_python3: +if which py3clean >/dev/null 2>&1; then + py3clean -p archivebox +else + dpkg -L archivebox | perl -ne 's,/([^/]*)\.py$,/__pycache__/\1.*, or next; unlink $_ or die $! foreach glob($_)' + find /usr/lib/python3/dist-packages/ -type d -name __pycache__ -empty -print0 | xargs --null --no-run-if-empty rmdir +fi + +# End automatically added section diff --git a/archivebox-0.5.3/debian/archivebox/usr/bin/archivebox b/archivebox-0.5.3/debian/archivebox/usr/bin/archivebox new file mode 100755 index 0000000..83c8b0b --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/bin/archivebox @@ -0,0 +1,12 @@ +#! /usr/bin/python3 +# EASY-INSTALL-ENTRY-SCRIPT: 'archivebox==0.5.3','console_scripts','archivebox' +__requires__ = 'archivebox==0.5.3' +import re +import sys +from pkg_resources import load_entry_point + +if __name__ == '__main__': + sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) + sys.exit( + load_entry_point('archivebox==0.5.3', 'console_scripts', 'archivebox')() + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/PKG-INFO b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/PKG-INFO new file mode 100644 index 0000000..b6534de --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/PKG-INFO @@ -0,0 +1,591 @@ +Metadata-Version: 2.1 +Name: archivebox +Version: 0.5.3 +Summary: The self-hosted internet archive. +Home-page: https://github.com/ArchiveBox/ArchiveBox +Author: Nick Sweeting +Author-email: git@nicksweeting.com +License: MIT +Project-URL: Source, https://github.com/ArchiveBox/ArchiveBox +Project-URL: Documentation, https://github.com/ArchiveBox/ArchiveBox/wiki +Project-URL: Bug Tracker, https://github.com/ArchiveBox/ArchiveBox/issues +Project-URL: Changelog, https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog +Project-URL: Roadmap, https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap +Project-URL: Community, https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community +Project-URL: Donate, https://github.com/ArchiveBox/ArchiveBox/wiki/Donations +Description:
    + +

    ArchiveBox
    The open-source self-hosted web archive.

    + + ▶️ Quickstart | + Demo | + Github | + Documentation | + Info & Motivation | + Community | + Roadmap + + 
    +        "Your own personal internet archive" (网站存档 / 爬虫)
+
    + + + + + + + + + + +
    +
    + + ArchiveBox is a powerful self-hosted internet archiving solution written in Python 3. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on the configuration and the content it detects. + + Your archive can be managed through the command line with commands like `archivebox add`, through the built-in Web UI `archivebox server`, or via the Python library API (beta). It can ingest bookmarks from a browser or service like Pocket/Pinboard, your entire browsing history, RSS feeds, or URLs one at a time. You can also schedule regular/realtime imports with `archivebox schedule`. + + The main index is a self-contained `index.sqlite3` file, and each snapshot is stored as a folder `data/archive//`, with an easy-to-read `index.html` and `index.json` within. For each page, ArchiveBox auto-extracts many types of assets/media and saves them in standard formats, with out-of-the-box support for: several types of HTML snapshots (wget, Chrome headless, singlefile), PDF snapshotting, screenshotting, WARC archiving, git repositories, images, audio, video, subtitles, article text, and more. The snapshots are browseable and managable offline through the filesystem, the built-in webserver, or the Python library API. + + ### Quickstart + + It works on Linux/BSD (Intel and ARM CPUs with `docker`/`apt`/`pip3`), macOS (with `docker`/`brew`/`pip3`), and Windows (beta with `docker`/`pip3`). + + ```bash + pip3 install archivebox + archivebox --version + # install extras as-needed, or use one of full setup methods below to get everything out-of-the-box + + mkdir ~/archivebox && cd ~/archivebox # this can be anywhere + archivebox init + + archivebox add 'https://example.com' + archivebox add --depth=1 'https://example.com' + archivebox schedule --every=day https://getpocket.com/users/USERNAME/feed/all + archivebox oneshot --extract=title,favicon,media https://www.youtube.com/watch?v=dQw4w9WgXcQ + archivebox help # to see more options + ``` + + *(click to expand the sections below for full setup instructions)* + +
    + Get ArchiveBox with docker-compose on any platform (recommended, everything included out-of-the-box) + + First make sure you have Docker installed: https://docs.docker.com/get-docker/ +

    + This is the recommended way to run ArchiveBox because it includes *all* the extractors like chrome, wget, youtube-dl, git, etc., as well as full-text search with sonic, and many other great features. + + ```bash + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml + docker-compose run archivebox init + docker-compose run archivebox --version + + # start the webserver and open the UI (optional) + docker-compose run archivebox manage createsuperuser + docker-compose up -d + open http://127.0.0.1:8000 + + # you can also add links and manage your archive via the CLI: + docker-compose run archivebox add 'https://example.com' + docker-compose run archivebox status + docker-compose run archivebox help # to see more options + ``` + +
    + +
    + Get ArchiveBox with docker on any platform + + First make sure you have Docker installed: https://docs.docker.com/get-docker/
    + ```bash + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + docker run -v $PWD:/data -it archivebox/archivebox init + docker run -v $PWD:/data -it archivebox/archivebox --version + + # start the webserver and open the UI (optional) + docker run -v $PWD:/data -it archivebox/archivebox manage createsuperuser + docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add links and manage your archive via the CLI: + docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com' + docker run -v $PWD:/data -it archivebox/archivebox status + docker run -v $PWD:/data -it archivebox/archivebox help # to see more options + ``` + +
    + +
    + Get ArchiveBox with apt on Ubuntu >=20.04 + + ```bash + sudo add-apt-repository -u ppa:archivebox/archivebox + sudo apt install archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + + For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`: + ```bash + deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main + deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main + ``` + (you may need to install some other dependencies manually however) + +
    + +
    + Get ArchiveBox with brew on macOS >=10.13 + + ```bash + brew install archivebox/archivebox/archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + +
    + +
    + Get ArchiveBox with pip on any platform + + ```bash + pip3 install archivebox + + # create a new empty directory and initalize your collection (can be anywhere) + mkdir ~/archivebox && cd ~/archivebox + npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' + archivebox init + archivebox --version + # Install any missing extras like wget/git/chrome/etc. manually as needed + + # start the webserver and open the web UI (optional) + archivebox manage createsuperuser + archivebox server 0.0.0.0:8000 + open http://127.0.0.1:8000 + + # you can also add URLs and manage the archive via the CLI and filesystem: + archivebox add 'https://example.com' + archivebox status + archivebox list --html --with-headers > index.html + archivebox list --json --with-headers > index.json + archivebox help # to see more options + ``` + +
    + + --- + +
    + +
    + + DEMO: archivebox.zervice.io/ + For more information, see the full Quickstart guide, Usage, and Configuration docs. +
    + + --- + + + # Overview + + ArchiveBox is a command line tool, self-hostable web-archiving server, and Python library all-in-one. It can be installed on Docker, macOS, and Linux/BSD, and Windows. You can download and install it as a Debian/Ubuntu package, Homebrew package, Python3 package, or a Docker image. No matter which install method you choose, they all provide the same CLI, Web UI, and on-disk data format. + + To use ArchiveBox you start by creating a folder for your data to live in (it can be anywhere on your system), and running `archivebox init` inside of it. That will create a sqlite3 index and an `ArchiveBox.conf` file. After that, you can continue to add/export/manage/etc using the CLI `archivebox help`, or you can run the Web UI (recommended). If you only want to archive a single site, you can run `archivebox oneshot` to avoid having to create a whole collection. + + The CLI is considered "stable", the ArchiveBox Python API and REST APIs are "beta", and the [desktop app](https://github.com/ArchiveBox/desktop) is "alpha". + + At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer). You can also self-host your archivebox server on a public domain to provide archive.org-style public access to your site snapshots. + +
    + CLI Screenshot + Desktop index screenshot + Desktop details page Screenshot + Desktop details page Screenshot
    + Demo | Usage | Screenshots +
    + . . . . . . . . . . . . . . . . . . . . . . . . . . . . +

    + + + ## Key Features + + - [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally + - [**Few dependencies**](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) + - [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - Easy to set up **[scheduled importing](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources** + - Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC + - ~~**Suitable for paywalled / [authenticated content](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes) + - **Doesn't require a constantly-running daemon**, proxy, or native app + - Provides a CLI, Python API, self-hosted web UI, and REST API (WIP) + - Architected to be able to run [**many varieties of scripts during archiving**](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. to extract media, summarize articles, [scroll pages](https://github.com/ArchiveBox/ArchiveBox/issues/80), [close modals](https://github.com/ArchiveBox/ArchiveBox/issues/175), expand comment threads, etc. + - Can also [**mirror content to 3rd-party archiving services**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy + + ## Input formats + + ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more! + + ```bash + echo 'http://example.com' | archivebox add + archivebox add 'https://example.com/some/page' + archivebox add < ~/Downloads/firefox_bookmarks_export.html + archivebox add < any_text_with_urls_in_it.txt + archivebox add --depth=1 'https://example.com/some/downloads.html' + archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12' + ``` + + - Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more) + - RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format + - Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more + + See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples. + + It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand. + + ## Output formats + + All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`. + + The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.) + + ```bash + ls ./archive// + ``` + + - **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details + - **Title:** `title` title of the site + - **Favicon:** `favicon.ico` favicon of the site + - **Headers:** `headers.json` Any HTTP headers the site returns are saved in a json file + - **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile + - **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present + - **WARC:** `warc/.gz` gzipped WARC of all the resources fetched while archiving + - **PDF:** `output.pdf` Printed PDF of site using headless chrome + - **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome + - **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome + - **Readability:** `article.html/json` Article text extraction using Readability + - **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org + - **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl + - **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links + - _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._ + + It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file. + + ## Dependencies + + You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled. + + If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) or the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install). + + ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more. + + ## Caveats + + If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process. + ```bash + # don't do this: + archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere' + archivebox add 'https://example.com/any/url/you/want/to/keep/secret/' + + # without first disabling share the URL with 3rd party APIs: + archivebox config --set SAVE_ARCHIVE_DOT_ORG=False # disable saving all URLs in Archive.org + archivebox config --set SAVE_FAVICON=False # optional: only the domain is leaked, not full URL + archivebox config --set CHROME_BINARY=chromium # optional: switch to chromium to avoid Chrome phoning home to Google + ``` + + Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details. + ```bash + # visiting an archived page with malicious JS: + https://127.0.0.1:8000/archive/1602401954/example.com/index.html + + # example.com/index.js can now make a request to read everything: + https://127.0.0.1:8000/index.html + https://127.0.0.1:8000/archive/* + # then example.com/index.js can send it off to some evil server + ``` + + Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash: + ```bash + archivebox add 'https://example.com#2020-10-24' + ... + archivebox add 'https://example.com#2020-10-25' + ``` + + --- + +
    + +
    + + --- + + # Background & Motivation + + Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity. + + Whether it's to resist censorship by saving articles before they get taken down or edited, or + just to save a collection of early 2010's flash games you love to play, having the tools to + archive internet content enables to you save the stuff you care most about before it disappears. + +
    +
    + Image from WTF is Link Rot?...
     +
    + + The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. + I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about. + + Because modern websites are complicated and often rely on dynamic content, + ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats. + + All the archived links are stored by date bookmarked in `./archive/`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM. + + ## Comparison to Other Projects + + ▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.** + + comparison The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations. + + #### User Interface & Intended Purpose + + ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend. + + #### Private Local Archives vs Centralized Public Archives + + Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle. + + #### Storage Requirements + + Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files. + + ## Learn more + + Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web! + + + + - [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists) + _Community-maintained indexes of archiving tools and institutions._ + - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) + _Open source tools and projects in the internet archiving space._ + - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List) + _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._ + - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities) + _A collection of the most active internet archiving communities and initiatives._ + - Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) + - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post. + - Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter. + + --- + + # Documentation + + + + We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation. + + You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder. + + ## Getting Started + + - [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) + - [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) + - [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + + ## Reference + + - [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) + - [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) + - [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) + - [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site) + - [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) + - [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) + - [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) + - [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) + - [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) + - [Python API](https://docs.archivebox.io/en/latest/modules.html) + - REST API (coming soon...) + + ## More Info + + - [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues) + - [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) + - [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) + - [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) + - [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation) + - [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + + --- + + # ArchiveBox Development + + All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap. + + ### Setup the dev environment + + First, install the system dependencies from the "Bare Metal" section above. + Then you can clone the ArchiveBox repo and install + ```python3 + git clone https://github.com/ArchiveBox/ArchiveBox && cd ArchiveBox + git checkout master # or the branch you want to test + git submodule update --init --recursive + git pull --recurse-submodules + + # Install ArchiveBox + python dependencies + python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev] + # or with pipenv: pipenv install --dev && pipenv shell + + # Install node dependencies + npm install + + # Optional: install extractor dependencies manually or with helper script + ./bin/setup.sh + + # Optional: develop via docker by mounting the code dir into the container + # if you edit e.g. ./archivebox/core/models.py on the docker host, runserver + # inside the container will reload and pick up your changes + docker build . -t archivebox + docker run -it -p 8000:8000 \ + -v $PWD/data:/data \ + -v $PWD/archivebox:/app/archivebox \ + archivebox server 0.0.0.0:8000 --debug --reload + ``` + + ### Common development tasks + + See the `./bin/` folder and read the source of the bash scripts within. + You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`. + + #### Run the linters + + ```bash + ./bin/lint.sh + ``` + (uses `flake8` and `mypy`) + + #### Run the integration tests + + ```bash + ./bin/test.sh + ``` + (uses `pytest -s`) + + #### Make migrations or enter a django shell + + ```bash + cd archivebox/ + ./manage.py makemigrations + + cd data/ + archivebox shell + ``` + (uses `pytest -s`) + + #### Build the docs, pip package, and docker image + + ```bash + ./bin/build.sh + + # or individually: + ./bin/build_docs.sh + ./bin/build_pip.sh + ./bin/build_deb.sh + ./bin/build_brew.sh + ./bin/build_docker.sh + ``` + + #### Roll a release + + ```bash + ./bin/release.sh + ``` + (bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages) + + + --- + +
    +

    + +
    + This project is maintained mostly in my spare time with the help from generous contributors and Monadical.com. +

    + +
    + Sponsor us on Github +
    +
    + +
    + + + + +

    + +
    + +Platform: UNKNOWN +Classifier: License :: OSI Approved :: MIT License +Classifier: Natural Language :: English +Classifier: Operating System :: OS Independent +Classifier: Development Status :: 4 - Beta +Classifier: Topic :: Utilities +Classifier: Topic :: System :: Archiving +Classifier: Topic :: System :: Archiving :: Backup +Classifier: Topic :: System :: Recovery Tools +Classifier: Topic :: Sociology :: History +Classifier: Topic :: Internet :: WWW/HTTP +Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search +Classifier: Topic :: Internet :: WWW/HTTP :: WSGI :: Application +Classifier: Topic :: Software Development :: Libraries :: Python Modules +Classifier: Intended Audience :: Developers +Classifier: Intended Audience :: Education +Classifier: Intended Audience :: End Users/Desktop +Classifier: Intended Audience :: Information Technology +Classifier: Intended Audience :: Legal Industry +Classifier: Intended Audience :: System Administrators +Classifier: Environment :: Console +Classifier: Environment :: Web Environment +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Framework :: Django +Classifier: Typing :: Typed +Requires-Python: >=3.7 +Description-Content-Type: text/markdown +Provides-Extra: dev diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/dependency_links.txt b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/dependency_links.txt new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/dependency_links.txt @@ -0,0 +1 @@ + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/entry_points.txt b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/entry_points.txt new file mode 100644 index 0000000..14fdb7e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/entry_points.txt @@ -0,0 +1,3 @@ +[console_scripts] +archivebox = archivebox.cli:main + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/requires.txt b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/requires.txt new file mode 100644 index 0000000..fccac24 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/requires.txt @@ -0,0 +1,15 @@ + +[dev] +bottle +django-stubs +flake8 +ipdb +mypy +pytest +recommonmark +setuptools +sphinx +sphinx-rtd-theme +stdeb +twine +wheel diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/top_level.txt b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/top_level.txt new file mode 100644 index 0000000..74056b6 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox-0.5.3.egg-info/top_level.txt @@ -0,0 +1 @@ +archivebox diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/.flake8 b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/.flake8 new file mode 100644 index 0000000..dd6ba8e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/.flake8 @@ -0,0 +1,6 @@ +[flake8] +ignore = D100,D101,D102,D103,D104,D105,D202,D203,D205,D400,E131,E241,E252,E266,E272,E701,E731,W293,W503,W291,W391 +select = F,E9,W +max-line-length = 130 +max-complexity = 10 +exclude = migrations,tests,node_modules,vendor,static,venv,.venv,.venv2,.docker-venv diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/LICENSE b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/LICENSE new file mode 100644 index 0000000..ea201f9 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2020 Nick Sweeting + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/README.md b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/README.md new file mode 100644 index 0000000..2e35783 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/README.md @@ -0,0 +1,545 @@ +
    + +

    ArchiveBox
    The open-source self-hosted web archive.

    + +▶️ Quickstart | +Demo | +Github | +Documentation | +Info & Motivation | +Community | +Roadmap + +
    +"Your own personal internet archive" (网站存档 / 爬虫)
+
    + + + + + + + + + + +
    +
    + +ArchiveBox is a powerful self-hosted internet archiving solution written in Python 3. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on the configuration and the content it detects. + +Your archive can be managed through the command line with commands like `archivebox add`, through the built-in Web UI `archivebox server`, or via the Python library API (beta). It can ingest bookmarks from a browser or service like Pocket/Pinboard, your entire browsing history, RSS feeds, or URLs one at a time. You can also schedule regular/realtime imports with `archivebox schedule`. + +The main index is a self-contained `index.sqlite3` file, and each snapshot is stored as a folder `data/archive//`, with an easy-to-read `index.html` and `index.json` within. For each page, ArchiveBox auto-extracts many types of assets/media and saves them in standard formats, with out-of-the-box support for: several types of HTML snapshots (wget, Chrome headless, singlefile), PDF snapshotting, screenshotting, WARC archiving, git repositories, images, audio, video, subtitles, article text, and more. The snapshots are browseable and managable offline through the filesystem, the built-in webserver, or the Python library API. + +### Quickstart + +It works on Linux/BSD (Intel and ARM CPUs with `docker`/`apt`/`pip3`), macOS (with `docker`/`brew`/`pip3`), and Windows (beta with `docker`/`pip3`). + +```bash +pip3 install archivebox +archivebox --version +# install extras as-needed, or use one of full setup methods below to get everything out-of-the-box + +mkdir ~/archivebox && cd ~/archivebox # this can be anywhere +archivebox init + +archivebox add 'https://example.com' +archivebox add --depth=1 'https://example.com' +archivebox schedule --every=day https://getpocket.com/users/USERNAME/feed/all +archivebox oneshot --extract=title,favicon,media https://www.youtube.com/watch?v=dQw4w9WgXcQ +archivebox help # to see more options +``` + +*(click to expand the sections below for full setup instructions)* + +
    +Get ArchiveBox with docker-compose on any platform (recommended, everything included out-of-the-box) + +First make sure you have Docker installed: https://docs.docker.com/get-docker/ +

    +This is the recommended way to run ArchiveBox because it includes *all* the extractors like chrome, wget, youtube-dl, git, etc., as well as full-text search with sonic, and many other great features. + +```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +curl -O https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml +docker-compose run archivebox init +docker-compose run archivebox --version + +# start the webserver and open the UI (optional) +docker-compose run archivebox manage createsuperuser +docker-compose up -d +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker-compose run archivebox add 'https://example.com' +docker-compose run archivebox status +docker-compose run archivebox help # to see more options +``` + +
    + +
    +Get ArchiveBox with docker on any platform + +First make sure you have Docker installed: https://docs.docker.com/get-docker/
    +```bash +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +docker run -v $PWD:/data -it archivebox/archivebox init +docker run -v $PWD:/data -it archivebox/archivebox --version + +# start the webserver and open the UI (optional) +docker run -v $PWD:/data -it archivebox/archivebox manage createsuperuser +docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add links and manage your archive via the CLI: +docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com' +docker run -v $PWD:/data -it archivebox/archivebox status +docker run -v $PWD:/data -it archivebox/archivebox help # to see more options +``` + +
    + +
    +Get ArchiveBox with apt on Ubuntu >=20.04 + +```bash +sudo add-apt-repository -u ppa:archivebox/archivebox +sudo apt install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`: +```bash +deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main +``` +(you may need to install some other dependencies manually however) + +
    + +
    +Get ArchiveBox with brew on macOS >=10.13 + +```bash +brew install archivebox/archivebox/archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
    + +
    +Get ArchiveBox with pip on any platform + +```bash +pip3 install archivebox + +# create a new empty directory and initalize your collection (can be anywhere) +mkdir ~/archivebox && cd ~/archivebox +npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git' +archivebox init +archivebox --version +# Install any missing extras like wget/git/chrome/etc. manually as needed + +# start the webserver and open the web UI (optional) +archivebox manage createsuperuser +archivebox server 0.0.0.0:8000 +open http://127.0.0.1:8000 + +# you can also add URLs and manage the archive via the CLI and filesystem: +archivebox add 'https://example.com' +archivebox status +archivebox list --html --with-headers > index.html +archivebox list --json --with-headers > index.json +archivebox help # to see more options +``` + +
    + +--- + +
    + +
    + +DEMO: archivebox.zervice.io/ +For more information, see the full Quickstart guide, Usage, and Configuration docs. +
    + +--- + + +# Overview + +ArchiveBox is a command line tool, self-hostable web-archiving server, and Python library all-in-one. It can be installed on Docker, macOS, and Linux/BSD, and Windows. You can download and install it as a Debian/Ubuntu package, Homebrew package, Python3 package, or a Docker image. No matter which install method you choose, they all provide the same CLI, Web UI, and on-disk data format. + +To use ArchiveBox you start by creating a folder for your data to live in (it can be anywhere on your system), and running `archivebox init` inside of it. That will create a sqlite3 index and an `ArchiveBox.conf` file. After that, you can continue to add/export/manage/etc using the CLI `archivebox help`, or you can run the Web UI (recommended). If you only want to archive a single site, you can run `archivebox oneshot` to avoid having to create a whole collection. + +The CLI is considered "stable", the ArchiveBox Python API and REST APIs are "beta", and the [desktop app](https://github.com/ArchiveBox/desktop) is "alpha". + +At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer). You can also self-host your archivebox server on a public domain to provide archive.org-style public access to your site snapshots. + +
    +CLI Screenshot +Desktop index screenshot +Desktop details page Screenshot +Desktop details page Screenshot
    +Demo | Usage | Screenshots +
    +. . . . . . . . . . . . . . . . . . . . . . . . . . . . +

    + + +## Key Features + +- [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally +- [**Few dependencies**](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) +- [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) +- Easy to set up **[scheduled importing](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources** +- Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC +- ~~**Suitable for paywalled / [authenticated content](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes) +- **Doesn't require a constantly-running daemon**, proxy, or native app +- Provides a CLI, Python API, self-hosted web UI, and REST API (WIP) +- Architected to be able to run [**many varieties of scripts during archiving**](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. to extract media, summarize articles, [scroll pages](https://github.com/ArchiveBox/ArchiveBox/issues/80), [close modals](https://github.com/ArchiveBox/ArchiveBox/issues/175), expand comment threads, etc. +- Can also [**mirror content to 3rd-party archiving services**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy + +## Input formats + +ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more! + +```bash +echo 'http://example.com' | archivebox add +archivebox add 'https://example.com/some/page' +archivebox add < ~/Downloads/firefox_bookmarks_export.html +archivebox add < any_text_with_urls_in_it.txt +archivebox add --depth=1 'https://example.com/some/downloads.html' +archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12' +``` + +- Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more) +- RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format +- Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more + +See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples. + +It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand. + +## Output formats + +All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`. + +The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.) + +```bash + ls ./archive// +``` + +- **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details +- **Title:** `title` title of the site +- **Favicon:** `favicon.ico` favicon of the site +- **Headers:** `headers.json` Any HTTP headers the site returns are saved in a json file +- **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile +- **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present +- **WARC:** `warc/.gz` gzipped WARC of all the resources fetched while archiving +- **PDF:** `output.pdf` Printed PDF of site using headless chrome +- **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome +- **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome +- **Readability:** `article.html/json` Article text extraction using Readability +- **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org +- **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl +- **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links +- _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._ + +It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file. + +## Dependencies + +You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled. + +If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) or the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install). + +ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more. + +## Caveats + +If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process. +```bash +# don't do this: +archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere' +archivebox add 'https://example.com/any/url/you/want/to/keep/secret/' + +# without first disabling share the URL with 3rd party APIs: +archivebox config --set SAVE_ARCHIVE_DOT_ORG=False # disable saving all URLs in Archive.org +archivebox config --set SAVE_FAVICON=False # optional: only the domain is leaked, not full URL +archivebox config --set CHROME_BINARY=chromium # optional: switch to chromium to avoid Chrome phoning home to Google +``` + +Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details. +```bash +# visiting an archived page with malicious JS: +https://127.0.0.1:8000/archive/1602401954/example.com/index.html + +# example.com/index.js can now make a request to read everything: +https://127.0.0.1:8000/index.html +https://127.0.0.1:8000/archive/* +# then example.com/index.js can send it off to some evil server +``` + +Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash: +```bash +archivebox add 'https://example.com#2020-10-24' +... +archivebox add 'https://example.com#2020-10-25' +``` + +--- + +
    + +
    + +--- + +# Background & Motivation + +Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity. + +Whether it's to resist censorship by saving articles before they get taken down or edited, or +just to save a collection of early 2010's flash games you love to play, having the tools to +archive internet content enables to you save the stuff you care most about before it disappears. + +
    +
    + Image from WTF is Link Rot?...
     +
    + +The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. +I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about. + +Because modern websites are complicated and often rely on dynamic content, +ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats. + +All the archived links are stored by date bookmarked in `./archive/`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM. + +## Comparison to Other Projects + +▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.** + +comparison The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations. + +#### User Interface & Intended Purpose + +ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend. + +#### Private Local Archives vs Centralized Public Archives + +Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle. + +#### Storage Requirements + +Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files. + +## Learn more + +Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web! + + + +- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists) + _Community-maintained indexes of archiving tools and institutions._ + - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) + _Open source tools and projects in the internet archiving space._ + - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List) + _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._ + - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities) + _A collection of the most active internet archiving communities and initiatives._ +- Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post. +- Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter. + +--- + +# Documentation + + + +We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation. + +You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder. + +## Getting Started + +- [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) +- [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) +- [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + +## Reference + +- [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) +- [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) +- [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) +- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site) +- [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) +- [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) +- [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) +- [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) +- [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) +- [Python API](https://docs.archivebox.io/en/latest/modules.html) +- REST API (coming soon...) + +## More Info + +- [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues) +- [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) +- [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) +- [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) +- [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation) +- [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) + +--- + +# ArchiveBox Development + +All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap. + +### Setup the dev environment + +First, install the system dependencies from the "Bare Metal" section above. +Then you can clone the ArchiveBox repo and install +```python3 +git clone https://github.com/ArchiveBox/ArchiveBox && cd ArchiveBox +git checkout master # or the branch you want to test +git submodule update --init --recursive +git pull --recurse-submodules + +# Install ArchiveBox + python dependencies +python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev] +# or with pipenv: pipenv install --dev && pipenv shell + +# Install node dependencies +npm install + +# Optional: install extractor dependencies manually or with helper script +./bin/setup.sh + +# Optional: develop via docker by mounting the code dir into the container +# if you edit e.g. ./archivebox/core/models.py on the docker host, runserver +# inside the container will reload and pick up your changes +docker build . -t archivebox +docker run -it -p 8000:8000 \ + -v $PWD/data:/data \ + -v $PWD/archivebox:/app/archivebox \ + archivebox server 0.0.0.0:8000 --debug --reload +``` + +### Common development tasks + +See the `./bin/` folder and read the source of the bash scripts within. +You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`. + +#### Run the linters + +```bash +./bin/lint.sh +``` +(uses `flake8` and `mypy`) + +#### Run the integration tests + +```bash +./bin/test.sh +``` +(uses `pytest -s`) + +#### Make migrations or enter a django shell + +```bash +cd archivebox/ +./manage.py makemigrations + +cd data/ +archivebox shell +``` +(uses `pytest -s`) + +#### Build the docs, pip package, and docker image + +```bash +./bin/build.sh + +# or individually: +./bin/build_docs.sh +./bin/build_pip.sh +./bin/build_deb.sh +./bin/build_brew.sh +./bin/build_docker.sh +``` + +#### Roll a release + +```bash +./bin/release.sh +``` +(bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages) + + +--- + +
    +

    + +
    +This project is maintained mostly in my spare time with the help from generous contributors and Monadical.com. +

    + +
    +Sponsor us on Github +
    +
    + +
    + + + + +

    + +
    diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/__init__.py new file mode 100644 index 0000000..b0c00b6 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/__init__.py @@ -0,0 +1 @@ +__package__ = 'archivebox' diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/__main__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/__main__.py new file mode 100644 index 0000000..8afaa27 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/__main__.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox' + +import sys + +from .cli import main + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/__init__.py new file mode 100644 index 0000000..f9a55ef --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/__init__.py @@ -0,0 +1,144 @@ +__package__ = 'archivebox.cli' +__command__ = 'archivebox' + +import os +import sys +import argparse + +from typing import Optional, Dict, List, IO, Union +from pathlib import Path + +from ..config import OUTPUT_DIR + +from importlib import import_module + +CLI_DIR = Path(__file__).resolve().parent + +# these common commands will appear sorted before any others for ease-of-use +meta_cmds = ('help', 'version') +main_cmds = ('init', 'info', 'config') +archive_cmds = ('add', 'remove', 'update', 'list', 'status') + +fake_db = ("oneshot",) + +display_first = (*meta_cmds, *main_cmds, *archive_cmds) + +# every imported command module must have these properties in order to be valid +required_attrs = ('__package__', '__command__', 'main') + +# basic checks to make sure imported files are valid subcommands +is_cli_module = lambda fname: fname.startswith('archivebox_') and fname.endswith('.py') +is_valid_cli_module = lambda module, subcommand: ( + all(hasattr(module, attr) for attr in required_attrs) + and module.__command__.split(' ')[-1] == subcommand +) + + +def list_subcommands() -> Dict[str, str]: + """find and import all valid archivebox_.py files in CLI_DIR""" + + COMMANDS = [] + for filename in os.listdir(CLI_DIR): + if is_cli_module(filename): + subcommand = filename.replace('archivebox_', '').replace('.py', '') + module = import_module('.archivebox_{}'.format(subcommand), __package__) + assert is_valid_cli_module(module, subcommand) + COMMANDS.append((subcommand, module.main.__doc__)) + globals()[subcommand] = module.main + + display_order = lambda cmd: ( + display_first.index(cmd[0]) + if cmd[0] in display_first else + 100 + len(cmd[0]) + ) + + return dict(sorted(COMMANDS, key=display_order)) + + +def run_subcommand(subcommand: str, + subcommand_args: List[str]=None, + stdin: Optional[IO]=None, + pwd: Union[Path, str, None]=None) -> None: + """Run a given ArchiveBox subcommand with the given list of args""" + + if subcommand not in meta_cmds: + from ..config import setup_django + setup_django(in_memory_db=subcommand in fake_db, check_db=subcommand in archive_cmds) + + module = import_module('.archivebox_{}'.format(subcommand), __package__) + module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore + + +SUBCOMMANDS = list_subcommands() + +class NotProvided: + pass + + +def main(args: Optional[List[str]]=NotProvided, stdin: Optional[IO]=NotProvided, pwd: Optional[str]=None) -> None: + args = sys.argv[1:] if args is NotProvided else args + stdin = sys.stdin if stdin is NotProvided else stdin + + subcommands = list_subcommands() + parser = argparse.ArgumentParser( + prog=__command__, + description='ArchiveBox: The self-hosted internet archive', + add_help=False, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--help', '-h', + action='store_true', + help=subcommands['help'], + ) + group.add_argument( + '--version', + action='store_true', + help=subcommands['version'], + ) + group.add_argument( + "subcommand", + type=str, + help= "The name of the subcommand to run", + nargs='?', + choices=subcommands.keys(), + default=None, + ) + parser.add_argument( + "subcommand_args", + help="Arguments for the subcommand", + nargs=argparse.REMAINDER, + ) + command = parser.parse_args(args or ()) + + if command.version: + command.subcommand = 'version' + elif command.help or command.subcommand is None: + command.subcommand = 'help' + + if command.subcommand not in ('help', 'version', 'status'): + from ..logging_util import log_cli_command + + log_cli_command( + subcommand=command.subcommand, + subcommand_args=command.subcommand_args, + stdin=stdin, + pwd=pwd or OUTPUT_DIR + ) + + run_subcommand( + subcommand=command.subcommand, + subcommand_args=command.subcommand_args, + stdin=stdin, + pwd=pwd or OUTPUT_DIR, + ) + + +__all__ = ( + 'SUBCOMMANDS', + 'list_subcommands', + 'run_subcommand', + *SUBCOMMANDS.keys(), +) + + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_add.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_add.py new file mode 100644 index 0000000..41c7554 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_add.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox add' + +import sys +import argparse + +from typing import List, Optional, IO + +from ..main import add +from ..util import docstring +from ..config import OUTPUT_DIR, ONLY_NEW +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(add.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=add.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--update-all', #'-n', + action='store_true', + default=not ONLY_NEW, # when ONLY_NEW=True we skip updating old links + help="Also retry previously skipped/failed links when adding new links", + ) + parser.add_argument( + '--index-only', #'-o', + action='store_true', + help="Add the links to the main index without archiving them", + ) + parser.add_argument( + 'urls', + nargs='*', + type=str, + default=None, + help=( + 'URLs or paths to archive e.g.:\n' + ' https://getpocket.com/users/USERNAME/feed/all\n' + ' https://example.com/some/rss/feed.xml\n' + ' https://example.com\n' + ' ~/Downloads/firefox_bookmarks_export.html\n' + ' ~/Desktop/sites_list.csv\n' + ) + ) + parser.add_argument( + "--depth", + action="store", + default=0, + choices=[0, 1], + type=int, + help="Recursively archive all linked pages up to this many hops away" + ) + parser.add_argument( + "--overwrite", + default=False, + action="store_true", + help="Re-archive URLs from scratch, overwriting any existing files" + ) + parser.add_argument( + "--init", #'-i', + action='store_true', + help="Init/upgrade the curent data directory before adding", + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + command = parser.parse_args(args or ()) + urls = command.urls + stdin_urls = accept_stdin(stdin) + if (stdin_urls and urls) or (not stdin and not urls): + stderr( + '[X] You must pass URLs/paths to add via stdin or CLI arguments.\n', + color='red', + ) + raise SystemExit(2) + add( + urls=stdin_urls or urls, + depth=command.depth, + update_all=command.update_all, + index_only=command.index_only, + overwrite=command.overwrite, + init=command.init, + extractors=command.extract, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) + + +# TODO: Implement these +# +# parser.add_argument( +# '--mirror', #'-m', +# action='store_true', +# help='Archive an entire site (finding all linked pages below it on the same domain)', +# ) +# parser.add_argument( +# '--crawler', #'-r', +# choices=('depth_first', 'breadth_first'), +# help='Controls which crawler to use in order to find outlinks in a given page', +# default=None, +# ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_config.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_config.py new file mode 100644 index 0000000..f81286c --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_config.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox config' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import config +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(config.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=config.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--get', #'-g', + action='store_true', + help="Get the value for the given config KEYs", + ) + group.add_argument( + '--set', #'-s', + action='store_true', + help="Set the given KEY=VALUE config values", + ) + group.add_argument( + '--reset', #'-s', + action='store_true', + help="Reset the given KEY config values to their defaults", + ) + parser.add_argument( + 'config_options', + nargs='*', + type=str, + help='KEY or KEY=VALUE formatted config values to get or set', + ) + command = parser.parse_args(args or ()) + config_options_str = accept_stdin(stdin) + + config( + config_options_str=config_options_str, + config_options=command.config_options, + get=command.get, + set=command.set, + reset=command.reset, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_help.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_help.py new file mode 100755 index 0000000..46f17cb --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_help.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox help' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import help +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(help.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=help.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + help(out_dir=pwd or OUTPUT_DIR) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_init.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_init.py new file mode 100755 index 0000000..6255ef2 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_init.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox init' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import init +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(init.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=init.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--force', # '-f', + action='store_true', + help='Ignore unrecognized files in current directory and initialize anyway', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + init( + force=command.force, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_list.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_list.py new file mode 100644 index 0000000..3838cf6 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_list.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox list' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import list_all +from ..util import docstring +from ..config import OUTPUT_DIR +from ..index import ( + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, +) +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(list_all.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=list_all.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--csv', #'-c', + type=str, + help="Print the output in CSV format with the given columns, e.g.: timestamp,url,extension", + default=None, + ) + group.add_argument( + '--json', #'-j', + action='store_true', + help="Print the output in JSON format with all columns included.", + ) + group.add_argument( + '--html', + action='store_true', + help="Print the output in HTML format" + ) + parser.add_argument( + '--with-headers', + action='store_true', + help='Include the headers in the output document' + ) + parser.add_argument( + '--sort', #'-s', + type=str, + help="List the links sorted using the given key, e.g. timestamp or updated.", + default=None, + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="List only links bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="List only links bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--status', + type=str, + choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'), + default='indexed', + help=( + 'List only links or data directories that have the given status\n' + f' indexed {get_indexed_folders.__doc__} (the default)\n' + f' archived {get_archived_folders.__doc__}\n' + f' unarchived {get_unarchived_folders.__doc__}\n' + '\n' + f' present {get_present_folders.__doc__}\n' + f' valid {get_valid_folders.__doc__}\n' + f' invalid {get_invalid_folders.__doc__}\n' + '\n' + f' duplicate {get_duplicate_folders.__doc__}\n' + f' orphaned {get_orphaned_folders.__doc__}\n' + f' corrupted {get_corrupted_folders.__doc__}\n' + f' unrecognized {get_unrecognized_folders.__doc__}\n' + ) + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + default=None, + help='List only URLs matching these filter patterns.' + ) + command = parser.parse_args(args or ()) + filter_patterns_str = accept_stdin(stdin) + + if command.with_headers and not (command.json or command.html or command.csv): + stderr( + '[X] --with-headers can only be used with --json, --html or --csv options.\n', + color='red', + ) + raise SystemExit(2) + + matching_folders = list_all( + filter_patterns_str=filter_patterns_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + status=command.status, + after=command.after, + before=command.before, + sort=command.sort, + csv=command.csv, + json=command.json, + html=command.html, + with_headers=command.with_headers, + out_dir=pwd or OUTPUT_DIR, + ) + raise SystemExit(not matching_folders) + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_manage.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_manage.py new file mode 100644 index 0000000..f05604e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_manage.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox manage' + +import sys + +from typing import Optional, List, IO + +from ..main import manage +from ..util import docstring +from ..config import OUTPUT_DIR + + +@docstring(manage.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + manage( + args=args, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_oneshot.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_oneshot.py new file mode 100644 index 0000000..af68bac --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_oneshot.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox oneshot' + +import sys +import argparse + +from pathlib import Path +from typing import List, Optional, IO + +from ..main import oneshot +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin, stderr + + +@docstring(oneshot.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=oneshot.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + 'url', + type=str, + default=None, + help=( + 'URLs or paths to archive e.g.:\n' + ' https://getpocket.com/users/USERNAME/feed/all\n' + ' https://example.com/some/rss/feed.xml\n' + ' https://example.com\n' + ' ~/Downloads/firefox_bookmarks_export.html\n' + ' ~/Desktop/sites_list.csv\n' + ) + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + parser.add_argument( + '--out-dir', + type=str, + default=OUTPUT_DIR, + help= "Path to save the single archive folder to, e.g. ./example.com_archive" + ) + command = parser.parse_args(args or ()) + url = command.url + stdin_url = accept_stdin(stdin) + if (stdin_url and url) or (not stdin and not url): + stderr( + '[X] You must pass a URL/path to add via stdin or CLI arguments.\n', + color='red', + ) + raise SystemExit(2) + + oneshot( + url=stdin_url or url, + out_dir=Path(command.out_dir).resolve(), + extractors=command.extract, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_remove.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_remove.py new file mode 100644 index 0000000..cb073e9 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_remove.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox remove' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import remove +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(remove.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=remove.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--yes', # '-y', + action='store_true', + help='Remove links instantly without prompting to confirm.', + ) + parser.add_argument( + '--delete', # '-r', + action='store_true', + help=( + "In addition to removing the link from the index, " + "also delete its archived content and metadata folder." + ), + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="List only URLs bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="List only URLs bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex','tag'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + help='URLs matching this filter pattern will be removed from the index.' + ) + command = parser.parse_args(args or ()) + filter_str = accept_stdin(stdin) + + remove( + filter_str=filter_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + before=command.before, + after=command.after, + yes=command.yes, + delete=command.delete, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_schedule.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_schedule.py new file mode 100644 index 0000000..ec5e914 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_schedule.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox schedule' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import schedule +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(schedule.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=schedule.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--quiet', '-q', + action='store_true', + help=("Don't warn about storage space."), + ) + group = parser.add_mutually_exclusive_group() + group.add_argument( + '--add', # '-a', + action='store_true', + help='Add a new scheduled ArchiveBox update job to cron', + ) + parser.add_argument( + '--every', # '-e', + type=str, + default=None, + help='Run ArchiveBox once every [timeperiod] (hour/day/month/year or cron format e.g. "0 0 * * *")', + ) + parser.add_argument( + '--depth', # '-d', + type=int, + default=0, + help='Depth to archive to [0] or 1, see "add" command help for more info.', + ) + group.add_argument( + '--clear', # '-c' + action='store_true', + help=("Stop all ArchiveBox scheduled runs (remove cron jobs)"), + ) + group.add_argument( + '--show', # '-s' + action='store_true', + help=("Print a list of currently active ArchiveBox cron jobs"), + ) + group.add_argument( + '--foreground', '-f', + action='store_true', + help=("Launch ArchiveBox scheduler as a long-running foreground task " + "instead of using cron."), + ) + group.add_argument( + '--run-all', # '-a', + action='store_true', + help=("Run all the scheduled jobs once immediately, independent of " + "their configured schedules, can be used together with --foreground"), + ) + parser.add_argument( + 'import_path', + nargs='?', + type=str, + default=None, + help=("Check this path and import any new links on every run " + "(can be either local file or remote URL)"), + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + schedule( + add=command.add, + show=command.show, + clear=command.clear, + foreground=command.foreground, + run_all=command.run_all, + quiet=command.quiet, + every=command.every, + depth=command.depth, + import_path=command.import_path, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_server.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_server.py new file mode 100644 index 0000000..dbacf7e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_server.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox server' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import server +from ..util import docstring +from ..config import OUTPUT_DIR, BIND_ADDR +from ..logging_util import SmartFormatter, reject_stdin + +@docstring(server.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=server.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + 'runserver_args', + nargs='*', + type=str, + default=[BIND_ADDR], + help='Arguments to pass to Django runserver' + ) + parser.add_argument( + '--reload', + action='store_true', + help='Enable auto-reloading when code or templates change', + ) + parser.add_argument( + '--debug', + action='store_true', + help='Enable DEBUG=True mode with more verbose errors', + ) + parser.add_argument( + '--init', + action='store_true', + help='Run archivebox init before starting the server', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + server( + runserver_args=command.runserver_args, + reload=command.reload, + debug=command.debug, + init=command.init, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_shell.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_shell.py new file mode 100644 index 0000000..bcd5fdd --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_shell.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox shell' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import shell +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(shell.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=shell.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + shell( + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_status.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_status.py new file mode 100644 index 0000000..2bef19c --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_status.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox status' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import status +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(status.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=status.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + status(out_dir=pwd or OUTPUT_DIR) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_update.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_update.py new file mode 100644 index 0000000..6748096 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_update.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox update' + +import sys +import argparse + +from typing import List, Optional, IO + +from ..main import update +from ..util import docstring +from ..config import OUTPUT_DIR +from ..index import ( + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, +) +from ..logging_util import SmartFormatter, accept_stdin + + +@docstring(update.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=update.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--only-new', #'-n', + action='store_true', + help="Don't attempt to retry previously skipped/failed links when updating", + ) + parser.add_argument( + '--index-only', #'-o', + action='store_true', + help="Update the main index without archiving any content", + ) + parser.add_argument( + '--resume', #'-r', + type=float, + help='Resume the update process from a given timestamp', + default=None, + ) + parser.add_argument( + '--overwrite', #'-x', + action='store_true', + help='Ignore existing archived content and overwrite with new versions (DANGEROUS)', + ) + parser.add_argument( + '--before', #'-b', + type=float, + help="Update only links bookmarked before the given timestamp.", + default=None, + ) + parser.add_argument( + '--after', #'-a', + type=float, + help="Update only links bookmarked after the given timestamp.", + default=None, + ) + parser.add_argument( + '--status', + type=str, + choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'), + default='indexed', + help=( + 'Update only links or data directories that have the given status\n' + f' indexed {get_indexed_folders.__doc__} (the default)\n' + f' archived {get_archived_folders.__doc__}\n' + f' unarchived {get_unarchived_folders.__doc__}\n' + '\n' + f' present {get_present_folders.__doc__}\n' + f' valid {get_valid_folders.__doc__}\n' + f' invalid {get_invalid_folders.__doc__}\n' + '\n' + f' duplicate {get_duplicate_folders.__doc__}\n' + f' orphaned {get_orphaned_folders.__doc__}\n' + f' corrupted {get_corrupted_folders.__doc__}\n' + f' unrecognized {get_unrecognized_folders.__doc__}\n' + ) + ) + parser.add_argument( + '--filter-type', + type=str, + choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'), + default='exact', + help='Type of pattern matching to use when filtering URLs', + ) + parser.add_argument( + 'filter_patterns', + nargs='*', + type=str, + default=None, + help='Update only URLs matching these filter patterns.' + ) + parser.add_argument( + "--extract", + type=str, + help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \ + This does not take precedence over the configuration", + default="" + ) + command = parser.parse_args(args or ()) + filter_patterns_str = accept_stdin(stdin) + + update( + resume=command.resume, + only_new=command.only_new, + index_only=command.index_only, + overwrite=command.overwrite, + filter_patterns_str=filter_patterns_str, + filter_patterns=command.filter_patterns, + filter_type=command.filter_type, + status=command.status, + after=command.after, + before=command.before, + out_dir=pwd or OUTPUT_DIR, + extractors=command.extract, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_version.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_version.py new file mode 100755 index 0000000..e7922f3 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/archivebox_version.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' +__command__ = 'archivebox version' + +import sys +import argparse + +from typing import Optional, List, IO + +from ..main import version +from ..util import docstring +from ..config import OUTPUT_DIR +from ..logging_util import SmartFormatter, reject_stdin + + +@docstring(version.__doc__) +def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None: + parser = argparse.ArgumentParser( + prog=__command__, + description=version.__doc__, + add_help=True, + formatter_class=SmartFormatter, + ) + parser.add_argument( + '--quiet', '-q', + action='store_true', + help='Only print ArchiveBox version number and nothing else.', + ) + command = parser.parse_args(args or ()) + reject_stdin(__command__, stdin) + + version( + quiet=command.quiet, + out_dir=pwd or OUTPUT_DIR, + ) + + +if __name__ == '__main__': + main(args=sys.argv[1:], stdin=sys.stdin) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/tests.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/tests.py new file mode 100755 index 0000000..4d7016a --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/cli/tests.py @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 + +__package__ = 'archivebox.cli' + + +import os +import sys +import shutil +import unittest +from pathlib import Path + +from contextlib import contextmanager + +TEST_CONFIG = { + 'USE_COLOR': 'False', + 'SHOW_PROGRESS': 'False', + + 'OUTPUT_DIR': 'data.tests', + + 'SAVE_ARCHIVE_DOT_ORG': 'False', + 'SAVE_TITLE': 'False', + + 'USE_CURL': 'False', + 'USE_WGET': 'False', + 'USE_GIT': 'False', + 'USE_CHROME': 'False', + 'USE_YOUTUBEDL': 'False', +} + +OUTPUT_DIR = 'data.tests' +os.environ.update(TEST_CONFIG) + +from ..main import init +from ..index import load_main_index +from ..config import ( + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, +) + +from . import ( + archivebox_init, + archivebox_add, + archivebox_remove, +) + +HIDE_CLI_OUTPUT = True + +test_urls = ''' +https://example1.com/what/is/happening.html?what=1#how-about-this=1 +https://example2.com/what/is/happening/?what=1#how-about-this=1 +HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f +https://example4.com/what/is/happening.html +https://example5.com/ +https://example6.com + +http://example7.com +[https://example8.com/what/is/this.php?what=1] +[and http://example9.com?what=1&other=3#and-thing=2] +https://example10.com#and-thing=2 " +abcdef +sdflkf[what](https://subb.example12.com/who/what.php?whoami=1#whatami=2)?am=hi +example13.bada +and example14.badb +htt://example15.badc +''' + +stdout = sys.stdout +stderr = sys.stderr + + +@contextmanager +def output_hidden(show_failing=True): + if not HIDE_CLI_OUTPUT: + yield + return + + sys.stdout = open('stdout.txt', 'w+') + sys.stderr = open('stderr.txt', 'w+') + try: + yield + sys.stdout.close() + sys.stderr.close() + sys.stdout = stdout + sys.stderr = stderr + except: + sys.stdout.close() + sys.stderr.close() + sys.stdout = stdout + sys.stderr = stderr + if show_failing: + with open('stdout.txt', 'r') as f: + print(f.read()) + with open('stderr.txt', 'r') as f: + print(f.read()) + raise + finally: + os.remove('stdout.txt') + os.remove('stderr.txt') + + +class TestInit(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + + def tearDown(self): + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + def test_basic_init(self): + with output_hidden(): + archivebox_init.main([]) + + assert (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists() + assert (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists() + assert (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists() + assert len(load_main_index(out_dir=OUTPUT_DIR)) == 0 + + def test_conflicting_init(self): + with open(Path(OUTPUT_DIR) / 'test_conflict.txt', 'w+') as f: + f.write('test') + + try: + with output_hidden(show_failing=False): + archivebox_init.main([]) + assert False, 'Init should have exited with an exception' + except SystemExit: + pass + + assert not (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists() + assert not (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists() + assert not (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists() + try: + load_main_index(out_dir=OUTPUT_DIR) + assert False, 'load_main_index should raise an exception when no index is present' + except: + pass + + def test_no_dirty_state(self): + with output_hidden(): + init() + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + with output_hidden(): + init() + + +class TestAdd(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + with output_hidden(): + init() + + def tearDown(self): + shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + def test_add_arg_url(self): + with output_hidden(): + archivebox_add.main(['https://getpocket.com/users/nikisweeting/feed/all']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 30 + + def test_add_arg_file(self): + test_file = Path(OUTPUT_DIR) / 'test.txt' + with open(test_file, 'w+') as f: + f.write(test_urls) + + with output_hidden(): + archivebox_add.main([test_file]) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 12 + os.remove(test_file) + + def test_add_stdin_url(self): + with output_hidden(): + archivebox_add.main([], stdin=test_urls) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 12 + + +class TestRemove(unittest.TestCase): + def setUp(self): + os.makedirs(OUTPUT_DIR, exist_ok=True) + with output_hidden(): + init() + archivebox_add.main([], stdin=test_urls) + + # def tearDown(self): + # shutil.rmtree(OUTPUT_DIR, ignore_errors=True) + + + def test_remove_exact(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', 'https://example5.com/']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 11 + + def test_remove_regex(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', '--filter-type=regex', r'http(s)?:\/\/(.+\.)?(example\d\.com)']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 4 + + def test_remove_domain(self): + with output_hidden(): + archivebox_remove.main(['--yes', '--delete', '--filter-type=domain', 'example5.com', 'example6.com']) + + all_links = load_main_index(out_dir=OUTPUT_DIR) + assert len(all_links) == 10 + + def test_remove_none(self): + try: + with output_hidden(show_failing=False): + archivebox_remove.main(['--yes', '--delete', 'https://doesntexist.com']) + assert False, 'Should raise if no URLs match' + except: + pass + + +if __name__ == '__main__': + if '--verbose' in sys.argv or '-v' in sys.argv: + HIDE_CLI_OUTPUT = False + + unittest.main() diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/config.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/config.py new file mode 100644 index 0000000..9a3f9a7 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/config.py @@ -0,0 +1,1081 @@ +""" +ArchiveBox config definitons (including defaults and dynamic config options). + +Config Usage Example: + + archivebox config --set MEDIA_TIMEOUT=600 + env MEDIA_TIMEOUT=600 USE_COLOR=False ... archivebox [subcommand] ... + +Config Precedence Order: + + 1. cli args (--update-all / --index-only / etc.) + 2. shell environment vars (env USE_COLOR=False archivebox add '...') + 3. config file (echo "SAVE_FAVICON=False" >> ArchiveBox.conf) + 4. defaults (defined below in Python) + +Documentation: + + https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration + +""" + +__package__ = 'archivebox' + +import os +import io +import re +import sys +import json +import getpass +import shutil +import django + +from hashlib import md5 +from pathlib import Path +from typing import Optional, Type, Tuple, Dict, Union, List +from subprocess import run, PIPE, DEVNULL +from configparser import ConfigParser +from collections import defaultdict + +from .config_stubs import ( + SimpleConfigValueDict, + ConfigValue, + ConfigDict, + ConfigDefaultValue, + ConfigDefaultDict, +) + +############################### Config Schema ################################## + +CONFIG_SCHEMA: Dict[str, ConfigDefaultDict] = { + 'SHELL_CONFIG': { + 'IS_TTY': {'type': bool, 'default': lambda _: sys.stdout.isatty()}, + 'USE_COLOR': {'type': bool, 'default': lambda c: c['IS_TTY']}, + 'SHOW_PROGRESS': {'type': bool, 'default': lambda c: c['IS_TTY']}, + 'IN_DOCKER': {'type': bool, 'default': False}, + # TODO: 'SHOW_HINTS': {'type: bool, 'default': True}, + }, + + 'GENERAL_CONFIG': { + 'OUTPUT_DIR': {'type': str, 'default': None}, + 'CONFIG_FILE': {'type': str, 'default': None}, + 'ONLY_NEW': {'type': bool, 'default': True}, + 'TIMEOUT': {'type': int, 'default': 60}, + 'MEDIA_TIMEOUT': {'type': int, 'default': 3600}, + 'OUTPUT_PERMISSIONS': {'type': str, 'default': '755'}, + 'RESTRICT_FILE_NAMES': {'type': str, 'default': 'windows'}, + 'URL_BLACKLIST': {'type': str, 'default': r'\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$'}, # to avoid downloading code assets as their own pages + }, + + 'SERVER_CONFIG': { + 'SECRET_KEY': {'type': str, 'default': None}, + 'BIND_ADDR': {'type': str, 'default': lambda c: ['127.0.0.1:8000', '0.0.0.0:8000'][c['IN_DOCKER']]}, + 'ALLOWED_HOSTS': {'type': str, 'default': '*'}, + 'DEBUG': {'type': bool, 'default': False}, + 'PUBLIC_INDEX': {'type': bool, 'default': True}, + 'PUBLIC_SNAPSHOTS': {'type': bool, 'default': True}, + 'PUBLIC_ADD_VIEW': {'type': bool, 'default': False}, + 'FOOTER_INFO': {'type': str, 'default': 'Content is hosted for personal archiving purposes only. Contact server owner for any takedown requests.'}, + 'ACTIVE_THEME': {'type': str, 'default': 'default'}, + }, + + 'ARCHIVE_METHOD_TOGGLES': { + 'SAVE_TITLE': {'type': bool, 'default': True, 'aliases': ('FETCH_TITLE',)}, + 'SAVE_FAVICON': {'type': bool, 'default': True, 'aliases': ('FETCH_FAVICON',)}, + 'SAVE_WGET': {'type': bool, 'default': True, 'aliases': ('FETCH_WGET',)}, + 'SAVE_WGET_REQUISITES': {'type': bool, 'default': True, 'aliases': ('FETCH_WGET_REQUISITES',)}, + 'SAVE_SINGLEFILE': {'type': bool, 'default': True, 'aliases': ('FETCH_SINGLEFILE',)}, + 'SAVE_READABILITY': {'type': bool, 'default': True, 'aliases': ('FETCH_READABILITY',)}, + 'SAVE_MERCURY': {'type': bool, 'default': True, 'aliases': ('FETCH_MERCURY',)}, + 'SAVE_PDF': {'type': bool, 'default': True, 'aliases': ('FETCH_PDF',)}, + 'SAVE_SCREENSHOT': {'type': bool, 'default': True, 'aliases': ('FETCH_SCREENSHOT',)}, + 'SAVE_DOM': {'type': bool, 'default': True, 'aliases': ('FETCH_DOM',)}, + 'SAVE_HEADERS': {'type': bool, 'default': True, 'aliases': ('FETCH_HEADERS',)}, + 'SAVE_WARC': {'type': bool, 'default': True, 'aliases': ('FETCH_WARC',)}, + 'SAVE_GIT': {'type': bool, 'default': True, 'aliases': ('FETCH_GIT',)}, + 'SAVE_MEDIA': {'type': bool, 'default': True, 'aliases': ('FETCH_MEDIA',)}, + 'SAVE_ARCHIVE_DOT_ORG': {'type': bool, 'default': True, 'aliases': ('SUBMIT_ARCHIVE_DOT_ORG',)}, + }, + + 'ARCHIVE_METHOD_OPTIONS': { + 'RESOLUTION': {'type': str, 'default': '1440,2000', 'aliases': ('SCREENSHOT_RESOLUTION',)}, + 'GIT_DOMAINS': {'type': str, 'default': 'github.com,bitbucket.org,gitlab.com'}, + 'CHECK_SSL_VALIDITY': {'type': bool, 'default': True}, + + 'CURL_USER_AGENT': {'type': str, 'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) curl/{CURL_VERSION}'}, + 'WGET_USER_AGENT': {'type': str, 'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) wget/{WGET_VERSION}'}, + 'CHROME_USER_AGENT': {'type': str, 'default': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36'}, + + 'COOKIES_FILE': {'type': str, 'default': None}, + 'CHROME_USER_DATA_DIR': {'type': str, 'default': None}, + + 'CHROME_HEADLESS': {'type': bool, 'default': True}, + 'CHROME_SANDBOX': {'type': bool, 'default': lambda c: not c['IN_DOCKER']}, + 'YOUTUBEDL_ARGS': {'type': list, 'default': ['--write-description', + '--write-info-json', + '--write-annotations', + '--write-thumbnail', + '--no-call-home', + '--user-agent', + '--all-subs', + '--extract-audio', + '--keep-video', + '--ignore-errors', + '--geo-bypass', + '--audio-format', 'mp3', + '--audio-quality', '320K', + '--embed-thumbnail', + '--add-metadata']}, + + 'WGET_ARGS': {'type': list, 'default': ['--no-verbose', + '--adjust-extension', + '--convert-links', + '--force-directories', + '--backup-converted', + '--span-hosts', + '--no-parent', + '-e', 'robots=off', + ]}, + 'CURL_ARGS': {'type': list, 'default': ['--silent', + '--location', + '--compressed' + ]}, + 'GIT_ARGS': {'type': list, 'default': ['--recursive']}, + }, + + 'SEARCH_BACKEND_CONFIG' : { + 'USE_INDEXING_BACKEND': {'type': bool, 'default': True}, + 'USE_SEARCHING_BACKEND': {'type': bool, 'default': True}, + 'SEARCH_BACKEND_ENGINE': {'type': str, 'default': 'ripgrep'}, + 'SEARCH_BACKEND_HOST_NAME': {'type': str, 'default': 'localhost'}, + 'SEARCH_BACKEND_PORT': {'type': int, 'default': 1491}, + 'SEARCH_BACKEND_PASSWORD': {'type': str, 'default': 'SecretPassword'}, + # SONIC + 'SONIC_COLLECTION': {'type': str, 'default': 'archivebox'}, + 'SONIC_BUCKET': {'type': str, 'default': 'snapshots'}, + }, + + 'DEPENDENCY_CONFIG': { + 'USE_CURL': {'type': bool, 'default': True}, + 'USE_WGET': {'type': bool, 'default': True}, + 'USE_SINGLEFILE': {'type': bool, 'default': True}, + 'USE_READABILITY': {'type': bool, 'default': True}, + 'USE_MERCURY': {'type': bool, 'default': True}, + 'USE_GIT': {'type': bool, 'default': True}, + 'USE_CHROME': {'type': bool, 'default': True}, + 'USE_NODE': {'type': bool, 'default': True}, + 'USE_YOUTUBEDL': {'type': bool, 'default': True}, + 'USE_RIPGREP': {'type': bool, 'default': True}, + + 'CURL_BINARY': {'type': str, 'default': 'curl'}, + 'GIT_BINARY': {'type': str, 'default': 'git'}, + 'WGET_BINARY': {'type': str, 'default': 'wget'}, + 'SINGLEFILE_BINARY': {'type': str, 'default': 'single-file'}, + 'READABILITY_BINARY': {'type': str, 'default': 'readability-extractor'}, + 'MERCURY_BINARY': {'type': str, 'default': 'mercury-parser'}, + 'YOUTUBEDL_BINARY': {'type': str, 'default': 'youtube-dl'}, + 'NODE_BINARY': {'type': str, 'default': 'node'}, + 'RIPGREP_BINARY': {'type': str, 'default': 'rg'}, + 'CHROME_BINARY': {'type': str, 'default': None}, + + 'POCKET_CONSUMER_KEY': {'type': str, 'default': None}, + 'POCKET_ACCESS_TOKENS': {'type': dict, 'default': {}}, + }, +} + + +########################## Backwards-Compatibility ############################# + + +# for backwards compatibility with old config files, check old/deprecated names for each key +CONFIG_ALIASES = { + alias: key + for section in CONFIG_SCHEMA.values() + for key, default in section.items() + for alias in default.get('aliases', ()) +} +USER_CONFIG = {key for section in CONFIG_SCHEMA.values() for key in section.keys()} + +def get_real_name(key: str) -> str: + """get the current canonical name for a given deprecated config key""" + return CONFIG_ALIASES.get(key.upper().strip(), key.upper().strip()) + + + +################################ Constants ##################################### + +PACKAGE_DIR_NAME = 'archivebox' +TEMPLATES_DIR_NAME = 'themes' + +ARCHIVE_DIR_NAME = 'archive' +SOURCES_DIR_NAME = 'sources' +LOGS_DIR_NAME = 'logs' +STATIC_DIR_NAME = 'static' +SQL_INDEX_FILENAME = 'index.sqlite3' +JSON_INDEX_FILENAME = 'index.json' +HTML_INDEX_FILENAME = 'index.html' +ROBOTS_TXT_FILENAME = 'robots.txt' +FAVICON_FILENAME = 'favicon.ico' +CONFIG_FILENAME = 'ArchiveBox.conf' + +DEFAULT_CLI_COLORS = { + 'reset': '\033[00;00m', + 'lightblue': '\033[01;30m', + 'lightyellow': '\033[01;33m', + 'lightred': '\033[01;35m', + 'red': '\033[01;31m', + 'green': '\033[01;32m', + 'blue': '\033[01;34m', + 'white': '\033[01;37m', + 'black': '\033[01;30m', +} +ANSI = {k: '' for k in DEFAULT_CLI_COLORS.keys()} + +COLOR_DICT = defaultdict(lambda: [(0, 0, 0), (0, 0, 0)], { + '00': [(0, 0, 0), (0, 0, 0)], + '30': [(0, 0, 0), (0, 0, 0)], + '31': [(255, 0, 0), (128, 0, 0)], + '32': [(0, 200, 0), (0, 128, 0)], + '33': [(255, 255, 0), (128, 128, 0)], + '34': [(0, 0, 255), (0, 0, 128)], + '35': [(255, 0, 255), (128, 0, 128)], + '36': [(0, 255, 255), (0, 128, 128)], + '37': [(255, 255, 255), (255, 255, 255)], +}) + +STATICFILE_EXTENSIONS = { + # 99.999% of the time, URLs ending in these extensions are static files + # that can be downloaded as-is, not html pages that need to be rendered + 'gif', 'jpeg', 'jpg', 'png', 'tif', 'tiff', 'wbmp', 'ico', 'jng', 'bmp', + 'svg', 'svgz', 'webp', 'ps', 'eps', 'ai', + 'mp3', 'mp4', 'm4a', 'mpeg', 'mpg', 'mkv', 'mov', 'webm', 'm4v', + 'flv', 'wmv', 'avi', 'ogg', 'ts', 'm3u8', + 'pdf', 'txt', 'rtf', 'rtfd', 'doc', 'docx', 'ppt', 'pptx', 'xls', 'xlsx', + 'atom', 'rss', 'css', 'js', 'json', + 'dmg', 'iso', 'img', + 'rar', 'war', 'hqx', 'zip', 'gz', 'bz2', '7z', + + # Less common extensions to consider adding later + # jar, swf, bin, com, exe, dll, deb + # ear, hqx, eot, wmlc, kml, kmz, cco, jardiff, jnlp, run, msi, msp, msm, + # pl pm, prc pdb, rar, rpm, sea, sit, tcl tk, der, pem, crt, xpi, xspf, + # ra, mng, asx, asf, 3gpp, 3gp, mid, midi, kar, jad, wml, htc, mml + + # These are always treated as pages, not as static files, never add them: + # html, htm, shtml, xhtml, xml, aspx, php, cgi +} + + + +############################## Derived Config ################################## + + +DYNAMIC_CONFIG_SCHEMA: ConfigDefaultDict = { + 'TERM_WIDTH': {'default': lambda c: lambda: shutil.get_terminal_size((100, 10)).columns}, + 'USER': {'default': lambda c: getpass.getuser() or os.getlogin()}, + 'ANSI': {'default': lambda c: DEFAULT_CLI_COLORS if c['USE_COLOR'] else {k: '' for k in DEFAULT_CLI_COLORS.keys()}}, + + 'PACKAGE_DIR': {'default': lambda c: Path(__file__).resolve().parent}, + 'TEMPLATES_DIR': {'default': lambda c: c['PACKAGE_DIR'] / TEMPLATES_DIR_NAME}, + + 'OUTPUT_DIR': {'default': lambda c: Path(c['OUTPUT_DIR']).resolve() if c['OUTPUT_DIR'] else Path(os.curdir).resolve()}, + 'ARCHIVE_DIR': {'default': lambda c: c['OUTPUT_DIR'] / ARCHIVE_DIR_NAME}, + 'SOURCES_DIR': {'default': lambda c: c['OUTPUT_DIR'] / SOURCES_DIR_NAME}, + 'LOGS_DIR': {'default': lambda c: c['OUTPUT_DIR'] / LOGS_DIR_NAME}, + 'CONFIG_FILE': {'default': lambda c: Path(c['CONFIG_FILE']).resolve() if c['CONFIG_FILE'] else c['OUTPUT_DIR'] / CONFIG_FILENAME}, + 'COOKIES_FILE': {'default': lambda c: c['COOKIES_FILE'] and Path(c['COOKIES_FILE']).resolve()}, + 'CHROME_USER_DATA_DIR': {'default': lambda c: find_chrome_data_dir() if c['CHROME_USER_DATA_DIR'] is None else (Path(c['CHROME_USER_DATA_DIR']).resolve() if c['CHROME_USER_DATA_DIR'] else None)}, # None means unset, so we autodetect it with find_chrome_Data_dir(), but emptystring '' means user manually set it to '', and we should store it as None + 'URL_BLACKLIST_PTN': {'default': lambda c: c['URL_BLACKLIST'] and re.compile(c['URL_BLACKLIST'] or '', re.IGNORECASE | re.UNICODE | re.MULTILINE)}, + + 'ARCHIVEBOX_BINARY': {'default': lambda c: sys.argv[0]}, + 'VERSION': {'default': lambda c: json.loads((Path(c['PACKAGE_DIR']) / 'package.json').read_text().strip())['version']}, + 'GIT_SHA': {'default': lambda c: c['VERSION'].split('+')[-1] or 'unknown'}, + + 'PYTHON_BINARY': {'default': lambda c: sys.executable}, + 'PYTHON_ENCODING': {'default': lambda c: sys.stdout.encoding.upper()}, + 'PYTHON_VERSION': {'default': lambda c: '{}.{}.{}'.format(*sys.version_info[:3])}, + + 'DJANGO_BINARY': {'default': lambda c: django.__file__.replace('__init__.py', 'bin/django-admin.py')}, + 'DJANGO_VERSION': {'default': lambda c: '{}.{}.{} {} ({})'.format(*django.VERSION)}, + + 'USE_CURL': {'default': lambda c: c['USE_CURL'] and (c['SAVE_FAVICON'] or c['SAVE_TITLE'] or c['SAVE_ARCHIVE_DOT_ORG'])}, + 'CURL_VERSION': {'default': lambda c: bin_version(c['CURL_BINARY']) if c['USE_CURL'] else None}, + 'CURL_USER_AGENT': {'default': lambda c: c['CURL_USER_AGENT'].format(**c)}, + 'CURL_ARGS': {'default': lambda c: c['CURL_ARGS'] or []}, + 'SAVE_FAVICON': {'default': lambda c: c['USE_CURL'] and c['SAVE_FAVICON']}, + 'SAVE_ARCHIVE_DOT_ORG': {'default': lambda c: c['USE_CURL'] and c['SAVE_ARCHIVE_DOT_ORG']}, + + 'USE_WGET': {'default': lambda c: c['USE_WGET'] and (c['SAVE_WGET'] or c['SAVE_WARC'])}, + 'WGET_VERSION': {'default': lambda c: bin_version(c['WGET_BINARY']) if c['USE_WGET'] else None}, + 'WGET_AUTO_COMPRESSION': {'default': lambda c: wget_supports_compression(c) if c['USE_WGET'] else False}, + 'WGET_USER_AGENT': {'default': lambda c: c['WGET_USER_AGENT'].format(**c)}, + 'SAVE_WGET': {'default': lambda c: c['USE_WGET'] and c['SAVE_WGET']}, + 'SAVE_WARC': {'default': lambda c: c['USE_WGET'] and c['SAVE_WARC']}, + 'WGET_ARGS': {'default': lambda c: c['WGET_ARGS'] or []}, + + 'RIPGREP_VERSION': {'default': lambda c: bin_version(c['RIPGREP_BINARY']) if c['USE_RIPGREP'] else None}, + + 'USE_SINGLEFILE': {'default': lambda c: c['USE_SINGLEFILE'] and c['SAVE_SINGLEFILE']}, + 'SINGLEFILE_VERSION': {'default': lambda c: bin_version(c['SINGLEFILE_BINARY']) if c['USE_SINGLEFILE'] else None}, + + 'USE_READABILITY': {'default': lambda c: c['USE_READABILITY'] and c['SAVE_READABILITY']}, + 'READABILITY_VERSION': {'default': lambda c: bin_version(c['READABILITY_BINARY']) if c['USE_READABILITY'] else None}, + + 'USE_MERCURY': {'default': lambda c: c['USE_MERCURY'] and c['SAVE_MERCURY']}, + 'MERCURY_VERSION': {'default': lambda c: '1.0.0' if shutil.which(str(bin_path(c['MERCURY_BINARY']))) else None}, # mercury is unversioned + + 'USE_GIT': {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']}, + 'GIT_VERSION': {'default': lambda c: bin_version(c['GIT_BINARY']) if c['USE_GIT'] else None}, + 'SAVE_GIT': {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']}, + + 'USE_YOUTUBEDL': {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']}, + 'YOUTUBEDL_VERSION': {'default': lambda c: bin_version(c['YOUTUBEDL_BINARY']) if c['USE_YOUTUBEDL'] else None}, + 'SAVE_MEDIA': {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']}, + 'YOUTUBEDL_ARGS': {'default': lambda c: c['YOUTUBEDL_ARGS'] or []}, + + 'USE_CHROME': {'default': lambda c: c['USE_CHROME'] and (c['SAVE_PDF'] or c['SAVE_SCREENSHOT'] or c['SAVE_DOM'] or c['SAVE_SINGLEFILE'])}, + 'CHROME_BINARY': {'default': lambda c: c['CHROME_BINARY'] if c['CHROME_BINARY'] else find_chrome_binary()}, + 'CHROME_VERSION': {'default': lambda c: bin_version(c['CHROME_BINARY']) if c['USE_CHROME'] else None}, + + 'SAVE_PDF': {'default': lambda c: c['USE_CHROME'] and c['SAVE_PDF']}, + 'SAVE_SCREENSHOT': {'default': lambda c: c['USE_CHROME'] and c['SAVE_SCREENSHOT']}, + 'SAVE_DOM': {'default': lambda c: c['USE_CHROME'] and c['SAVE_DOM']}, + 'SAVE_SINGLEFILE': {'default': lambda c: c['USE_CHROME'] and c['SAVE_SINGLEFILE'] and c['USE_NODE']}, + 'SAVE_READABILITY': {'default': lambda c: c['USE_READABILITY'] and c['USE_NODE']}, + 'SAVE_MERCURY': {'default': lambda c: c['USE_MERCURY'] and c['USE_NODE']}, + + 'USE_NODE': {'default': lambda c: c['USE_NODE'] and (c['SAVE_READABILITY'] or c['SAVE_SINGLEFILE'] or c['SAVE_MERCURY'])}, + 'NODE_VERSION': {'default': lambda c: bin_version(c['NODE_BINARY']) if c['USE_NODE'] else None}, + + 'DEPENDENCIES': {'default': lambda c: get_dependency_info(c)}, + 'CODE_LOCATIONS': {'default': lambda c: get_code_locations(c)}, + 'EXTERNAL_LOCATIONS': {'default': lambda c: get_external_locations(c)}, + 'DATA_LOCATIONS': {'default': lambda c: get_data_locations(c)}, + 'CHROME_OPTIONS': {'default': lambda c: get_chrome_info(c)}, +} + + + +################################### Helpers #################################### + + +def load_config_val(key: str, + default: ConfigDefaultValue=None, + type: Optional[Type]=None, + aliases: Optional[Tuple[str, ...]]=None, + config: Optional[ConfigDict]=None, + env_vars: Optional[os._Environ]=None, + config_file_vars: Optional[Dict[str, str]]=None) -> ConfigValue: + """parse bool, int, and str key=value pairs from env""" + + + config_keys_to_check = (key, *(aliases or ())) + for key in config_keys_to_check: + if env_vars: + val = env_vars.get(key) + if val: + break + if config_file_vars: + val = config_file_vars.get(key) + if val: + break + + if type is None or val is None: + if callable(default): + assert isinstance(config, dict) + return default(config) + + return default + + elif type is bool: + if val.lower() in ('true', 'yes', '1'): + return True + elif val.lower() in ('false', 'no', '0'): + return False + else: + raise ValueError(f'Invalid configuration option {key}={val} (expected a boolean: True/False)') + + elif type is str: + if val.lower() in ('true', 'false', 'yes', 'no', '1', '0'): + raise ValueError(f'Invalid configuration option {key}={val} (expected a string)') + return val.strip() + + elif type is int: + if not val.isdigit(): + raise ValueError(f'Invalid configuration option {key}={val} (expected an integer)') + return int(val) + + elif type is list or type is dict: + return json.loads(val) + + raise Exception('Config values can only be str, bool, int or json') + + +def load_config_file(out_dir: str=None) -> Optional[Dict[str, str]]: + """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf""" + + out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve() + config_path = Path(out_dir) / CONFIG_FILENAME + if config_path.exists(): + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(config_path) + # flatten into one namespace + config_file_vars = { + key.upper(): val + for section, options in config_file.items() + for key, val in options.items() + } + # print('[i] Loaded config file', os.path.abspath(config_path)) + # print(config_file_vars) + return config_file_vars + return None + + +def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict: + """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf""" + + from .system import atomic_write + + CONFIG_HEADER = ( + """# This is the config file for your ArchiveBox collection. + # + # You can add options here manually in INI format, or automatically by running: + # archivebox config --set KEY=VALUE + # + # If you modify this file manually, make sure to update your archive after by running: + # archivebox init + # + # A list of all possible config with documentation and examples can be found here: + # https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration + + """) + + out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve() + config_path = Path(out_dir) / CONFIG_FILENAME + + if not config_path.exists(): + atomic_write(config_path, CONFIG_HEADER) + + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(config_path) + + with open(config_path, 'r') as old: + atomic_write(f'{config_path}.bak', old.read()) + + find_section = lambda key: [name for name, opts in CONFIG_SCHEMA.items() if key in opts][0] + + # Set up sections in empty config file + for key, val in config.items(): + section = find_section(key) + if section in config_file: + existing_config = dict(config_file[section]) + else: + existing_config = {} + config_file[section] = {**existing_config, key: val} + + # always make sure there's a SECRET_KEY defined for Django + existing_secret_key = None + if 'SERVER_CONFIG' in config_file and 'SECRET_KEY' in config_file['SERVER_CONFIG']: + existing_secret_key = config_file['SERVER_CONFIG']['SECRET_KEY'] + + if (not existing_secret_key) or ('not a valid secret' in existing_secret_key): + from django.utils.crypto import get_random_string + chars = 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.' + random_secret_key = get_random_string(50, chars) + if 'SERVER_CONFIG' in config_file: + config_file['SERVER_CONFIG']['SECRET_KEY'] = random_secret_key + else: + config_file['SERVER_CONFIG'] = {'SECRET_KEY': random_secret_key} + + with open(config_path, 'w+') as new: + config_file.write(new) + + try: + # validate the config by attempting to re-parse it + CONFIG = load_all_config() + return { + key.upper(): CONFIG.get(key.upper()) + for key in config.keys() + } + except: + # something went horribly wrong, rever to the previous version + with open(f'{config_path}.bak', 'r') as old: + atomic_write(config_path, old.read()) + + if Path(f'{config_path}.bak').exists(): + os.remove(f'{config_path}.bak') + + return {} + + + +def load_config(defaults: ConfigDefaultDict, + config: Optional[ConfigDict]=None, + out_dir: Optional[str]=None, + env_vars: Optional[os._Environ]=None, + config_file_vars: Optional[Dict[str, str]]=None) -> ConfigDict: + + env_vars = env_vars or os.environ + config_file_vars = config_file_vars or load_config_file(out_dir=out_dir) + + extended_config: ConfigDict = config.copy() if config else {} + for key, default in defaults.items(): + try: + extended_config[key] = load_config_val( + key, + default=default['default'], + type=default.get('type'), + aliases=default.get('aliases'), + config=extended_config, + env_vars=env_vars, + config_file_vars=config_file_vars, + ) + except KeyboardInterrupt: + raise SystemExit(0) + except Exception as e: + stderr() + stderr(f'[X] Error while loading configuration value: {key}', color='red', config=extended_config) + stderr(' {}: {}'.format(e.__class__.__name__, e)) + stderr() + stderr(' Check your config for mistakes and try again (your archive data is unaffected).') + stderr() + stderr(' For config documentation and examples see:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration') + stderr() + raise + raise SystemExit(2) + + return extended_config + +# def write_config(config: ConfigDict): + +# with open(os.path.join(config['OUTPUT_DIR'], CONFIG_FILENAME), 'w+') as f: + + +# Logging Helpers +def stdout(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if color: + strs = [ansi[color], ' '.join(str(a) for a in args), ansi['reset'], '\n'] + else: + strs = [' '.join(str(a) for a in args), '\n'] + + sys.stdout.write(prefix + ''.join(strs)) + +def stderr(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if color: + strs = [ansi[color], ' '.join(str(a) for a in args), ansi['reset'], '\n'] + else: + strs = [' '.join(str(a) for a in args), '\n'] + + sys.stderr.write(prefix + ''.join(strs)) + +def hint(text: Union[Tuple[str, ...], List[str], str], prefix=' ', config: Optional[ConfigDict]=None) -> None: + ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI + + if isinstance(text, str): + stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text, **ansi)) + else: + stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text[0], **ansi)) + for line in text[1:]: + stderr('{} {}'.format(prefix, line)) + + +# Dependency Metadata Helpers +def bin_version(binary: Optional[str]) -> Optional[str]: + """check the presence and return valid version line of a specified binary""" + + abspath = bin_path(binary) + if not binary or not abspath: + return None + + try: + version_str = run([abspath, "--version"], stdout=PIPE).stdout.strip().decode() + # take first 3 columns of first line of version info + return ' '.join(version_str.split('\n')[0].strip().split()[:3]) + except OSError: + pass + # stderr(f'[X] Unable to find working version of dependency: {binary}', color='red') + # stderr(' Make sure it\'s installed, then confirm it\'s working by running:') + # stderr(f' {binary} --version') + # stderr() + # stderr(' If you don\'t want to install it, you can disable it via config. See here for more info:') + # stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Install') + return None + +def bin_path(binary: Optional[str]) -> Optional[str]: + if binary is None: + return None + + node_modules_bin = Path('.') / 'node_modules' / '.bin' / binary + if node_modules_bin.exists(): + return str(node_modules_bin.resolve()) + + return shutil.which(str(Path(binary).expanduser())) or shutil.which(str(binary)) or binary + +def bin_hash(binary: Optional[str]) -> Optional[str]: + if binary is None: + return None + abs_path = bin_path(binary) + if abs_path is None or not Path(abs_path).exists(): + return None + + file_hash = md5() + with io.open(abs_path, mode='rb') as f: + for chunk in iter(lambda: f.read(io.DEFAULT_BUFFER_SIZE), b''): + file_hash.update(chunk) + + return f'md5:{file_hash.hexdigest()}' + +def find_chrome_binary() -> Optional[str]: + """find any installed chrome binaries in the default locations""" + # Precedence: Chromium, Chrome, Beta, Canary, Unstable, Dev + # make sure data dir finding precedence order always matches binary finding order + default_executable_paths = ( + 'chromium-browser', + 'chromium', + '/Applications/Chromium.app/Contents/MacOS/Chromium', + 'chrome', + 'google-chrome', + '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', + 'google-chrome-stable', + 'google-chrome-beta', + 'google-chrome-canary', + '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary', + 'google-chrome-unstable', + 'google-chrome-dev', + ) + for name in default_executable_paths: + full_path_exists = shutil.which(name) + if full_path_exists: + return name + + return None + +def find_chrome_data_dir() -> Optional[str]: + """find any installed chrome user data directories in the default locations""" + # Precedence: Chromium, Chrome, Beta, Canary, Unstable, Dev + # make sure data dir finding precedence order always matches binary finding order + default_profile_paths = ( + '~/.config/chromium', + '~/Library/Application Support/Chromium', + '~/AppData/Local/Chromium/User Data', + '~/.config/chrome', + '~/.config/google-chrome', + '~/Library/Application Support/Google/Chrome', + '~/AppData/Local/Google/Chrome/User Data', + '~/.config/google-chrome-stable', + '~/.config/google-chrome-beta', + '~/Library/Application Support/Google/Chrome Canary', + '~/AppData/Local/Google/Chrome SxS/User Data', + '~/.config/google-chrome-unstable', + '~/.config/google-chrome-dev', + ) + for path in default_profile_paths: + full_path = Path(path).resolve() + if full_path.exists(): + return full_path + return None + +def wget_supports_compression(config): + try: + cmd = [ + config['WGET_BINARY'], + "--compression=auto", + "--help", + ] + return not run(cmd, stdout=DEVNULL, stderr=DEVNULL).returncode + except (FileNotFoundError, OSError): + return False + +def get_code_locations(config: ConfigDict) -> SimpleConfigValueDict: + return { + 'PACKAGE_DIR': { + 'path': (config['PACKAGE_DIR']).resolve(), + 'enabled': True, + 'is_valid': (config['PACKAGE_DIR'] / '__main__.py').exists(), + }, + 'TEMPLATES_DIR': { + 'path': (config['TEMPLATES_DIR']).resolve(), + 'enabled': True, + 'is_valid': (config['TEMPLATES_DIR'] / config['ACTIVE_THEME'] / 'static').exists(), + }, + # 'NODE_MODULES_DIR': { + # 'path': , + # 'enabled': , + # 'is_valid': (...).exists(), + # }, + } + +def get_external_locations(config: ConfigDict) -> ConfigValue: + abspath = lambda path: None if path is None else Path(path).resolve() + return { + 'CHROME_USER_DATA_DIR': { + 'path': abspath(config['CHROME_USER_DATA_DIR']), + 'enabled': config['USE_CHROME'] and config['CHROME_USER_DATA_DIR'], + 'is_valid': False if config['CHROME_USER_DATA_DIR'] is None else (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(), + }, + 'COOKIES_FILE': { + 'path': abspath(config['COOKIES_FILE']), + 'enabled': config['USE_WGET'] and config['COOKIES_FILE'], + 'is_valid': False if config['COOKIES_FILE'] is None else Path(config['COOKIES_FILE']).exists(), + }, + } + +def get_data_locations(config: ConfigDict) -> ConfigValue: + return { + 'OUTPUT_DIR': { + 'path': config['OUTPUT_DIR'].resolve(), + 'enabled': True, + 'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(), + }, + 'SOURCES_DIR': { + 'path': config['SOURCES_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['SOURCES_DIR'].exists(), + }, + 'LOGS_DIR': { + 'path': config['LOGS_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['LOGS_DIR'].exists(), + }, + 'ARCHIVE_DIR': { + 'path': config['ARCHIVE_DIR'].resolve(), + 'enabled': True, + 'is_valid': config['ARCHIVE_DIR'].exists(), + }, + 'CONFIG_FILE': { + 'path': config['CONFIG_FILE'].resolve(), + 'enabled': True, + 'is_valid': config['CONFIG_FILE'].exists(), + }, + 'SQL_INDEX': { + 'path': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).resolve(), + 'enabled': True, + 'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(), + }, + } + +def get_dependency_info(config: ConfigDict) -> ConfigValue: + return { + 'ARCHIVEBOX_BINARY': { + 'path': bin_path(config['ARCHIVEBOX_BINARY']), + 'version': config['VERSION'], + 'hash': bin_hash(config['ARCHIVEBOX_BINARY']), + 'enabled': True, + 'is_valid': True, + }, + 'PYTHON_BINARY': { + 'path': bin_path(config['PYTHON_BINARY']), + 'version': config['PYTHON_VERSION'], + 'hash': bin_hash(config['PYTHON_BINARY']), + 'enabled': True, + 'is_valid': bool(config['DJANGO_VERSION']), + }, + 'DJANGO_BINARY': { + 'path': bin_path(config['DJANGO_BINARY']), + 'version': config['DJANGO_VERSION'], + 'hash': bin_hash(config['DJANGO_BINARY']), + 'enabled': True, + 'is_valid': bool(config['DJANGO_VERSION']), + }, + 'CURL_BINARY': { + 'path': bin_path(config['CURL_BINARY']), + 'version': config['CURL_VERSION'], + 'hash': bin_hash(config['PYTHON_BINARY']), + 'enabled': config['USE_CURL'], + 'is_valid': bool(config['CURL_VERSION']), + }, + 'WGET_BINARY': { + 'path': bin_path(config['WGET_BINARY']), + 'version': config['WGET_VERSION'], + 'hash': bin_hash(config['WGET_BINARY']), + 'enabled': config['USE_WGET'], + 'is_valid': bool(config['WGET_VERSION']), + }, + 'NODE_BINARY': { + 'path': bin_path(config['NODE_BINARY']), + 'version': config['NODE_VERSION'], + 'hash': bin_hash(config['NODE_BINARY']), + 'enabled': config['USE_NODE'], + 'is_valid': bool(config['SINGLEFILE_VERSION']), + }, + 'SINGLEFILE_BINARY': { + 'path': bin_path(config['SINGLEFILE_BINARY']), + 'version': config['SINGLEFILE_VERSION'], + 'hash': bin_hash(config['SINGLEFILE_BINARY']), + 'enabled': config['USE_SINGLEFILE'], + 'is_valid': bool(config['SINGLEFILE_VERSION']), + }, + 'READABILITY_BINARY': { + 'path': bin_path(config['READABILITY_BINARY']), + 'version': config['READABILITY_VERSION'], + 'hash': bin_hash(config['READABILITY_BINARY']), + 'enabled': config['USE_READABILITY'], + 'is_valid': bool(config['READABILITY_VERSION']), + }, + 'MERCURY_BINARY': { + 'path': bin_path(config['MERCURY_BINARY']), + 'version': config['MERCURY_VERSION'], + 'hash': bin_hash(config['MERCURY_BINARY']), + 'enabled': config['USE_MERCURY'], + 'is_valid': bool(config['MERCURY_VERSION']), + }, + 'GIT_BINARY': { + 'path': bin_path(config['GIT_BINARY']), + 'version': config['GIT_VERSION'], + 'hash': bin_hash(config['GIT_BINARY']), + 'enabled': config['USE_GIT'], + 'is_valid': bool(config['GIT_VERSION']), + }, + 'YOUTUBEDL_BINARY': { + 'path': bin_path(config['YOUTUBEDL_BINARY']), + 'version': config['YOUTUBEDL_VERSION'], + 'hash': bin_hash(config['YOUTUBEDL_BINARY']), + 'enabled': config['USE_YOUTUBEDL'], + 'is_valid': bool(config['YOUTUBEDL_VERSION']), + }, + 'CHROME_BINARY': { + 'path': bin_path(config['CHROME_BINARY']), + 'version': config['CHROME_VERSION'], + 'hash': bin_hash(config['CHROME_BINARY']), + 'enabled': config['USE_CHROME'], + 'is_valid': bool(config['CHROME_VERSION']), + }, + 'RIPGREP_BINARY': { + 'path': bin_path(config['RIPGREP_BINARY']), + 'version': config['RIPGREP_VERSION'], + 'hash': bin_hash(config['RIPGREP_BINARY']), + 'enabled': config['USE_RIPGREP'], + 'is_valid': bool(config['RIPGREP_VERSION']), + }, + # TODO: add an entry for the sonic search backend? + # 'SONIC_BINARY': { + # 'path': bin_path(config['SONIC_BINARY']), + # 'version': config['SONIC_VERSION'], + # 'hash': bin_hash(config['SONIC_BINARY']), + # 'enabled': config['USE_SONIC'], + # 'is_valid': bool(config['SONIC_VERSION']), + # }, + } + +def get_chrome_info(config: ConfigDict) -> ConfigValue: + return { + 'TIMEOUT': config['TIMEOUT'], + 'RESOLUTION': config['RESOLUTION'], + 'CHECK_SSL_VALIDITY': config['CHECK_SSL_VALIDITY'], + 'CHROME_BINARY': config['CHROME_BINARY'], + 'CHROME_HEADLESS': config['CHROME_HEADLESS'], + 'CHROME_SANDBOX': config['CHROME_SANDBOX'], + 'CHROME_USER_AGENT': config['CHROME_USER_AGENT'], + 'CHROME_USER_DATA_DIR': config['CHROME_USER_DATA_DIR'], + } + + +# ****************************************************************************** +# ****************************************************************************** +# ******************************** Load Config ********************************* +# ******* (compile the defaults, configs, and metadata all into CONFIG) ******** +# ****************************************************************************** +# ****************************************************************************** + + +def load_all_config(): + CONFIG: ConfigDict = {} + for section_name, section_config in CONFIG_SCHEMA.items(): + CONFIG = load_config(section_config, CONFIG) + + return load_config(DYNAMIC_CONFIG_SCHEMA, CONFIG) + +# add all final config values in CONFIG to globals in this file +CONFIG = load_all_config() +globals().update(CONFIG) +# this lets us do: from .config import DEBUG, MEDIA_TIMEOUT, ... + + +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** +# ****************************************************************************** + + + +########################### System Environment Setup ########################### + + +# Set timezone to UTC and umask to OUTPUT_PERMISSIONS +os.environ["TZ"] = 'UTC' +os.umask(0o777 - int(OUTPUT_PERMISSIONS, base=8)) # noqa: F821 + +# add ./node_modules/.bin to $PATH so we can use node scripts in extractors +NODE_BIN_PATH = str((Path(CONFIG["OUTPUT_DIR"]).absolute() / 'node_modules' / '.bin')) +sys.path.append(NODE_BIN_PATH) + + + + +########################### Config Validity Checkers ########################### + + +def check_system_config(config: ConfigDict=CONFIG) -> None: + ### Check system environment + if config['USER'] == 'root': + stderr('[!] ArchiveBox should never be run as root!', color='red') + stderr(' For more information, see the security overview documentation:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root') + raise SystemExit(2) + + ### Check Python environment + if sys.version_info[:3] < (3, 6, 0): + stderr(f'[X] Python version is not new enough: {config["PYTHON_VERSION"]} (>3.6 is required)', color='red') + stderr(' See https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#python for help upgrading your Python installation.') + raise SystemExit(2) + + if config['PYTHON_ENCODING'] not in ('UTF-8', 'UTF8'): + stderr(f'[X] Your system is running python3 scripts with a bad locale setting: {config["PYTHON_ENCODING"]} (it should be UTF-8).', color='red') + stderr(' To fix it, add the line "export PYTHONIOENCODING=UTF-8" to your ~/.bashrc file (without quotes)') + stderr(' Or if you\'re using ubuntu/debian, run "dpkg-reconfigure locales"') + stderr('') + stderr(' Confirm that it\'s fixed by opening a new shell and running:') + stderr(' python3 -c "import sys; print(sys.stdout.encoding)" # should output UTF-8') + raise SystemExit(2) + + # stderr('[i] Using Chrome binary: {}'.format(shutil.which(CHROME_BINARY) or CHROME_BINARY)) + # stderr('[i] Using Chrome data dir: {}'.format(os.path.abspath(CHROME_USER_DATA_DIR))) + if config['CHROME_USER_DATA_DIR'] is not None: + if not (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(): + stderr('[X] Could not find profile "Default" in CHROME_USER_DATA_DIR.', color='red') + stderr(f' {config["CHROME_USER_DATA_DIR"]}') + stderr(' Make sure you set it to a Chrome user data directory containing a Default profile folder.') + stderr(' For more info see:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#CHROME_USER_DATA_DIR') + if '/Default' in str(config['CHROME_USER_DATA_DIR']): + stderr() + stderr(' Try removing /Default from the end e.g.:') + stderr(' CHROME_USER_DATA_DIR="{}"'.format(config['CHROME_USER_DATA_DIR'].split('/Default')[0])) + raise SystemExit(2) + + +def check_dependencies(config: ConfigDict=CONFIG, show_help: bool=True) -> None: + invalid_dependencies = [ + (name, info) for name, info in config['DEPENDENCIES'].items() + if info['enabled'] and not info['is_valid'] + ] + if invalid_dependencies and show_help: + stderr(f'[!] Warning: Missing {len(invalid_dependencies)} recommended dependencies', color='lightyellow') + for dependency, info in invalid_dependencies: + stderr( + ' ! {}: {} ({})'.format( + dependency, + info['path'] or 'unable to find binary', + info['version'] or 'unable to detect version', + ) + ) + if dependency in ('SINGLEFILE_BINARY', 'READABILITY_BINARY', 'MERCURY_BINARY'): + hint(('npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"', + f'or archivebox config --set SAVE_{dependency.rsplit("_", 1)[0]}=False to silence this warning', + ''), prefix=' ') + stderr('') + + if config['TIMEOUT'] < 5: + stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red') + stderr(' You must allow *at least* 5 seconds for indexing and archive methods to run succesfully.') + stderr(' (Setting it to somewhere between 30 and 3000 seconds is recommended)') + stderr() + stderr(' If you want to make ArchiveBox run faster, disable specific archive methods instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles') + stderr() + + elif config['USE_CHROME'] and config['TIMEOUT'] < 15: + stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red') + stderr(' Chrome will fail to archive all sites if set to less than ~15 seconds.') + stderr(' (Setting it to somewhere between 30 and 300 seconds is recommended)') + stderr() + stderr(' If you want to make ArchiveBox run faster, disable specific archive methods instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles') + stderr() + + if config['USE_YOUTUBEDL'] and config['MEDIA_TIMEOUT'] < 20: + stderr(f'[!] Warning: MEDIA_TIMEOUT is set too low! (currently set to MEDIA_TIMEOUT={config["MEDIA_TIMEOUT"]} seconds)', color='red') + stderr(' Youtube-dl will fail to archive all media if set to less than ~20 seconds.') + stderr(' (Setting it somewhere over 60 seconds is recommended)') + stderr() + stderr(' If you want to disable media archiving entirely, set SAVE_MEDIA=False instead:') + stderr(' https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_media') + stderr() + +def check_data_folder(out_dir: Union[str, Path, None]=None, config: ConfigDict=CONFIG) -> None: + output_dir = out_dir or config['OUTPUT_DIR'] + assert isinstance(output_dir, (str, Path)) + + sql_index_exists = (Path(output_dir) / SQL_INDEX_FILENAME).exists() + if not sql_index_exists: + stderr('[X] No archivebox index found in the current directory.', color='red') + stderr(f' {output_dir}', color='lightyellow') + stderr() + stderr(' {lightred}Hint{reset}: Are you running archivebox in the right folder?'.format(**config['ANSI'])) + stderr(' cd path/to/your/archive/folder') + stderr(' archivebox [command]') + stderr() + stderr(' {lightred}Hint{reset}: To create a new archive collection or import existing data in this folder, run:'.format(**config['ANSI'])) + stderr(' archivebox init') + raise SystemExit(2) + + from .index.sql import list_migrations + + pending_migrations = [name for status, name in list_migrations() if not status] + + if (not sql_index_exists) or pending_migrations: + if sql_index_exists: + pending_operation = f'apply the {len(pending_migrations)} pending migrations' + else: + pending_operation = 'generate the new SQL main index' + + stderr('[X] This collection was created with an older version of ArchiveBox and must be upgraded first.', color='lightyellow') + stderr(f' {output_dir}') + stderr() + stderr(f' To upgrade it to the latest version and {pending_operation} run:') + stderr(' archivebox init') + raise SystemExit(3) + + sources_dir = Path(output_dir) / SOURCES_DIR_NAME + if not sources_dir.exists(): + sources_dir.mkdir() + + + +def setup_django(out_dir: Path=None, check_db=False, config: ConfigDict=CONFIG, in_memory_db=False) -> None: + check_system_config() + + output_dir = out_dir or Path(config['OUTPUT_DIR']) + + assert isinstance(output_dir, Path) and isinstance(config['PACKAGE_DIR'], Path) + + try: + import django + sys.path.append(str(config['PACKAGE_DIR'])) + os.environ.setdefault('OUTPUT_DIR', str(output_dir)) + assert (config['PACKAGE_DIR'] / 'core' / 'settings.py').exists(), 'settings.py was not found at archivebox/core/settings.py' + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings') + + if in_memory_db: + # Put the db in memory and run migrations in case any command requires it + from django.core.management import call_command + os.environ.setdefault("ARCHIVEBOX_DATABASE_NAME", ":memory:") + django.setup() + call_command("migrate", interactive=False, verbosity=0) + else: + django.setup() + + if check_db: + sql_index_path = Path(output_dir) / SQL_INDEX_FILENAME + assert sql_index_path.exists(), ( + f'No database file {SQL_INDEX_FILENAME} found in OUTPUT_DIR: {config["OUTPUT_DIR"]}') + except KeyboardInterrupt: + raise SystemExit(2) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/config_stubs.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/config_stubs.py new file mode 100644 index 0000000..988f58a --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/config_stubs.py @@ -0,0 +1,113 @@ +from pathlib import Path +from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any, List +from mypy_extensions import TypedDict + + + +SimpleConfigValue = Union[str, bool, int, None, Pattern, Dict[str, Any]] +SimpleConfigValueDict = Dict[str, SimpleConfigValue] +SimpleConfigValueGetter = Callable[[], SimpleConfigValue] +ConfigValue = Union[SimpleConfigValue, SimpleConfigValueDict, SimpleConfigValueGetter] + + +class BaseConfig(TypedDict): + pass + +class ConfigDict(BaseConfig, total=False): + """ + # Regenerate by pasting this quine into `archivebox shell` 🥚 + from archivebox.config import ConfigDict, CONFIG_DEFAULTS + print('class ConfigDict(BaseConfig, total=False):') + print(' ' + '"'*3 + ConfigDict.__doc__ + '"'*3) + for section, configs in CONFIG_DEFAULTS.items(): + for key, attrs in configs.items(): + Type, default = attrs['type'], attrs['default'] + if default is None: + print(f' {key}: Optional[{Type.__name__}]') + else: + print(f' {key}: {Type.__name__}') + print() + """ + IS_TTY: bool + USE_COLOR: bool + SHOW_PROGRESS: bool + IN_DOCKER: bool + + PACKAGE_DIR: Path + OUTPUT_DIR: Path + CONFIG_FILE: Path + ONLY_NEW: bool + TIMEOUT: int + MEDIA_TIMEOUT: int + OUTPUT_PERMISSIONS: str + RESTRICT_FILE_NAMES: str + URL_BLACKLIST: str + + SECRET_KEY: Optional[str] + BIND_ADDR: str + ALLOWED_HOSTS: str + DEBUG: bool + PUBLIC_INDEX: bool + PUBLIC_SNAPSHOTS: bool + FOOTER_INFO: str + ACTIVE_THEME: str + + SAVE_TITLE: bool + SAVE_FAVICON: bool + SAVE_WGET: bool + SAVE_WGET_REQUISITES: bool + SAVE_SINGLEFILE: bool + SAVE_READABILITY: bool + SAVE_MERCURY: bool + SAVE_PDF: bool + SAVE_SCREENSHOT: bool + SAVE_DOM: bool + SAVE_WARC: bool + SAVE_GIT: bool + SAVE_MEDIA: bool + SAVE_ARCHIVE_DOT_ORG: bool + + RESOLUTION: str + GIT_DOMAINS: str + CHECK_SSL_VALIDITY: bool + CURL_USER_AGENT: str + WGET_USER_AGENT: str + CHROME_USER_AGENT: str + COOKIES_FILE: Union[str, Path, None] + CHROME_USER_DATA_DIR: Union[str, Path, None] + CHROME_HEADLESS: bool + CHROME_SANDBOX: bool + + USE_CURL: bool + USE_WGET: bool + USE_SINGLEFILE: bool + USE_READABILITY: bool + USE_MERCURY: bool + USE_GIT: bool + USE_CHROME: bool + USE_YOUTUBEDL: bool + CURL_BINARY: str + GIT_BINARY: str + WGET_BINARY: str + SINGLEFILE_BINARY: str + READABILITY_BINARY: str + MERCURY_BINARY: str + YOUTUBEDL_BINARY: str + CHROME_BINARY: Optional[str] + + YOUTUBEDL_ARGS: List[str] + WGET_ARGS: List[str] + CURL_ARGS: List[str] + GIT_ARGS: List[str] + + +ConfigDefaultValueGetter = Callable[[ConfigDict], ConfigValue] +ConfigDefaultValue = Union[ConfigValue, ConfigDefaultValueGetter] + +ConfigDefault = TypedDict('ConfigDefault', { + 'default': ConfigDefaultValue, + 'type': Optional[Type], + 'aliases': Optional[Tuple[str, ...]], +}, total=False) + +ConfigDefaultDict = Dict[str, ConfigDefault] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/__init__.py new file mode 100644 index 0000000..3e1d607 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/__init__.py @@ -0,0 +1 @@ +__package__ = 'archivebox.core' diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/admin.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/admin.py new file mode 100644 index 0000000..832bea3 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/admin.py @@ -0,0 +1,257 @@ +__package__ = 'archivebox.core' + +from io import StringIO +from contextlib import redirect_stdout + +from django.contrib import admin +from django.urls import path +from django.utils.html import format_html +from django.utils.safestring import mark_safe +from django.shortcuts import render, redirect +from django.contrib.auth import get_user_model +from django import forms + +from core.models import Snapshot, Tag +from core.forms import AddLinkForm, TagField + +from core.mixins import SearchResultsAdminMixin + +from index.html import snapshot_icons +from util import htmldecode, urldecode, ansi_to_html +from logging_util import printable_filesize +from main import add, remove +from config import OUTPUT_DIR +from extractors import archive_links + +# TODO: https://stackoverflow.com/questions/40760880/add-custom-button-to-django-admin-panel + +def update_snapshots(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], out_dir=OUTPUT_DIR) +update_snapshots.short_description = "Archive" + +def update_titles(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], overwrite=True, methods=('title','favicon'), out_dir=OUTPUT_DIR) +update_titles.short_description = "Pull title" + +def overwrite_snapshots(modeladmin, request, queryset): + archive_links([ + snapshot.as_link() + for snapshot in queryset + ], overwrite=True, out_dir=OUTPUT_DIR) +overwrite_snapshots.short_description = "Re-archive (overwrite)" + +def verify_snapshots(modeladmin, request, queryset): + for snapshot in queryset: + print(snapshot.timestamp, snapshot.url, snapshot.is_archived, snapshot.archive_size, len(snapshot.history)) + +verify_snapshots.short_description = "Check" + +def delete_snapshots(modeladmin, request, queryset): + remove(snapshots=queryset, yes=True, delete=True, out_dir=OUTPUT_DIR) + +delete_snapshots.short_description = "Delete" + + +class SnapshotAdminForm(forms.ModelForm): + tags = TagField(required=False) + + class Meta: + model = Snapshot + fields = "__all__" + + def save(self, commit=True): + # Based on: https://stackoverflow.com/a/49933068/3509554 + + # Get the unsave instance + instance = forms.ModelForm.save(self, False) + tags = self.cleaned_data.pop("tags") + + #update save_m2m + def new_save_m2m(): + instance.save_tags(tags) + + # Do we need to save all changes now? + self.save_m2m = new_save_m2m + if commit: + instance.save() + + return instance + + +class SnapshotAdmin(SearchResultsAdminMixin, admin.ModelAdmin): + list_display = ('added', 'title_str', 'url_str', 'files', 'size') + sort_fields = ('title_str', 'url_str', 'added') + readonly_fields = ('id', 'url', 'timestamp', 'num_outputs', 'is_archived', 'url_hash', 'added', 'updated') + search_fields = ['url', 'timestamp', 'title', 'tags__name'] + fields = (*readonly_fields, 'title', 'tags') + list_filter = ('added', 'updated', 'tags') + ordering = ['-added'] + actions = [delete_snapshots, overwrite_snapshots, update_snapshots, update_titles, verify_snapshots] + actions_template = 'admin/actions_as_select.html' + form = SnapshotAdminForm + + def get_urls(self): + urls = super().get_urls() + custom_urls = [ + path('grid/', self.admin_site.admin_view(self.grid_view),name='grid') + ] + return custom_urls + urls + + def get_queryset(self, request): + return super().get_queryset(request).prefetch_related('tags') + + def tag_list(self, obj): + return ', '.join(obj.tags.values_list('name', flat=True)) + + def id_str(self, obj): + return format_html( + '{}', + obj.url_hash[:8], + ) + + def title_str(self, obj): + canon = obj.as_link().canonical_outputs() + tags = ''.join( + format_html('{} ', tag.id, tag) + for tag in obj.tags.all() + if str(tag).strip() + ) + return format_html( + '' + '' + '' + '' + '{}' + '', + obj.archive_path, + obj.archive_path, canon['favicon_path'], + obj.archive_path, + 'fetched' if obj.latest_title or obj.title else 'pending', + urldecode(htmldecode(obj.latest_title or obj.title or ''))[:128] or 'Pending...' + ) + mark_safe(f' {tags}') + + def files(self, obj): + return snapshot_icons(obj) + + def size(self, obj): + archive_size = obj.archive_size + if archive_size: + size_txt = printable_filesize(archive_size) + if archive_size > 52428800: + size_txt = mark_safe(f'{size_txt}') + else: + size_txt = mark_safe('...') + return format_html( + '{}', + obj.archive_path, + size_txt, + ) + + def url_str(self, obj): + return format_html( + '{}', + obj.url, + obj.url.split('://www.', 1)[-1].split('://', 1)[-1][:64], + ) + + def grid_view(self, request): + + # cl = self.get_changelist_instance(request) + + # Save before monkey patching to restore for changelist list view + saved_change_list_template = self.change_list_template + saved_list_per_page = self.list_per_page + saved_list_max_show_all = self.list_max_show_all + + # Monkey patch here plus core_tags.py + self.change_list_template = 'admin/grid_change_list.html' + self.list_per_page = 20 + self.list_max_show_all = self.list_per_page + + # Call monkey patched view + rendered_response = self.changelist_view(request) + + # Restore values + self.change_list_template = saved_change_list_template + self.list_per_page = saved_list_per_page + self.list_max_show_all = saved_list_max_show_all + + return rendered_response + + + id_str.short_description = 'ID' + title_str.short_description = 'Title' + url_str.short_description = 'Original URL' + + id_str.admin_order_field = 'id' + title_str.admin_order_field = 'title' + url_str.admin_order_field = 'url' + +class TagAdmin(admin.ModelAdmin): + list_display = ('slug', 'name', 'id') + sort_fields = ('id', 'name', 'slug') + readonly_fields = ('id',) + search_fields = ('id', 'name', 'slug') + fields = (*readonly_fields, 'name', 'slug') + + +class ArchiveBoxAdmin(admin.AdminSite): + site_header = 'ArchiveBox' + index_title = 'Links' + site_title = 'Index' + + def get_urls(self): + return [ + path('core/snapshot/add/', self.add_view, name='Add'), + ] + super().get_urls() + + def add_view(self, request): + if not request.user.is_authenticated: + return redirect(f'/admin/login/?next={request.path}') + + request.current_app = self.name + context = { + **self.each_context(request), + 'title': 'Add URLs', + } + + if request.method == 'GET': + context['form'] = AddLinkForm() + + elif request.method == 'POST': + form = AddLinkForm(request.POST) + if form.is_valid(): + url = form.cleaned_data["url"] + print(f'[+] Adding URL: {url}') + depth = 0 if form.cleaned_data["depth"] == "0" else 1 + input_kwargs = { + "urls": url, + "depth": depth, + "update_all": False, + "out_dir": OUTPUT_DIR, + } + add_stdout = StringIO() + with redirect_stdout(add_stdout): + add(**input_kwargs) + print(add_stdout.getvalue()) + + context.update({ + "stdout": ansi_to_html(add_stdout.getvalue().strip()), + "form": AddLinkForm() + }) + else: + context["form"] = form + + return render(template_name='add_links.html', request=request, context=context) + +admin.site = ArchiveBoxAdmin() +admin.site.register(get_user_model()) +admin.site.register(Snapshot, SnapshotAdmin) +admin.site.register(Tag, TagAdmin) +admin.site.disable_action('delete_selected') diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/apps.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/apps.py new file mode 100644 index 0000000..26f78a8 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class CoreConfig(AppConfig): + name = 'core' diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/forms.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/forms.py new file mode 100644 index 0000000..86b29bb --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/forms.py @@ -0,0 +1,67 @@ +__package__ = 'archivebox.core' + +from django import forms + +from ..util import URL_REGEX +from ..vendor.taggit_utils import edit_string_for_tags, parse_tags + +CHOICES = ( + ('0', 'depth = 0 (archive just these URLs)'), + ('1', 'depth = 1 (archive these URLs and all URLs one hop away)'), +) + +from ..extractors import get_default_archive_methods + +ARCHIVE_METHODS = [ + (name, name) + for name, _, _ in get_default_archive_methods() +] + + +class AddLinkForm(forms.Form): + url = forms.RegexField(label="URLs (one per line)", regex=URL_REGEX, min_length='6', strip=True, widget=forms.Textarea, required=True) + depth = forms.ChoiceField(label="Archive depth", choices=CHOICES, widget=forms.RadioSelect, initial='0') + archive_methods = forms.MultipleChoiceField( + required=False, + widget=forms.SelectMultiple, + choices=ARCHIVE_METHODS, + ) +class TagWidgetMixin: + def format_value(self, value): + if value is not None and not isinstance(value, str): + value = edit_string_for_tags(value) + return super().format_value(value) + +class TagWidget(TagWidgetMixin, forms.TextInput): + pass + +class TagField(forms.CharField): + widget = TagWidget + + def clean(self, value): + value = super().clean(value) + try: + return parse_tags(value) + except ValueError: + raise forms.ValidationError( + "Please provide a comma-separated list of tags." + ) + + def has_changed(self, initial_value, data_value): + # Always return False if the field is disabled since self.bound_data + # always uses the initial value in this case. + if self.disabled: + return False + + try: + data_value = self.clean(data_value) + except forms.ValidationError: + pass + + if initial_value is None: + initial_value = [] + + initial_value = [tag.name for tag in initial_value] + initial_value.sort() + + return initial_value != data_value diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/management/commands/archivebox.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/management/commands/archivebox.py new file mode 100644 index 0000000..a68b5d9 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/management/commands/archivebox.py @@ -0,0 +1,18 @@ +__package__ = 'archivebox' + +from django.core.management.base import BaseCommand + + +from .cli import run_subcommand + + +class Command(BaseCommand): + help = 'Run an ArchiveBox CLI subcommand (e.g. add, remove, list, etc)' + + def add_arguments(self, parser): + parser.add_argument('subcommand', type=str, help='The subcommand you want to run') + parser.add_argument('command_args', nargs='*', help='Arguments to pass to the subcommand') + + + def handle(self, *args, **kwargs): + run_subcommand(kwargs['subcommand'], args=kwargs['command_args']) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0001_initial.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0001_initial.py new file mode 100644 index 0000000..73ac78e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0001_initial.py @@ -0,0 +1,27 @@ +# Generated by Django 2.2 on 2019-05-01 03:27 + +from django.db import migrations, models +import uuid + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ] + + operations = [ + migrations.CreateModel( + name='Snapshot', + fields=[ + ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)), + ('url', models.URLField(unique=True)), + ('timestamp', models.CharField(default=None, max_length=32, null=True, unique=True)), + ('title', models.CharField(default=None, max_length=128, null=True)), + ('tags', models.CharField(default=None, max_length=256, null=True)), + ('added', models.DateTimeField(auto_now_add=True)), + ('updated', models.DateTimeField(default=None, null=True)), + ], + ), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0002_auto_20200625_1521.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0002_auto_20200625_1521.py new file mode 100644 index 0000000..4811282 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0002_auto_20200625_1521.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.7 on 2020-06-25 15:21 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0001_initial'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(default=None, max_length=32, null=True), + ), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0003_auto_20200630_1034.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0003_auto_20200630_1034.py new file mode 100644 index 0000000..61fd472 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0003_auto_20200630_1034.py @@ -0,0 +1,38 @@ +# Generated by Django 3.0.7 on 2020-06-30 10:34 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0002_auto_20200625_1521'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='added', + field=models.DateTimeField(auto_now_add=True, db_index=True), + ), + migrations.AlterField( + model_name='snapshot', + name='tags', + field=models.CharField(db_index=True, default=None, max_length=256, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(db_index=True, default=None, max_length=32, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='title', + field=models.CharField(db_index=True, default=None, max_length=128, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='updated', + field=models.DateTimeField(db_index=True, default=None, null=True), + ), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0004_auto_20200713_1552.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0004_auto_20200713_1552.py new file mode 100644 index 0000000..6983662 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0004_auto_20200713_1552.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.7 on 2020-07-13 15:52 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0003_auto_20200630_1034'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='timestamp', + field=models.CharField(db_index=True, default=None, max_length=32, unique=True), + preserve_default=False, + ), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0005_auto_20200728_0326.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0005_auto_20200728_0326.py new file mode 100644 index 0000000..f367aeb --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0005_auto_20200728_0326.py @@ -0,0 +1,28 @@ +# Generated by Django 3.0.7 on 2020-07-28 03:26 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0004_auto_20200713_1552'), + ] + + operations = [ + migrations.AlterField( + model_name='snapshot', + name='tags', + field=models.CharField(blank=True, db_index=True, max_length=256, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='title', + field=models.CharField(blank=True, db_index=True, max_length=128, null=True), + ), + migrations.AlterField( + model_name='snapshot', + name='updated', + field=models.DateTimeField(blank=True, db_index=True, null=True), + ), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0006_auto_20201012_1520.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0006_auto_20201012_1520.py new file mode 100644 index 0000000..694c990 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0006_auto_20201012_1520.py @@ -0,0 +1,70 @@ +# Generated by Django 3.0.8 on 2020-10-12 15:20 + +from django.db import migrations, models +from django.utils.text import slugify + +def forwards_func(apps, schema_editor): + SnapshotModel = apps.get_model("core", "Snapshot") + TagModel = apps.get_model("core", "Tag") + + db_alias = schema_editor.connection.alias + snapshots = SnapshotModel.objects.all() + for snapshot in snapshots: + tags = snapshot.tags + tag_set = ( + set(tag.strip() for tag in (snapshot.tags_old or '').split(',')) + ) + tag_set.discard("") + + for tag in tag_set: + to_add, _ = TagModel.objects.get_or_create(name=tag, slug=slugify(tag)) + snapshot.tags.add(to_add) + + +def reverse_func(apps, schema_editor): + SnapshotModel = apps.get_model("core", "Snapshot") + TagModel = apps.get_model("core", "Tag") + + db_alias = schema_editor.connection.alias + snapshots = SnapshotModel.objects.all() + for snapshot in snapshots: + tags = snapshot.tags.values_list("name", flat=True) + snapshot.tags_old = ",".join([tag for tag in tags]) + snapshot.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0005_auto_20200728_0326'), + ] + + operations = [ + migrations.RenameField( + model_name='snapshot', + old_name='tags', + new_name='tags_old', + ), + migrations.CreateModel( + name='Tag', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('name', models.CharField(max_length=100, unique=True, verbose_name='name')), + ('slug', models.SlugField(max_length=100, unique=True, verbose_name='slug')), + ], + options={ + 'verbose_name': 'Tag', + 'verbose_name_plural': 'Tags', + }, + ), + migrations.AddField( + model_name='snapshot', + name='tags', + field=models.ManyToManyField(to='core.Tag'), + ), + migrations.RunPython(forwards_func, reverse_func), + migrations.RemoveField( + model_name='snapshot', + name='tags_old', + ), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0007_archiveresult.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0007_archiveresult.py new file mode 100644 index 0000000..a780376 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0007_archiveresult.py @@ -0,0 +1,97 @@ +# Generated by Django 3.0.8 on 2020-11-04 12:25 + +import json +from pathlib import Path + +from django.db import migrations, models +import django.db.models.deletion + +from config import CONFIG +from index.json import to_json + +try: + JSONField = models.JSONField +except AttributeError: + import jsonfield + JSONField = jsonfield.JSONField + + +def forwards_func(apps, schema_editor): + from core.models import EXTRACTORS + + Snapshot = apps.get_model("core", "Snapshot") + ArchiveResult = apps.get_model("core", "ArchiveResult") + + snapshots = Snapshot.objects.all() + for snapshot in snapshots: + out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp + + try: + with open(out_dir / "index.json", "r") as f: + fs_index = json.load(f) + except Exception as e: + continue + + history = fs_index["history"] + + for extractor in history: + for result in history[extractor]: + ArchiveResult.objects.create(extractor=extractor, snapshot=snapshot, cmd=result["cmd"], cmd_version=result["cmd_version"], + start_ts=result["start_ts"], end_ts=result["end_ts"], status=result["status"], pwd=result["pwd"], output=result["output"]) + + +def verify_json_index_integrity(snapshot): + results = snapshot.archiveresult_set.all() + out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp + with open(out_dir / "index.json", "r") as f: + index = json.load(f) + + history = index["history"] + index_results = [result for extractor in history for result in history[extractor]] + flattened_results = [result["start_ts"] for result in index_results] + + missing_results = [result for result in results if result.start_ts.isoformat() not in flattened_results] + + for missing in missing_results: + index["history"][missing.extractor].append({"cmd": missing.cmd, "cmd_version": missing.cmd_version, "end_ts": missing.end_ts.isoformat(), + "start_ts": missing.start_ts.isoformat(), "pwd": missing.pwd, "output": missing.output, + "schema": "ArchiveResult", "status": missing.status}) + + json_index = to_json(index) + with open(out_dir / "index.json", "w") as f: + f.write(json_index) + + +def reverse_func(apps, schema_editor): + Snapshot = apps.get_model("core", "Snapshot") + ArchiveResult = apps.get_model("core", "ArchiveResult") + for snapshot in Snapshot.objects.all(): + verify_json_index_integrity(snapshot) + + ArchiveResult.objects.all().delete() + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0006_auto_20201012_1520'), + ] + + operations = [ + migrations.CreateModel( + name='ArchiveResult', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('cmd', JSONField()), + ('pwd', models.CharField(max_length=256)), + ('cmd_version', models.CharField(max_length=32)), + ('status', models.CharField(choices=[('succeeded', 'succeeded'), ('failed', 'failed'), ('skipped', 'skipped')], max_length=16)), + ('output', models.CharField(max_length=512)), + ('start_ts', models.DateTimeField()), + ('end_ts', models.DateTimeField()), + ('extractor', models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('wget', 'wget'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('headers', 'headers'), ('archive_org', 'archive_org')], max_length=32)), + ('snapshot', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Snapshot')), + ], + ), + migrations.RunPython(forwards_func, reverse_func), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0008_auto_20210105_1421.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0008_auto_20210105_1421.py new file mode 100644 index 0000000..e5b3387 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/0008_auto_20210105_1421.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.3 on 2021-01-05 14:21 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0007_archiveresult'), + ] + + operations = [ + migrations.AlterField( + model_name='archiveresult', + name='cmd_version', + field=models.CharField(blank=True, default=None, max_length=32, null=True), + ), + ] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/mixins.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/mixins.py new file mode 100644 index 0000000..538ca1e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/mixins.py @@ -0,0 +1,23 @@ +from django.contrib import messages + +from archivebox.search import query_search_index + +class SearchResultsAdminMixin(object): + def get_search_results(self, request, queryset, search_term): + ''' Enhances the search queryset with results from the search backend. + ''' + qs, use_distinct = \ + super(SearchResultsAdminMixin, self).get_search_results( + request, queryset, search_term) + + search_term = search_term.strip() + if not search_term: + return qs, use_distinct + try: + qsearch = query_search_index(search_term) + except Exception as err: + messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}') + else: + qs = queryset & qsearch + finally: + return qs, use_distinct diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/models.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/models.py new file mode 100644 index 0000000..13d75b6 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/models.py @@ -0,0 +1,194 @@ +__package__ = 'archivebox.core' + +import uuid + +from django.db import models, transaction +from django.utils.functional import cached_property +from django.utils.text import slugify +from django.db.models import Case, When, Value, IntegerField + +from ..util import parse_date +from ..index.schema import Link +from ..extractors import get_default_archive_methods, ARCHIVE_METHODS_INDEXING_PRECEDENCE + +EXTRACTORS = [(extractor[0], extractor[0]) for extractor in get_default_archive_methods()] +STATUS_CHOICES = [ + ("succeeded", "succeeded"), + ("failed", "failed"), + ("skipped", "skipped") +] + +try: + JSONField = models.JSONField +except AttributeError: + import jsonfield + JSONField = jsonfield.JSONField + + +class Tag(models.Model): + """ + Based on django-taggit model + """ + name = models.CharField(verbose_name="name", unique=True, blank=False, max_length=100) + slug = models.SlugField(verbose_name="slug", unique=True, max_length=100) + + class Meta: + verbose_name = "Tag" + verbose_name_plural = "Tags" + + def __str__(self): + return self.name + + def slugify(self, tag, i=None): + slug = slugify(tag) + if i is not None: + slug += "_%d" % i + return slug + + def save(self, *args, **kwargs): + if self._state.adding and not self.slug: + self.slug = self.slugify(self.name) + + with transaction.atomic(): + slugs = set( + type(self) + ._default_manager.filter(slug__startswith=self.slug) + .values_list("slug", flat=True) + ) + + i = None + while True: + slug = self.slugify(self.name, i) + if slug not in slugs: + self.slug = slug + return super().save(*args, **kwargs) + i = 1 if i is None else i+1 + else: + return super().save(*args, **kwargs) + + +class Snapshot(models.Model): + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) + + url = models.URLField(unique=True) + timestamp = models.CharField(max_length=32, unique=True, db_index=True) + + title = models.CharField(max_length=128, null=True, blank=True, db_index=True) + + added = models.DateTimeField(auto_now_add=True, db_index=True) + updated = models.DateTimeField(null=True, blank=True, db_index=True) + tags = models.ManyToManyField(Tag) + + keys = ('url', 'timestamp', 'title', 'tags', 'updated') + + def __repr__(self) -> str: + title = self.title or '-' + return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})' + + def __str__(self) -> str: + title = self.title or '-' + return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})' + + @classmethod + def from_json(cls, info: dict): + info = {k: v for k, v in info.items() if k in cls.keys} + return cls(**info) + + def as_json(self, *args) -> dict: + args = args or self.keys + return { + key: getattr(self, key) + if key != 'tags' else self.tags_str() + for key in args + } + + def as_link(self) -> Link: + return Link.from_json(self.as_json()) + + def as_link_with_details(self) -> Link: + from ..index import load_link_details + return load_link_details(self.as_link()) + + def tags_str(self) -> str: + return ','.join(self.tags.order_by('name').values_list('name', flat=True)) + + @cached_property + def bookmarked(self): + return parse_date(self.timestamp) + + @cached_property + def is_archived(self): + return self.as_link().is_archived + + @cached_property + def num_outputs(self): + return self.archiveresult_set.filter(status='succeeded').count() + + @cached_property + def url_hash(self): + return self.as_link().url_hash + + @cached_property + def base_url(self): + return self.as_link().base_url + + @cached_property + def link_dir(self): + return self.as_link().link_dir + + @cached_property + def archive_path(self): + return self.as_link().archive_path + + @cached_property + def archive_size(self): + return self.as_link().archive_size + + @cached_property + def history(self): + # TODO: use ArchiveResult for this instead of json + return self.as_link_with_details().history + + @cached_property + def latest_title(self): + if ('title' in self.history + and self.history['title'] + and (self.history['title'][-1].status == 'succeeded') + and self.history['title'][-1].output.strip()): + return self.history['title'][-1].output.strip() + return None + + def save_tags(self, tags=()): + tags_id = [] + for tag in tags: + tags_id.append(Tag.objects.get_or_create(name=tag)[0].id) + self.tags.clear() + self.tags.add(*tags_id) + + +class ArchiveResultManager(models.Manager): + def indexable(self, sorted: bool = True): + INDEXABLE_METHODS = [ r[0] for r in ARCHIVE_METHODS_INDEXING_PRECEDENCE ] + qs = self.get_queryset().filter(extractor__in=INDEXABLE_METHODS,status='succeeded') + + if sorted: + precedence = [ When(extractor=method, then=Value(precedence)) for method, precedence in ARCHIVE_METHODS_INDEXING_PRECEDENCE ] + qs = qs.annotate(indexing_precedence=Case(*precedence, default=Value(1000),output_field=IntegerField())).order_by('indexing_precedence') + return qs + + +class ArchiveResult(models.Model): + snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE) + cmd = JSONField() + pwd = models.CharField(max_length=256) + cmd_version = models.CharField(max_length=32, default=None, null=True, blank=True) + output = models.CharField(max_length=512) + start_ts = models.DateTimeField() + end_ts = models.DateTimeField() + status = models.CharField(max_length=16, choices=STATUS_CHOICES) + extractor = models.CharField(choices=EXTRACTORS, max_length=32) + + objects = ArchiveResultManager() + + def __str__(self): + return self.extractor diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/settings.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/settings.py new file mode 100644 index 0000000..e8ed6b1 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/settings.py @@ -0,0 +1,165 @@ +__package__ = 'archivebox.core' + +import os +import sys + +from pathlib import Path +from django.utils.crypto import get_random_string + +from ..config import ( # noqa: F401 + DEBUG, + SECRET_KEY, + ALLOWED_HOSTS, + PACKAGE_DIR, + ACTIVE_THEME, + TEMPLATES_DIR_NAME, + SQL_INDEX_FILENAME, + OUTPUT_DIR, +) + + +IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3] +IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ +IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3] + +################################################################################ +### Django Core Settings +################################################################################ + +WSGI_APPLICATION = 'core.wsgi.application' +ROOT_URLCONF = 'core.urls' + +LOGIN_URL = '/accounts/login/' +LOGOUT_REDIRECT_URL = '/' +PASSWORD_RESET_URL = '/accounts/password_reset/' +APPEND_SLASH = True + +INSTALLED_APPS = [ + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', + 'django.contrib.admin', + + 'core', + + 'django_extensions', +] + + +MIDDLEWARE = [ + 'django.middleware.security.SecurityMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.common.CommonMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', +] + +AUTHENTICATION_BACKENDS = [ + 'django.contrib.auth.backends.ModelBackend', +] + + +################################################################################ +### Staticfile and Template Settings +################################################################################ + +STATIC_URL = '/static/' + +STATICFILES_DIRS = [ + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / ACTIVE_THEME / 'static'), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'default' / 'static'), +] + +TEMPLATE_DIRS = [ + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / ACTIVE_THEME), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'default'), + str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME), +] + +TEMPLATES = [ + { + 'BACKEND': 'django.template.backends.django.DjangoTemplates', + 'DIRS': TEMPLATE_DIRS, + 'APP_DIRS': True, + 'OPTIONS': { + 'context_processors': [ + 'django.template.context_processors.debug', + 'django.template.context_processors.request', + 'django.contrib.auth.context_processors.auth', + 'django.contrib.messages.context_processors.messages', + ], + }, + }, +] + + +################################################################################ +### External Service Settings +################################################################################ + +DATABASE_FILE = Path(OUTPUT_DIR) / SQL_INDEX_FILENAME +DATABASE_NAME = os.environ.get("ARCHIVEBOX_DATABASE_NAME", DATABASE_FILE) + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': DATABASE_NAME, + } +} + +EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' + + +################################################################################ +### Security Settings +################################################################################ + +SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.') + +ALLOWED_HOSTS = ALLOWED_HOSTS.split(',') + +SECURE_BROWSER_XSS_FILTER = True +SECURE_CONTENT_TYPE_NOSNIFF = True + +CSRF_COOKIE_SECURE = False +SESSION_COOKIE_SECURE = False +SESSION_COOKIE_DOMAIN = None +SESSION_COOKIE_AGE = 1209600 # 2 weeks +SESSION_EXPIRE_AT_BROWSER_CLOSE = False +SESSION_SAVE_EVERY_REQUEST = True + +AUTH_PASSWORD_VALIDATORS = [ + {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'}, + {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'}, + {'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator'}, + {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'}, +] + + +################################################################################ +### Shell Settings +################################################################################ + +SHELL_PLUS = 'ipython' +SHELL_PLUS_PRINT_SQL = False +IPYTHON_ARGUMENTS = ['--no-confirm-exit', '--no-banner'] +IPYTHON_KERNEL_DISPLAY_NAME = 'ArchiveBox Django Shell' +if IS_SHELL: + os.environ['PYTHONSTARTUP'] = str(Path(PACKAGE_DIR) / 'core' / 'welcome_message.py') + + +################################################################################ +### Internationalization & Localization Settings +################################################################################ + +LANGUAGE_CODE = 'en-us' +TIME_ZONE = 'UTC' +USE_I18N = False +USE_L10N = False +USE_TZ = False + +DATETIME_FORMAT = 'Y-m-d g:iA' +SHORT_DATETIME_FORMAT = 'Y-m-d h:iA' diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/templatetags/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/templatetags/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/templatetags/core_tags.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/templatetags/core_tags.py new file mode 100644 index 0000000..25f0685 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/templatetags/core_tags.py @@ -0,0 +1,47 @@ +from django import template +from django.urls import reverse +from django.contrib.admin.templatetags.base import InclusionAdminNode +from django.templatetags.static import static + + +from typing import Union + +from core.models import ArchiveResult + +register = template.Library() + +@register.simple_tag +def snapshot_image(snapshot): + result = ArchiveResult.objects.filter(snapshot=snapshot, extractor='screenshot', status='succeeded').first() + if result: + return reverse('LinkAssets', args=[f'{str(snapshot.timestamp)}/{result.output}']) + + return static('archive.png') + +@register.filter +def file_size(num_bytes: Union[int, float]) -> str: + for count in ['Bytes','KB','MB','GB']: + if num_bytes > -1024.0 and num_bytes < 1024.0: + return '%3.1f %s' % (num_bytes, count) + num_bytes /= 1024.0 + return '%3.1f %s' % (num_bytes, 'TB') + +def result_list(cl): + """ + Monkey patched result + """ + num_sorted_fields = 0 + return { + 'cl': cl, + 'num_sorted_fields': num_sorted_fields, + 'results': cl.result_list, + } + +@register.tag(name='snapshots_grid') +def result_list_tag(parser, token): + return InclusionAdminNode( + parser, token, + func=result_list, + template_name='snapshots_grid.html', + takes_context=False, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/tests.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/tests.py new file mode 100644 index 0000000..4d66077 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/tests.py @@ -0,0 +1,3 @@ +#from django.test import TestCase + +# Create your tests here. diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/urls.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/urls.py new file mode 100644 index 0000000..b8e4baf --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/urls.py @@ -0,0 +1,36 @@ +from django.contrib import admin + +from django.urls import path, include +from django.views import static +from django.conf import settings +from django.views.generic.base import RedirectView + +from core.views import MainIndex, LinkDetails, PublicArchiveView, AddView + + +# print('DEBUG', settings.DEBUG) + +urlpatterns = [ + path('robots.txt', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'robots.txt'}), + path('favicon.ico', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'favicon.ico'}), + + path('docs/', RedirectView.as_view(url='https://github.com/ArchiveBox/ArchiveBox/wiki'), name='Docs'), + + path('archive/', RedirectView.as_view(url='/')), + path('archive/', LinkDetails.as_view(), name='LinkAssets'), + + path('admin/core/snapshot/add/', RedirectView.as_view(url='/add/')), + path('add/', AddView.as_view()), + + path('accounts/login/', RedirectView.as_view(url='/admin/login/')), + path('accounts/logout/', RedirectView.as_view(url='/admin/logout/')), + + + path('accounts/', include('django.contrib.auth.urls')), + path('admin/', admin.site.urls), + + path('index.html', RedirectView.as_view(url='/')), + path('index.json', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'index.json'}), + path('', MainIndex.as_view(), name='Home'), + path('public/', PublicArchiveView.as_view(), name='public-index'), +] diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/views.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/views.py new file mode 100644 index 0000000..b46e364 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/views.py @@ -0,0 +1,173 @@ +__package__ = 'archivebox.core' + +from io import StringIO +from contextlib import redirect_stdout + +from django.shortcuts import render, redirect + +from django.http import HttpResponse +from django.views import View, static +from django.views.generic.list import ListView +from django.views.generic import FormView +from django.contrib.auth.mixins import UserPassesTestMixin + +from core.models import Snapshot +from core.forms import AddLinkForm + +from ..config import ( + OUTPUT_DIR, + PUBLIC_INDEX, + PUBLIC_SNAPSHOTS, + PUBLIC_ADD_VIEW, + VERSION, + FOOTER_INFO, +) +from main import add +from ..util import base_url, ansi_to_html +from ..index.html import snapshot_icons + + +class MainIndex(View): + template = 'main_index.html' + + def get(self, request): + if request.user.is_authenticated: + return redirect('/admin/core/snapshot/') + + if PUBLIC_INDEX: + return redirect('public-index') + + return redirect(f'/admin/login/?next={request.path}') + + +class LinkDetails(View): + def get(self, request, path): + # missing trailing slash -> redirect to index + if '/' not in path: + return redirect(f'{path}/index.html') + + if not request.user.is_authenticated and not PUBLIC_SNAPSHOTS: + return redirect(f'/admin/login/?next={request.path}') + + try: + slug, archivefile = path.split('/', 1) + except (IndexError, ValueError): + slug, archivefile = path.split('/', 1)[0], 'index.html' + + all_pages = list(Snapshot.objects.all()) + + # slug is a timestamp + by_ts = {page.timestamp: page for page in all_pages} + try: + # print('SERVING STATICFILE', by_ts[slug].link_dir, request.path, path) + response = static.serve(request, archivefile, document_root=by_ts[slug].link_dir, show_indexes=True) + response["Link"] = f'<{by_ts[slug].url}>; rel="canonical"' + return response + except KeyError: + pass + + # slug is a hash + by_hash = {page.url_hash: page for page in all_pages} + try: + timestamp = by_hash[slug].timestamp + return redirect(f'/archive/{timestamp}/{archivefile}') + except KeyError: + pass + + # slug is a URL + by_url = {page.base_url: page for page in all_pages} + try: + # TODO: add multiple snapshot support by showing index of all snapshots + # for given url instead of redirecting to timestamp index + timestamp = by_url[base_url(path)].timestamp + return redirect(f'/archive/{timestamp}/index.html') + except KeyError: + pass + + return HttpResponse( + 'No archived link matches the given timestamp or hash.', + content_type="text/plain", + status=404, + ) + +class PublicArchiveView(ListView): + template = 'snapshot_list.html' + model = Snapshot + paginate_by = 100 + ordering = ['title'] + + def get_context_data(self, **kwargs): + return { + **super().get_context_data(**kwargs), + 'VERSION': VERSION, + 'FOOTER_INFO': FOOTER_INFO, + } + + def get_queryset(self, **kwargs): + qs = super().get_queryset(**kwargs) + query = self.request.GET.get('q') + if query: + qs = qs.filter(title__icontains=query) + for snapshot in qs: + snapshot.icons = snapshot_icons(snapshot) + return qs + + def get(self, *args, **kwargs): + if PUBLIC_INDEX or self.request.user.is_authenticated: + response = super().get(*args, **kwargs) + return response + else: + return redirect(f'/admin/login/?next={self.request.path}') + + +class AddView(UserPassesTestMixin, FormView): + template_name = "add_links.html" + form_class = AddLinkForm + + def get_initial(self): + """Prefill the AddLinkForm with the 'url' GET parameter""" + if self.request.method == 'GET': + url = self.request.GET.get('url', None) + if url: + return {'url': url} + else: + return super().get_initial() + + def test_func(self): + return PUBLIC_ADD_VIEW or self.request.user.is_authenticated + + def get_context_data(self, **kwargs): + return { + **super().get_context_data(**kwargs), + 'title': "Add URLs", + # We can't just call request.build_absolute_uri in the template, because it would include query parameters + 'absolute_add_path': self.request.build_absolute_uri(self.request.path), + 'VERSION': VERSION, + 'FOOTER_INFO': FOOTER_INFO, + } + + def form_valid(self, form): + url = form.cleaned_data["url"] + print(f'[+] Adding URL: {url}') + depth = 0 if form.cleaned_data["depth"] == "0" else 1 + extractors = ','.join(form.cleaned_data["archive_methods"]) + input_kwargs = { + "urls": url, + "depth": depth, + "update_all": False, + "out_dir": OUTPUT_DIR, + } + if extractors: + input_kwargs.update({"extractors": extractors}) + add_stdout = StringIO() + with redirect_stdout(add_stdout): + add(**input_kwargs) + print(add_stdout.getvalue()) + + context = self.get_context_data() + + context.update({ + "stdout": ansi_to_html(add_stdout.getvalue().strip()), + "form": AddLinkForm() + }) + return render(template_name=self.template_name, request=self.request, context=context) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/welcome_message.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/welcome_message.py new file mode 100644 index 0000000..ed5d2d7 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/welcome_message.py @@ -0,0 +1,5 @@ +from archivebox.logging_util import log_shell_welcome_msg + + +if __name__ == '__main__': + log_shell_welcome_msg() diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/wsgi.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/wsgi.py new file mode 100644 index 0000000..f933afa --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/core/wsgi.py @@ -0,0 +1,16 @@ +""" +WSGI config for archivebox project. + +It exposes the WSGI callable as a module-level variable named ``application``. + +For more information on this file, see +https://docs.djangoproject.com/en/2.1/howto/deployment/wsgi/ +""" + +import os + +from django.core.wsgi import get_wsgi_application + +os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'archivebox.settings') + +application = get_wsgi_application() diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/__init__.py new file mode 100644 index 0000000..a4acef0 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/__init__.py @@ -0,0 +1,182 @@ +__package__ = 'archivebox.extractors' + +import os +from pathlib import Path + +from typing import Optional, List, Iterable, Union +from datetime import datetime +from django.db.models import QuerySet + +from ..index.schema import Link +from ..index.sql import write_link_to_sql_index +from ..index import ( + load_link_details, + write_link_details, +) +from ..util import enforce_types +from ..logging_util import ( + log_archiving_started, + log_archiving_paused, + log_archiving_finished, + log_link_archiving_started, + log_link_archiving_finished, + log_archive_method_started, + log_archive_method_finished, +) +from ..search import write_search_index + +from .title import should_save_title, save_title +from .favicon import should_save_favicon, save_favicon +from .wget import should_save_wget, save_wget +from .singlefile import should_save_singlefile, save_singlefile +from .readability import should_save_readability, save_readability +from .mercury import should_save_mercury, save_mercury +from .pdf import should_save_pdf, save_pdf +from .screenshot import should_save_screenshot, save_screenshot +from .dom import should_save_dom, save_dom +from .git import should_save_git, save_git +from .media import should_save_media, save_media +from .archive_org import should_save_archive_dot_org, save_archive_dot_org +from .headers import should_save_headers, save_headers + + +def get_default_archive_methods(): + return [ + ('title', should_save_title, save_title), + ('favicon', should_save_favicon, save_favicon), + ('wget', should_save_wget, save_wget), + ('singlefile', should_save_singlefile, save_singlefile), + ('pdf', should_save_pdf, save_pdf), + ('screenshot', should_save_screenshot, save_screenshot), + ('dom', should_save_dom, save_dom), + ('readability', should_save_readability, save_readability), #keep readability below wget and singlefile, as it depends on them + ('mercury', should_save_mercury, save_mercury), + ('git', should_save_git, save_git), + ('media', should_save_media, save_media), + ('headers', should_save_headers, save_headers), + ('archive_org', should_save_archive_dot_org, save_archive_dot_org), + ] + +ARCHIVE_METHODS_INDEXING_PRECEDENCE = [('readability', 1), ('singlefile', 2), ('dom', 3), ('wget', 4)] + +@enforce_types +def ignore_methods(to_ignore: List[str]): + ARCHIVE_METHODS = get_default_archive_methods() + methods = filter(lambda x: x[0] not in to_ignore, ARCHIVE_METHODS) + methods = map(lambda x: x[0], methods) + return list(methods) + +@enforce_types +def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> Link: + """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp""" + + # TODO: Remove when the input is changed to be a snapshot. Suboptimal approach. + from core.models import Snapshot, ArchiveResult + try: + snapshot = Snapshot.objects.get(url=link.url) # TODO: This will be unnecessary once everything is a snapshot + except Snapshot.DoesNotExist: + snapshot = write_link_to_sql_index(link) + + ARCHIVE_METHODS = get_default_archive_methods() + + if methods: + ARCHIVE_METHODS = [ + method for method in ARCHIVE_METHODS + if method[0] in methods + ] + + out_dir = out_dir or Path(link.link_dir) + try: + is_new = not Path(out_dir).exists() + if is_new: + os.makedirs(out_dir) + + link = load_link_details(link, out_dir=out_dir) + write_link_details(link, out_dir=out_dir, skip_sql_index=False) + log_link_archiving_started(link, out_dir, is_new) + link = link.overwrite(updated=datetime.now()) + stats = {'skipped': 0, 'succeeded': 0, 'failed': 0} + + for method_name, should_run, method_function in ARCHIVE_METHODS: + try: + if method_name not in link.history: + link.history[method_name] = [] + + if should_run(link, out_dir) or overwrite: + log_archive_method_started(method_name) + + result = method_function(link=link, out_dir=out_dir) + + link.history[method_name].append(result) + + stats[result.status] += 1 + log_archive_method_finished(result) + write_search_index(link=link, texts=result.index_texts) + ArchiveResult.objects.create(snapshot=snapshot, extractor=method_name, cmd=result.cmd, cmd_version=result.cmd_version, + output=result.output, pwd=result.pwd, start_ts=result.start_ts, end_ts=result.end_ts, status=result.status) + + else: + # print('{black} X {}{reset}'.format(method_name, **ANSI)) + stats['skipped'] += 1 + except Exception as e: + raise Exception('Exception in archive_methods.save_{}(Link(url={}))'.format( + method_name, + link.url, + )) from e + + # print(' ', stats) + + try: + latest_title = link.history['title'][-1].output.strip() + if latest_title and len(latest_title) >= len(link.title or ''): + link = link.overwrite(title=latest_title) + except Exception: + pass + + write_link_details(link, out_dir=out_dir, skip_sql_index=False) + + log_link_archiving_finished(link, link.link_dir, is_new, stats) + + except KeyboardInterrupt: + try: + write_link_details(link, out_dir=link.link_dir) + except: + pass + raise + + except Exception as err: + print(' ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err)) + raise + + return link + +@enforce_types +def archive_links(all_links: Union[Iterable[Link], QuerySet], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> List[Link]: + + if type(all_links) is QuerySet: + num_links: int = all_links.count() + get_link = lambda x: x.as_link() + all_links = all_links.iterator() + else: + num_links: int = len(all_links) + get_link = lambda x: x + + if num_links == 0: + return [] + + log_archiving_started(num_links) + idx: int = 0 + try: + for link in all_links: + idx += 1 + to_archive = get_link(link) + archive_link(to_archive, overwrite=overwrite, methods=methods, out_dir=Path(link.link_dir)) + except KeyboardInterrupt: + log_archiving_paused(num_links, idx, link.timestamp) + raise SystemExit(0) + except BaseException: + print() + raise + + log_archiving_finished(num_links) + return all_links diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/archive_org.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/archive_org.py new file mode 100644 index 0000000..f5598d6 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/archive_org.py @@ -0,0 +1,112 @@ +__package__ = 'archivebox.extractors' + + +from pathlib import Path +from typing import Optional, List, Dict, Tuple +from collections import defaultdict + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, +) +from ..config import ( + TIMEOUT, + CURL_ARGS, + CHECK_SSL_VALIDITY, + SAVE_ARCHIVE_DOT_ORG, + CURL_BINARY, + CURL_VERSION, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_archive_dot_org(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "archive.org.txt").exists(): + # if open(path, 'r').read().strip() != 'None': + return False + + return SAVE_ARCHIVE_DOT_ORG + +@enforce_types +def save_archive_dot_org(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """submit site to archive.org for archiving via their service, save returned archive url""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'archive.org.txt' + archive_org_url = None + submit_url = 'https://web.archive.org/save/{}'.format(link.url) + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--head', + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + submit_url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + content_location, errors = parse_archive_dot_org_response(result.stdout) + if content_location: + archive_org_url = content_location[0] + elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]: + archive_org_url = None + # raise ArchiveError('Archive.org denied by {}/robots.txt'.format(domain(link.url))) + elif errors: + raise ArchiveError(', '.join(errors)) + else: + raise ArchiveError('Failed to find "content-location" URL header in Archive.org response.') + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + if output and not isinstance(output, Exception): + # instead of writing None when archive.org rejects the url write the + # url to resubmit it to archive.org. This is so when the user visits + # the URL in person, it will attempt to re-archive it, and it'll show the + # nicer error message explaining why the url was rejected if it fails. + archive_org_url = archive_org_url or submit_url + with open(str(out_dir / output), 'w', encoding='utf-8') as f: + f.write(archive_org_url) + chmod_file('archive.org.txt', cwd=str(out_dir)) + output = archive_org_url + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) + +@enforce_types +def parse_archive_dot_org_response(response: bytes) -> Tuple[List[str], List[str]]: + # Parse archive.org response headers + headers: Dict[str, List[str]] = defaultdict(list) + + # lowercase all the header names and store in dict + for header in response.splitlines(): + if b':' not in header or not header.strip(): + continue + name, val = header.decode().split(':', 1) + headers[name.lower().strip()].append(val.strip()) + + # Get successful archive url in "content-location" header or any errors + content_location = headers.get('content-location', headers['location']) + errors = headers['x-archive-wayback-runtime-error'] + return content_location, errors + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/dom.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/dom.py new file mode 100644 index 0000000..babbe71 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/dom.py @@ -0,0 +1,69 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file, atomic_write +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_DOM, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_dom(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / 'output.html').exists(): + return False + + return SAVE_DOM + +@enforce_types +def save_dom(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """print HTML of site to file using chrome --dump-html""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'output.html' + output_path = out_dir / output + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--dump-dom', + link.url + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + atomic_write(output_path, result.stdout) + + if result.returncode: + hints = result.stderr.decode() + raise ArchiveError('Failed to save DOM', hints) + + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/favicon.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/favicon.py new file mode 100644 index 0000000..5e7c1fb --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/favicon.py @@ -0,0 +1,64 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput +from ..system import chmod_file, run +from ..util import enforce_types, domain +from ..config import ( + TIMEOUT, + SAVE_FAVICON, + CURL_BINARY, + CURL_ARGS, + CURL_VERSION, + CHECK_SSL_VALIDITY, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_favicon(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if (Path(out_dir) / 'favicon.ico').exists(): + return False + + return SAVE_FAVICON + +@enforce_types +def save_favicon(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download site favicon from google's favicon api""" + + out_dir = out_dir or link.link_dir + output: ArchiveOutput = 'favicon.ico' + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--max-time', str(timeout), + '--output', str(output), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + 'https://www.google.com/s2/favicons?domain={}'.format(domain(link.url)), + ] + status = 'pending' + timer = TimedProgress(timeout, prefix=' ') + try: + run(cmd, cwd=str(out_dir), timeout=timeout) + chmod_file(output, cwd=str(out_dir)) + status = 'succeeded' + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/git.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/git.py new file mode 100644 index 0000000..fd20d4b --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/git.py @@ -0,0 +1,90 @@ +__package__ = 'archivebox.extractors' + + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + domain, + extension, + without_query, + without_fragment, +) +from ..config import ( + TIMEOUT, + SAVE_GIT, + GIT_BINARY, + GIT_ARGS, + GIT_VERSION, + GIT_DOMAINS, + CHECK_SSL_VALIDITY +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_git(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + if (out_dir / "git").exists(): + return False + + is_clonable_url = ( + (domain(link.url) in GIT_DOMAINS) + or (extension(link.url) == 'git') + ) + if not is_clonable_url: + return False + + return SAVE_GIT + + +@enforce_types +def save_git(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using git""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'git' + output_path = out_dir / output + output_path.mkdir(exist_ok=True) + cmd = [ + GIT_BINARY, + 'clone', + *GIT_ARGS, + *([] if CHECK_SSL_VALIDITY else ['-c', 'http.sslVerify=false']), + without_query(without_fragment(link.url)), + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(output_path), timeout=timeout + 1) + if result.returncode == 128: + # ignore failed re-download when the folder already exists + pass + elif result.returncode > 0: + hints = 'Got git response code: {}.'.format(result.returncode) + raise ArchiveError('Failed to save git clone', hints) + + chmod_file(output, cwd=str(out_dir)) + + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=GIT_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/headers.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/headers.py new file mode 100644 index 0000000..4e69dec --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/headers.py @@ -0,0 +1,69 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput +from ..system import atomic_write +from ..util import ( + enforce_types, + get_headers, +) +from ..config import ( + TIMEOUT, + CURL_BINARY, + CURL_ARGS, + CURL_USER_AGENT, + CURL_VERSION, + CHECK_SSL_VALIDITY, + SAVE_HEADERS +) +from ..logging_util import TimedProgress + +@enforce_types +def should_save_headers(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + + output = Path(out_dir or link.link_dir) / 'headers.json' + return not output.exists() and SAVE_HEADERS + + +@enforce_types +def save_headers(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """Download site headers""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() + output: ArchiveOutput = 'headers.json' + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--head', + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + link.url, + ] + try: + json_headers = get_headers(link.url, timeout=timeout) + output_folder.mkdir(exist_ok=True) + atomic_write(str(output_folder / "headers.json"), json_headers) + except (Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/media.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/media.py new file mode 100644 index 0000000..3792fd2 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/media.py @@ -0,0 +1,81 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, +) +from ..config import ( + MEDIA_TIMEOUT, + SAVE_MEDIA, + YOUTUBEDL_ARGS, + YOUTUBEDL_BINARY, + YOUTUBEDL_VERSION, + CHECK_SSL_VALIDITY +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_media(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or link.link_dir + + if is_static_file(link.url): + return False + + if (out_dir / "media").exists(): + return False + + return SAVE_MEDIA + +@enforce_types +def save_media(link: Link, out_dir: Optional[Path]=None, timeout: int=MEDIA_TIMEOUT) -> ArchiveResult: + """Download playlists or individual video, audio, and subtitles using youtube-dl""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'media' + output_path = out_dir / output + output_path.mkdir(exist_ok=True) + cmd = [ + YOUTUBEDL_BINARY, + *YOUTUBEDL_ARGS, + *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate']), + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(output_path), timeout=timeout + 1) + chmod_file(output, cwd=str(out_dir)) + if result.returncode: + if (b'ERROR: Unsupported URL' in result.stderr + or b'HTTP Error 404' in result.stderr + or b'HTTP Error 403' in result.stderr + or b'URL could be a direct video link' in result.stderr + or b'Unable to extract container ID' in result.stderr): + # These happen too frequently on non-media pages to warrant printing to console + pass + else: + hints = ( + 'Got youtube-dl response code: {}.'.format(result.returncode), + *result.stderr.decode().split('\n'), + ) + raise ArchiveError('Failed to save media', hints) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=YOUTUBEDL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/mercury.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/mercury.py new file mode 100644 index 0000000..741c329 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/mercury.py @@ -0,0 +1,104 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from subprocess import CompletedProcess +from typing import Optional, List +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, atomic_write +from ..util import ( + enforce_types, + is_static_file, + +) +from ..config import ( + TIMEOUT, + SAVE_MERCURY, + DEPENDENCIES, + MERCURY_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def ShellError(cmd: List[str], result: CompletedProcess, lines: int=20) -> ArchiveError: + # parse out last line of stderr + return ArchiveError( + f'Got {cmd[0]} response code: {result.returncode}).', + *( + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', lines)[-lines:] + if line.strip() + ), + ) + + +@enforce_types +def should_save_mercury(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + output = Path(out_dir or link.link_dir) / 'mercury' + return SAVE_MERCURY and MERCURY_VERSION and (not output.exists()) + + +@enforce_types +def save_mercury(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download reader friendly version using @postlight/mercury-parser""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() / "mercury" + output = str(output_folder) + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + # Get plain text version of article + cmd = [ + DEPENDENCIES['MERCURY_BINARY']['path'], + link.url, + "--format=text" + ] + result = run(cmd, cwd=out_dir, timeout=timeout) + try: + article_text = json.loads(result.stdout) + except json.JSONDecodeError: + raise ShellError(cmd, result) + + # Get HTML version of article + cmd = [ + DEPENDENCIES['MERCURY_BINARY']['path'], + link.url + ] + result = run(cmd, cwd=out_dir, timeout=timeout) + try: + article_json = json.loads(result.stdout) + except json.JSONDecodeError: + raise ShellError(cmd, result) + + output_folder.mkdir(exist_ok=True) + atomic_write(str(output_folder / "content.html"), article_json.pop("content")) + atomic_write(str(output_folder / "content.txt"), article_text["content"]) + atomic_write(str(output_folder / "article.json"), article_json) + + # Check for common failure cases + if (result.returncode > 0): + raise ShellError(cmd, result) + except (ArchiveError, Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=MERCURY_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/pdf.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/pdf.py new file mode 100644 index 0000000..1b0201e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/pdf.py @@ -0,0 +1,68 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_PDF, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_pdf(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "output.pdf").exists(): + return False + + return SAVE_PDF + + +@enforce_types +def save_pdf(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """print PDF of site to file using chrome --headless""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'output.pdf' + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--print-to-pdf', + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + if result.returncode: + hints = (result.stderr or result.stdout).decode() + raise ArchiveError('Failed to save PDF', hints) + + chmod_file('output.pdf', cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/readability.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/readability.py new file mode 100644 index 0000000..9da620b --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/readability.py @@ -0,0 +1,124 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from tempfile import NamedTemporaryFile + +from typing import Optional +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, atomic_write +from ..util import ( + enforce_types, + download_url, + is_static_file, + +) +from ..config import ( + TIMEOUT, + CURL_BINARY, + SAVE_READABILITY, + DEPENDENCIES, + READABILITY_VERSION, +) +from ..logging_util import TimedProgress + +@enforce_types +def get_html(link: Link, path: Path) -> str: + """ + Try to find wget, singlefile and then dom files. + If none is found, download the url again. + """ + canonical = link.canonical_outputs() + abs_path = path.absolute() + sources = [canonical["singlefile_path"], canonical["wget_path"], canonical["dom_path"]] + document = None + for source in sources: + try: + with open(abs_path / source, "r") as f: + document = f.read() + break + except (FileNotFoundError, TypeError): + continue + if document is None: + return download_url(link.url) + else: + return document + +@enforce_types +def should_save_readability(link: Link, out_dir: Optional[str]=None) -> bool: + out_dir = out_dir or link.link_dir + if is_static_file(link.url): + return False + + output = Path(out_dir or link.link_dir) / 'readability' + return SAVE_READABILITY and READABILITY_VERSION and (not output.exists()) + + +@enforce_types +def save_readability(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download reader friendly version using @mozilla/readability""" + + out_dir = Path(out_dir or link.link_dir) + output_folder = out_dir.absolute() / "readability" + output = str(output_folder) + + # Readability Docs: https://github.com/mozilla/readability + + status = 'succeeded' + # fake command to show the user so they have something to try debugging if get_html fails + cmd = [ + CURL_BINARY, + link.url + ] + readability_content = None + timer = TimedProgress(timeout, prefix=' ') + try: + document = get_html(link, out_dir) + temp_doc = NamedTemporaryFile(delete=False) + temp_doc.write(document.encode("utf-8")) + temp_doc.close() + + cmd = [ + DEPENDENCIES['READABILITY_BINARY']['path'], + temp_doc.name + ] + + result = run(cmd, cwd=out_dir, timeout=timeout) + result_json = json.loads(result.stdout) + output_folder.mkdir(exist_ok=True) + readability_content = result_json.pop("textContent") + atomic_write(str(output_folder / "content.html"), result_json.pop("content")) + atomic_write(str(output_folder / "content.txt"), readability_content) + atomic_write(str(output_folder / "article.json"), result_json) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + hints = ( + 'Got readability response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0): + raise ArchiveError('Readability was not able to archive the page', hints) + except (Exception, OSError) as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=READABILITY_VERSION, + output=output, + status=status, + index_texts= [readability_content] if readability_content else [], + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/screenshot.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/screenshot.py new file mode 100644 index 0000000..325584e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/screenshot.py @@ -0,0 +1,67 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_SCREENSHOT, + CHROME_VERSION, +) +from ..logging_util import TimedProgress + + + +@enforce_types +def should_save_screenshot(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + if (out_dir / "screenshot.png").exists(): + return False + + return SAVE_SCREENSHOT + +@enforce_types +def save_screenshot(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """take screenshot of site using chrome --headless""" + + out_dir = out_dir or Path(link.link_dir) + output: ArchiveOutput = 'screenshot.png' + cmd = [ + *chrome_args(TIMEOUT=timeout), + '--screenshot', + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + if result.returncode: + hints = (result.stderr or result.stdout).decode() + raise ArchiveError('Failed to save screenshot', hints) + + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CHROME_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/singlefile.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/singlefile.py new file mode 100644 index 0000000..2e5c389 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/singlefile.py @@ -0,0 +1,90 @@ +__package__ = 'archivebox.extractors' + +from pathlib import Path + +from typing import Optional +import json + +from ..index.schema import Link, ArchiveResult, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + chrome_args, +) +from ..config import ( + TIMEOUT, + SAVE_SINGLEFILE, + DEPENDENCIES, + SINGLEFILE_VERSION, + CHROME_BINARY, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_singlefile(link: Link, out_dir: Optional[Path]=None) -> bool: + out_dir = out_dir or Path(link.link_dir) + if is_static_file(link.url): + return False + + output = out_dir / 'singlefile.html' + return SAVE_SINGLEFILE and SINGLEFILE_VERSION and (not output.exists()) + + +@enforce_types +def save_singlefile(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using single-file""" + + out_dir = out_dir or Path(link.link_dir) + output = str(out_dir.absolute() / "singlefile.html") + + browser_args = chrome_args(TIMEOUT=0) + + # SingleFile CLI Docs: https://github.com/gildas-lormeau/SingleFile/tree/master/cli + browser_args = '--browser-args={}'.format(json.dumps(browser_args[1:])) + cmd = [ + DEPENDENCIES['SINGLEFILE_BINARY']['path'], + '--browser-executable-path={}'.format(CHROME_BINARY), + browser_args, + link.url, + output + ] + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + hints = ( + 'Got single-file response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0): + raise ArchiveError('SingleFile was not able to archive the page', hints) + chmod_file(output) + except (Exception, OSError) as err: + status = 'failed' + # TODO: Make this prettier. This is necessary to run the command (escape JSON internal quotes). + cmd[2] = browser_args.replace('"', "\\\"") + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=SINGLEFILE_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/title.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/title.py new file mode 100644 index 0000000..28cb128 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/title.py @@ -0,0 +1,130 @@ +__package__ = 'archivebox.extractors' + +import re +from html.parser import HTMLParser +from pathlib import Path +from typing import Optional + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..util import ( + enforce_types, + is_static_file, + download_url, + htmldecode, +) +from ..config import ( + TIMEOUT, + CHECK_SSL_VALIDITY, + SAVE_TITLE, + CURL_BINARY, + CURL_ARGS, + CURL_VERSION, + CURL_USER_AGENT, +) +from ..logging_util import TimedProgress + + + +HTML_TITLE_REGEX = re.compile( + r'' # start matching text after tag + r'(.[^<>]+)', # get everything up to these symbols + re.IGNORECASE | re.MULTILINE | re.DOTALL | re.UNICODE, +) + + +class TitleParser(HTMLParser): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.title_tag = "" + self.title_og = "" + self.inside_title_tag = False + + @property + def title(self): + return self.title_tag or self.title_og or None + + def handle_starttag(self, tag, attrs): + if tag.lower() == "title" and not self.title_tag: + self.inside_title_tag = True + elif tag.lower() == "meta" and not self.title_og: + attrs = dict(attrs) + if attrs.get("property") == "og:title" and attrs.get("content"): + self.title_og = attrs.get("content") + + def handle_data(self, data): + if self.inside_title_tag and data: + self.title_tag += data.strip() + + def handle_endtag(self, tag): + if tag.lower() == "title": + self.inside_title_tag = False + + +@enforce_types +def should_save_title(link: Link, out_dir: Optional[str]=None) -> bool: + # if link already has valid title, skip it + if link.title and not link.title.lower().startswith('http'): + return False + + if is_static_file(link.url): + return False + + return SAVE_TITLE + +def extract_title_with_regex(html): + match = re.search(HTML_TITLE_REGEX, html) + output = htmldecode(match.group(1).strip()) if match else None + return output + +@enforce_types +def save_title(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """try to guess the page's title from its content""" + + from core.models import Snapshot + + output: ArchiveOutput = None + cmd = [ + CURL_BINARY, + *CURL_ARGS, + '--max-time', str(timeout), + *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []), + *([] if CHECK_SSL_VALIDITY else ['--insecure']), + link.url, + ] + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + html = download_url(link.url, timeout=timeout) + try: + # try using relatively strict html parser first + parser = TitleParser() + parser.feed(html) + output = parser.title + if output is None: + raise + except Exception: + # fallback to regex that can handle broken/malformed html + output = extract_title_with_regex(html) + + # if title is better than the one in the db, update db with new title + if isinstance(output, str) and output: + if not link.title or len(output) >= len(link.title): + Snapshot.objects.filter(url=link.url, + timestamp=link.timestamp)\ + .update(title=output) + else: + raise ArchiveError('Unable to detect page title') + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=CURL_VERSION, + output=output, + status=status, + **timer.stats, + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/wget.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/wget.py new file mode 100644 index 0000000..331f636 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/extractors/wget.py @@ -0,0 +1,184 @@ +__package__ = 'archivebox.extractors' + +import re +from pathlib import Path + +from typing import Optional +from datetime import datetime + +from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError +from ..system import run, chmod_file +from ..util import ( + enforce_types, + is_static_file, + without_scheme, + without_fragment, + without_query, + path, + domain, + urldecode, +) +from ..config import ( + WGET_ARGS, + TIMEOUT, + SAVE_WGET, + SAVE_WARC, + WGET_BINARY, + WGET_VERSION, + RESTRICT_FILE_NAMES, + CHECK_SSL_VALIDITY, + SAVE_WGET_REQUISITES, + WGET_AUTO_COMPRESSION, + WGET_USER_AGENT, + COOKIES_FILE, +) +from ..logging_util import TimedProgress + + +@enforce_types +def should_save_wget(link: Link, out_dir: Optional[Path]=None) -> bool: + output_path = wget_output_path(link) + out_dir = out_dir or Path(link.link_dir) + if output_path and (out_dir / output_path).exists(): + return False + + return SAVE_WGET + + +@enforce_types +def save_wget(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult: + """download full site using wget""" + + out_dir = out_dir or link.link_dir + if SAVE_WARC: + warc_dir = out_dir / "warc" + warc_dir.mkdir(exist_ok=True) + warc_path = warc_dir / str(int(datetime.now().timestamp())) + + # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html + output: ArchiveOutput = None + cmd = [ + WGET_BINARY, + # '--server-response', # print headers for better error parsing + *WGET_ARGS, + '--timeout={}'.format(timeout), + *(['--restrict-file-names={}'.format(RESTRICT_FILE_NAMES)] if RESTRICT_FILE_NAMES else []), + *(['--warc-file={}'.format(str(warc_path))] if SAVE_WARC else []), + *(['--page-requisites'] if SAVE_WGET_REQUISITES else []), + *(['--user-agent={}'.format(WGET_USER_AGENT)] if WGET_USER_AGENT else []), + *(['--load-cookies', COOKIES_FILE] if COOKIES_FILE else []), + *(['--compression=auto'] if WGET_AUTO_COMPRESSION else []), + *([] if SAVE_WARC else ['--timestamping']), + *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate', '--no-hsts']), + link.url, + ] + + status = 'succeeded' + timer = TimedProgress(timeout, prefix=' ') + try: + result = run(cmd, cwd=str(out_dir), timeout=timeout) + output = wget_output_path(link) + + # parse out number of files downloaded from last line of stderr: + # "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)" + output_tail = [ + line.strip() + for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] + if line.strip() + ] + files_downloaded = ( + int(output_tail[-1].strip().split(' ', 2)[1] or 0) + if 'Downloaded:' in output_tail[-1] + else 0 + ) + hints = ( + 'Got wget response code: {}.'.format(result.returncode), + *output_tail, + ) + + # Check for common failure cases + if (result.returncode > 0 and files_downloaded < 1) or output is None: + if b'403: Forbidden' in result.stderr: + raise ArchiveError('403 Forbidden (try changing WGET_USER_AGENT)', hints) + if b'404: Not Found' in result.stderr: + raise ArchiveError('404 Not Found', hints) + if b'ERROR 500: Internal Server Error' in result.stderr: + raise ArchiveError('500 Internal Server Error', hints) + raise ArchiveError('Wget failed or got an error from the server', hints) + chmod_file(output, cwd=str(out_dir)) + except Exception as err: + status = 'failed' + output = err + finally: + timer.end() + + return ArchiveResult( + cmd=cmd, + pwd=str(out_dir), + cmd_version=WGET_VERSION, + output=output, + status=status, + **timer.stats, + ) + + +@enforce_types +def wget_output_path(link: Link) -> Optional[str]: + """calculate the path to the wgetted .html file, since wget may + adjust some paths to be different than the base_url path. + + See docs on wget --adjust-extension (-E) + """ + if is_static_file(link.url): + return without_scheme(without_fragment(link.url)) + + # Wget downloads can save in a number of different ways depending on the url: + # https://example.com + # > example.com/index.html + # https://example.com?v=zzVa_tX1OiI + # > example.com/index.html?v=zzVa_tX1OiI.html + # https://www.example.com/?v=zzVa_tX1OiI + # > example.com/index.html?v=zzVa_tX1OiI.html + + # https://example.com/abc + # > example.com/abc.html + # https://example.com/abc/ + # > example.com/abc/index.html + # https://example.com/abc?v=zzVa_tX1OiI.html + # > example.com/abc?v=zzVa_tX1OiI.html + # https://example.com/abc/?v=zzVa_tX1OiI.html + # > example.com/abc/index.html?v=zzVa_tX1OiI.html + + # https://example.com/abc/test.html + # > example.com/abc/test.html + # https://example.com/abc/test?v=zzVa_tX1OiI + # > example.com/abc/test?v=zzVa_tX1OiI.html + # https://example.com/abc/test/?v=zzVa_tX1OiI + # > example.com/abc/test/index.html?v=zzVa_tX1OiI.html + + # There's also lots of complexity around how the urlencoding and renaming + # is done for pages with query and hash fragments or extensions like shtml / htm / php / etc + + # Since the wget algorithm for -E (appending .html) is incredibly complex + # and there's no way to get the computed output path from wget + # in order to avoid having to reverse-engineer how they calculate it, + # we just look in the output folder read the filename wget used from the filesystem + full_path = without_fragment(without_query(path(link.url))).strip('/') + search_dir = Path(link.link_dir) / domain(link.url).replace(":", "+") / urldecode(full_path) + for _ in range(4): + if search_dir.exists(): + if search_dir.is_dir(): + html_files = [ + f for f in search_dir.iterdir() + if re.search(".+\\.[Ss]?[Hh][Tt][Mm][Ll]?$", str(f), re.I | re.M) + ] + if html_files: + return str(html_files[0].relative_to(link.link_dir)) + + # Move up one directory level + search_dir = search_dir.parent + + if str(search_dir) == link.link_dir: + break + + return None diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/__init__.py new file mode 100644 index 0000000..8eab1d3 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/__init__.py @@ -0,0 +1,617 @@ +__package__ = 'archivebox.index' + +import os +import shutil +import json as pyjson +from pathlib import Path + +from itertools import chain +from typing import List, Tuple, Dict, Optional, Iterable +from collections import OrderedDict +from contextlib import contextmanager +from urllib.parse import urlparse +from django.db.models import QuerySet, Q + +from ..util import ( + scheme, + enforce_types, + ExtendedEncoder, +) +from ..config import ( + ARCHIVE_DIR_NAME, + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + OUTPUT_DIR, + TIMEOUT, + URL_BLACKLIST_PTN, + stderr, + OUTPUT_PERMISSIONS +) +from ..logging_util import ( + TimedProgress, + log_indexing_process_started, + log_indexing_process_finished, + log_indexing_started, + log_indexing_finished, + log_parsing_finished, + log_deduping_finished, +) + +from .schema import Link, ArchiveResult +from .html import ( + write_html_link_details, +) +from .json import ( + parse_json_link_details, + write_json_link_details, +) +from .sql import ( + write_sql_main_index, + write_sql_link_details, +) + +from ..search import search_backend_enabled, query_search_index + +### Link filtering and checking + +@enforce_types +def merge_links(a: Link, b: Link) -> Link: + """deterministially merge two links, favoring longer field values over shorter, + and "cleaner" values over worse ones. + """ + assert a.base_url == b.base_url, f'Cannot merge two links with different URLs ({a.base_url} != {b.base_url})' + + # longest url wins (because a fuzzy url will always be shorter) + url = a.url if len(a.url) > len(b.url) else b.url + + # best title based on length and quality + possible_titles = [ + title + for title in (a.title, b.title) + if title and title.strip() and '://' not in title + ] + title = None + if len(possible_titles) == 2: + title = max(possible_titles, key=lambda t: len(t)) + elif len(possible_titles) == 1: + title = possible_titles[0] + + # earliest valid timestamp + timestamp = ( + a.timestamp + if float(a.timestamp or 0) < float(b.timestamp or 0) else + b.timestamp + ) + + # all unique, truthy tags + tags_set = ( + set(tag.strip() for tag in (a.tags or '').split(',')) + | set(tag.strip() for tag in (b.tags or '').split(',')) + ) + tags = ','.join(tags_set) or None + + # all unique source entries + sources = list(set(a.sources + b.sources)) + + # all unique history entries for the combined archive methods + all_methods = set(list(a.history.keys()) + list(a.history.keys())) + history = { + method: (a.history.get(method) or []) + (b.history.get(method) or []) + for method in all_methods + } + for method in all_methods: + deduped_jsons = { + pyjson.dumps(result, sort_keys=True, cls=ExtendedEncoder) + for result in history[method] + } + history[method] = list(reversed(sorted( + (ArchiveResult.from_json(pyjson.loads(result)) for result in deduped_jsons), + key=lambda result: result.start_ts, + ))) + + return Link( + url=url, + timestamp=timestamp, + title=title, + tags=tags, + sources=sources, + history=history, + ) + + +@enforce_types +def validate_links(links: Iterable[Link]) -> List[Link]: + timer = TimedProgress(TIMEOUT * 4) + try: + links = archivable_links(links) # remove chrome://, about:, mailto: etc. + links = sorted_links(links) # deterministically sort the links based on timestamp, url + links = fix_duplicate_links(links) # merge/dedupe duplicate timestamps & urls + finally: + timer.end() + + return list(links) + +@enforce_types +def archivable_links(links: Iterable[Link]) -> Iterable[Link]: + """remove chrome://, about:// or other schemed links that cant be archived""" + for link in links: + try: + urlparse(link.url) + except ValueError: + continue + if scheme(link.url) not in ('http', 'https', 'ftp'): + continue + if URL_BLACKLIST_PTN and URL_BLACKLIST_PTN.search(link.url): + continue + + yield link + + +@enforce_types +def fix_duplicate_links(sorted_links: Iterable[Link]) -> Iterable[Link]: + """ + ensures that all non-duplicate links have monotonically increasing timestamps + """ + # from core.models import Snapshot + + unique_urls: OrderedDict[str, Link] = OrderedDict() + + for link in sorted_links: + if link.url in unique_urls: + # merge with any other links that share the same url + link = merge_links(unique_urls[link.url], link) + unique_urls[link.url] = link + + return unique_urls.values() + + +@enforce_types +def sorted_links(links: Iterable[Link]) -> Iterable[Link]: + sort_func = lambda link: (link.timestamp.split('.', 1)[0], link.url) + return sorted(links, key=sort_func, reverse=True) + + +@enforce_types +def links_after_timestamp(links: Iterable[Link], resume: Optional[float]=None) -> Iterable[Link]: + if not resume: + yield from links + return + + for link in links: + try: + if float(link.timestamp) <= resume: + yield link + except (ValueError, TypeError): + print('Resume value and all timestamp values must be valid numbers.') + + +@enforce_types +def lowest_uniq_timestamp(used_timestamps: OrderedDict, timestamp: str) -> str: + """resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2""" + + timestamp = timestamp.split('.')[0] + nonce = 0 + + # first try 152323423 before 152323423.0 + if timestamp not in used_timestamps: + return timestamp + + new_timestamp = '{}.{}'.format(timestamp, nonce) + while new_timestamp in used_timestamps: + nonce += 1 + new_timestamp = '{}.{}'.format(timestamp, nonce) + + return new_timestamp + + + +### Main Links Index + +@contextmanager +@enforce_types +def timed_index_update(out_path: Path): + log_indexing_started(out_path) + timer = TimedProgress(TIMEOUT * 2, prefix=' ') + try: + yield + finally: + timer.end() + + assert out_path.exists(), f'Failed to write index file: {out_path}' + log_indexing_finished(out_path) + + +@enforce_types +def write_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None: + """Writes links to sqlite3 file for a given list of links""" + + log_indexing_process_started(len(links)) + + try: + with timed_index_update(out_dir / SQL_INDEX_FILENAME): + write_sql_main_index(links, out_dir=out_dir) + os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes + + except (KeyboardInterrupt, SystemExit): + stderr('[!] Warning: Still writing index to disk...', color='lightyellow') + stderr(' Run archivebox init to fix any inconsistencies from an ungraceful exit.') + with timed_index_update(out_dir / SQL_INDEX_FILENAME): + write_sql_main_index(links, out_dir=out_dir) + os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes + raise SystemExit(0) + + log_indexing_process_finished() + +@enforce_types +def load_main_index(out_dir: Path=OUTPUT_DIR, warn: bool=True) -> List[Link]: + """parse and load existing index with any new links from import_path merged in""" + from core.models import Snapshot + try: + return Snapshot.objects.all() + + except (KeyboardInterrupt, SystemExit): + raise SystemExit(0) + +@enforce_types +def load_main_index_meta(out_dir: Path=OUTPUT_DIR) -> Optional[dict]: + index_path = out_dir / JSON_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + meta_dict = pyjson.load(f) + meta_dict.pop('links') + return meta_dict + + return None + + +@enforce_types +def parse_links_from_source(source_path: str, root_url: Optional[str]=None) -> Tuple[List[Link], List[Link]]: + + from ..parsers import parse_links + + new_links: List[Link] = [] + + # parse and validate the import file + raw_links, parser_name = parse_links(source_path, root_url=root_url) + new_links = validate_links(raw_links) + + if parser_name: + num_parsed = len(raw_links) + log_parsing_finished(num_parsed, parser_name) + + return new_links + +@enforce_types +def fix_duplicate_links_in_index(snapshots: QuerySet, links: Iterable[Link]) -> Iterable[Link]: + """ + Given a list of in-memory Links, dedupe and merge them with any conflicting Snapshots in the DB. + """ + unique_urls: OrderedDict[str, Link] = OrderedDict() + + for link in links: + index_link = snapshots.filter(url=link.url) + if index_link: + link = merge_links(index_link[0].as_link(), link) + + unique_urls[link.url] = link + + return unique_urls.values() + +@enforce_types +def dedupe_links(snapshots: QuerySet, + new_links: List[Link]) -> List[Link]: + """ + The validation of links happened at a different stage. This method will + focus on actual deduplication and timestamp fixing. + """ + + # merge existing links in out_dir and new links + dedup_links = fix_duplicate_links_in_index(snapshots, new_links) + + new_links = [ + link for link in new_links + if not snapshots.filter(url=link.url).exists() + ] + + dedup_links_dict = {link.url: link for link in dedup_links} + + # Replace links in new_links with the dedup version + for i in range(len(new_links)): + if new_links[i].url in dedup_links_dict.keys(): + new_links[i] = dedup_links_dict[new_links[i].url] + log_deduping_finished(len(new_links)) + + return new_links + +### Link Details Index + +@enforce_types +def write_link_details(link: Link, out_dir: Optional[str]=None, skip_sql_index: bool=False) -> None: + out_dir = out_dir or link.link_dir + + write_json_link_details(link, out_dir=out_dir) + write_html_link_details(link, out_dir=out_dir) + if not skip_sql_index: + write_sql_link_details(link) + + +@enforce_types +def load_link_details(link: Link, out_dir: Optional[str]=None) -> Link: + """check for an existing link archive in the given directory, + and load+merge it into the given link dict + """ + out_dir = out_dir or link.link_dir + + existing_link = parse_json_link_details(out_dir) + if existing_link: + return merge_links(existing_link, link) + + return link + + + +LINK_FILTERS = { + 'exact': lambda pattern: Q(url=pattern), + 'substring': lambda pattern: Q(url__icontains=pattern), + 'regex': lambda pattern: Q(url__iregex=pattern), + 'domain': lambda pattern: Q(url__istartswith=f"http://{pattern}") | Q(url__istartswith=f"https://{pattern}") | Q(url__istartswith=f"ftp://{pattern}"), + 'tag': lambda pattern: Q(tags__name=pattern), +} + +@enforce_types +def q_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet: + q_filter = Q() + for pattern in filter_patterns: + try: + q_filter = q_filter | LINK_FILTERS[filter_type](pattern) + except KeyError: + stderr() + stderr( + f'[X] Got invalid pattern for --filter-type={filter_type}:', + color='red', + ) + stderr(f' {pattern}') + raise SystemExit(2) + return snapshots.filter(q_filter) + +def search_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='search') -> QuerySet: + if not search_backend_enabled(): + stderr() + stderr( + '[X] The search backend is not enabled, set config.USE_SEARCHING_BACKEND = True', + color='red', + ) + raise SystemExit(2) + from core.models import Snapshot + + qsearch = Snapshot.objects.none() + for pattern in filter_patterns: + try: + qsearch |= query_search_index(pattern) + except: + raise SystemExit(2) + + return snapshots & qsearch + +@enforce_types +def snapshot_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet: + if filter_type != 'search': + return q_filter(snapshots, filter_patterns, filter_type) + else: + return search_filter(snapshots, filter_patterns, filter_type) + + +def get_indexed_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links without checking archive status or data directory validity""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in links + } + +def get_archived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links that are archived with a valid data directory""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_archived, links) + } + +def get_unarchived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """indexed links that are unarchived with no data directory or an empty data directory""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_unarchived, links) + } + +def get_present_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that actually exist in the archive/ folder""" + + all_folders = {} + + for entry in (out_dir / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + link = None + try: + link = parse_json_link_details(entry.path) + except Exception: + pass + + all_folders[entry.name] = link + + return all_folders + +def get_valid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs with a valid index matched to the main index and archived content""" + links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()] + return { + link.link_dir: link + for link in filter(is_valid, links) + } + +def get_invalid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized""" + duplicate = get_duplicate_folders(snapshots, out_dir=OUTPUT_DIR) + orphaned = get_orphaned_folders(snapshots, out_dir=OUTPUT_DIR) + corrupted = get_corrupted_folders(snapshots, out_dir=OUTPUT_DIR) + unrecognized = get_unrecognized_folders(snapshots, out_dir=OUTPUT_DIR) + return {**duplicate, **orphaned, **corrupted, **unrecognized} + + +def get_duplicate_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that conflict with other directories that have the same link URL or timestamp""" + by_url = {} + by_timestamp = {} + duplicate_folders = {} + + data_folders = ( + str(entry) + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir() + if entry.is_dir() and not snapshots.filter(timestamp=entry.name).exists() + ) + + for path in chain(snapshots.iterator(), data_folders): + link = None + if type(path) is not str: + path = path.as_link().link_dir + + try: + link = parse_json_link_details(path) + except Exception: + pass + + if link: + # link folder has same timestamp as different link folder + by_timestamp[link.timestamp] = by_timestamp.get(link.timestamp, 0) + 1 + if by_timestamp[link.timestamp] > 1: + duplicate_folders[path] = link + + # link folder has same url as different link folder + by_url[link.url] = by_url.get(link.url, 0) + 1 + if by_url[link.url] > 1: + duplicate_folders[path] = link + return duplicate_folders + +def get_orphaned_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that contain a valid index but aren't listed in the main index""" + orphaned_folders = {} + + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + link = None + try: + link = parse_json_link_details(str(entry)) + except Exception: + pass + + if link and not snapshots.filter(timestamp=entry.name).exists(): + # folder is a valid link data dir with index details, but it's not in the main index + orphaned_folders[str(entry)] = link + + return orphaned_folders + +def get_corrupted_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that don't contain a valid index and aren't listed in the main index""" + corrupted = {} + for snapshot in snapshots.iterator(): + link = snapshot.as_link() + if is_corrupt(link): + corrupted[link.link_dir] = link + return corrupted + +def get_unrecognized_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + """dirs that don't contain recognizable archive data and aren't listed in the main index""" + unrecognized_folders: Dict[str, Optional[Link]] = {} + + for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir(): + if entry.is_dir(): + index_exists = (entry / "index.json").exists() + link = None + try: + link = parse_json_link_details(str(entry)) + except KeyError: + # Try to fix index + if index_exists: + try: + # Last attempt to repair the detail index + link_guessed = parse_json_link_details(str(entry), guess=True) + write_json_link_details(link_guessed, out_dir=str(entry)) + link = parse_json_link_details(str(entry)) + except Exception: + pass + + if index_exists and link is None: + # index exists but it's corrupted or unparseable + unrecognized_folders[str(entry)] = link + + elif not index_exists: + # link details index doesn't exist and the folder isn't in the main index + timestamp = entry.name + if not snapshots.filter(timestamp=timestamp).exists(): + unrecognized_folders[str(entry)] = link + + return unrecognized_folders + + +def is_valid(link: Link) -> bool: + dir_exists = Path(link.link_dir).exists() + index_exists = (Path(link.link_dir) / "index.json").exists() + if not dir_exists: + # unarchived links are not included in the valid list + return False + if dir_exists and not index_exists: + return False + if dir_exists and index_exists: + try: + parsed_link = parse_json_link_details(link.link_dir, guess=True) + return link.url == parsed_link.url + except Exception: + pass + return False + +def is_corrupt(link: Link) -> bool: + if not Path(link.link_dir).exists(): + # unarchived links are not considered corrupt + return False + + if is_valid(link): + return False + + return True + +def is_archived(link: Link) -> bool: + return is_valid(link) and link.is_archived + +def is_unarchived(link: Link) -> bool: + if not Path(link.link_dir).exists(): + return True + return not link.is_archived + + +def fix_invalid_folder_locations(out_dir: Path=OUTPUT_DIR) -> Tuple[List[str], List[str]]: + fixed = [] + cant_fix = [] + for entry in os.scandir(out_dir / ARCHIVE_DIR_NAME): + if entry.is_dir(follow_symlinks=True): + if (Path(entry.path) / 'index.json').exists(): + try: + link = parse_json_link_details(entry.path) + except KeyError: + link = None + if not link: + continue + + if not entry.path.endswith(f'/{link.timestamp}'): + dest = out_dir / ARCHIVE_DIR_NAME / link.timestamp + if dest.exists(): + cant_fix.append(entry.path) + else: + shutil.move(entry.path, dest) + fixed.append(dest) + timestamp = entry.path.rsplit('/', 1)[-1] + assert link.link_dir == entry.path + assert link.timestamp == timestamp + write_json_link_details(link, out_dir=entry.path) + + return fixed, cant_fix diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/csv.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/csv.py new file mode 100644 index 0000000..804e646 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/csv.py @@ -0,0 +1,37 @@ +__package__ = 'archivebox.index' + +from typing import List, Optional, Any + +from ..util import enforce_types +from .schema import Link + + +@enforce_types +def links_to_csv(links: List[Link], + cols: Optional[List[str]]=None, + header: bool=True, + separator: str=',', + ljust: int=0) -> str: + + cols = cols or ['timestamp', 'is_archived', 'url'] + + header_str = '' + if header: + header_str = separator.join(col.ljust(ljust) for col in cols) + + row_strs = ( + link.to_csv(cols=cols, ljust=ljust, separator=separator) + for link in links + ) + + return '\n'.join((header_str, *row_strs)) + + +@enforce_types +def to_csv(obj: Any, cols: List[str], separator: str=',', ljust: int=0) -> str: + from .json import to_json + + return separator.join( + to_json(getattr(obj, col), indent=None).ljust(ljust) + for col in cols + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/html.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/html.py new file mode 100644 index 0000000..a62e2c7 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/html.py @@ -0,0 +1,164 @@ +__package__ = 'archivebox.index' + +from datetime import datetime +from typing import List, Optional, Iterator, Mapping +from pathlib import Path + +from django.utils.html import format_html +from collections import defaultdict + +from .schema import Link +from ..system import atomic_write +from ..logging_util import printable_filesize +from ..util import ( + enforce_types, + ts_to_date, + urlencode, + htmlencode, + urldecode, +) +from ..config import ( + OUTPUT_DIR, + VERSION, + GIT_SHA, + FOOTER_INFO, + HTML_INDEX_FILENAME, +) + +MAIN_INDEX_TEMPLATE = 'main_index.html' +MINIMAL_INDEX_TEMPLATE = 'main_index_minimal.html' +LINK_DETAILS_TEMPLATE = 'link_details.html' +TITLE_LOADING_MSG = 'Not yet archived...' + + +### Main Links Index + +@enforce_types +def parse_html_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[str]: + """parse an archive index html file and return the list of urls""" + + index_path = Path(out_dir) / HTML_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + for line in f: + if 'class="link-url"' in line: + yield line.split('"')[1] + return () + +@enforce_types +def generate_index_from_links(links: List[Link], with_headers: bool): + if with_headers: + output = main_index_template(links) + else: + output = main_index_template(links, template=MINIMAL_INDEX_TEMPLATE) + return output + +@enforce_types +def main_index_template(links: List[Link], template: str=MAIN_INDEX_TEMPLATE) -> str: + """render the template for the entire main index""" + + return render_django_template(template, { + 'version': VERSION, + 'git_sha': GIT_SHA, + 'num_links': str(len(links)), + 'date_updated': datetime.now().strftime('%Y-%m-%d'), + 'time_updated': datetime.now().strftime('%Y-%m-%d %H:%M'), + 'links': [link._asdict(extended=True) for link in links], + 'FOOTER_INFO': FOOTER_INFO, + }) + + +### Link Details Index + +@enforce_types +def write_html_link_details(link: Link, out_dir: Optional[str]=None) -> None: + out_dir = out_dir or link.link_dir + + rendered_html = link_details_template(link) + atomic_write(str(Path(out_dir) / HTML_INDEX_FILENAME), rendered_html) + + +@enforce_types +def link_details_template(link: Link) -> str: + + from ..extractors.wget import wget_output_path + + link_info = link._asdict(extended=True) + + return render_django_template(LINK_DETAILS_TEMPLATE, { + **link_info, + **link_info['canonical'], + 'title': htmlencode( + link.title + or (link.base_url if link.is_archived else TITLE_LOADING_MSG) + ), + 'url_str': htmlencode(urldecode(link.base_url)), + 'archive_url': urlencode( + wget_output_path(link) + or (link.domain if link.is_archived else '') + ) or 'about:blank', + 'extension': link.extension or 'html', + 'tags': link.tags or 'untagged', + 'size': printable_filesize(link.archive_size) if link.archive_size else 'pending', + 'status': 'archived' if link.is_archived else 'not yet archived', + 'status_color': 'success' if link.is_archived else 'danger', + 'oldest_archive_date': ts_to_date(link.oldest_archive_date), + }) + +@enforce_types +def render_django_template(template: str, context: Mapping[str, str]) -> str: + """render a given html template string with the given template content""" + from django.template.loader import render_to_string + + return render_to_string(template, context) + + +def snapshot_icons(snapshot) -> str: + from core.models import EXTRACTORS + + archive_results = snapshot.archiveresult_set.filter(status="succeeded") + link = snapshot.as_link() + path = link.archive_path + canon = link.canonical_outputs() + output = "" + output_template = '<a href="/{}/{}" class="exists-{}" title="{}">{} </a>' + icons = { + "singlefile": "❶", + "wget": "🆆", + "dom": "🅷", + "pdf": "📄", + "screenshot": "💻", + "media": "📼", + "git": "🅶", + "archive_org": "🏛", + "readability": "🆁", + "mercury": "🅼", + "warc": "📦" + } + exclude = ["favicon", "title", "headers", "archive_org"] + # Missing specific entry for WARC + + extractor_items = defaultdict(lambda: None) + for extractor, _ in EXTRACTORS: + for result in archive_results: + if result.extractor == extractor: + extractor_items[extractor] = result + + for extractor, _ in EXTRACTORS: + if extractor not in exclude: + exists = extractor_items[extractor] is not None + output += output_template.format(path, canon[f"{extractor}_path"], str(exists), + extractor, icons.get(extractor, "?")) + if extractor == "wget": + # warc isn't technically it's own extractor, so we have to add it after wget + exists = list((Path(path) / canon["warc_path"]).glob("*.warc.gz")) + output += output_template.format(exists[0] if exists else '#', canon["warc_path"], str(bool(exists)), "warc", icons.get("warc", "?")) + + if extractor == "archive_org": + # The check for archive_org is different, so it has to be handled separately + target_path = Path(path) / "archive.org.txt" + exists = target_path.exists() + output += '<a href="{}" class="exists-{}" title="{}">{}</a> '.format(canon["archive_org_path"], str(exists), + "archive_org", icons.get("archive_org", "?")) + + return format_html(f'<span class="files-icons" style="font-size: 1.1em; opacity: 0.8">{output}<span>') diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/json.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/json.py new file mode 100644 index 0000000..f24b969 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/json.py @@ -0,0 +1,154 @@ +__package__ = 'archivebox.index' + +import os +import sys +import json as pyjson +from pathlib import Path + +from datetime import datetime +from typing import List, Optional, Iterator, Any, Union + +from .schema import Link +from ..system import atomic_write +from ..util import enforce_types +from ..config import ( + VERSION, + OUTPUT_DIR, + FOOTER_INFO, + GIT_SHA, + DEPENDENCIES, + JSON_INDEX_FILENAME, + ARCHIVE_DIR_NAME, + ANSI +) + + +MAIN_INDEX_HEADER = { + 'info': 'This is an index of site data archived by ArchiveBox: The self-hosted web archive.', + 'schema': 'archivebox.index.json', + 'copyright_info': FOOTER_INFO, + 'meta': { + 'project': 'ArchiveBox', + 'version': VERSION, + 'git_sha': GIT_SHA, + 'website': 'https://ArchiveBox.io', + 'docs': 'https://github.com/ArchiveBox/ArchiveBox/wiki', + 'source': 'https://github.com/ArchiveBox/ArchiveBox', + 'issues': 'https://github.com/ArchiveBox/ArchiveBox/issues', + 'dependencies': DEPENDENCIES, + }, +} + +@enforce_types +def generate_json_index_from_links(links: List[Link], with_headers: bool): + if with_headers: + output = { + **MAIN_INDEX_HEADER, + 'num_links': len(links), + 'updated': datetime.now(), + 'last_run_cmd': sys.argv, + 'links': links, + } + else: + output = links + return to_json(output, indent=4, sort_keys=True) + + +@enforce_types +def parse_json_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]: + """parse an archive index json file and return the list of links""" + + index_path = Path(out_dir) / JSON_INDEX_FILENAME + if index_path.exists(): + with open(index_path, 'r', encoding='utf-8') as f: + links = pyjson.load(f)['links'] + for link_json in links: + try: + yield Link.from_json(link_json) + except KeyError: + try: + detail_index_path = Path(OUTPUT_DIR) / ARCHIVE_DIR_NAME / link_json['timestamp'] + yield parse_json_link_details(str(detail_index_path)) + except KeyError: + # as a last effort, try to guess the missing values out of existing ones + try: + yield Link.from_json(link_json, guess=True) + except KeyError: + print(" {lightyellow}! Failed to load the index.json from {}".format(detail_index_path, **ANSI)) + continue + return () + +### Link Details Index + +@enforce_types +def write_json_link_details(link: Link, out_dir: Optional[str]=None) -> None: + """write a json file with some info about the link""" + + out_dir = out_dir or link.link_dir + path = Path(out_dir) / JSON_INDEX_FILENAME + atomic_write(str(path), link._asdict(extended=True)) + + +@enforce_types +def parse_json_link_details(out_dir: Union[Path, str], guess: Optional[bool]=False) -> Optional[Link]: + """load the json link index from a given directory""" + existing_index = Path(out_dir) / JSON_INDEX_FILENAME + if existing_index.exists(): + with open(existing_index, 'r', encoding='utf-8') as f: + try: + link_json = pyjson.load(f) + return Link.from_json(link_json, guess) + except pyjson.JSONDecodeError: + pass + return None + + +@enforce_types +def parse_json_links_details(out_dir: Union[Path, str]) -> Iterator[Link]: + """read through all the archive data folders and return the parsed links""" + + for entry in os.scandir(Path(out_dir) / ARCHIVE_DIR_NAME): + if entry.is_dir(follow_symlinks=True): + if (Path(entry.path) / 'index.json').exists(): + try: + link = parse_json_link_details(entry.path) + except KeyError: + link = None + if link: + yield link + + + +### Helpers + +class ExtendedEncoder(pyjson.JSONEncoder): + """ + Extended json serializer that supports serializing several model + fields and objects + """ + + def default(self, obj): + cls_name = obj.__class__.__name__ + + if hasattr(obj, '_asdict'): + return obj._asdict() + + elif isinstance(obj, bytes): + return obj.decode() + + elif isinstance(obj, datetime): + return obj.isoformat() + + elif isinstance(obj, Exception): + return '{}: {}'.format(obj.__class__.__name__, obj) + + elif cls_name in ('dict_items', 'dict_keys', 'dict_values'): + return tuple(obj) + + return pyjson.JSONEncoder.default(self, obj) + + +@enforce_types +def to_json(obj: Any, indent: Optional[int]=4, sort_keys: bool=True, cls=ExtendedEncoder) -> str: + return pyjson.dumps(obj, indent=indent, sort_keys=sort_keys, cls=ExtendedEncoder) + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/schema.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/schema.py new file mode 100644 index 0000000..bc3a25d --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/schema.py @@ -0,0 +1,448 @@ +""" + +WARNING: THIS FILE IS ALL LEGACY CODE TO BE REMOVED. + +DO NOT ADD ANY NEW FEATURES TO THIS FILE, NEW CODE GOES HERE: core/models.py + +""" + +__package__ = 'archivebox.index' + +from pathlib import Path + +from datetime import datetime, timedelta + +from typing import List, Dict, Any, Optional, Union + +from dataclasses import dataclass, asdict, field, fields + + +from ..system import get_dir_size + +from ..config import OUTPUT_DIR, ARCHIVE_DIR_NAME + +class ArchiveError(Exception): + def __init__(self, message, hints=None): + super().__init__(message) + self.hints = hints + +LinkDict = Dict[str, Any] + +ArchiveOutput = Union[str, Exception, None] + +@dataclass(frozen=True) +class ArchiveResult: + cmd: List[str] + pwd: Optional[str] + cmd_version: Optional[str] + output: ArchiveOutput + status: str + start_ts: datetime + end_ts: datetime + index_texts: Union[List[str], None] = None + schema: str = 'ArchiveResult' + + def __post_init__(self): + self.typecheck() + + def _asdict(self): + return asdict(self) + + def typecheck(self) -> None: + assert self.schema == self.__class__.__name__ + assert isinstance(self.status, str) and self.status + assert isinstance(self.start_ts, datetime) + assert isinstance(self.end_ts, datetime) + assert isinstance(self.cmd, list) + assert all(isinstance(arg, str) and arg for arg in self.cmd) + assert self.pwd is None or isinstance(self.pwd, str) and self.pwd + assert self.cmd_version is None or isinstance(self.cmd_version, str) and self.cmd_version + assert self.output is None or isinstance(self.output, (str, Exception)) + if isinstance(self.output, str): + assert self.output + + @classmethod + def guess_ts(_cls, dict_info): + from ..util import parse_date + parsed_timestamp = parse_date(dict_info["timestamp"]) + start_ts = parsed_timestamp + end_ts = parsed_timestamp + timedelta(seconds=int(dict_info["duration"])) + return start_ts, end_ts + + @classmethod + def from_json(cls, json_info, guess=False): + from ..util import parse_date + + info = { + key: val + for key, val in json_info.items() + if key in cls.field_names() + } + if guess: + keys = info.keys() + if "start_ts" not in keys: + info["start_ts"], info["end_ts"] = cls.guess_ts(json_info) + else: + info['start_ts'] = parse_date(info['start_ts']) + info['end_ts'] = parse_date(info['end_ts']) + if "pwd" not in keys: + info["pwd"] = str(Path(OUTPUT_DIR) / ARCHIVE_DIR_NAME / json_info["timestamp"]) + if "cmd_version" not in keys: + info["cmd_version"] = "Undefined" + if "cmd" not in keys: + info["cmd"] = [] + else: + info['start_ts'] = parse_date(info['start_ts']) + info['end_ts'] = parse_date(info['end_ts']) + info['cmd_version'] = info.get('cmd_version') + if type(info["cmd"]) is str: + info["cmd"] = [info["cmd"]] + return cls(**info) + + def to_dict(self, *keys) -> dict: + if keys: + return {k: v for k, v in asdict(self).items() if k in keys} + return asdict(self) + + def to_json(self, indent=4, sort_keys=True) -> str: + from .json import to_json + + return to_json(self, indent=indent, sort_keys=sort_keys) + + def to_csv(self, cols: Optional[List[str]]=None, separator: str=',', ljust: int=0) -> str: + from .csv import to_csv + + return to_csv(self, csv_col=cols or self.field_names(), separator=separator, ljust=ljust) + + @classmethod + def field_names(cls): + return [f.name for f in fields(cls)] + + @property + def duration(self) -> int: + return (self.end_ts - self.start_ts).seconds + +@dataclass(frozen=True) +class Link: + timestamp: str + url: str + title: Optional[str] + tags: Optional[str] + sources: List[str] + history: Dict[str, List[ArchiveResult]] = field(default_factory=lambda: {}) + updated: Optional[datetime] = None + schema: str = 'Link' + + + def __str__(self) -> str: + return f'[{self.timestamp}] {self.url} "{self.title}"' + + def __post_init__(self): + self.typecheck() + + def overwrite(self, **kwargs): + """pure functional version of dict.update that returns a new instance""" + return Link(**{**self._asdict(), **kwargs}) + + def __eq__(self, other): + if not isinstance(other, Link): + return NotImplemented + return self.url == other.url + + def __gt__(self, other): + if not isinstance(other, Link): + return NotImplemented + if not self.timestamp or not other.timestamp: + return + return float(self.timestamp) > float(other.timestamp) + + def typecheck(self) -> None: + from ..config import stderr, ANSI + try: + assert self.schema == self.__class__.__name__ + assert isinstance(self.timestamp, str) and self.timestamp + assert self.timestamp.replace('.', '').isdigit() + assert isinstance(self.url, str) and '://' in self.url + assert self.updated is None or isinstance(self.updated, datetime) + assert self.title is None or (isinstance(self.title, str) and self.title) + assert self.tags is None or isinstance(self.tags, str) + assert isinstance(self.sources, list) + assert all(isinstance(source, str) and source for source in self.sources) + assert isinstance(self.history, dict) + for method, results in self.history.items(): + assert isinstance(method, str) and method + assert isinstance(results, list) + assert all(isinstance(result, ArchiveResult) for result in results) + except Exception: + stderr('{red}[X] Error while loading link! [{}] {} "{}"{reset}'.format(self.timestamp, self.url, self.title, **ANSI)) + raise + + def _asdict(self, extended=False): + info = { + 'schema': 'Link', + 'url': self.url, + 'title': self.title or None, + 'timestamp': self.timestamp, + 'updated': self.updated or None, + 'tags': self.tags or None, + 'sources': self.sources or [], + 'history': self.history or {}, + } + if extended: + info.update({ + 'link_dir': self.link_dir, + 'archive_path': self.archive_path, + + 'hash': self.url_hash, + 'base_url': self.base_url, + 'scheme': self.scheme, + 'domain': self.domain, + 'path': self.path, + 'basename': self.basename, + 'extension': self.extension, + 'is_static': self.is_static, + + 'bookmarked_date': self.bookmarked_date, + 'updated_date': self.updated_date, + 'oldest_archive_date': self.oldest_archive_date, + 'newest_archive_date': self.newest_archive_date, + + 'is_archived': self.is_archived, + 'num_outputs': self.num_outputs, + 'num_failures': self.num_failures, + + 'latest': self.latest_outputs(), + 'canonical': self.canonical_outputs(), + }) + return info + + def as_snapshot(self): + from core.models import Snapshot + return Snapshot.objects.get(url=self.url) + + @classmethod + def from_json(cls, json_info, guess=False): + from ..util import parse_date + + info = { + key: val + for key, val in json_info.items() + if key in cls.field_names() + } + info['updated'] = parse_date(info.get('updated')) + info['sources'] = info.get('sources') or [] + + json_history = info.get('history') or {} + cast_history = {} + + for method, method_history in json_history.items(): + cast_history[method] = [] + for json_result in method_history: + assert isinstance(json_result, dict), 'Items in Link["history"][method] must be dicts' + cast_result = ArchiveResult.from_json(json_result, guess) + cast_history[method].append(cast_result) + + info['history'] = cast_history + return cls(**info) + + def to_json(self, indent=4, sort_keys=True) -> str: + from .json import to_json + + return to_json(self, indent=indent, sort_keys=sort_keys) + + def to_csv(self, cols: Optional[List[str]]=None, separator: str=',', ljust: int=0) -> str: + from .csv import to_csv + + return to_csv(self, cols=cols or self.field_names(), separator=separator, ljust=ljust) + + @classmethod + def field_names(cls): + return [f.name for f in fields(cls)] + + @property + def link_dir(self) -> str: + from ..config import CONFIG + return str(Path(CONFIG['ARCHIVE_DIR']) / self.timestamp) + + @property + def archive_path(self) -> str: + from ..config import ARCHIVE_DIR_NAME + return '{}/{}'.format(ARCHIVE_DIR_NAME, self.timestamp) + + @property + def archive_size(self) -> float: + try: + return get_dir_size(self.archive_path)[0] + except Exception: + return 0 + + ### URL Helpers + @property + def url_hash(self): + from ..util import hashurl + + return hashurl(self.url) + + @property + def scheme(self) -> str: + from ..util import scheme + return scheme(self.url) + + @property + def extension(self) -> str: + from ..util import extension + return extension(self.url) + + @property + def domain(self) -> str: + from ..util import domain + return domain(self.url) + + @property + def path(self) -> str: + from ..util import path + return path(self.url) + + @property + def basename(self) -> str: + from ..util import basename + return basename(self.url) + + @property + def base_url(self) -> str: + from ..util import base_url + return base_url(self.url) + + ### Pretty Printing Helpers + @property + def bookmarked_date(self) -> Optional[str]: + from ..util import ts_to_date + + max_ts = (datetime.now() + timedelta(days=30)).timestamp() + + if self.timestamp and self.timestamp.replace('.', '').isdigit(): + if 0 < float(self.timestamp) < max_ts: + return ts_to_date(datetime.fromtimestamp(float(self.timestamp))) + else: + return str(self.timestamp) + return None + + + @property + def updated_date(self) -> Optional[str]: + from ..util import ts_to_date + return ts_to_date(self.updated) if self.updated else None + + @property + def archive_dates(self) -> List[datetime]: + return [ + result.start_ts + for method in self.history.keys() + for result in self.history[method] + ] + + @property + def oldest_archive_date(self) -> Optional[datetime]: + return min(self.archive_dates, default=None) + + @property + def newest_archive_date(self) -> Optional[datetime]: + return max(self.archive_dates, default=None) + + ### Archive Status Helpers + @property + def num_outputs(self) -> int: + return self.as_snapshot().num_outputs + + @property + def num_failures(self) -> int: + return sum(1 + for method in self.history.keys() + for result in self.history[method] + if result.status == 'failed') + + @property + def is_static(self) -> bool: + from ..util import is_static_file + return is_static_file(self.url) + + @property + def is_archived(self) -> bool: + from ..config import ARCHIVE_DIR + from ..util import domain + + output_paths = ( + domain(self.url), + 'output.pdf', + 'screenshot.png', + 'output.html', + 'media', + 'singlefile.html' + ) + + return any( + (Path(ARCHIVE_DIR) / self.timestamp / path).exists() + for path in output_paths + ) + + def latest_outputs(self, status: str=None) -> Dict[str, ArchiveOutput]: + """get the latest output that each archive method produced for link""" + + ARCHIVE_METHODS = ( + 'title', 'favicon', 'wget', 'warc', 'singlefile', 'pdf', + 'screenshot', 'dom', 'git', 'media', 'archive_org', + ) + latest: Dict[str, ArchiveOutput] = {} + for archive_method in ARCHIVE_METHODS: + # get most recent succesful result in history for each archive method + history = self.history.get(archive_method) or [] + history = list(filter(lambda result: result.output, reversed(history))) + if status is not None: + history = list(filter(lambda result: result.status == status, history)) + + history = list(history) + if history: + latest[archive_method] = history[0].output + else: + latest[archive_method] = None + return latest + + + def canonical_outputs(self) -> Dict[str, Optional[str]]: + """predict the expected output paths that should be present after archiving""" + + from ..extractors.wget import wget_output_path + canonical = { + 'index_path': 'index.html', + 'favicon_path': 'favicon.ico', + 'google_favicon_path': 'https://www.google.com/s2/favicons?domain={}'.format(self.domain), + 'wget_path': wget_output_path(self), + 'warc_path': 'warc', + 'singlefile_path': 'singlefile.html', + 'readability_path': 'readability/content.html', + 'mercury_path': 'mercury/content.html', + 'pdf_path': 'output.pdf', + 'screenshot_path': 'screenshot.png', + 'dom_path': 'output.html', + 'archive_org_path': 'https://web.archive.org/web/{}'.format(self.base_url), + 'git_path': 'git', + 'media_path': 'media', + } + if self.is_static: + # static binary files like PDF and images are handled slightly differently. + # they're just downloaded once and aren't archived separately multiple times, + # so the wget, screenshot, & pdf urls should all point to the same file + + static_path = wget_output_path(self) + canonical.update({ + 'title': self.basename, + 'wget_path': static_path, + 'pdf_path': static_path, + 'screenshot_path': static_path, + 'dom_path': static_path, + 'singlefile_path': static_path, + 'readability_path': static_path, + 'mercury_path': static_path, + }) + return canonical + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/sql.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/sql.py new file mode 100644 index 0000000..1e99f67 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/index/sql.py @@ -0,0 +1,106 @@ +__package__ = 'archivebox.index' + +from io import StringIO +from pathlib import Path +from typing import List, Tuple, Iterator +from django.db.models import QuerySet +from django.db import transaction + +from .schema import Link +from ..util import enforce_types +from ..config import OUTPUT_DIR + + +### Main Links Index + +@enforce_types +def parse_sql_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]: + from core.models import Snapshot + + return ( + Link.from_json(page.as_json(*Snapshot.keys)) + for page in Snapshot.objects.all() + ) + +@enforce_types +def remove_from_sql_main_index(snapshots: QuerySet, out_dir: Path=OUTPUT_DIR) -> None: + with transaction.atomic(): + snapshots.delete() + +@enforce_types +def write_link_to_sql_index(link: Link): + from core.models import Snapshot + info = {k: v for k, v in link._asdict().items() if k in Snapshot.keys} + tags = info.pop("tags") + if tags is None: + tags = [] + + try: + info["timestamp"] = Snapshot.objects.get(url=link.url).timestamp + except Snapshot.DoesNotExist: + while Snapshot.objects.filter(timestamp=info["timestamp"]).exists(): + info["timestamp"] = str(float(info["timestamp"]) + 1.0) + + snapshot, _ = Snapshot.objects.update_or_create(url=link.url, defaults=info) + snapshot.save_tags(tags) + return snapshot + + +@enforce_types +def write_sql_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None: + with transaction.atomic(): + for link in links: + write_link_to_sql_index(link) + + +@enforce_types +def write_sql_link_details(link: Link, out_dir: Path=OUTPUT_DIR) -> None: + from core.models import Snapshot + + with transaction.atomic(): + try: + snap = Snapshot.objects.get(url=link.url) + except Snapshot.DoesNotExist: + snap = write_link_to_sql_index(link) + snap.title = link.title + + tag_set = ( + set(tag.strip() for tag in (link.tags or '').split(',')) + ) + tag_list = list(tag_set) or [] + + snap.save() + snap.save_tags(tag_list) + + + +@enforce_types +def list_migrations(out_dir: Path=OUTPUT_DIR) -> List[Tuple[bool, str]]: + from django.core.management import call_command + out = StringIO() + call_command("showmigrations", list=True, stdout=out) + out.seek(0) + migrations = [] + for line in out.readlines(): + if line.strip() and ']' in line: + status_str, name_str = line.strip().split(']', 1) + is_applied = 'X' in status_str + migration_name = name_str.strip() + migrations.append((is_applied, migration_name)) + + return migrations + +@enforce_types +def apply_migrations(out_dir: Path=OUTPUT_DIR) -> List[str]: + from django.core.management import call_command + null, out = StringIO(), StringIO() + call_command("makemigrations", interactive=False, stdout=null) + call_command("migrate", interactive=False, stdout=out) + out.seek(0) + + return [line.strip() for line in out.readlines() if line.strip()] + +@enforce_types +def get_admins(out_dir: Path=OUTPUT_DIR) -> List[str]: + from django.contrib.auth.models import User + return User.objects.filter(is_superuser=True) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/logging_util.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/logging_util.py new file mode 100644 index 0000000..f2b8673 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/logging_util.py @@ -0,0 +1,569 @@ +__package__ = 'archivebox' + +import re +import os +import sys +import time +import argparse +from math import log +from multiprocessing import Process +from pathlib import Path + +from datetime import datetime +from dataclasses import dataclass +from typing import Optional, List, Dict, Union, IO, TYPE_CHECKING + +if TYPE_CHECKING: + from .index.schema import Link, ArchiveResult + +from .util import enforce_types +from .config import ( + ConfigDict, + OUTPUT_DIR, + PYTHON_ENCODING, + ANSI, + IS_TTY, + TERM_WIDTH, + SHOW_PROGRESS, + SOURCES_DIR_NAME, + stderr, +) + +@dataclass +class RuntimeStats: + """mutable stats counter for logging archiving timing info to CLI output""" + + skipped: int = 0 + succeeded: int = 0 + failed: int = 0 + + parse_start_ts: Optional[datetime] = None + parse_end_ts: Optional[datetime] = None + + index_start_ts: Optional[datetime] = None + index_end_ts: Optional[datetime] = None + + archiving_start_ts: Optional[datetime] = None + archiving_end_ts: Optional[datetime] = None + +# globals are bad, mmkay +_LAST_RUN_STATS = RuntimeStats() + + + +class SmartFormatter(argparse.HelpFormatter): + """Patched formatter that prints newlines in argparse help strings""" + def _split_lines(self, text, width): + if '\n' in text: + return text.splitlines() + return argparse.HelpFormatter._split_lines(self, text, width) + + +def reject_stdin(caller: str, stdin: Optional[IO]=sys.stdin) -> None: + """Tell the user they passed stdin to a command that doesn't accept it""" + + if stdin and not stdin.isatty(): + stdin_raw_text = stdin.read().strip() + if stdin_raw_text: + stderr(f'[X] The "{caller}" command does not accept stdin.', color='red') + stderr(f' Run archivebox "{caller} --help" to see usage and examples.') + stderr() + raise SystemExit(1) + + +def accept_stdin(stdin: Optional[IO]=sys.stdin) -> Optional[str]: + """accept any standard input and return it as a string or None""" + if not stdin: + return None + elif stdin and not stdin.isatty(): + stdin_str = stdin.read().strip() + return stdin_str or None + return None + + +class TimedProgress: + """Show a progress bar and measure elapsed time until .end() is called""" + + def __init__(self, seconds, prefix=''): + self.SHOW_PROGRESS = SHOW_PROGRESS + if self.SHOW_PROGRESS: + self.p = Process(target=progress_bar, args=(seconds, prefix)) + self.p.start() + + self.stats = {'start_ts': datetime.now(), 'end_ts': None} + + def end(self): + """immediately end progress, clear the progressbar line, and save end_ts""" + + end_ts = datetime.now() + self.stats['end_ts'] = end_ts + + if self.SHOW_PROGRESS: + # terminate if we havent already terminated + try: + # kill the progress bar subprocess + try: + self.p.close() # must be closed *before* its terminnated + except: + pass + self.p.terminate() + self.p.join() + + + # clear whole terminal line + try: + sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset'])) + except (IOError, BrokenPipeError): + # ignore when the parent proc has stopped listening to our stdout + pass + except ValueError: + pass + + +@enforce_types +def progress_bar(seconds: int, prefix: str='') -> None: + """show timer in the form of progress bar, with percentage and seconds remaining""" + chunk = '█' if PYTHON_ENCODING == 'UTF-8' else '#' + last_width = TERM_WIDTH() + chunks = last_width - len(prefix) - 20 # number of progress chunks to show (aka max bar width) + try: + for s in range(seconds * chunks): + max_width = TERM_WIDTH() + if max_width < last_width: + # when the terminal size is shrunk, we have to write a newline + # otherwise the progress bar will keep wrapping incorrectly + sys.stdout.write('\r\n') + sys.stdout.flush() + chunks = max_width - len(prefix) - 20 + pct_complete = s / chunks / seconds * 100 + log_pct = (log(pct_complete or 1, 10) / 2) * 100 # everyone likes faster progress bars ;) + bar_width = round(log_pct/(100/chunks)) + last_width = max_width + + # ████████████████████ 0.9% (1/60sec) + sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format( + prefix, + ANSI['green' if pct_complete < 80 else 'lightyellow'], + (chunk * bar_width).ljust(chunks), + ANSI['reset'], + round(pct_complete, 1), + round(s/chunks), + seconds, + )) + sys.stdout.flush() + time.sleep(1 / chunks) + + # ██████████████████████████████████ 100.0% (60/60sec) + sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format( + prefix, + ANSI['red'], + chunk * chunks, + ANSI['reset'], + 100.0, + seconds, + seconds, + )) + sys.stdout.flush() + # uncomment to have it disappear when it hits 100% instead of staying full red: + # time.sleep(0.5) + # sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset'])) + # sys.stdout.flush() + except (KeyboardInterrupt, BrokenPipeError): + print() + pass + + +def log_cli_command(subcommand: str, subcommand_args: List[str], stdin: Optional[str], pwd: str): + from .config import VERSION, ANSI + cmd = ' '.join(('archivebox', subcommand, *subcommand_args)) + stderr('{black}[i] [{now}] ArchiveBox v{VERSION}: {cmd}{reset}'.format( + now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'), + VERSION=VERSION, + cmd=cmd, + **ANSI, + )) + stderr('{black} > {pwd}{reset}'.format(pwd=pwd, **ANSI)) + stderr() + +### Parsing Stage + + +def log_importing_started(urls: Union[str, List[str]], depth: int, index_only: bool): + _LAST_RUN_STATS.parse_start_ts = datetime.now() + print('{green}[+] [{}] Adding {} links to index (crawl depth={}){}...{reset}'.format( + _LAST_RUN_STATS.parse_start_ts.strftime('%Y-%m-%d %H:%M:%S'), + len(urls) if isinstance(urls, list) else len(urls.split('\n')), + depth, + ' (index only)' if index_only else '', + **ANSI, + )) + +def log_source_saved(source_file: str): + print(' > Saved verbatim input to {}/{}'.format(SOURCES_DIR_NAME, source_file.rsplit('/', 1)[-1])) + +def log_parsing_finished(num_parsed: int, parser_name: str): + _LAST_RUN_STATS.parse_end_ts = datetime.now() + print(' > Parsed {} URLs from input ({})'.format(num_parsed, parser_name)) + +def log_deduping_finished(num_new_links: int): + print(' > Found {} new URLs not already in index'.format(num_new_links)) + + +def log_crawl_started(new_links): + print() + print('{green}[*] Starting crawl of {} sites 1 hop out from starting point{reset}'.format(len(new_links), **ANSI)) + +### Indexing Stage + +def log_indexing_process_started(num_links: int): + start_ts = datetime.now() + _LAST_RUN_STATS.index_start_ts = start_ts + print() + print('{black}[*] [{}] Writing {} links to main index...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + **ANSI, + )) + + +def log_indexing_process_finished(): + end_ts = datetime.now() + _LAST_RUN_STATS.index_end_ts = end_ts + + +def log_indexing_started(out_path: str): + if IS_TTY: + sys.stdout.write(f' > {out_path}') + + +def log_indexing_finished(out_path: str): + print(f'\r √ {out_path}') + + +### Archiving Stage + +def log_archiving_started(num_links: int, resume: Optional[float]=None): + start_ts = datetime.now() + _LAST_RUN_STATS.archiving_start_ts = start_ts + print() + if resume: + print('{green}[▶] [{}] Resuming archive updating for {} pages starting from {}...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + resume, + **ANSI, + )) + else: + print('{green}[▶] [{}] Starting archiving of {} snapshots in index...{reset}'.format( + start_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + **ANSI, + )) + +def log_archiving_paused(num_links: int, idx: int, timestamp: str): + end_ts = datetime.now() + _LAST_RUN_STATS.archiving_end_ts = end_ts + print() + print('\n{lightyellow}[X] [{now}] Downloading paused on link {timestamp} ({idx}/{total}){reset}'.format( + **ANSI, + now=end_ts.strftime('%Y-%m-%d %H:%M:%S'), + idx=idx+1, + timestamp=timestamp, + total=num_links, + )) + print() + print(' {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI)) + print(' archivebox server # then visit http://127.0.0.1:8000') + print(' Continue archiving where you left off by running:') + print(' archivebox update --resume={}'.format(timestamp)) + +def log_archiving_finished(num_links: int): + end_ts = datetime.now() + _LAST_RUN_STATS.archiving_end_ts = end_ts + assert _LAST_RUN_STATS.archiving_start_ts is not None + seconds = end_ts.timestamp() - _LAST_RUN_STATS.archiving_start_ts.timestamp() + if seconds > 60: + duration = '{0:.2f} min'.format(seconds / 60) + else: + duration = '{0:.2f} sec'.format(seconds) + + print() + print('{}[√] [{}] Update of {} pages complete ({}){}'.format( + ANSI['green'], + end_ts.strftime('%Y-%m-%d %H:%M:%S'), + num_links, + duration, + ANSI['reset'], + )) + print(' - {} links skipped'.format(_LAST_RUN_STATS.skipped)) + print(' - {} links updated'.format(_LAST_RUN_STATS.succeeded + _LAST_RUN_STATS.failed)) + print(' - {} links had errors'.format(_LAST_RUN_STATS.failed)) + print() + print(' {lightred}Hint:{reset} To manage your archive in a Web UI, run:'.format(**ANSI)) + print(' archivebox server 0.0.0.0:8000') + + +def log_link_archiving_started(link: "Link", link_dir: str, is_new: bool): + # [*] [2019-03-22 13:46:45] "Log Structured Merge Trees - ben stopford" + # http://www.benstopford.com/2015/02/14/log-structured-merge-trees/ + # > output/archive/1478739709 + + print('\n[{symbol_color}{symbol}{reset}] [{symbol_color}{now}{reset}] "{title}"'.format( + symbol_color=ANSI['green' if is_new else 'black'], + symbol='+' if is_new else '√', + now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'), + title=link.title or link.base_url, + **ANSI, + )) + print(' {blue}{url}{reset}'.format(url=link.url, **ANSI)) + print(' {} {}'.format( + '>' if is_new else '√', + pretty_path(link_dir), + )) + +def log_link_archiving_finished(link: "Link", link_dir: str, is_new: bool, stats: dict): + total = sum(stats.values()) + + if stats['failed'] > 0 : + _LAST_RUN_STATS.failed += 1 + elif stats['skipped'] == total: + _LAST_RUN_STATS.skipped += 1 + else: + _LAST_RUN_STATS.succeeded += 1 + + +def log_archive_method_started(method: str): + print(' > {}'.format(method)) + + +def log_archive_method_finished(result: "ArchiveResult"): + """quote the argument with whitespace in a command so the user can + copy-paste the outputted string directly to run the cmd + """ + # Prettify CMD string and make it safe to copy-paste by quoting arguments + quoted_cmd = ' '.join( + '"{}"'.format(arg) if ' ' in arg else arg + for arg in result.cmd + ) + + if result.status == 'failed': + if result.output.__class__.__name__ == 'TimeoutExpired': + duration = (result.end_ts - result.start_ts).seconds + hint_header = [ + '{lightyellow}Extractor timed out after {}s.{reset}'.format(duration, **ANSI), + ] + else: + hint_header = [ + '{lightyellow}Extractor failed:{reset}'.format(**ANSI), + ' {reset}{} {red}{}{reset}'.format( + result.output.__class__.__name__.replace('ArchiveError', ''), + result.output, + **ANSI, + ), + ] + + # Prettify error output hints string and limit to five lines + hints = getattr(result.output, 'hints', None) or () + if hints: + hints = hints if isinstance(hints, (list, tuple)) else hints.split('\n') + hints = ( + ' {}{}{}'.format(ANSI['lightyellow'], line.strip(), ANSI['reset']) + for line in hints[:5] if line.strip() + ) + + + # Collect and prefix output lines with indentation + output_lines = [ + *hint_header, + *hints, + '{}Run to see full output:{}'.format(ANSI['lightred'], ANSI['reset']), + *([' cd {};'.format(result.pwd)] if result.pwd else []), + ' {}'.format(quoted_cmd), + ] + print('\n'.join( + ' {}'.format(line) + for line in output_lines + if line + )) + print() + + +def log_list_started(filter_patterns: Optional[List[str]], filter_type: str): + print('{green}[*] Finding links in the archive index matching these {} patterns:{reset}'.format( + filter_type, + **ANSI, + )) + print(' {}'.format(' '.join(filter_patterns or ()))) + +def log_list_finished(links): + from .index.csv import links_to_csv + print() + print('---------------------------------------------------------------------------------------------------') + print(links_to_csv(links, cols=['timestamp', 'is_archived', 'num_outputs', 'url'], header=True, ljust=16, separator=' | ')) + print('---------------------------------------------------------------------------------------------------') + print() + + +def log_removal_started(links: List["Link"], yes: bool, delete: bool): + print('{lightyellow}[i] Found {} matching URLs to remove.{reset}'.format(len(links), **ANSI)) + if delete: + file_counts = [link.num_outputs for link in links if Path(link.link_dir).exists()] + print( + f' {len(links)} Links will be de-listed from the main index, and their archived content folders will be deleted from disk.\n' + f' ({len(file_counts)} data folders with {sum(file_counts)} archived files will be deleted!)' + ) + else: + print( + ' Matching links will be de-listed from the main index, but their archived content folders will remain in place on disk.\n' + ' (Pass --delete if you also want to permanently delete the data folders)' + ) + + if not yes: + print() + print('{lightyellow}[?] Do you want to proceed with removing these {} links?{reset}'.format(len(links), **ANSI)) + try: + assert input(' y/[n]: ').lower() == 'y' + except (KeyboardInterrupt, EOFError, AssertionError): + raise SystemExit(0) + +def log_removal_finished(all_links: int, to_remove: int): + if all_links == 0: + print() + print('{red}[X] No matching links found.{reset}'.format(**ANSI)) + else: + print() + print('{red}[√] Removed {} out of {} links from the archive index.{reset}'.format( + to_remove, + all_links, + **ANSI, + )) + print(' Index now contains {} links.'.format(all_links - to_remove)) + + +def log_shell_welcome_msg(): + from .cli import list_subcommands + + print('{green}# ArchiveBox Imports{reset}'.format(**ANSI)) + print('{green}from core.models import Snapshot, User{reset}'.format(**ANSI)) + print('{green}from archivebox import *\n {}{reset}'.format("\n ".join(list_subcommands().keys()), **ANSI)) + print() + print('[i] Welcome to the ArchiveBox Shell!') + print(' https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Shell-Usage') + print() + print(' {lightred}Hint:{reset} Example use:'.format(**ANSI)) + print(' print(Snapshot.objects.filter(is_archived=True).count())') + print(' Snapshot.objects.get(url="https://example.com").as_json()') + print(' add("https://example.com/some/new/url")') + + + +### Helpers + +@enforce_types +def pretty_path(path: Union[Path, str]) -> str: + """convert paths like .../ArchiveBox/archivebox/../output/abc into output/abc""" + pwd = Path('.').resolve() + # parent = os.path.abspath(os.path.join(pwd, os.path.pardir)) + return str(path).replace(str(pwd) + '/', './') + + +@enforce_types +def printable_filesize(num_bytes: Union[int, float]) -> str: + for count in ['Bytes','KB','MB','GB']: + if num_bytes > -1024.0 and num_bytes < 1024.0: + return '%3.1f %s' % (num_bytes, count) + num_bytes /= 1024.0 + return '%3.1f %s' % (num_bytes, 'TB') + + +@enforce_types +def printable_folders(folders: Dict[str, Optional["Link"]], + with_headers: bool=False) -> str: + return '\n'.join( + f'{folder} {link and link.url} "{link and link.title}"' + for folder, link in folders.items() + ) + + + +@enforce_types +def printable_config(config: ConfigDict, prefix: str='') -> str: + return f'\n{prefix}'.join( + f'{key}={val}' + for key, val in config.items() + if not (isinstance(val, dict) or callable(val)) + ) + + +@enforce_types +def printable_folder_status(name: str, folder: Dict) -> str: + if folder['enabled']: + if folder['is_valid']: + color, symbol, note = 'green', '√', 'valid' + else: + color, symbol, note, num_files = 'red', 'X', 'invalid', '?' + else: + color, symbol, note, num_files = 'lightyellow', '-', 'disabled', '-' + + if folder['path']: + if Path(folder['path']).exists(): + num_files = ( + f'{len(os.listdir(folder["path"]))} files' + if Path(folder['path']).is_dir() else + printable_filesize(Path(folder['path']).stat().st_size) + ) + else: + num_files = 'missing' + + path = str(folder['path']).replace(str(OUTPUT_DIR), '.') if folder['path'] else '' + if path and ' ' in path: + path = f'"{path}"' + + # if path is just a plain dot, replace it back with the full path for clarity + if path == '.': + path = str(OUTPUT_DIR) + + return ' '.join(( + ANSI[color], + symbol, + ANSI['reset'], + name.ljust(21), + num_files.ljust(14), + ANSI[color], + note.ljust(8), + ANSI['reset'], + path.ljust(76), + )) + + +@enforce_types +def printable_dependency_version(name: str, dependency: Dict) -> str: + version = None + if dependency['enabled']: + if dependency['is_valid']: + color, symbol, note, version = 'green', '√', 'valid', '' + + parsed_version_num = re.search(r'[\d\.]+', dependency['version']) + if parsed_version_num: + version = f'v{parsed_version_num[0]}' + + if not version: + color, symbol, note, version = 'red', 'X', 'invalid', '?' + else: + color, symbol, note, version = 'lightyellow', '-', 'disabled', '-' + + path = str(dependency["path"]).replace(str(OUTPUT_DIR), '.') if dependency["path"] else '' + if path and ' ' in path: + path = f'"{path}"' + + return ' '.join(( + ANSI[color], + symbol, + ANSI['reset'], + name.ljust(21), + version.ljust(14), + ANSI[color], + note.ljust(8), + ANSI['reset'], + path.ljust(76), + )) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/main.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/main.py new file mode 100644 index 0000000..eb8cd6a --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/main.py @@ -0,0 +1,1131 @@ +__package__ = 'archivebox' + +import os +import sys +import shutil +import platform +from pathlib import Path +from datetime import date + +from typing import Dict, List, Optional, Iterable, IO, Union +from crontab import CronTab, CronSlices +from django.db.models import QuerySet + +from .cli import ( + list_subcommands, + run_subcommand, + display_first, + meta_cmds, + main_cmds, + archive_cmds, +) +from .parsers import ( + save_text_as_source, + save_file_as_source, + parse_links_memory, +) +from .index.schema import Link +from .util import enforce_types # type: ignore +from .system import get_dir_size, dedupe_cron_jobs, CRON_COMMENT +from .index import ( + load_main_index, + parse_links_from_source, + dedupe_links, + write_main_index, + snapshot_filter, + get_indexed_folders, + get_archived_folders, + get_unarchived_folders, + get_present_folders, + get_valid_folders, + get_invalid_folders, + get_duplicate_folders, + get_orphaned_folders, + get_corrupted_folders, + get_unrecognized_folders, + fix_invalid_folder_locations, + write_link_details, +) +from .index.json import ( + parse_json_main_index, + parse_json_links_details, + generate_json_index_from_links, +) +from .index.sql import ( + get_admins, + apply_migrations, + remove_from_sql_main_index, +) +from .index.html import ( + generate_index_from_links, +) +from .index.csv import links_to_csv +from .extractors import archive_links, archive_link, ignore_methods +from .config import ( + stderr, + hint, + ConfigDict, + ANSI, + IS_TTY, + IN_DOCKER, + USER, + ARCHIVEBOX_BINARY, + ONLY_NEW, + OUTPUT_DIR, + SOURCES_DIR, + ARCHIVE_DIR, + LOGS_DIR, + CONFIG_FILE, + ARCHIVE_DIR_NAME, + SOURCES_DIR_NAME, + LOGS_DIR_NAME, + STATIC_DIR_NAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, + SQL_INDEX_FILENAME, + ROBOTS_TXT_FILENAME, + FAVICON_FILENAME, + check_dependencies, + check_data_folder, + write_config_file, + VERSION, + CODE_LOCATIONS, + EXTERNAL_LOCATIONS, + DATA_LOCATIONS, + DEPENDENCIES, + load_all_config, + CONFIG, + USER_CONFIG, + get_real_name, +) +from .logging_util import ( + TERM_WIDTH, + TimedProgress, + log_importing_started, + log_crawl_started, + log_removal_started, + log_removal_finished, + log_list_started, + log_list_finished, + printable_config, + printable_folders, + printable_filesize, + printable_folder_status, + printable_dependency_version, +) + +from .search import flush_search_index, index_links + +ALLOWED_IN_OUTPUT_DIR = { + 'lost+found', + '.DS_Store', + '.venv', + 'venv', + 'virtualenv', + '.virtualenv', + 'node_modules', + 'package-lock.json', + ARCHIVE_DIR_NAME, + SOURCES_DIR_NAME, + LOGS_DIR_NAME, + STATIC_DIR_NAME, + SQL_INDEX_FILENAME, + JSON_INDEX_FILENAME, + HTML_INDEX_FILENAME, + ROBOTS_TXT_FILENAME, + FAVICON_FILENAME, +} + +@enforce_types +def help(out_dir: Path=OUTPUT_DIR) -> None: + """Print the ArchiveBox help message and usage""" + + all_subcommands = list_subcommands() + COMMANDS_HELP_TEXT = '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in meta_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in main_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd in archive_cmds + ) + '\n\n ' + '\n '.join( + f'{cmd.ljust(20)} {summary}' + for cmd, summary in all_subcommands.items() + if cmd not in display_first + ) + + + if (Path(out_dir) / SQL_INDEX_FILENAME).exists(): + print('''{green}ArchiveBox v{}: The self-hosted internet archive.{reset} + +{lightred}Active data directory:{reset} + {} + +{lightred}Usage:{reset} + archivebox [command] [--help] [--version] [...args] + +{lightred}Commands:{reset} + {} + +{lightred}Example Use:{reset} + mkdir my-archive; cd my-archive/ + archivebox init + archivebox status + + archivebox add https://example.com/some/page + archivebox add --depth=1 ~/Downloads/bookmarks_export.html + + archivebox list --sort=timestamp --csv=timestamp,url,is_archived + archivebox schedule --every=day https://example.com/some/feed.rss + archivebox update --resume=15109948213.123 + +{lightred}Documentation:{reset} + https://github.com/ArchiveBox/ArchiveBox/wiki +'''.format(VERSION, out_dir, COMMANDS_HELP_TEXT, **ANSI)) + + else: + print('{green}Welcome to ArchiveBox v{}!{reset}'.format(VERSION, **ANSI)) + print() + if IN_DOCKER: + print('When using Docker, you need to mount a volume to use as your data dir:') + print(' docker run -v /some/path:/data archivebox ...') + print() + print('To import an existing archive (from a previous version of ArchiveBox):') + print(' 1. cd into your data dir OUTPUT_DIR (usually ArchiveBox/output) and run:') + print(' 2. archivebox init') + print() + print('To start a new archive:') + print(' 1. Create an empty directory, then cd into it and run:') + print(' 2. archivebox init') + print() + print('For more information, see the documentation here:') + print(' https://github.com/ArchiveBox/ArchiveBox/wiki') + + +@enforce_types +def version(quiet: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Print the ArchiveBox version and dependency information""" + + if quiet: + print(VERSION) + else: + print('ArchiveBox v{}'.format(VERSION)) + p = platform.uname() + print(sys.implementation.name.title(), p.system, platform.platform(), p.machine, '(in Docker)' if IN_DOCKER else '(not in Docker)') + print() + + print('{white}[i] Dependency versions:{reset}'.format(**ANSI)) + for name, dependency in DEPENDENCIES.items(): + print(printable_dependency_version(name, dependency)) + + print() + print('{white}[i] Source-code locations:{reset}'.format(**ANSI)) + for name, folder in CODE_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + + print() + print('{white}[i] Secrets locations:{reset}'.format(**ANSI)) + for name, folder in EXTERNAL_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + + print() + if DATA_LOCATIONS['OUTPUT_DIR']['is_valid']: + print('{white}[i] Data locations:{reset}'.format(**ANSI)) + for name, folder in DATA_LOCATIONS.items(): + print(printable_folder_status(name, folder)) + else: + print() + print('{white}[i] Data locations:{reset}'.format(**ANSI)) + + print() + check_dependencies() + + +@enforce_types +def run(subcommand: str, + subcommand_args: Optional[List[str]], + stdin: Optional[IO]=None, + out_dir: Path=OUTPUT_DIR) -> None: + """Run a given ArchiveBox subcommand with the given list of args""" + run_subcommand( + subcommand=subcommand, + subcommand_args=subcommand_args, + stdin=stdin, + pwd=out_dir, + ) + + +@enforce_types +def init(force: bool=False, out_dir: Path=OUTPUT_DIR) -> None: + """Initialize a new ArchiveBox collection in the current directory""" + from core.models import Snapshot + Path(out_dir).mkdir(exist_ok=True) + is_empty = not len(set(os.listdir(out_dir)) - ALLOWED_IN_OUTPUT_DIR) + + if (Path(out_dir) / JSON_INDEX_FILENAME).exists(): + stderr("[!] This folder contains a JSON index. It is deprecated, and will no longer be kept up to date automatically.", color="lightyellow") + stderr(" You can run `archivebox list --json --with-headers > index.json` to manually generate it.", color="lightyellow") + + existing_index = (Path(out_dir) / SQL_INDEX_FILENAME).exists() + + if is_empty and not existing_index: + print('{green}[+] Initializing a new ArchiveBox collection in this folder...{reset}'.format(**ANSI)) + print(f' {out_dir}') + print('{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + elif existing_index: + print('{green}[*] Updating existing ArchiveBox collection in this folder...{reset}'.format(**ANSI)) + print(f' {out_dir}') + print('{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + else: + if force: + stderr('[!] This folder appears to already have files in it, but no index.sqlite3 is present.', color='lightyellow') + stderr(' Because --force was passed, ArchiveBox will initialize anyway (which may overwrite existing files).') + else: + stderr( + ("{red}[X] This folder appears to already have files in it, but no index.sqlite3 present.{reset}\n\n" + " You must run init in a completely empty directory, or an existing data folder.\n\n" + " {lightred}Hint:{reset} To import an existing data folder make sure to cd into the folder first, \n" + " then run and run 'archivebox init' to pick up where you left off.\n\n" + " (Always make sure your data folder is backed up first before updating ArchiveBox)" + ).format(out_dir, **ANSI) + ) + raise SystemExit(2) + + if existing_index: + print('\n{green}[*] Verifying archive folder structure...{reset}'.format(**ANSI)) + else: + print('\n{green}[+] Building archive folder structure...{reset}'.format(**ANSI)) + + Path(SOURCES_DIR).mkdir(exist_ok=True) + print(f' √ {SOURCES_DIR}') + + Path(ARCHIVE_DIR).mkdir(exist_ok=True) + print(f' √ {ARCHIVE_DIR}') + + Path(LOGS_DIR).mkdir(exist_ok=True) + print(f' √ {LOGS_DIR}') + + write_config_file({}, out_dir=out_dir) + print(f' √ {CONFIG_FILE}') + if (Path(out_dir) / SQL_INDEX_FILENAME).exists(): + print('\n{green}[*] Verifying main SQL index and running migrations...{reset}'.format(**ANSI)) + else: + print('\n{green}[+] Building main SQL index and running migrations...{reset}'.format(**ANSI)) + + DATABASE_FILE = Path(out_dir) / SQL_INDEX_FILENAME + print(f' √ {DATABASE_FILE}') + print() + for migration_line in apply_migrations(out_dir): + print(f' {migration_line}') + + + assert DATABASE_FILE.exists() + + # from django.contrib.auth.models import User + # if IS_TTY and not User.objects.filter(is_superuser=True).exists(): + # print('{green}[+] Creating admin user account...{reset}'.format(**ANSI)) + # call_command("createsuperuser", interactive=True) + + print() + print('{green}[*] Collecting links from any existing indexes and archive folders...{reset}'.format(**ANSI)) + + all_links = Snapshot.objects.none() + pending_links: Dict[str, Link] = {} + + if existing_index: + all_links = load_main_index(out_dir=out_dir, warn=False) + print(' √ Loaded {} links from existing main index.'.format(all_links.count())) + + # Links in data folders that dont match their timestamp + fixed, cant_fix = fix_invalid_folder_locations(out_dir=out_dir) + if fixed: + print(' {lightyellow}√ Fixed {} data directory locations that didn\'t match their link timestamps.{reset}'.format(len(fixed), **ANSI)) + if cant_fix: + print(' {lightyellow}! Could not fix {} data directory locations due to conflicts with existing folders.{reset}'.format(len(cant_fix), **ANSI)) + + # Links in JSON index but not in main index + orphaned_json_links = { + link.url: link + for link in parse_json_main_index(out_dir) + if not all_links.filter(url=link.url).exists() + } + if orphaned_json_links: + pending_links.update(orphaned_json_links) + print(' {lightyellow}√ Added {} orphaned links from existing JSON index...{reset}'.format(len(orphaned_json_links), **ANSI)) + + # Links in data dir indexes but not in main index + orphaned_data_dir_links = { + link.url: link + for link in parse_json_links_details(out_dir) + if not all_links.filter(url=link.url).exists() + } + if orphaned_data_dir_links: + pending_links.update(orphaned_data_dir_links) + print(' {lightyellow}√ Added {} orphaned links from existing archive directories.{reset}'.format(len(orphaned_data_dir_links), **ANSI)) + + # Links in invalid/duplicate data dirs + invalid_folders = { + folder: link + for folder, link in get_invalid_folders(all_links, out_dir=out_dir).items() + } + if invalid_folders: + print(' {lightyellow}! Skipped adding {} invalid link data directories.{reset}'.format(len(invalid_folders), **ANSI)) + print(' X ' + '\n X '.join(f'{folder} {link}' for folder, link in invalid_folders.items())) + print() + print(' {lightred}Hint:{reset} For more information about the link data directories that were skipped, run:'.format(**ANSI)) + print(' archivebox status') + print(' archivebox list --status=invalid') + + + write_main_index(list(pending_links.values()), out_dir=out_dir) + + print('\n{green}------------------------------------------------------------------{reset}'.format(**ANSI)) + if existing_index: + print('{green}[√] Done. Verified and updated the existing ArchiveBox collection.{reset}'.format(**ANSI)) + else: + print('{green}[√] Done. A new ArchiveBox collection was initialized ({} links).{reset}'.format(len(all_links), **ANSI)) + print() + print(' {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI)) + print(' archivebox server # then visit http://127.0.0.1:8000') + print() + print(' To add new links, you can run:') + print(" archivebox add ~/some/path/or/url/to/list_of_links.txt") + print() + print(' For more usage and examples, run:') + print(' archivebox help') + + json_index = Path(out_dir) / JSON_INDEX_FILENAME + html_index = Path(out_dir) / HTML_INDEX_FILENAME + index_name = f"{date.today()}_index_old" + if json_index.exists(): + json_index.rename(f"{index_name}.json") + if html_index.exists(): + html_index.rename(f"{index_name}.html") + + + +@enforce_types +def status(out_dir: Path=OUTPUT_DIR) -> None: + """Print out some info and statistics about the archive collection""" + + check_data_folder(out_dir=out_dir) + + from core.models import Snapshot + from django.contrib.auth import get_user_model + User = get_user_model() + + print('{green}[*] Scanning archive main index...{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {out_dir}/*', ANSI['reset']) + num_bytes, num_dirs, num_files = get_dir_size(out_dir, recursive=False, pattern='index.') + size = printable_filesize(num_bytes) + print(f' Index size: {size} across {num_files} files') + print() + + links = load_main_index(out_dir=out_dir) + num_sql_links = links.count() + num_link_details = sum(1 for link in parse_json_links_details(out_dir=out_dir)) + print(f' > SQL Main Index: {num_sql_links} links'.ljust(36), f'(found in {SQL_INDEX_FILENAME})') + print(f' > JSON Link Details: {num_link_details} links'.ljust(36), f'(found in {ARCHIVE_DIR_NAME}/*/index.json)') + print() + print('{green}[*] Scanning archive data directories...{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {ARCHIVE_DIR}/*', ANSI['reset']) + num_bytes, num_dirs, num_files = get_dir_size(ARCHIVE_DIR) + size = printable_filesize(num_bytes) + print(f' Size: {size} across {num_files} files in {num_dirs} directories') + print(ANSI['black']) + num_indexed = len(get_indexed_folders(links, out_dir=out_dir)) + num_archived = len(get_archived_folders(links, out_dir=out_dir)) + num_unarchived = len(get_unarchived_folders(links, out_dir=out_dir)) + print(f' > indexed: {num_indexed}'.ljust(36), f'({get_indexed_folders.__doc__})') + print(f' > archived: {num_archived}'.ljust(36), f'({get_archived_folders.__doc__})') + print(f' > unarchived: {num_unarchived}'.ljust(36), f'({get_unarchived_folders.__doc__})') + + num_present = len(get_present_folders(links, out_dir=out_dir)) + num_valid = len(get_valid_folders(links, out_dir=out_dir)) + print() + print(f' > present: {num_present}'.ljust(36), f'({get_present_folders.__doc__})') + print(f' > valid: {num_valid}'.ljust(36), f'({get_valid_folders.__doc__})') + + duplicate = get_duplicate_folders(links, out_dir=out_dir) + orphaned = get_orphaned_folders(links, out_dir=out_dir) + corrupted = get_corrupted_folders(links, out_dir=out_dir) + unrecognized = get_unrecognized_folders(links, out_dir=out_dir) + num_invalid = len({**duplicate, **orphaned, **corrupted, **unrecognized}) + print(f' > invalid: {num_invalid}'.ljust(36), f'({get_invalid_folders.__doc__})') + print(f' > duplicate: {len(duplicate)}'.ljust(36), f'({get_duplicate_folders.__doc__})') + print(f' > orphaned: {len(orphaned)}'.ljust(36), f'({get_orphaned_folders.__doc__})') + print(f' > corrupted: {len(corrupted)}'.ljust(36), f'({get_corrupted_folders.__doc__})') + print(f' > unrecognized: {len(unrecognized)}'.ljust(36), f'({get_unrecognized_folders.__doc__})') + + print(ANSI['reset']) + + if num_indexed: + print(' {lightred}Hint:{reset} You can list link data directories by status like so:'.format(**ANSI)) + print(' archivebox list --status=<status> (e.g. indexed, corrupted, archived, etc.)') + + if orphaned: + print(' {lightred}Hint:{reset} To automatically import orphaned data directories into the main index, run:'.format(**ANSI)) + print(' archivebox init') + + if num_invalid: + print(' {lightred}Hint:{reset} You may need to manually remove or fix some invalid data directories, afterwards make sure to run:'.format(**ANSI)) + print(' archivebox init') + + print() + print('{green}[*] Scanning recent archive changes and user logins:{reset}'.format(**ANSI)) + print(ANSI['lightyellow'], f' {LOGS_DIR}/*', ANSI['reset']) + users = get_admins().values_list('username', flat=True) + print(f' UI users {len(users)}: {", ".join(users)}') + last_login = User.objects.order_by('last_login').last() + if last_login: + print(f' Last UI login: {last_login.username} @ {str(last_login.last_login)[:16]}') + last_updated = Snapshot.objects.order_by('updated').last() + if last_updated: + print(f' Last changes: {str(last_updated.updated)[:16]}') + + if not users: + print() + print(' {lightred}Hint:{reset} You can create an admin user by running:'.format(**ANSI)) + print(' archivebox manage createsuperuser') + + print() + for snapshot in links.order_by('-updated')[:10]: + if not snapshot.updated: + continue + print( + ANSI['black'], + ( + f' > {str(snapshot.updated)[:16]} ' + f'[{snapshot.num_outputs} {("X", "√")[snapshot.is_archived]} {printable_filesize(snapshot.archive_size)}] ' + f'"{snapshot.title}": {snapshot.url}' + )[:TERM_WIDTH()], + ANSI['reset'], + ) + print(ANSI['black'], ' ...', ANSI['reset']) + + +@enforce_types +def oneshot(url: str, extractors: str="", out_dir: Path=OUTPUT_DIR): + """ + Create a single URL archive folder with an index.json and index.html, and all the archive method outputs. + You can run this to archive single pages without needing to create a whole collection with archivebox init. + """ + oneshot_link, _ = parse_links_memory([url]) + if len(oneshot_link) > 1: + stderr( + '[X] You should pass a single url to the oneshot command', + color='red' + ) + raise SystemExit(2) + + methods = extractors.split(",") if extractors else ignore_methods(['title']) + archive_link(oneshot_link[0], out_dir=out_dir, methods=methods) + return oneshot_link + +@enforce_types +def add(urls: Union[str, List[str]], + depth: int=0, + update_all: bool=not ONLY_NEW, + index_only: bool=False, + overwrite: bool=False, + init: bool=False, + extractors: str="", + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Add a new URL or list of URLs to your archive""" + + assert depth in (0, 1), 'Depth must be 0 or 1 (depth >1 is not supported yet)' + + extractors = extractors.split(",") if extractors else [] + + if init: + run_subcommand('init', stdin=None, pwd=out_dir) + + # Load list of links from the existing index + check_data_folder(out_dir=out_dir) + check_dependencies() + new_links: List[Link] = [] + all_links = load_main_index(out_dir=out_dir) + + log_importing_started(urls=urls, depth=depth, index_only=index_only) + if isinstance(urls, str): + # save verbatim stdin to sources + write_ahead_log = save_text_as_source(urls, filename='{ts}-import.txt', out_dir=out_dir) + elif isinstance(urls, list): + # save verbatim args to sources + write_ahead_log = save_text_as_source('\n'.join(urls), filename='{ts}-import.txt', out_dir=out_dir) + + new_links += parse_links_from_source(write_ahead_log, root_url=None) + + # If we're going one level deeper, download each link and look for more links + new_links_depth = [] + if new_links and depth == 1: + log_crawl_started(new_links) + for new_link in new_links: + downloaded_file = save_file_as_source(new_link.url, filename=f'{new_link.timestamp}-crawl-{new_link.domain}.txt', out_dir=out_dir) + new_links_depth += parse_links_from_source(downloaded_file, root_url=new_link.url) + + imported_links = list({link.url: link for link in (new_links + new_links_depth)}.values()) + new_links = dedupe_links(all_links, imported_links) + + write_main_index(links=new_links, out_dir=out_dir) + all_links = load_main_index(out_dir=out_dir) + + if index_only: + return all_links + + # Run the archive methods for each link + archive_kwargs = { + "out_dir": out_dir, + } + if extractors: + archive_kwargs["methods"] = extractors + if update_all: + archive_links(all_links, overwrite=overwrite, **archive_kwargs) + elif overwrite: + archive_links(imported_links, overwrite=True, **archive_kwargs) + elif new_links: + archive_links(new_links, overwrite=False, **archive_kwargs) + + return all_links + +@enforce_types +def remove(filter_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + snapshots: Optional[QuerySet]=None, + after: Optional[float]=None, + before: Optional[float]=None, + yes: bool=False, + delete: bool=False, + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Remove the specified URLs from the archive""" + + check_data_folder(out_dir=out_dir) + + if snapshots is None: + if filter_str and filter_patterns: + stderr( + '[X] You should pass either a pattern as an argument, ' + 'or pass a list of patterns via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif not (filter_str or filter_patterns): + stderr( + '[X] You should pass either a pattern as an argument, ' + 'or pass a list of patterns via stdin.', + color='red', + ) + stderr() + hint(('To remove all urls you can run:', + 'archivebox remove --filter-type=regex ".*"')) + stderr() + raise SystemExit(2) + elif filter_str: + filter_patterns = [ptn.strip() for ptn in filter_str.split('\n')] + + list_kwargs = { + "filter_patterns": filter_patterns, + "filter_type": filter_type, + "after": after, + "before": before, + } + if snapshots: + list_kwargs["snapshots"] = snapshots + + log_list_started(filter_patterns, filter_type) + timer = TimedProgress(360, prefix=' ') + try: + snapshots = list_links(**list_kwargs) + finally: + timer.end() + + + if not snapshots.exists(): + log_removal_finished(0, 0) + raise SystemExit(1) + + + log_links = [link.as_link() for link in snapshots] + log_list_finished(log_links) + log_removal_started(log_links, yes=yes, delete=delete) + + timer = TimedProgress(360, prefix=' ') + try: + for snapshot in snapshots: + if delete: + shutil.rmtree(snapshot.as_link().link_dir, ignore_errors=True) + finally: + timer.end() + + to_remove = snapshots.count() + + flush_search_index(snapshots=snapshots) + remove_from_sql_main_index(snapshots=snapshots, out_dir=out_dir) + all_snapshots = load_main_index(out_dir=out_dir) + log_removal_finished(all_snapshots.count(), to_remove) + + return all_snapshots + +@enforce_types +def update(resume: Optional[float]=None, + only_new: bool=ONLY_NEW, + index_only: bool=False, + overwrite: bool=False, + filter_patterns_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: Optional[str]=None, + status: Optional[str]=None, + after: Optional[str]=None, + before: Optional[str]=None, + extractors: str="", + out_dir: Path=OUTPUT_DIR) -> List[Link]: + """Import any new links from subscriptions and retry any previously failed/skipped links""" + + check_data_folder(out_dir=out_dir) + check_dependencies() + new_links: List[Link] = [] # TODO: Remove input argument: only_new + + extractors = extractors.split(",") if extractors else [] + + # Step 1: Filter for selected_links + matching_snapshots = list_links( + filter_patterns=filter_patterns, + filter_type=filter_type, + before=before, + after=after, + ) + + matching_folders = list_folders( + links=matching_snapshots, + status=status, + out_dir=out_dir, + ) + all_links = [link for link in matching_folders.values() if link] + + if index_only: + for link in all_links: + write_link_details(link, out_dir=out_dir, skip_sql_index=True) + index_links(all_links, out_dir=out_dir) + return all_links + + # Step 2: Run the archive methods for each link + to_archive = new_links if only_new else all_links + if resume: + to_archive = [ + link for link in to_archive + if link.timestamp >= str(resume) + ] + if not to_archive: + stderr('') + stderr(f'[√] Nothing found to resume after {resume}', color='green') + return all_links + + archive_kwargs = { + "out_dir": out_dir, + } + if extractors: + archive_kwargs["methods"] = extractors + + archive_links(to_archive, overwrite=overwrite, **archive_kwargs) + + # Step 4: Re-write links index with updated titles, icons, and resources + all_links = load_main_index(out_dir=out_dir) + return all_links + +@enforce_types +def list_all(filter_patterns_str: Optional[str]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + status: Optional[str]=None, + after: Optional[float]=None, + before: Optional[float]=None, + sort: Optional[str]=None, + csv: Optional[str]=None, + json: bool=False, + html: bool=False, + with_headers: bool=False, + out_dir: Path=OUTPUT_DIR) -> Iterable[Link]: + """List, filter, and export information about archive entries""" + + check_data_folder(out_dir=out_dir) + + if filter_patterns and filter_patterns_str: + stderr( + '[X] You should either pass filter patterns as an arguments ' + 'or via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif filter_patterns_str: + filter_patterns = filter_patterns_str.split('\n') + + snapshots = list_links( + filter_patterns=filter_patterns, + filter_type=filter_type, + before=before, + after=after, + ) + + if sort: + snapshots = snapshots.order_by(sort) + + folders = list_folders( + links=snapshots, + status=status, + out_dir=out_dir, + ) + + if json: + output = generate_json_index_from_links(folders.values(), with_headers) + elif html: + output = generate_index_from_links(folders.values(), with_headers) + elif csv: + output = links_to_csv(folders.values(), cols=csv.split(','), header=with_headers) + else: + output = printable_folders(folders, with_headers=with_headers) + print(output) + return folders + + +@enforce_types +def list_links(snapshots: Optional[QuerySet]=None, + filter_patterns: Optional[List[str]]=None, + filter_type: str='exact', + after: Optional[float]=None, + before: Optional[float]=None, + out_dir: Path=OUTPUT_DIR) -> Iterable[Link]: + + check_data_folder(out_dir=out_dir) + + if snapshots: + all_snapshots = snapshots + else: + all_snapshots = load_main_index(out_dir=out_dir) + + if after is not None: + all_snapshots = all_snapshots.filter(timestamp__lt=after) + if before is not None: + all_snapshots = all_snapshots.filter(timestamp__gt=before) + if filter_patterns: + all_snapshots = snapshot_filter(all_snapshots, filter_patterns, filter_type) + return all_snapshots + +@enforce_types +def list_folders(links: List[Link], + status: str, + out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]: + + check_data_folder(out_dir=out_dir) + + STATUS_FUNCTIONS = { + "indexed": get_indexed_folders, + "archived": get_archived_folders, + "unarchived": get_unarchived_folders, + "present": get_present_folders, + "valid": get_valid_folders, + "invalid": get_invalid_folders, + "duplicate": get_duplicate_folders, + "orphaned": get_orphaned_folders, + "corrupted": get_corrupted_folders, + "unrecognized": get_unrecognized_folders, + } + + try: + return STATUS_FUNCTIONS[status](links, out_dir=out_dir) + except KeyError: + raise ValueError('Status not recognized.') + + +@enforce_types +def config(config_options_str: Optional[str]=None, + config_options: Optional[List[str]]=None, + get: bool=False, + set: bool=False, + reset: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Get and set your ArchiveBox project configuration values""" + + check_data_folder(out_dir=out_dir) + + if config_options and config_options_str: + stderr( + '[X] You should either pass config values as an arguments ' + 'or via stdin, but not both.\n', + color='red', + ) + raise SystemExit(2) + elif config_options_str: + config_options = config_options_str.split('\n') + + config_options = config_options or [] + + no_args = not (get or set or reset or config_options) + + matching_config: ConfigDict = {} + if get or no_args: + if config_options: + config_options = [get_real_name(key) for key in config_options] + matching_config = {key: CONFIG[key] for key in config_options if key in CONFIG} + failed_config = [key for key in config_options if key not in CONFIG] + if failed_config: + stderr() + stderr('[X] These options failed to get', color='red') + stderr(' {}'.format('\n '.join(config_options))) + raise SystemExit(1) + else: + matching_config = CONFIG + + print(printable_config(matching_config)) + raise SystemExit(not matching_config) + elif set: + new_config = {} + failed_options = [] + for line in config_options: + if line.startswith('#') or not line.strip(): + continue + if '=' not in line: + stderr('[X] Config KEY=VALUE must have an = sign in it', color='red') + stderr(f' {line}') + raise SystemExit(2) + + raw_key, val = line.split('=', 1) + raw_key = raw_key.upper().strip() + key = get_real_name(raw_key) + if key != raw_key: + stderr(f'[i] Note: The config option {raw_key} has been renamed to {key}, please use the new name going forwards.', color='lightyellow') + + if key in CONFIG: + new_config[key] = val.strip() + else: + failed_options.append(line) + + if new_config: + before = CONFIG + matching_config = write_config_file(new_config, out_dir=OUTPUT_DIR) + after = load_all_config() + print(printable_config(matching_config)) + + side_effect_changes: ConfigDict = {} + for key, val in after.items(): + if key in USER_CONFIG and (before[key] != after[key]) and (key not in matching_config): + side_effect_changes[key] = after[key] + + if side_effect_changes: + stderr() + stderr('[i] Note: This change also affected these other options that depended on it:', color='lightyellow') + print(' {}'.format(printable_config(side_effect_changes, prefix=' '))) + if failed_options: + stderr() + stderr('[X] These options failed to set (check for typos):', color='red') + stderr(' {}'.format('\n '.join(failed_options))) + raise SystemExit(bool(failed_options)) + elif reset: + stderr('[X] This command is not implemented yet.', color='red') + stderr(' Please manually remove the relevant lines from your config file:') + stderr(f' {CONFIG_FILE}') + raise SystemExit(2) + else: + stderr('[X] You must pass either --get or --set, or no arguments to get the whole config.', color='red') + stderr(' archivebox config') + stderr(' archivebox config --get SOME_KEY') + stderr(' archivebox config --set SOME_KEY=SOME_VALUE') + raise SystemExit(2) + + +@enforce_types +def schedule(add: bool=False, + show: bool=False, + clear: bool=False, + foreground: bool=False, + run_all: bool=False, + quiet: bool=False, + every: Optional[str]=None, + depth: int=0, + import_path: Optional[str]=None, + out_dir: Path=OUTPUT_DIR): + """Set ArchiveBox to regularly import URLs at specific times using cron""" + + check_data_folder(out_dir=out_dir) + + (Path(out_dir) / LOGS_DIR_NAME).mkdir(exist_ok=True) + + cron = CronTab(user=True) + cron = dedupe_cron_jobs(cron) + + if clear: + print(cron.remove_all(comment=CRON_COMMENT)) + cron.write() + raise SystemExit(0) + + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + if every or add: + every = every or 'day' + quoted = lambda s: f'"{s}"' if s and ' ' in str(s) else str(s) + cmd = [ + 'cd', + quoted(out_dir), + '&&', + quoted(ARCHIVEBOX_BINARY), + *(['add', f'--depth={depth}', f'"{import_path}"'] if import_path else ['update']), + '>', + quoted(Path(LOGS_DIR) / 'archivebox.log'), + '2>&1', + + ] + new_job = cron.new(command=' '.join(cmd), comment=CRON_COMMENT) + + if every in ('minute', 'hour', 'day', 'month', 'year'): + set_every = getattr(new_job.every(), every) + set_every() + elif CronSlices.is_valid(every): + new_job.setall(every) + else: + stderr('{red}[X] Got invalid timeperiod for cron task.{reset}'.format(**ANSI)) + stderr(' It must be one of minute/hour/day/month') + stderr(' or a quoted cron-format schedule like:') + stderr(' archivebox init --every=day https://example.com/some/rss/feed.xml') + stderr(' archivebox init --every="0/5 * * * *" https://example.com/some/rss/feed.xml') + raise SystemExit(1) + + cron = dedupe_cron_jobs(cron) + cron.write() + + total_runs = sum(j.frequency_per_year() for j in cron) + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + print() + print('{green}[√] Scheduled new ArchiveBox cron job for user: {} ({} jobs are active).{reset}'.format(USER, len(existing_jobs), **ANSI)) + print('\n'.join(f' > {cmd}' if str(cmd) == str(new_job) else f' {cmd}' for cmd in existing_jobs)) + if total_runs > 60 and not quiet: + stderr() + stderr('{lightyellow}[!] With the current cron config, ArchiveBox is estimated to run >{} times per year.{reset}'.format(total_runs, **ANSI)) + stderr(' Congrats on being an enthusiastic internet archiver! 👌') + stderr() + stderr(' Make sure you have enough storage space available to hold all the data.') + stderr(' Using a compressed/deduped filesystem like ZFS is recommended if you plan on archiving a lot.') + stderr('') + elif show: + if existing_jobs: + print('\n'.join(str(cmd) for cmd in existing_jobs)) + else: + stderr('{red}[X] There are no ArchiveBox cron jobs scheduled for your user ({}).{reset}'.format(USER, **ANSI)) + stderr(' To schedule a new job, run:') + stderr(' archivebox schedule --every=[timeperiod] https://example.com/some/rss/feed.xml') + raise SystemExit(0) + + cron = CronTab(user=True) + cron = dedupe_cron_jobs(cron) + existing_jobs = list(cron.find_comment(CRON_COMMENT)) + + if foreground or run_all: + if not existing_jobs: + stderr('{red}[X] You must schedule some jobs first before running in foreground mode.{reset}'.format(**ANSI)) + stderr(' archivebox schedule --every=hour https://example.com/some/rss/feed.xml') + raise SystemExit(1) + + print('{green}[*] Running {} ArchiveBox jobs in foreground task scheduler...{reset}'.format(len(existing_jobs), **ANSI)) + if run_all: + try: + for job in existing_jobs: + sys.stdout.write(f' > {job.command.split("/archivebox ")[0].split(" && ")[0]}\n') + sys.stdout.write(f' > {job.command.split("/archivebox ")[-1].split(" > ")[0]}') + sys.stdout.flush() + job.run() + sys.stdout.write(f'\r √ {job.command.split("/archivebox ")[-1]}\n') + except KeyboardInterrupt: + print('\n{green}[√] Stopped.{reset}'.format(**ANSI)) + raise SystemExit(1) + + if foreground: + try: + for job in existing_jobs: + print(f' > {job.command.split("/archivebox ")[-1].split(" > ")[0]}') + for result in cron.run_scheduler(): + print(result) + except KeyboardInterrupt: + print('\n{green}[√] Stopped.{reset}'.format(**ANSI)) + raise SystemExit(1) + + +@enforce_types +def server(runserver_args: Optional[List[str]]=None, + reload: bool=False, + debug: bool=False, + init: bool=False, + out_dir: Path=OUTPUT_DIR) -> None: + """Run the ArchiveBox HTTP server""" + + runserver_args = runserver_args or [] + + if init: + run_subcommand('init', stdin=None, pwd=out_dir) + + # setup config for django runserver + from . import config + config.SHOW_PROGRESS = False + config.DEBUG = config.DEBUG or debug + + check_data_folder(out_dir=out_dir) + + from django.core.management import call_command + from django.contrib.auth.models import User + + admin_user = User.objects.filter(is_superuser=True).order_by('date_joined').only('username').last() + + print('{green}[+] Starting ArchiveBox webserver...{reset}'.format(**ANSI)) + if admin_user: + hint('The admin username is{lightblue} {}{reset}\n'.format(admin_user.username, **ANSI)) + else: + print('{lightyellow}[!] No admin users exist yet, you will not be able to edit links in the UI.{reset}'.format(**ANSI)) + print() + print(' To create an admin user, run:') + print(' archivebox manage createsuperuser') + print() + + # fallback to serving staticfiles insecurely with django when DEBUG=False + if not config.DEBUG: + runserver_args.append('--insecure') # TODO: serve statics w/ nginx instead + + # toggle autoreloading when archivebox code changes (it's on by default) + if not reload: + runserver_args.append('--noreload') + + config.SHOW_PROGRESS = False + config.DEBUG = config.DEBUG or debug + + + call_command("runserver", *runserver_args) + + +@enforce_types +def manage(args: Optional[List[str]]=None, out_dir: Path=OUTPUT_DIR) -> None: + """Run an ArchiveBox Django management command""" + + check_data_folder(out_dir=out_dir) + from django.core.management import execute_from_command_line + + if (args and "createsuperuser" in args) and (IN_DOCKER and not IS_TTY): + stderr('[!] Warning: you need to pass -it to use interactive commands in docker', color='lightyellow') + stderr(' docker run -it archivebox manage {}'.format(' '.join(args or ['...'])), color='lightyellow') + stderr() + + execute_from_command_line([f'{ARCHIVEBOX_BINARY} manage', *(args or ['help'])]) + + +@enforce_types +def shell(out_dir: Path=OUTPUT_DIR) -> None: + """Enter an interactive ArchiveBox Django shell""" + + check_data_folder(out_dir=out_dir) + + from django.core.management import call_command + call_command("shell_plus") + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/manage.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/manage.py new file mode 100644 index 0000000..1a9b297 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/manage.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python +import os +import sys + +if __name__ == '__main__': + # if you're a developer working on archivebox, still prefer the archivebox + # versions of ./manage.py commands whenever possible. When that's not possible + # (e.g. makemigrations), you can comment out this check temporarily + + if not ('makemigrations' in sys.argv or 'migrate' in sys.argv): + print("[X] Don't run ./manage.py directly (unless you are a developer running makemigrations):") + print() + print(' Hint: Use these archivebox CLI commands instead of the ./manage.py equivalents:') + print(' archivebox init (migrates the databse to latest version)') + print(' archivebox server (runs the Django web server)') + print(' archivebox shell (opens an iPython Django shell with all models imported)') + print(' archivebox manage [cmd] (any other management commands)') + raise SystemExit(2) + + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings') + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/mypy.ini b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/mypy.ini new file mode 100644 index 0000000..b1b4489 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/mypy.ini @@ -0,0 +1,3 @@ +[mypy] +plugins = + mypy_django_plugin.main diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/package.json b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/package.json new file mode 100644 index 0000000..7f8bf66 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/package.json @@ -0,0 +1,21 @@ +{ + "name": "archivebox", + "version": "0.5.3", + "description": "ArchiveBox: The self-hosted internet archive", + "author": "Nick Sweeting <archivebox-npm@sweeting.me>", + "license": "MIT", + "scripts": { + "archivebox": "./bin/archive" + }, + "bin": { + "archivebox-node": "./bin/archive", + "single-file": "./node_modules/.bin/single-file", + "readability-extractor": "./node_modules/.bin/readability-extractor", + "mercury-parser": "./node_modules/.bin/mercury-parser" + }, + "dependencies": { + "@postlight/mercury-parser": "^2.2.0", + "readability-extractor": "git+https://github.com/pirate/readability-extractor.git", + "single-file": "git+https://github.com/gildas-lormeau/SingleFile.git" + } +} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/__init__.py new file mode 100644 index 0000000..441c08a --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/__init__.py @@ -0,0 +1,203 @@ +""" +Everything related to parsing links from input sources. + +For a list of supported services, see the README.md. +For examples of supported import formats see tests/. +""" + +__package__ = 'archivebox.parsers' + +import re +from io import StringIO + +from typing import IO, Tuple, List, Optional +from datetime import datetime +from pathlib import Path + +from ..system import atomic_write +from ..config import ( + ANSI, + OUTPUT_DIR, + SOURCES_DIR_NAME, + TIMEOUT, +) +from ..util import ( + basename, + htmldecode, + download_url, + enforce_types, + URL_REGEX, +) +from ..index.schema import Link +from ..logging_util import TimedProgress, log_source_saved + +from .pocket_html import parse_pocket_html_export +from .pocket_api import parse_pocket_api_export +from .pinboard_rss import parse_pinboard_rss_export +from .wallabag_atom import parse_wallabag_atom_export +from .shaarli_rss import parse_shaarli_rss_export +from .medium_rss import parse_medium_rss_export +from .netscape_html import parse_netscape_html_export +from .generic_rss import parse_generic_rss_export +from .generic_json import parse_generic_json_export +from .generic_html import parse_generic_html_export +from .generic_txt import parse_generic_txt_export + +PARSERS = ( + # Specialized parsers + ('Pocket API', parse_pocket_api_export), + ('Wallabag ATOM', parse_wallabag_atom_export), + ('Pocket HTML', parse_pocket_html_export), + ('Pinboard RSS', parse_pinboard_rss_export), + ('Shaarli RSS', parse_shaarli_rss_export), + ('Medium RSS', parse_medium_rss_export), + + # General parsers + ('Netscape HTML', parse_netscape_html_export), + ('Generic RSS', parse_generic_rss_export), + ('Generic JSON', parse_generic_json_export), + ('Generic HTML', parse_generic_html_export), + + # Fallback parser + ('Plain Text', parse_generic_txt_export), +) + + +@enforce_types +def parse_links_memory(urls: List[str], root_url: Optional[str]=None): + """ + parse a list of URLS without touching the filesystem + """ + check_url_parsing_invariants() + + timer = TimedProgress(TIMEOUT * 4) + #urls = list(map(lambda x: x + "\n", urls)) + file = StringIO() + file.writelines(urls) + file.name = "io_string" + links, parser = run_parser_functions(file, timer, root_url=root_url) + timer.end() + + if parser is None: + return [], 'Failed to parse' + return links, parser + + +@enforce_types +def parse_links(source_file: str, root_url: Optional[str]=None) -> Tuple[List[Link], str]: + """parse a list of URLs with their metadata from an + RSS feed, bookmarks export, or text file + """ + + check_url_parsing_invariants() + + timer = TimedProgress(TIMEOUT * 4) + with open(source_file, 'r', encoding='utf-8') as file: + links, parser = run_parser_functions(file, timer, root_url=root_url) + + timer.end() + if parser is None: + return [], 'Failed to parse' + return links, parser + + +def run_parser_functions(to_parse: IO[str], timer, root_url: Optional[str]=None) -> Tuple[List[Link], Optional[str]]: + most_links: List[Link] = [] + best_parser_name = None + + for parser_name, parser_func in PARSERS: + try: + parsed_links = list(parser_func(to_parse, root_url=root_url)) + if not parsed_links: + raise Exception('no links found') + + # print(f'[√] Parser {parser_name} succeeded: {len(parsed_links)} links parsed') + if len(parsed_links) > len(most_links): + most_links = parsed_links + best_parser_name = parser_name + + except Exception as err: # noqa + # Parsers are tried one by one down the list, and the first one + # that succeeds is used. To see why a certain parser was not used + # due to error or format incompatibility, uncomment this line: + + # print('[!] Parser {} failed: {} {}'.format(parser_name, err.__class__.__name__, err)) + # raise + pass + timer.end() + return most_links, best_parser_name + + +@enforce_types +def save_text_as_source(raw_text: str, filename: str='{ts}-stdin.txt', out_dir: Path=OUTPUT_DIR) -> str: + ts = str(datetime.now().timestamp()).split('.', 1)[0] + source_path = str(out_dir / SOURCES_DIR_NAME / filename.format(ts=ts)) + atomic_write(source_path, raw_text) + log_source_saved(source_file=source_path) + return source_path + + +@enforce_types +def save_file_as_source(path: str, timeout: int=TIMEOUT, filename: str='{ts}-{basename}.txt', out_dir: Path=OUTPUT_DIR) -> str: + """download a given url's content into output/sources/domain-<timestamp>.txt""" + ts = str(datetime.now().timestamp()).split('.', 1)[0] + source_path = str(OUTPUT_DIR / SOURCES_DIR_NAME / filename.format(basename=basename(path), ts=ts)) + + if any(path.startswith(s) for s in ('http://', 'https://', 'ftp://')): + # Source is a URL that needs to be downloaded + print(f' > Downloading {path} contents') + timer = TimedProgress(timeout, prefix=' ') + try: + raw_source_text = download_url(path, timeout=timeout) + raw_source_text = htmldecode(raw_source_text) + timer.end() + except Exception as e: + timer.end() + print('{}[!] Failed to download {}{}\n'.format( + ANSI['red'], + path, + ANSI['reset'], + )) + print(' ', e) + raise SystemExit(1) + + else: + # Source is a path to a local file on the filesystem + with open(path, 'r') as f: + raw_source_text = f.read() + + atomic_write(source_path, raw_source_text) + + log_source_saved(source_file=source_path) + + return source_path + + +def check_url_parsing_invariants() -> None: + """Check that plain text regex URL parsing works as expected""" + + # this is last-line-of-defense to make sure the URL_REGEX isn't + # misbehaving, as the consequences could be disastrous and lead to many + # incorrect/badly parsed links being added to the archive + + test_urls = ''' + https://example1.com/what/is/happening.html?what=1#how-about-this=1 + https://example2.com/what/is/happening/?what=1#how-about-this=1 + HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f + https://example4.com/what/is/happening.html + https://example5.com/ + https://example6.com + + <test>http://example7.com</test> + [https://example8.com/what/is/this.php?what=1] + [and http://example9.com?what=1&other=3#and-thing=2] + <what>https://example10.com#and-thing=2 "</about> + abc<this["https://example11.com/what/is#and-thing=2?whoami=23&where=1"]that>def + sdflkf[what](https://example12.com/who/what.php?whoami=1#whatami=2)?am=hi + example13.bada + and example14.badb + <or>htt://example15.badc</that> + ''' + # print('\n'.join(re.findall(URL_REGEX, test_urls))) + assert len(re.findall(URL_REGEX, test_urls)) == 12 + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_html.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_html.py new file mode 100644 index 0000000..74b3d1f --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_html.py @@ -0,0 +1,53 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable, Optional +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + URL_REGEX, +) +from html.parser import HTMLParser +from urllib.parse import urljoin + + +class HrefParser(HTMLParser): + def __init__(self): + super().__init__() + self.urls = [] + + def handle_starttag(self, tag, attrs): + if tag == "a": + for attr, value in attrs: + if attr == "href": + self.urls.append(value) + + +@enforce_types +def parse_generic_html_export(html_file: IO[str], root_url: Optional[str]=None, **_kwargs) -> Iterable[Link]: + """Parse Generic HTML for href tags and use only the url (support for title coming later)""" + + html_file.seek(0) + for line in html_file: + parser = HrefParser() + # example line + # <li><a href="http://example.com/ time_added="1478739709" tags="tag1,tag2">example title</a></li> + parser.feed(line) + for url in parser.urls: + if root_url: + # resolve relative urls /home.html -> https://example.com/home.html + url = urljoin(root_url, url) + + for archivable_url in re.findall(URL_REGEX, url): + yield Link( + url=htmldecode(archivable_url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[html_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_json.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_json.py new file mode 100644 index 0000000..e6ed677 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_json.py @@ -0,0 +1,65 @@ +__package__ = 'archivebox.parsers' + +import json + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_generic_json_export(json_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)""" + + json_file.seek(0) + links = json.load(json_file) + json_date = lambda s: datetime.strptime(s, '%Y-%m-%dT%H:%M:%S%z') + + for link in links: + # example line + # {"href":"http:\/\/www.reddit.com\/r\/example","description":"title here","extended":"","meta":"18a973f09c9cc0608c116967b64e0419","hash":"910293f019c2f4bb1a749fb937ba58e3","time":"2014-06-14T15:51:42Z","shared":"no","toread":"no","tags":"reddit android"}] + if link: + # Parse URL + url = link.get('href') or link.get('url') or link.get('URL') + if not url: + raise Exception('JSON must contain URL in each entry [{"url": "http://...", ...}, ...]') + + # Parse the timestamp + ts_str = str(datetime.now().timestamp()) + if link.get('timestamp'): + # chrome/ff histories use a very precise timestamp + ts_str = str(link['timestamp'] / 10000000) + elif link.get('time'): + ts_str = str(json_date(link['time'].split(',', 1)[0]).timestamp()) + elif link.get('created_at'): + ts_str = str(json_date(link['created_at']).timestamp()) + elif link.get('created'): + ts_str = str(json_date(link['created']).timestamp()) + elif link.get('date'): + ts_str = str(json_date(link['date']).timestamp()) + elif link.get('bookmarked'): + ts_str = str(json_date(link['bookmarked']).timestamp()) + elif link.get('saved'): + ts_str = str(json_date(link['saved']).timestamp()) + + # Parse the title + title = None + if link.get('title'): + title = link['title'].strip() + elif link.get('description'): + title = link['description'].replace(' — Readability', '').strip() + elif link.get('name'): + title = link['name'].strip() + + yield Link( + url=htmldecode(url), + timestamp=ts_str, + title=htmldecode(title) or None, + tags=htmldecode(link.get('tags')) or '', + sources=[json_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_rss.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_rss.py new file mode 100644 index 0000000..2831844 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/generic_rss.py @@ -0,0 +1,49 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + +@enforce_types +def parse_generic_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse RSS XML-format files into links""" + + rss_file.seek(0) + items = rss_file.read().split('<item>') + items = items[1:] if items else [] + for item in items: + # example item: + # <item> + # <title><![CDATA[How JavaScript works: inside the V8 engine]]> + # Unread + # https://blog.sessionstack.com/how-javascript-works-inside + # https://blog.sessionstack.com/how-javascript-works-inside + # Mon, 21 Aug 2017 14:21:58 -0500 + # + + trailing_removed = item.split('', 1)[0] + leading_removed = trailing_removed.split('', 1)[-1].strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r for r in rows if r.strip().startswith('<{}>'.format(key))][0] + + url = str_between(get_row('link'), '', '') + ts_str = str_between(get_row('pubDate'), '', '') + time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %z") + title = str_between(get_row('title'), ' Iterable[Link]: + """Parse raw links from each line in a text file""" + + text_file.seek(0) + for line in text_file.readlines(): + if not line.strip(): + continue + + # if the line is a local file path that resolves, then we can archive it + try: + if Path(line).exists(): + yield Link( + url=line, + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) + except (OSError, PermissionError): + # nvm, not a valid path... + pass + + # otherwise look for anything that looks like a URL in the line + for url in re.findall(URL_REGEX, line): + yield Link( + url=htmldecode(url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) + + # look inside the URL for any sub-urls, e.g. for archive.org links + # https://web.archive.org/web/20200531203453/https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/ + # -> https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/ + for url in re.findall(URL_REGEX, line[1:]): + yield Link( + url=htmldecode(url), + timestamp=str(datetime.now().timestamp()), + title=None, + tags=None, + sources=[text_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/medium_rss.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/medium_rss.py new file mode 100644 index 0000000..8f14f77 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/medium_rss.py @@ -0,0 +1,35 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from xml.etree import ElementTree + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_medium_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Medium RSS feed files into links""" + + rss_file.seek(0) + root = ElementTree.parse(rss_file).getroot() + items = root.find("channel").findall("item") # type: ignore + for item in items: + url = item.find("link").text # type: ignore + title = item.find("title").text.strip() # type: ignore + ts_str = item.find("pubDate").text # type: ignore + time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %Z") # type: ignore + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/netscape_html.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/netscape_html.py new file mode 100644 index 0000000..a063023 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/netscape_html.py @@ -0,0 +1,39 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_netscape_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse netscape-format bookmarks export files (produced by all browsers)""" + + html_file.seek(0) + pattern = re.compile("]*>(.+)", re.UNICODE | re.IGNORECASE) + for line in html_file: + # example line + #
    example bookmark title + + match = pattern.search(line) + if match: + url = match.group(1) + time = datetime.fromtimestamp(float(match.group(2))) + title = match.group(3).strip() + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[html_file.name], + ) + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pinboard_rss.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pinboard_rss.py new file mode 100644 index 0000000..98ff14a --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pinboard_rss.py @@ -0,0 +1,47 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from xml.etree import ElementTree + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_pinboard_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Pinboard RSS feed files into links""" + + rss_file.seek(0) + root = ElementTree.parse(rss_file).getroot() + items = root.findall("{http://purl.org/rss/1.0/}item") + for item in items: + find = lambda p: item.find(p).text.strip() if item.find(p) else None # type: ignore + + url = find("{http://purl.org/rss/1.0/}link") + tags = find("{http://purl.org/dc/elements/1.1/}subject") + title = find("{http://purl.org/rss/1.0/}title") + ts_str = find("{http://purl.org/dc/elements/1.1/}date") + + # Pinboard includes a colon in its date stamp timezone offsets, which + # Python can't parse. Remove it: + if ts_str and ts_str[-3:-2] == ":": + ts_str = ts_str[:-3]+ts_str[-2:] + + if ts_str: + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + else: + time = datetime.now() + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=htmldecode(tags) or None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pocket_api.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pocket_api.py new file mode 100644 index 0000000..bf3a292 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pocket_api.py @@ -0,0 +1,113 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable, Optional +from configparser import ConfigParser + +from pathlib import Path +from ..vendor.pocket import Pocket + +from ..index.schema import Link +from ..util import enforce_types +from ..system import atomic_write +from ..config import ( + SOURCES_DIR, + POCKET_CONSUMER_KEY, + POCKET_ACCESS_TOKENS, +) + + +COUNT_PER_PAGE = 500 +API_DB_PATH = Path(SOURCES_DIR) / 'pocket_api.db' + +# search for broken protocols that sometimes come from the Pocket API +_BROKEN_PROTOCOL_RE = re.compile('^(http[s]?)(:/(?!/))') + + +def get_pocket_articles(api: Pocket, since=None, page=0): + body, headers = api.get( + state='archive', + sort='oldest', + since=since, + count=COUNT_PER_PAGE, + offset=page * COUNT_PER_PAGE, + ) + + articles = body['list'].values() if isinstance(body['list'], dict) else body['list'] + returned_count = len(articles) + + yield from articles + + if returned_count == COUNT_PER_PAGE: + yield from get_pocket_articles(api, since=since, page=page + 1) + else: + api.last_since = body['since'] + + +def link_from_article(article: dict, sources: list): + url: str = article['resolved_url'] or article['given_url'] + broken_protocol = _BROKEN_PROTOCOL_RE.match(url) + if broken_protocol: + url = url.replace(f'{broken_protocol.group(1)}:/', f'{broken_protocol.group(1)}://') + title = article['resolved_title'] or article['given_title'] or url + + return Link( + url=url, + timestamp=article['time_read'], + title=title, + tags=article.get('tags'), + sources=sources + ) + + +def write_since(username: str, since: str): + if not API_DB_PATH.exists(): + atomic_write(API_DB_PATH, '') + + since_file = ConfigParser() + since_file.optionxform = str + since_file.read(API_DB_PATH) + + since_file[username] = { + 'since': since + } + + with open(API_DB_PATH, 'w+') as new: + since_file.write(new) + + +def read_since(username: str) -> Optional[str]: + if not API_DB_PATH.exists(): + atomic_write(API_DB_PATH, '') + + config_file = ConfigParser() + config_file.optionxform = str + config_file.read(API_DB_PATH) + + return config_file.get(username, 'since', fallback=None) + + +@enforce_types +def should_parse_as_pocket_api(text: str) -> bool: + return text.startswith('pocket://') + + +@enforce_types +def parse_pocket_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Link]: + """Parse bookmarks from the Pocket API""" + + input_buffer.seek(0) + pattern = re.compile(r"^pocket:\/\/(\w+)") + for line in input_buffer: + if should_parse_as_pocket_api(line): + + username = pattern.search(line).group(1) + api = Pocket(POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKENS[username]) + api.last_since = None + + for article in get_pocket_articles(api, since=read_since(username)): + yield link_from_article(article, sources=[line]) + + write_since(username, api.last_since) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pocket_html.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pocket_html.py new file mode 100644 index 0000000..653f21b --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/pocket_html.py @@ -0,0 +1,38 @@ +__package__ = 'archivebox.parsers' + + +import re + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, +) + + +@enforce_types +def parse_pocket_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)""" + + html_file.seek(0) + pattern = re.compile("^\\s*
  • (.+)
    • ", re.UNICODE) + for line in html_file: + # example line + #
  • example title
    • + match = pattern.search(line) + if match: + url = match.group(1).replace('http://www.readability.com/read?url=', '') # remove old readability prefixes to get original url + time = datetime.fromtimestamp(float(match.group(2))) + tags = match.group(3) + title = match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=tags or '', + sources=[html_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/shaarli_rss.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/shaarli_rss.py new file mode 100644 index 0000000..4a925f4 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/shaarli_rss.py @@ -0,0 +1,50 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + + +@enforce_types +def parse_shaarli_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Shaarli-specific RSS XML-format files into links""" + + rss_file.seek(0) + entries = rss_file.read().split('')[1:] + for entry in entries: + # example entry: + # + # Aktuelle Trojaner-Welle: Emotet lauert in gefälschten Rechnungsmails | heise online + # + # https://demo.shaarli.org/?cEV4vw + # 2019-01-30T06:06:01+00:00 + # 2019-01-30T06:06:01+00:00 + #

    — Permalink

    ]]>     + #     + + trailing_removed = entry.split('    ', 1)[0] + leading_removed = trailing_removed.strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0] + + title = str_between(get_row('title'), '', '').strip() + url = str_between(get_row('link'), '') + ts_str = str_between(get_row('published'), '', '') + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=None, + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/wallabag_atom.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/wallabag_atom.py new file mode 100644 index 0000000..0d77869 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/parsers/wallabag_atom.py @@ -0,0 +1,57 @@ +__package__ = 'archivebox.parsers' + + +from typing import IO, Iterable +from datetime import datetime + +from ..index.schema import Link +from ..util import ( + htmldecode, + enforce_types, + str_between, +) + + +@enforce_types +def parse_wallabag_atom_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]: + """Parse Wallabag Atom files into links""" + + rss_file.seek(0) + entries = rss_file.read().split('')[1:] + for entry in entries: + # example entry: + # + # <![CDATA[Orient Ray vs Mako: Is There Much Difference? - iknowwatches.com]]> + # + # https://iknowwatches.com/orient-ray-vs-mako/ + # wallabag:wallabag.drycat.fr:milosh:entry:14041 + # 2020-10-18T09:14:02+02:00 + # 2020-10-18T09:13:56+02:00 + # + # + # + + trailing_removed = entry.split('', 1)[0] + leading_removed = trailing_removed.strip() + rows = leading_removed.split('\n') + + def get_row(key): + return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0] + + title = str_between(get_row('title'), '<![CDATA[', ']]>').strip() + url = str_between(get_row('link rel="via"'), '', '') + ts_str = str_between(get_row('published'), '', '') + time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z") + try: + tags = str_between(get_row('category'), 'label="', '" />') + except: + tags = None + + yield Link( + url=htmldecode(url), + timestamp=str(time.timestamp()), + title=htmldecode(title) or None, + tags=tags or '', + sources=[rss_file.name], + ) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/__init__.py new file mode 100644 index 0000000..6191ede --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/__init__.py @@ -0,0 +1,108 @@ +from typing import List, Union +from pathlib import Path +from importlib import import_module + +from django.db.models import QuerySet + +from archivebox.index.schema import Link +from archivebox.util import enforce_types +from archivebox.config import stderr, OUTPUT_DIR, USE_INDEXING_BACKEND, USE_SEARCHING_BACKEND, SEARCH_BACKEND_ENGINE + +from .utils import get_indexable_content, log_index_started + +def indexing_enabled(): + return USE_INDEXING_BACKEND + +def search_backend_enabled(): + return USE_SEARCHING_BACKEND + +def get_backend(): + return f'search.backends.{SEARCH_BACKEND_ENGINE}' + +def import_backend(): + backend_string = get_backend() + try: + backend = import_module(backend_string) + except Exception as err: + raise Exception("Could not load '%s' as a backend: %s" % (backend_string, err)) + return backend + +@enforce_types +def write_search_index(link: Link, texts: Union[List[str], None]=None, out_dir: Path=OUTPUT_DIR, skip_text_index: bool=False) -> None: + if not indexing_enabled(): + return + + if not skip_text_index and texts: + from core.models import Snapshot + + snap = Snapshot.objects.filter(url=link.url).first() + backend = import_backend() + if snap: + try: + backend.index(snapshot_id=str(snap.id), texts=texts) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + +@enforce_types +def query_search_index(query: str, out_dir: Path=OUTPUT_DIR) -> QuerySet: + from core.models import Snapshot + + if search_backend_enabled(): + backend = import_backend() + try: + snapshot_ids = backend.search(query) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + raise + else: + # TODO preserve ordering from backend + qsearch = Snapshot.objects.filter(pk__in=snapshot_ids) + return qsearch + + return Snapshot.objects.none() + +@enforce_types +def flush_search_index(snapshots: QuerySet): + if not indexing_enabled() or not snapshots: + return + backend = import_backend() + snapshot_ids=(str(pk) for pk in snapshots.values_list('pk',flat=True)) + try: + backend.flush(snapshot_ids) + except Exception as err: + stderr() + stderr( + f'[X] The search backend threw an exception={err}:', + color='red', + ) + +@enforce_types +def index_links(links: Union[List[Link],None], out_dir: Path=OUTPUT_DIR): + if not links: + return + + from core.models import Snapshot, ArchiveResult + + for link in links: + snap = Snapshot.objects.filter(url=link.url).first() + if snap: + results = ArchiveResult.objects.indexable().filter(snapshot=snap) + log_index_started(link.url) + try: + texts = get_indexable_content(results) + except Exception as err: + stderr() + stderr( + f'[X] An Exception ocurred reading the indexable content={err}:', + color='red', + ) + else: + write_search_index(link, texts, out_dir=out_dir) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/ripgrep.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/ripgrep.py new file mode 100644 index 0000000..840d2d2 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/ripgrep.py @@ -0,0 +1,45 @@ +import re +from subprocess import run, PIPE +from typing import List, Generator + +from archivebox.config import ARCHIVE_DIR, RIPGREP_VERSION +from archivebox.util import enforce_types + +RG_IGNORE_EXTENSIONS = ('css','js','orig','svg') + +RG_ADD_TYPE = '--type-add' +RG_IGNORE_ARGUMENTS = f"ignore:*.{{{','.join(RG_IGNORE_EXTENSIONS)}}}" +RG_DEFAULT_ARGUMENTS = "-ilTignore" # Case insensitive(i), matching files results(l) +RG_REGEX_ARGUMENT = '-e' + +TIMESTAMP_REGEX = r'\/([\d]+\.[\d]+)\/' + +ts_regex = re.compile(TIMESTAMP_REGEX) + +@enforce_types +def index(snapshot_id: str, texts: List[str]): + return + +@enforce_types +def flush(snapshot_ids: Generator[str, None, None]): + return + +@enforce_types +def search(text: str) -> List[str]: + if not RIPGREP_VERSION: + raise Exception("ripgrep binary not found, install ripgrep to use this search backend") + + from core.models import Snapshot + + rg_cmd = ['rg', RG_ADD_TYPE, RG_IGNORE_ARGUMENTS, RG_DEFAULT_ARGUMENTS, RG_REGEX_ARGUMENT, text, str(ARCHIVE_DIR)] + rg = run(rg_cmd, stdout=PIPE, stderr=PIPE, timeout=60) + file_paths = [p.decode() for p in rg.stdout.splitlines()] + timestamps = set() + for path in file_paths: + ts = ts_regex.findall(path) + if ts: + timestamps.add(ts[0]) + + snap_ids = [str(id) for id in Snapshot.objects.filter(timestamp__in=timestamps).values_list('pk', flat=True)] + + return snap_ids diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/sonic.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/sonic.py new file mode 100644 index 0000000..f0beadd --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/backends/sonic.py @@ -0,0 +1,28 @@ +from typing import List, Generator + +from sonic import IngestClient, SearchClient + +from archivebox.util import enforce_types +from archivebox.config import SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD, SONIC_BUCKET, SONIC_COLLECTION + +MAX_SONIC_TEXT_LENGTH = 20000 + +@enforce_types +def index(snapshot_id: str, texts: List[str]): + with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl: + for text in texts: + chunks = [text[i:i+MAX_SONIC_TEXT_LENGTH] for i in range(0, len(text), MAX_SONIC_TEXT_LENGTH)] + for chunk in chunks: + ingestcl.push(SONIC_COLLECTION, SONIC_BUCKET, snapshot_id, str(chunk)) + +@enforce_types +def search(text: str) -> List[str]: + with SearchClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as querycl: + snap_ids = querycl.query(SONIC_COLLECTION, SONIC_BUCKET, text) + return snap_ids + +@enforce_types +def flush(snapshot_ids: Generator[str, None, None]): + with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl: + for id in snapshot_ids: + ingestcl.flush_object(SONIC_COLLECTION, SONIC_BUCKET, str(id)) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/utils.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/utils.py new file mode 100644 index 0000000..55c97e7 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/search/utils.py @@ -0,0 +1,44 @@ +from django.db.models import QuerySet + +from archivebox.util import enforce_types +from archivebox.config import ANSI + +def log_index_started(url): + print('{green}[*] Indexing url: {} in the search index {reset}'.format(url, **ANSI)) + print( ) + +def get_file_result_content(res, extra_path, use_pwd=False): + if use_pwd: + fpath = f'{res.pwd}/{res.output}' + else: + fpath = f'{res.output}' + + if extra_path: + fpath = f'{fpath}/{extra_path}' + + with open(fpath, 'r') as file: + data = file.read() + if data: + return [data] + return [] + + +# This should be abstracted by a plugin interface for extractors +@enforce_types +def get_indexable_content(results: QuerySet): + if not results: + return [] + # Only use the first method available + res, method = results.first(), results.first().extractor + if method not in ('readability', 'singlefile', 'dom', 'wget'): + return [] + # This should come from a plugin interface + + if method == 'readability': + return get_file_result_content(res, 'content.txt') + elif method == 'singlefile': + return get_file_result_content(res, '') + elif method == 'dom': + return get_file_result_content(res,'',use_pwd=True) + elif method == 'wget': + return get_file_result_content(res,'',use_pwd=True) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/system.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/system.py new file mode 100644 index 0000000..b27c5e4 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/system.py @@ -0,0 +1,163 @@ +__package__ = 'archivebox' + + +import os +import shutil + +from json import dump +from pathlib import Path +from typing import Optional, Union, Set, Tuple +from subprocess import run as subprocess_run + +from crontab import CronTab +from atomicwrites import atomic_write as lib_atomic_write + +from .util import enforce_types, ExtendedEncoder +from .config import OUTPUT_PERMISSIONS + + + +def run(*args, input=None, capture_output=True, text=False, **kwargs): + """Patched of subprocess.run to fix blocking io making timeout=innefective""" + + if input is not None: + if 'stdin' in kwargs: + raise ValueError('stdin and input arguments may not both be used.') + + if capture_output: + if ('stdout' in kwargs) or ('stderr' in kwargs): + raise ValueError('stdout and stderr arguments may not be used ' + 'with capture_output.') + + return subprocess_run(*args, input=input, capture_output=capture_output, text=text, **kwargs) + + +@enforce_types +def atomic_write(path: Union[Path, str], contents: Union[dict, str, bytes], overwrite: bool=True) -> None: + """Safe atomic write to filesystem by writing to temp file + atomic rename""" + + mode = 'wb+' if isinstance(contents, bytes) else 'w' + + # print('\n> Atomic Write:', mode, path, len(contents), f'overwrite={overwrite}') + try: + with lib_atomic_write(path, mode=mode, overwrite=overwrite) as f: + if isinstance(contents, dict): + dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder) + elif isinstance(contents, (bytes, str)): + f.write(contents) + except OSError as e: + print(f"[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC. ({e})") + print(" For data integrity, ArchiveBox requires a filesystem that supports atomic writes.") + print(" Filesystems and network drives that don't implement FSYNC are incompatible and require workarounds.") + raise SystemExit(1) + os.chmod(path, int(OUTPUT_PERMISSIONS, base=8)) + +@enforce_types +def chmod_file(path: str, cwd: str='.', permissions: str=OUTPUT_PERMISSIONS) -> None: + """chmod -R /""" + + root = Path(cwd) / path + if not root.exists(): + raise Exception('Failed to chmod: {} does not exist (did the previous step fail?)'.format(path)) + + if not root.is_dir(): + os.chmod(root, int(OUTPUT_PERMISSIONS, base=8)) + else: + for subpath in Path(path).glob('**/*'): + os.chmod(subpath, int(OUTPUT_PERMISSIONS, base=8)) + + +@enforce_types +def copy_and_overwrite(from_path: Union[str, Path], to_path: Union[str, Path]): + """copy a given file or directory to a given path, overwriting the destination""" + if Path(from_path).is_dir(): + shutil.rmtree(to_path, ignore_errors=True) + shutil.copytree(from_path, to_path) + else: + with open(from_path, 'rb') as src: + contents = src.read() + atomic_write(to_path, contents) + + +@enforce_types +def get_dir_size(path: Union[str, Path], recursive: bool=True, pattern: Optional[str]=None) -> Tuple[int, int, int]: + """get the total disk size of a given directory, optionally summing up + recursively and limiting to a given filter list + """ + num_bytes, num_dirs, num_files = 0, 0, 0 + for entry in os.scandir(path): + if (pattern is not None) and (pattern not in entry.path): + continue + if entry.is_dir(follow_symlinks=False): + if not recursive: + continue + num_dirs += 1 + bytes_inside, dirs_inside, files_inside = get_dir_size(entry.path) + num_bytes += bytes_inside + num_dirs += dirs_inside + num_files += files_inside + else: + num_bytes += entry.stat(follow_symlinks=False).st_size + num_files += 1 + return num_bytes, num_dirs, num_files + + +CRON_COMMENT = 'archivebox_schedule' + + +@enforce_types +def dedupe_cron_jobs(cron: CronTab) -> CronTab: + deduped: Set[Tuple[str, str]] = set() + + for job in list(cron): + unique_tuple = (str(job.slices), job.command) + if unique_tuple not in deduped: + deduped.add(unique_tuple) + cron.remove(job) + + for schedule, command in deduped: + job = cron.new(command=command, comment=CRON_COMMENT) + job.setall(schedule) + job.enable() + + return cron + + +class suppress_output(object): + ''' + A context manager for doing a "deep suppression" of stdout and stderr in + Python, i.e. will suppress all print, even if the print originates in a + compiled C/Fortran sub-function. + This will not suppress raised exceptions, since exceptions are printed + to stderr just before a script exits, and after the context manager has + exited (at least, I think that is why it lets exceptions through). + + with suppress_stdout_stderr(): + rogue_function() + ''' + def __init__(self, stdout=True, stderr=True): + # Open a pair of null files + # Save the actual stdout (1) and stderr (2) file descriptors. + self.stdout, self.stderr = stdout, stderr + if stdout: + self.null_stdout = os.open(os.devnull, os.O_RDWR) + self.real_stdout = os.dup(1) + if stderr: + self.null_stderr = os.open(os.devnull, os.O_RDWR) + self.real_stderr = os.dup(2) + + def __enter__(self): + # Assign the null pointers to stdout and stderr. + if self.stdout: + os.dup2(self.null_stdout, 1) + if self.stderr: + os.dup2(self.null_stderr, 2) + + def __exit__(self, *_): + # Re-assign the real stdout/stderr back to (1) and (2) + if self.stdout: + os.dup2(self.real_stdout, 1) + os.close(self.null_stdout) + if self.stderr: + os.dup2(self.real_stderr, 2) + os.close(self.null_stderr) diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/actions_as_select.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/actions_as_select.html new file mode 100644 index 0000000..86a7719 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/actions_as_select.html @@ -0,0 +1 @@ +actions_as_select diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/app_index.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/app_index.html new file mode 100644 index 0000000..6868b49 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/app_index.html @@ -0,0 +1,18 @@ +{% extends "admin/index.html" %} +{% load i18n %} + +{% block bodyclass %}{{ block.super }} app-{{ app_label }}{% endblock %} + +{% if not is_popup %} +{% block breadcrumbs %} +
    +{% trans 'Home' %} +› +{% for app in app_list %} +{{ app.name }} +{% endfor %} +
    +{% endblock %} +{% endif %} + +{% block sidebar %}{% endblock %} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/base.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/base.html new file mode 100644 index 0000000..d8ad8d0 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/base.html @@ -0,0 +1,246 @@ +{% load i18n static %} +{% get_current_language as LANGUAGE_CODE %}{% get_current_language_bidi as LANGUAGE_BIDI %} + + +{% block title %}{% endblock %} | ArchiveBox + +{% block extrastyle %}{% endblock %} +{% if LANGUAGE_BIDI %}{% endif %} +{% block extrahead %}{% endblock %} +{% block responsive %} + + + {% if LANGUAGE_BIDI %}{% endif %} +{% endblock %} +{% block blockbots %}{% endblock %} + + +{% load i18n %} + + + + + + + + + +
    + + {% if not is_popup %} + + + + {% block breadcrumbs %} +
    + {% trans 'Home' %} + {% if title %} › {{ title }}{% endif %} +
    + {% endblock %} + {% endif %} + + {% block messages %} + {% if messages %} +
      {% for message in messages %} + {{ message|capfirst }} + {% endfor %}
    + {% endif %} + {% endblock messages %} + + +
    + {% block pretitle %}{% endblock %} + {% block content_title %}{# {% if title %}

    {{ title }}

    {% endif %} #}{% endblock %} + {% block content %} + {% block object-tools %}{% endblock %} + {{ content }} + {% endblock %} + {% block sidebar %}{% endblock %} +
    +
    + + + {% block footer %}{% endblock %} +
    + + + + + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/grid_change_list.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/grid_change_list.html new file mode 100644 index 0000000..6894efd --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/grid_change_list.html @@ -0,0 +1,91 @@ +{% extends "admin/base_site.html" %} +{% load i18n admin_urls static admin_list %} +{% load core_tags %} + +{% block extrastyle %} + {{ block.super }} + + {% if cl.formset %} + + {% endif %} + {% if cl.formset or action_form %} + + {% endif %} + {{ media.css }} + {% if not actions_on_top and not actions_on_bottom %} + + {% endif %} +{% endblock %} + +{% block extrahead %} +{{ block.super }} +{{ media.js }} +{% endblock %} + +{% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-list{% endblock %} + +{% if not is_popup %} +{% block breadcrumbs %} +
    +{% translate 'Home' %} +› {{ cl.opts.app_config.verbose_name }} +› {{ cl.opts.verbose_name_plural|capfirst }} +
    +{% endblock %} +{% endif %} + +{% block coltype %}{% endblock %} + +{% block content %} +
    + {% block object-tools %} +
      + {% block object-tools-items %} + {% change_list_object_tools %} + {% endblock %} +
    + {% endblock %} + {% if cl.formset and cl.formset.errors %} +

    + {% if cl.formset.total_error_count == 1 %}{% translate "Please correct the error below." %}{% else %}{% translate "Please correct the errors below." %}{% endif %} +

    + {{ cl.formset.non_form_errors }} + {% endif %} +
    +
    + {% block search %}{% search_form cl %}{% endblock %} + {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %} + +
    {% csrf_token %} + {% if cl.formset %} +
    {{ cl.formset.management_form }}
    + {% endif %} + + {% block result_list %} + {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %} + {% comment %} + Table grid + {% result_list cl %} + {% endcomment %} + {% snapshots_grid cl %} + {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %} + {% endblock %} + {% block pagination %}{% pagination cl %}{% endblock %} +
    +
    + {% block filters %} + {% if cl.has_filters %} +
    +

    {% translate 'Filter' %}

    + {% if cl.has_active_filters %}

    + ✖ {% translate "Clear all filters" %} +

    {% endif %} + {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %} +
    + {% endif %} + {% endblock %} +
    +
    +{% endblock %} \ No newline at end of file diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/login.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/login.html new file mode 100644 index 0000000..98283f8 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/login.html @@ -0,0 +1,100 @@ +{% extends "admin/base_site.html" %} +{% load i18n static %} + +{% block extrastyle %}{{ block.super }} +{{ form.media }} +{% endblock %} + +{% block bodyclass %}{{ block.super }} login{% endblock %} + +{% block branding %}

    ArchiveBox Admin

    {% endblock %} + +{% block usertools %} +
    + Back to Main Index +{% endblock %} + +{% block nav-global %}{% endblock %} + +{% block content_title %} +
    + Log in to add, edit, and remove links from your archive. +


    +
    +{% endblock %} + +{% block breadcrumbs %}{% endblock %} + +{% block content %} +{% if form.errors and not form.non_field_errors %} +

    +{% if form.errors.items|length == 1 %}{% trans "Please correct the error below." %}{% else %}{% trans "Please correct the errors below." %}{% endif %} +

    +{% endif %} + +{% if form.non_field_errors %} +{% for error in form.non_field_errors %} +

    + {{ error }} +

    +{% endfor %} +{% endif %} + +
    + +{% if user.is_authenticated %} +

    +{% blocktrans trimmed %} + You are authenticated as {{ username }}, but are not authorized to + access this page. Would you like to login to a different account? +{% endblocktrans %} +

    +{% endif %} + +
    +
    {% csrf_token %} +
    + {{ form.username.errors }} + {{ form.username.label_tag }} {{ form.username }} +
    +
    + {{ form.password.errors }} + {{ form.password.label_tag }} {{ form.password }} + +
    + {% url 'admin_password_reset' as password_reset_url %} + {% if password_reset_url %} +
    + {% trans 'Forgotten your password or username?' %} +
    + {% endif %} +
    + +
    +
    + +
    +

    +
    +
    + If you forgot your password, reset it here or run:
    +
    +archivebox manage changepassword USERNAME
+
    + +

    +
    +
    + To create a new admin user, run the following: +
    +archivebox manage createsuperuser
+
    +
    +
    + + (cd into your archive folder before running commands) +
    + + +
    +{% endblock %} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/snapshots_grid.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/snapshots_grid.html new file mode 100644 index 0000000..a7a2d4f --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/admin/snapshots_grid.html @@ -0,0 +1,162 @@ +{% load i18n admin_urls static admin_list %} +{% load core_tags %} + +{% block extrastyle %} + + +{% endblock %} + +{% block content %} +
    + {% for obj in results %} +
    + + + + + +
    + {% if obj.tags_str %} +

    {{obj.tags_str}}

    + {% endif %} + {% if obj.title %} + +

    {{obj.title|truncatechars:55 }}

    +     + {% endif %} + {% comment %}

    TEXT If needed.

    {% endcomment %} +
    +
    +
    + + 🕑 {{obj.added}} + 📖{{obj.num_outputs}} + 🗄️{{ obj.archive_size | file_size }} +
    +
    +
    + {% endfor %} +
    + +{% endblock %} \ No newline at end of file diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/add_links.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/add_links.html new file mode 100644 index 0000000..0b384f5 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/add_links.html @@ -0,0 +1,71 @@ +{% extends "base.html" %} + +{% load static %} +{% load i18n %} + +{% block breadcrumbs %} +
    + {% trans 'Home' %} + {% if title %} › {{ title }}{% endif %} +
    +{% endblock %} + +{% block extra_head %} + +{% endblock %} + +{% block body %} +
    +

    + {% if stdout %} +

    Add new URLs to your archive: results

    + 
    +                {{ stdout | safe }}
+                

    
+
    +
    +
    +   Add more URLs ➕ +
    + {% else %} +
    {% csrf_token %} +

    Add new URLs to your archive

    +
    + {{ form.as_p }} +
    + +
    +
    +


    + + {% if absolute_add_path %} +
    +

    Bookmark this link to quickly add to your archive: + Add to ArchiveBox

    +
    + {% endif %} + + {% endif %} +
    +{% endblock %} + +{% block sidebar %}{% endblock %} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/base.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/base.html new file mode 100644 index 0000000..a70430e --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/base.html @@ -0,0 +1,284 @@ +{% load static %} + + + + + + Archived Sites + + + + + + {% block extra_head %} + {% endblock %} + + + + + + +
    +
    +
    +
    + + Logo + ArchiveBox: Index + +
    +
    + Add Links   |   + Admin   |   + Docs +
    +
    +
    +
    + {% block body %} + {% endblock %} +
    + + + + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/core/snapshot_list.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/core/snapshot_list.html new file mode 100644 index 0000000..ce2b2fa --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/core/snapshot_list.html @@ -0,0 +1,51 @@ +{% extends "base.html" %} +{% load static %} + +{% block body %} +
    +
    + + + +
    + + + + + + + + + + + {% for link in object_list %} + {% include 'main_index_row.html' with link=link %} + {% endfor %} + +
    BookmarkedSnapshot ({{object_list|length}})FilesOriginal URL
    +
    + + {% if page_obj.has_previous %} + « first + previous + {% endif %} + + + Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}. + + + {% if page_obj.has_next %} + next + last » + {% endif %} + + + {% if page_obj.has_next %} + next + last » + {% endif %} + +
    +
    +{% endblock %} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/link_details.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/link_details.html new file mode 100644 index 0000000..b1edcfe --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/link_details.html @@ -0,0 +1,488 @@ + + + + {{title}} + + + + + +
    +
    +
    +
    + + + Archive Icon + ArchiveBox + +
    +
    + Favicon +    + {{title}} +    + ▾ +
    + + + {{url_str}} + + +
    +
    +
    +
    +
    +
    +
    +
    Added
    + {{bookmarked_date}} +
    +
    +
    First Archived
    + {{oldest_archive_date}} +
    +
    +
    Last Checked
    + {{updated_date}} +
    +
    +
    +
    +
    Type
    +
    {{extension}}
    +
    +
    +
    Tags
    +
    {{tags}}
    +
    +
    +
    Status
    +
    {{status}}
    +
    +
    +
    Saved
    + ✅ {{num_outputs}} +
    +
    +
    Errors
    + ❌ {{num_failures}} +
    +
    +
    Size
    + {{size}} +
    +
    +
    +
    +
    🗃 Files
    + JSON | + WARC | + Media | + Git | + Favicon | + See all... +
    +
    +
    +
    +
    +
    + +
    + + + +

    Wget > WARC

     +

    archive/{{domain}}

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > SingleFile

     +

    archive/singlefile.html

    +
    +
    +
    +
    +
    + +
    + + + +

    Archive.Org

     +

    web.archive.org/web/...

    +
    +
    +
    +
    +
    + +
    + + + +

    Original

     +

    {{domain}}

    +
    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > PDF

     +

    archive/output.pdf

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > Screenshot

     +

    archive/screenshot.png

    +
    +
    +
    +
    +
    + +
    + + + +

    Chrome > HTML

     +

    archive/output.html

    +
    +
    +
    +
    +
    + +
    + + + +

    Readability

     +

    archive/readability/...

    +
    +
    +
    +
    +
    +
    + +
    + + + +

    mercury

     +

    archive/mercury/...

    +
    +
    +
    +
    +
    +
    + + + + + + + + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index.html new file mode 100644 index 0000000..95af196 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index.html @@ -0,0 +1,255 @@ +{% load static %} + + + + + Archived Sites + + + + + + + + + +
    +
    +
    +
    + + Logo + ArchiveBox: Index + +
    +
    + Add Links   |   + Admin   |   + Docs +
    +
    +
    +
    + + + + + + + + + + + + {% for link in links %} + {% include 'main_index_row.html' with link=link %} + {% endfor %} + +
    BookmarkedSaved Link ({{num_links}})FilesOriginal URL
    + + + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index_minimal.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index_minimal.html new file mode 100644 index 0000000..dcfaa23 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index_minimal.html @@ -0,0 +1,24 @@ + + + + Archived Sites + + + + + + + + + + + + + + {% for link in links %} + {% include "main_index_row.html" with link=link %} + {% endfor %} + +
    BookmarkedSaved Link ({{num_links}})FilesOriginal URL
    + + \ No newline at end of file diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index_row.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index_row.html new file mode 100644 index 0000000..5e21a8c --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/main_index_row.html @@ -0,0 +1,22 @@ +{% load static %} + + + {% if link.bookmarked_date %} {{ link.bookmarked_date }} {% else %} {{ link.added }} {% endif %} + + {% if link.is_archived %} + + {% else %} + + {% endif %} + + {{link.title|default:'Loading...'}} + {% if link.tags_str != None %} {{link.tags_str|default:''}} {% else %} {{ link.tags|default:'' }} {% endif %} + + + + 📄 + {% if link.icons %} {{link.icons}} {% else %} {{ link.num_outputs}} {% endif %} + + + {{link.url}} + \ No newline at end of file diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/add.css b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/add.css new file mode 100644 index 0000000..b128bf4 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/add.css @@ -0,0 +1,62 @@ +.dashboard #content { + width: 100%; + margin-right: 0px; + margin-left: 0px; +} +#submit { + border: 1px solid rgba(0, 0, 0, 0.2); + padding: 10px; + border-radius: 4px; + background-color: #f5dd5d; + color: #333; + font-size: 18px; + font-weight: 800; +} +#add-form button[role="submit"]:hover { + background-color: #e5cd4d; +} +#add-form label { + display: block; + font-size: 16px; +} +#add-form textarea { + width: 100%; + min-height: 300px; +} +#delay-warning div { + border: 1px solid red; + border-radius: 4px; + margin: 10px; + padding: 10px; + font-size: 15px; + background-color: #f5dd5d; +} +#stdout { + background-color: #ded; + padding: 10px 10px; + border-radius: 4px; + white-space: normal; +} +ul#id_depth { + list-style-type: none; + padding: 0; +} + +@keyframes spin { + 0% { + transform: rotate(0deg); + } + 100% { + transform: rotate(360deg); + } +} + +.loader { + border: 16px solid #f3f3f3; /* Light grey */ + border-top: 16px solid #3498db; /* Blue */ + border-radius: 50%; + width: 30px; + height: 30px; + box-sizing: border-box; + animation: spin 2s linear infinite; +} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/admin.css b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/admin.css new file mode 100644 index 0000000..181c06d --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/admin.css @@ -0,0 +1,234 @@ +#logo { + height: 30px; + vertical-align: -6px; + padding-right: 5px; +} +#site-name:hover a { + opacity: 0.9; +} +#site-name .loader { + height: 25px; + width: 25px; + display: inline-block; + border-width: 3px; + vertical-align: -3px; + margin-right: 5px; + margin-top: 2px; +} +#branding h1, #branding h1 a:link, #branding h1 a:visited { + color: mintcream; +} +#header { + background: #aa1e55; + padding: 6px 14px; +} +#content { + padding: 8px 8px; +} +#user-tools { + font-size: 13px; + +} + +div.breadcrumbs { + background: #772948; + color: #f5dd5d; + padding: 6px 15px; +} + +body.model-snapshot.change-list div.breadcrumbs, +body.model-snapshot.change-list #content .object-tools { + display: none; +} + +.module h2, .module caption, .inline-group h2 { + background: #772948; +} + +#content .object-tools { + margin-top: -35px; + margin-right: -10px; + float: right; +} + +#content .object-tools a:link, #content .object-tools a:visited { + border-radius: 0px; + background-color: #f5dd5d; + color: #333; + font-size: 12px; + font-weight: 800; +} + +#content .object-tools a.addlink { + background-blend-mode: difference; +} + +#content #changelist #toolbar { + padding: 0px; + background: none; + margin-bottom: 10px; + border-top: 0px; + border-bottom: 0px; +} + +#content #changelist #toolbar form input[type="submit"] { + border-color: #aa1e55; +} + +#content #changelist-filter li.selected a { + color: #aa1e55; +} + + +/*#content #changelist .actions { + position: fixed; + bottom: 0px; + z-index: 800; +}*/ +#content #changelist .actions { + float: right; + margin-top: -34px; + padding: 0px; + background: none; + margin-right: 0px; + width: auto; +} + +#content #changelist .actions .button { + border-radius: 2px; + background-color: #f5dd5d; + color: #333; + font-size: 12px; + font-weight: 800; + margin-right: 4px; + box-shadow: 4px 4px 4px rgba(0,0,0,0.02); + border: 1px solid rgba(0,0,0,0.08); +} +#content #changelist .actions .button:hover { + border: 1px solid rgba(0,0,0,0.2); + opacity: 0.9; +} +#content #changelist .actions .button[name=verify_snapshots], #content #changelist .actions .button[name=update_titles] { + background-color: #dedede; + color: #333; +} +#content #changelist .actions .button[name=update_snapshots] { + background-color:lightseagreen; + color: #333; +} +#content #changelist .actions .button[name=overwrite_snapshots] { + background-color: #ffaa31; + color: #333; +} +#content #changelist .actions .button[name=delete_snapshots] { + background-color: #f91f74; + color: rgb(255 248 252 / 64%); +} + + +#content #changelist-filter h2 { + border-radius: 4px 4px 0px 0px; +} + +@media (min-width: 767px) { + #content #changelist-filter { + top: 35px; + width: 110px; + margin-bottom: 35px; + } + + .change-list .filtered .results, + .change-list .filtered .paginator, + .filtered #toolbar, + .filtered div.xfull { + margin-right: 115px; + } +} + +@media (max-width: 1127px) { + #content #changelist .actions { + position: fixed; + bottom: 6px; + left: 10px; + float: left; + z-index: 1000; + } +} + +#content a img.favicon { + height: 20px; + width: 20px; + vertical-align: -5px; + padding-right: 6px; +} + +#content td, #content th { + vertical-align: middle; + padding: 4px; +} + +#content #changelist table input { + vertical-align: -2px; +} + +#content thead th .text a { + padding: 8px 4px; +} + +#content th.field-added, #content td.field-updated { + word-break: break-word; + min-width: 128px; + white-space: normal; +} + +#content th.field-title_str { + min-width: 300px; +} + +#content td.field-files { + white-space: nowrap; +} +#content td.field-files .exists-True { + opacity: 1; +} +#content td.field-files .exists-False { + opacity: 0.1; + filter: grayscale(100%); +} +#content td.field-size { + white-space: nowrap; +} + +#content td.field-url_str { + word-break: break-all; + min-width: 200px; +} + +#content tr b.status-pending { + font-weight: 200; + opacity: 0.6; +} + +.loader { + border: 16px solid #f3f3f3; /* Light grey */ + border-top: 16px solid #3498db; /* Blue */ + border-radius: 50%; + width: 30px; + height: 30px; + box-sizing: border-box; + animation: spin 2s linear infinite; +} + +@keyframes spin { + 0% { transform: rotate(0deg); } + 100% { transform: rotate(360deg); } +} + +.tags > a > .tag { + float: right; + border-radius: 5px; + background-color: #bfdfff; + padding: 2px 5px; + margin-left: 4px; + margin-top: 1px; +} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/archive.png b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/archive.png new file mode 100644 index 0000000..307b450 Binary files /dev/null and b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/archive.png differ diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/bootstrap.min.css b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/bootstrap.min.css new file mode 100644 index 0000000..a8da074 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/bootstrap.min.css @@ -0,0 +1,6 @@ +/*! + * Bootstrap v4.0.0-alpha.6 (https://getbootstrap.com) + * Copyright 2011-2017 The Bootstrap Authors + * Copyright 2011-2017 Twitter, Inc. + * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE) + *//*! normalize.css v5.0.0 | MIT License | github.com/necolas/normalize.css */html{font-family:sans-serif;line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}figcaption,figure,main{display:block}figure{margin:1em 40px}hr{-webkit-box-sizing:content-box;box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1em}a{background-color:transparent;-webkit-text-decoration-skip:objects}a:active,a:hover{outline-width:0}abbr[title]{border-bottom:none;text-decoration:underline;text-decoration:underline dotted}b,strong{font-weight:inherit}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1em}dfn{font-style:italic}mark{background-color:#ff0;color:#000}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}audio,video{display:inline-block}audio:not([controls]){display:none;height:0}img{border-style:none}svg:not(:root){overflow:hidden}button,input,optgroup,select,textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}[type=reset],[type=submit],button,html [type=button]{-webkit-appearance:button}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}legend{-webkit-box-sizing:border-box;box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{display:inline-block;vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{-webkit-box-sizing:border-box;box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-cancel-button,[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details,menu{display:block}summary{display:list-item}canvas{display:inline-block}template{display:none}[hidden]{display:none}@media print{*,::after,::before,blockquote::first-letter,blockquote::first-line,div::first-letter,div::first-line,li::first-letter,li::first-line,p::first-letter,p::first-line{text-shadow:none!important;-webkit-box-shadow:none!important;box-shadow:none!important}a,a:visited{text-decoration:underline}abbr[title]::after{content:" (" attr(title) ")"}pre{white-space:pre-wrap!important}blockquote,pre{border:1px solid #999;page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}h2,h3,p{orphans:3;widows:3}h2,h3{page-break-after:avoid}.navbar{display:none}.badge{border:1px solid #000}.table{border-collapse:collapse!important}.table td,.table th{background-color:#fff!important}.table-bordered td,.table-bordered th{border:1px solid #ddd!important}}html{-webkit-box-sizing:border-box;box-sizing:border-box}*,::after,::before{-webkit-box-sizing:inherit;box-sizing:inherit}@-ms-viewport{width:device-width}html{-ms-overflow-style:scrollbar;-webkit-tap-highlight-color:transparent}body{font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-size:1rem;font-weight:400;line-height:1.5;color:#292b2c;background-color:#fff}[tabindex="-1"]:focus{outline:0!important}h1,h2,h3,h4,h5,h6{margin-top:0;margin-bottom:.5rem}p{margin-top:0;margin-bottom:1rem}abbr[data-original-title],abbr[title]{cursor:help}address{margin-bottom:1rem;font-style:normal;line-height:inherit}dl,ol,ul{margin-top:0;margin-bottom:1rem}ol ol,ol ul,ul ol,ul ul{margin-bottom:0}dt{font-weight:700}dd{margin-bottom:.5rem;margin-left:0}blockquote{margin:0 0 1rem}a{color:#0275d8;text-decoration:none}a:focus,a:hover{color:#014c8c;text-decoration:underline}a:not([href]):not([tabindex]){color:inherit;text-decoration:none}a:not([href]):not([tabindex]):focus,a:not([href]):not([tabindex]):hover{color:inherit;text-decoration:none}a:not([href]):not([tabindex]):focus{outline:0}pre{margin-top:0;margin-bottom:1rem;overflow:auto}figure{margin:0 0 1rem}img{vertical-align:middle}[role=button]{cursor:pointer}[role=button],a,area,button,input,label,select,summary,textarea{-ms-touch-action:manipulation;touch-action:manipulation}table{border-collapse:collapse;background-color:transparent}caption{padding-top:.75rem;padding-bottom:.75rem;color:#636c72;text-align:left;caption-side:bottom}th{text-align:left}label{display:inline-block;margin-bottom:.5rem}button:focus{outline:1px dotted;outline:5px auto -webkit-focus-ring-color}button,input,select,textarea{line-height:inherit}input[type=checkbox]:disabled,input[type=radio]:disabled{cursor:not-allowed}input[type=date],input[type=time],input[type=datetime-local],input[type=month]{-webkit-appearance:listbox}textarea{resize:vertical}fieldset{min-width:0;padding:0;margin:0;border:0}legend{display:block;width:100%;padding:0;margin-bottom:.5rem;font-size:1.5rem;line-height:inherit}input[type=search]{-webkit-appearance:none}output{display:inline-block}[hidden]{display:none!important}.h1,.h2,.h3,.h4,.h5,.h6,h1,h2,h3,h4,h5,h6{margin-bottom:.5rem;font-family:inherit;font-weight:500;line-height:1.1;color:inherit}.h1,h1{font-size:2.5rem}.h2,h2{font-size:2rem}.h3,h3{font-size:1.75rem}.h4,h4{font-size:1.5rem}.h5,h5{font-size:1.25rem}.h6,h6{font-size:1rem}.lead{font-size:1.25rem;font-weight:300}.display-1{font-size:6rem;font-weight:300;line-height:1.1}.display-2{font-size:5.5rem;font-weight:300;line-height:1.1}.display-3{font-size:4.5rem;font-weight:300;line-height:1.1}.display-4{font-size:3.5rem;font-weight:300;line-height:1.1}hr{margin-top:1rem;margin-bottom:1rem;border:0;border-top:1px solid rgba(0,0,0,.1)}.small,small{font-size:80%;font-weight:400}.mark,mark{padding:.2em;background-color:#fcf8e3}.list-unstyled{padding-left:0;list-style:none}.list-inline{padding-left:0;list-style:none}.list-inline-item{display:inline-block}.list-inline-item:not(:last-child){margin-right:5px}.initialism{font-size:90%;text-transform:uppercase}.blockquote{padding:.5rem 1rem;margin-bottom:1rem;font-size:1.25rem;border-left:.25rem solid #eceeef}.blockquote-footer{display:block;font-size:80%;color:#636c72}.blockquote-footer::before{content:"\2014 \00A0"}.blockquote-reverse{padding-right:1rem;padding-left:0;text-align:right;border-right:.25rem solid #eceeef;border-left:0}.blockquote-reverse .blockquote-footer::before{content:""}.blockquote-reverse .blockquote-footer::after{content:"\00A0 \2014"}.img-fluid{max-width:100%;height:auto}.img-thumbnail{padding:.25rem;background-color:#fff;border:1px solid #ddd;border-radius:.25rem;-webkit-transition:all .2s ease-in-out;-o-transition:all .2s ease-in-out;transition:all .2s ease-in-out;max-width:100%;height:auto}.figure{display:inline-block}.figure-img{margin-bottom:.5rem;line-height:1}.figure-caption{font-size:90%;color:#636c72}code,kbd,pre,samp{font-family:Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace}code{padding:.2rem .4rem;font-size:90%;color:#bd4147;background-color:#f7f7f9;border-radius:.25rem}a>code{padding:0;color:inherit;background-color:inherit}kbd{padding:.2rem .4rem;font-size:90%;color:#fff;background-color:#292b2c;border-radius:.2rem}kbd kbd{padding:0;font-size:100%;font-weight:700}pre{display:block;margin-top:0;margin-bottom:1rem;font-size:90%;color:#292b2c}pre code{padding:0;font-size:inherit;color:inherit;background-color:transparent;border-radius:0}.pre-scrollable{max-height:340px;overflow-y:scroll}.container{position:relative;margin-left:auto;margin-right:auto;padding-right:15px;padding-left:15px}@media (min-width:576px){.container{padding-right:15px;padding-left:15px}}@media (min-width:768px){.container{padding-right:15px;padding-left:15px}}@media (min-width:992px){.container{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.container{padding-right:15px;padding-left:15px}}@media (min-width:576px){.container{width:540px;max-width:100%}}@media (min-width:768px){.container{width:720px;max-width:100%}}@media (min-width:992px){.container{width:960px;max-width:100%}}@media (min-width:1200px){.container{width:1140px;max-width:100%}}.container-fluid{position:relative;margin-left:auto;margin-right:auto;padding-right:15px;padding-left:15px}@media (min-width:576px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:768px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:992px){.container-fluid{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.container-fluid{padding-right:15px;padding-left:15px}}.row{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;margin-right:-15px;margin-left:-15px}@media (min-width:576px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:768px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:992px){.row{margin-right:-15px;margin-left:-15px}}@media (min-width:1200px){.row{margin-right:-15px;margin-left:-15px}}.no-gutters{margin-right:0;margin-left:0}.no-gutters>.col,.no-gutters>[class*=col-]{padding-right:0;padding-left:0}.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{position:relative;width:100%;min-height:1px;padding-right:15px;padding-left:15px}@media (min-width:576px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:768px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:992px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}@media (min-width:1200px){.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9{padding-right:15px;padding-left:15px}}.col{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-0{right:auto}.pull-1{right:8.333333%}.pull-2{right:16.666667%}.pull-3{right:25%}.pull-4{right:33.333333%}.pull-5{right:41.666667%}.pull-6{right:50%}.pull-7{right:58.333333%}.pull-8{right:66.666667%}.pull-9{right:75%}.pull-10{right:83.333333%}.pull-11{right:91.666667%}.pull-12{right:100%}.push-0{left:auto}.push-1{left:8.333333%}.push-2{left:16.666667%}.push-3{left:25%}.push-4{left:33.333333%}.push-5{left:41.666667%}.push-6{left:50%}.push-7{left:58.333333%}.push-8{left:66.666667%}.push-9{left:75%}.push-10{left:83.333333%}.push-11{left:91.666667%}.push-12{left:100%}.offset-1{margin-left:8.333333%}.offset-2{margin-left:16.666667%}.offset-3{margin-left:25%}.offset-4{margin-left:33.333333%}.offset-5{margin-left:41.666667%}.offset-6{margin-left:50%}.offset-7{margin-left:58.333333%}.offset-8{margin-left:66.666667%}.offset-9{margin-left:75%}.offset-10{margin-left:83.333333%}.offset-11{margin-left:91.666667%}@media (min-width:576px){.col-sm{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-sm-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-sm-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-sm-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-sm-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-sm-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-sm-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-sm-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-sm-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-sm-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-sm-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-sm-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-sm-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-sm-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-sm-0{right:auto}.pull-sm-1{right:8.333333%}.pull-sm-2{right:16.666667%}.pull-sm-3{right:25%}.pull-sm-4{right:33.333333%}.pull-sm-5{right:41.666667%}.pull-sm-6{right:50%}.pull-sm-7{right:58.333333%}.pull-sm-8{right:66.666667%}.pull-sm-9{right:75%}.pull-sm-10{right:83.333333%}.pull-sm-11{right:91.666667%}.pull-sm-12{right:100%}.push-sm-0{left:auto}.push-sm-1{left:8.333333%}.push-sm-2{left:16.666667%}.push-sm-3{left:25%}.push-sm-4{left:33.333333%}.push-sm-5{left:41.666667%}.push-sm-6{left:50%}.push-sm-7{left:58.333333%}.push-sm-8{left:66.666667%}.push-sm-9{left:75%}.push-sm-10{left:83.333333%}.push-sm-11{left:91.666667%}.push-sm-12{left:100%}.offset-sm-0{margin-left:0}.offset-sm-1{margin-left:8.333333%}.offset-sm-2{margin-left:16.666667%}.offset-sm-3{margin-left:25%}.offset-sm-4{margin-left:33.333333%}.offset-sm-5{margin-left:41.666667%}.offset-sm-6{margin-left:50%}.offset-sm-7{margin-left:58.333333%}.offset-sm-8{margin-left:66.666667%}.offset-sm-9{margin-left:75%}.offset-sm-10{margin-left:83.333333%}.offset-sm-11{margin-left:91.666667%}}@media (min-width:768px){.col-md{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-md-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-md-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-md-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-md-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-md-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-md-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-md-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-md-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-md-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-md-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-md-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-md-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-md-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-md-0{right:auto}.pull-md-1{right:8.333333%}.pull-md-2{right:16.666667%}.pull-md-3{right:25%}.pull-md-4{right:33.333333%}.pull-md-5{right:41.666667%}.pull-md-6{right:50%}.pull-md-7{right:58.333333%}.pull-md-8{right:66.666667%}.pull-md-9{right:75%}.pull-md-10{right:83.333333%}.pull-md-11{right:91.666667%}.pull-md-12{right:100%}.push-md-0{left:auto}.push-md-1{left:8.333333%}.push-md-2{left:16.666667%}.push-md-3{left:25%}.push-md-4{left:33.333333%}.push-md-5{left:41.666667%}.push-md-6{left:50%}.push-md-7{left:58.333333%}.push-md-8{left:66.666667%}.push-md-9{left:75%}.push-md-10{left:83.333333%}.push-md-11{left:91.666667%}.push-md-12{left:100%}.offset-md-0{margin-left:0}.offset-md-1{margin-left:8.333333%}.offset-md-2{margin-left:16.666667%}.offset-md-3{margin-left:25%}.offset-md-4{margin-left:33.333333%}.offset-md-5{margin-left:41.666667%}.offset-md-6{margin-left:50%}.offset-md-7{margin-left:58.333333%}.offset-md-8{margin-left:66.666667%}.offset-md-9{margin-left:75%}.offset-md-10{margin-left:83.333333%}.offset-md-11{margin-left:91.666667%}}@media (min-width:992px){.col-lg{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-lg-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-lg-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-lg-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-lg-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-lg-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-lg-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-lg-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-lg-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-lg-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-lg-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-lg-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-lg-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-lg-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-lg-0{right:auto}.pull-lg-1{right:8.333333%}.pull-lg-2{right:16.666667%}.pull-lg-3{right:25%}.pull-lg-4{right:33.333333%}.pull-lg-5{right:41.666667%}.pull-lg-6{right:50%}.pull-lg-7{right:58.333333%}.pull-lg-8{right:66.666667%}.pull-lg-9{right:75%}.pull-lg-10{right:83.333333%}.pull-lg-11{right:91.666667%}.pull-lg-12{right:100%}.push-lg-0{left:auto}.push-lg-1{left:8.333333%}.push-lg-2{left:16.666667%}.push-lg-3{left:25%}.push-lg-4{left:33.333333%}.push-lg-5{left:41.666667%}.push-lg-6{left:50%}.push-lg-7{left:58.333333%}.push-lg-8{left:66.666667%}.push-lg-9{left:75%}.push-lg-10{left:83.333333%}.push-lg-11{left:91.666667%}.push-lg-12{left:100%}.offset-lg-0{margin-left:0}.offset-lg-1{margin-left:8.333333%}.offset-lg-2{margin-left:16.666667%}.offset-lg-3{margin-left:25%}.offset-lg-4{margin-left:33.333333%}.offset-lg-5{margin-left:41.666667%}.offset-lg-6{margin-left:50%}.offset-lg-7{margin-left:58.333333%}.offset-lg-8{margin-left:66.666667%}.offset-lg-9{margin-left:75%}.offset-lg-10{margin-left:83.333333%}.offset-lg-11{margin-left:91.666667%}}@media (min-width:1200px){.col-xl{-webkit-flex-basis:0;-ms-flex-preferred-size:0;flex-basis:0;-webkit-box-flex:1;-webkit-flex-grow:1;-ms-flex-positive:1;flex-grow:1;max-width:100%}.col-xl-auto{-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.col-xl-1{-webkit-box-flex:0;-webkit-flex:0 0 8.333333%;-ms-flex:0 0 8.333333%;flex:0 0 8.333333%;max-width:8.333333%}.col-xl-2{-webkit-box-flex:0;-webkit-flex:0 0 16.666667%;-ms-flex:0 0 16.666667%;flex:0 0 16.666667%;max-width:16.666667%}.col-xl-3{-webkit-box-flex:0;-webkit-flex:0 0 25%;-ms-flex:0 0 25%;flex:0 0 25%;max-width:25%}.col-xl-4{-webkit-box-flex:0;-webkit-flex:0 0 33.333333%;-ms-flex:0 0 33.333333%;flex:0 0 33.333333%;max-width:33.333333%}.col-xl-5{-webkit-box-flex:0;-webkit-flex:0 0 41.666667%;-ms-flex:0 0 41.666667%;flex:0 0 41.666667%;max-width:41.666667%}.col-xl-6{-webkit-box-flex:0;-webkit-flex:0 0 50%;-ms-flex:0 0 50%;flex:0 0 50%;max-width:50%}.col-xl-7{-webkit-box-flex:0;-webkit-flex:0 0 58.333333%;-ms-flex:0 0 58.333333%;flex:0 0 58.333333%;max-width:58.333333%}.col-xl-8{-webkit-box-flex:0;-webkit-flex:0 0 66.666667%;-ms-flex:0 0 66.666667%;flex:0 0 66.666667%;max-width:66.666667%}.col-xl-9{-webkit-box-flex:0;-webkit-flex:0 0 75%;-ms-flex:0 0 75%;flex:0 0 75%;max-width:75%}.col-xl-10{-webkit-box-flex:0;-webkit-flex:0 0 83.333333%;-ms-flex:0 0 83.333333%;flex:0 0 83.333333%;max-width:83.333333%}.col-xl-11{-webkit-box-flex:0;-webkit-flex:0 0 91.666667%;-ms-flex:0 0 91.666667%;flex:0 0 91.666667%;max-width:91.666667%}.col-xl-12{-webkit-box-flex:0;-webkit-flex:0 0 100%;-ms-flex:0 0 100%;flex:0 0 100%;max-width:100%}.pull-xl-0{right:auto}.pull-xl-1{right:8.333333%}.pull-xl-2{right:16.666667%}.pull-xl-3{right:25%}.pull-xl-4{right:33.333333%}.pull-xl-5{right:41.666667%}.pull-xl-6{right:50%}.pull-xl-7{right:58.333333%}.pull-xl-8{right:66.666667%}.pull-xl-9{right:75%}.pull-xl-10{right:83.333333%}.pull-xl-11{right:91.666667%}.pull-xl-12{right:100%}.push-xl-0{left:auto}.push-xl-1{left:8.333333%}.push-xl-2{left:16.666667%}.push-xl-3{left:25%}.push-xl-4{left:33.333333%}.push-xl-5{left:41.666667%}.push-xl-6{left:50%}.push-xl-7{left:58.333333%}.push-xl-8{left:66.666667%}.push-xl-9{left:75%}.push-xl-10{left:83.333333%}.push-xl-11{left:91.666667%}.push-xl-12{left:100%}.offset-xl-0{margin-left:0}.offset-xl-1{margin-left:8.333333%}.offset-xl-2{margin-left:16.666667%}.offset-xl-3{margin-left:25%}.offset-xl-4{margin-left:33.333333%}.offset-xl-5{margin-left:41.666667%}.offset-xl-6{margin-left:50%}.offset-xl-7{margin-left:58.333333%}.offset-xl-8{margin-left:66.666667%}.offset-xl-9{margin-left:75%}.offset-xl-10{margin-left:83.333333%}.offset-xl-11{margin-left:91.666667%}}.table{width:100%;max-width:100%;margin-bottom:1rem}.table td,.table th{padding:.75rem;vertical-align:top;border-top:1px solid #eceeef}.table thead th{vertical-align:bottom;border-bottom:2px solid #eceeef}.table tbody+tbody{border-top:2px solid #eceeef}.table .table{background-color:#fff}.table-sm td,.table-sm th{padding:.3rem}.table-bordered{border:1px solid #eceeef}.table-bordered td,.table-bordered th{border:1px solid #eceeef}.table-bordered thead td,.table-bordered thead th{border-bottom-width:2px}.table-striped tbody tr:nth-of-type(odd){background-color:rgba(0,0,0,.05)}.table-hover tbody tr:hover{background-color:rgba(0,0,0,.075)}.table-active,.table-active>td,.table-active>th{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover>td,.table-hover .table-active:hover>th{background-color:rgba(0,0,0,.075)}.table-success,.table-success>td,.table-success>th{background-color:#dff0d8}.table-hover .table-success:hover{background-color:#d0e9c6}.table-hover .table-success:hover>td,.table-hover .table-success:hover>th{background-color:#d0e9c6}.table-info,.table-info>td,.table-info>th{background-color:#d9edf7}.table-hover .table-info:hover{background-color:#c4e3f3}.table-hover .table-info:hover>td,.table-hover .table-info:hover>th{background-color:#c4e3f3}.table-warning,.table-warning>td,.table-warning>th{background-color:#fcf8e3}.table-hover .table-warning:hover{background-color:#faf2cc}.table-hover .table-warning:hover>td,.table-hover .table-warning:hover>th{background-color:#faf2cc}.table-danger,.table-danger>td,.table-danger>th{background-color:#f2dede}.table-hover .table-danger:hover{background-color:#ebcccc}.table-hover .table-danger:hover>td,.table-hover .table-danger:hover>th{background-color:#ebcccc}.thead-inverse th{color:#fff;background-color:#292b2c}.thead-default th{color:#464a4c;background-color:#eceeef}.table-inverse{color:#fff;background-color:#292b2c}.table-inverse td,.table-inverse th,.table-inverse thead th{border-color:#fff}.table-inverse.table-bordered{border:0}.table-responsive{display:block;width:100%;overflow-x:auto;-ms-overflow-style:-ms-autohiding-scrollbar}.table-responsive.table-bordered{border:0}.form-control{display:block;width:100%;padding:.5rem .75rem;font-size:1rem;line-height:1.25;color:#464a4c;background-color:#fff;background-image:none;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.15);border-radius:.25rem;-webkit-transition:border-color ease-in-out .15s,-webkit-box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,-webkit-box-shadow ease-in-out .15s;-o-transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s;transition:border-color ease-in-out .15s,box-shadow ease-in-out .15s,-webkit-box-shadow ease-in-out .15s}.form-control::-ms-expand{background-color:transparent;border:0}.form-control:focus{color:#464a4c;background-color:#fff;border-color:#5cb3fd;outline:0}.form-control::-webkit-input-placeholder{color:#636c72;opacity:1}.form-control::-moz-placeholder{color:#636c72;opacity:1}.form-control:-ms-input-placeholder{color:#636c72;opacity:1}.form-control::placeholder{color:#636c72;opacity:1}.form-control:disabled,.form-control[readonly]{background-color:#eceeef;opacity:1}.form-control:disabled{cursor:not-allowed}select.form-control:not([size]):not([multiple]){height:calc(2.25rem + 2px)}select.form-control:focus::-ms-value{color:#464a4c;background-color:#fff}.form-control-file,.form-control-range{display:block}.col-form-label{padding-top:calc(.5rem - 1px * 2);padding-bottom:calc(.5rem - 1px * 2);margin-bottom:0}.col-form-label-lg{padding-top:calc(.75rem - 1px * 2);padding-bottom:calc(.75rem - 1px * 2);font-size:1.25rem}.col-form-label-sm{padding-top:calc(.25rem - 1px * 2);padding-bottom:calc(.25rem - 1px * 2);font-size:.875rem}.col-form-legend{padding-top:.5rem;padding-bottom:.5rem;margin-bottom:0;font-size:1rem}.form-control-static{padding-top:.5rem;padding-bottom:.5rem;margin-bottom:0;line-height:1.25;border:solid transparent;border-width:1px 0}.form-control-static.form-control-lg,.form-control-static.form-control-sm,.input-group-lg>.form-control-static.form-control,.input-group-lg>.form-control-static.input-group-addon,.input-group-lg>.input-group-btn>.form-control-static.btn,.input-group-sm>.form-control-static.form-control,.input-group-sm>.form-control-static.input-group-addon,.input-group-sm>.input-group-btn>.form-control-static.btn{padding-right:0;padding-left:0}.form-control-sm,.input-group-sm>.form-control,.input-group-sm>.input-group-addon,.input-group-sm>.input-group-btn>.btn{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.input-group-sm>.input-group-btn>select.btn:not([size]):not([multiple]),.input-group-sm>select.form-control:not([size]):not([multiple]),.input-group-sm>select.input-group-addon:not([size]):not([multiple]),select.form-control-sm:not([size]):not([multiple]){height:1.8125rem}.form-control-lg,.input-group-lg>.form-control,.input-group-lg>.input-group-addon,.input-group-lg>.input-group-btn>.btn{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.input-group-lg>.input-group-btn>select.btn:not([size]):not([multiple]),.input-group-lg>select.form-control:not([size]):not([multiple]),.input-group-lg>select.input-group-addon:not([size]):not([multiple]),select.form-control-lg:not([size]):not([multiple]){height:3.166667rem}.form-group{margin-bottom:1rem}.form-text{display:block;margin-top:.25rem}.form-check{position:relative;display:block;margin-bottom:.5rem}.form-check.disabled .form-check-label{color:#636c72;cursor:not-allowed}.form-check-label{padding-left:1.25rem;margin-bottom:0;cursor:pointer}.form-check-input{position:absolute;margin-top:.25rem;margin-left:-1.25rem}.form-check-input:only-child{position:static}.form-check-inline{display:inline-block}.form-check-inline .form-check-label{vertical-align:middle}.form-check-inline+.form-check-inline{margin-left:.75rem}.form-control-feedback{margin-top:.25rem}.form-control-danger,.form-control-success,.form-control-warning{padding-right:2.25rem;background-repeat:no-repeat;background-position:center right .5625rem;-webkit-background-size:1.125rem 1.125rem;background-size:1.125rem 1.125rem}.has-success .col-form-label,.has-success .custom-control,.has-success .form-check-label,.has-success .form-control-feedback,.has-success .form-control-label{color:#5cb85c}.has-success .form-control{border-color:#5cb85c}.has-success .input-group-addon{color:#5cb85c;border-color:#5cb85c;background-color:#eaf6ea}.has-success .form-control-success{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%235cb85c' d='M2.3 6.73L.6 4.53c-.4-1.04.46-1.4 1.1-.8l1.1 1.4 3.4-3.8c.6-.63 1.6-.27 1.2.7l-4 4.6c-.43.5-.8.4-1.1.1z'/%3E%3C/svg%3E")}.has-warning .col-form-label,.has-warning .custom-control,.has-warning .form-check-label,.has-warning .form-control-feedback,.has-warning .form-control-label{color:#f0ad4e}.has-warning .form-control{border-color:#f0ad4e}.has-warning .input-group-addon{color:#f0ad4e;border-color:#f0ad4e;background-color:#fff}.has-warning .form-control-warning{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%23f0ad4e' d='M4.4 5.324h-.8v-2.46h.8zm0 1.42h-.8V5.89h.8zM3.76.63L.04 7.075c-.115.2.016.425.26.426h7.397c.242 0 .372-.226.258-.426C6.726 4.924 5.47 2.79 4.253.63c-.113-.174-.39-.174-.494 0z'/%3E%3C/svg%3E")}.has-danger .col-form-label,.has-danger .custom-control,.has-danger .form-check-label,.has-danger .form-control-feedback,.has-danger .form-control-label{color:#d9534f}.has-danger .form-control{border-color:#d9534f}.has-danger .input-group-addon{color:#d9534f;border-color:#d9534f;background-color:#fdf7f7}.has-danger .form-control-danger{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23d9534f' viewBox='-2 -2 7 7'%3E%3Cpath stroke='%23d9534f' d='M0 0l3 3m0-3L0 3'/%3E%3Ccircle r='.5'/%3E%3Ccircle cx='3' r='.5'/%3E%3Ccircle cy='3' r='.5'/%3E%3Ccircle cx='3' cy='3' r='.5'/%3E%3C/svg%3E")}.form-inline{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.form-inline .form-check{width:100%}@media (min-width:576px){.form-inline label{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;margin-bottom:0}.form-inline .form-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-flex:0;-webkit-flex:0 0 auto;-ms-flex:0 0 auto;flex:0 0 auto;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;margin-bottom:0}.form-inline .form-control{display:inline-block;width:auto;vertical-align:middle}.form-inline .form-control-static{display:inline-block}.form-inline .input-group{width:auto}.form-inline .form-control-label{margin-bottom:0;vertical-align:middle}.form-inline .form-check{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;width:auto;margin-top:0;margin-bottom:0}.form-inline .form-check-label{padding-left:0}.form-inline .form-check-input{position:relative;margin-top:0;margin-right:.25rem;margin-left:0}.form-inline .custom-control{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;padding-left:0}.form-inline .custom-control-indicator{position:static;display:inline-block;margin-right:.25rem;vertical-align:text-bottom}.form-inline .has-feedback .form-control-feedback{top:0}}.btn{display:inline-block;font-weight:400;line-height:1.25;text-align:center;white-space:nowrap;vertical-align:middle;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;border:1px solid transparent;padding:.5rem 1rem;font-size:1rem;border-radius:.25rem;-webkit-transition:all .2s ease-in-out;-o-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.btn:focus,.btn:hover{text-decoration:none}.btn.focus,.btn:focus{outline:0;-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.25);box-shadow:0 0 0 2px rgba(2,117,216,.25)}.btn.disabled,.btn:disabled{cursor:not-allowed;opacity:.65}.btn.active,.btn:active{background-image:none}a.btn.disabled,fieldset[disabled] a.btn{pointer-events:none}.btn-primary{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-primary:hover{color:#fff;background-color:#025aa5;border-color:#01549b}.btn-primary.focus,.btn-primary:focus{-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.5);box-shadow:0 0 0 2px rgba(2,117,216,.5)}.btn-primary.disabled,.btn-primary:disabled{background-color:#0275d8;border-color:#0275d8}.btn-primary.active,.btn-primary:active,.show>.btn-primary.dropdown-toggle{color:#fff;background-color:#025aa5;background-image:none;border-color:#01549b}.btn-secondary{color:#292b2c;background-color:#fff;border-color:#ccc}.btn-secondary:hover{color:#292b2c;background-color:#e6e6e6;border-color:#adadad}.btn-secondary.focus,.btn-secondary:focus{-webkit-box-shadow:0 0 0 2px rgba(204,204,204,.5);box-shadow:0 0 0 2px rgba(204,204,204,.5)}.btn-secondary.disabled,.btn-secondary:disabled{background-color:#fff;border-color:#ccc}.btn-secondary.active,.btn-secondary:active,.show>.btn-secondary.dropdown-toggle{color:#292b2c;background-color:#e6e6e6;background-image:none;border-color:#adadad}.btn-info{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-info:hover{color:#fff;background-color:#31b0d5;border-color:#2aabd2}.btn-info.focus,.btn-info:focus{-webkit-box-shadow:0 0 0 2px rgba(91,192,222,.5);box-shadow:0 0 0 2px rgba(91,192,222,.5)}.btn-info.disabled,.btn-info:disabled{background-color:#5bc0de;border-color:#5bc0de}.btn-info.active,.btn-info:active,.show>.btn-info.dropdown-toggle{color:#fff;background-color:#31b0d5;background-image:none;border-color:#2aabd2}.btn-success{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-success:hover{color:#fff;background-color:#449d44;border-color:#419641}.btn-success.focus,.btn-success:focus{-webkit-box-shadow:0 0 0 2px rgba(92,184,92,.5);box-shadow:0 0 0 2px rgba(92,184,92,.5)}.btn-success.disabled,.btn-success:disabled{background-color:#5cb85c;border-color:#5cb85c}.btn-success.active,.btn-success:active,.show>.btn-success.dropdown-toggle{color:#fff;background-color:#449d44;background-image:none;border-color:#419641}.btn-warning{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-warning:hover{color:#fff;background-color:#ec971f;border-color:#eb9316}.btn-warning.focus,.btn-warning:focus{-webkit-box-shadow:0 0 0 2px rgba(240,173,78,.5);box-shadow:0 0 0 2px rgba(240,173,78,.5)}.btn-warning.disabled,.btn-warning:disabled{background-color:#f0ad4e;border-color:#f0ad4e}.btn-warning.active,.btn-warning:active,.show>.btn-warning.dropdown-toggle{color:#fff;background-color:#ec971f;background-image:none;border-color:#eb9316}.btn-danger{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-danger:hover{color:#fff;background-color:#c9302c;border-color:#c12e2a}.btn-danger.focus,.btn-danger:focus{-webkit-box-shadow:0 0 0 2px rgba(217,83,79,.5);box-shadow:0 0 0 2px rgba(217,83,79,.5)}.btn-danger.disabled,.btn-danger:disabled{background-color:#d9534f;border-color:#d9534f}.btn-danger.active,.btn-danger:active,.show>.btn-danger.dropdown-toggle{color:#fff;background-color:#c9302c;background-image:none;border-color:#c12e2a}.btn-outline-primary{color:#0275d8;background-image:none;background-color:transparent;border-color:#0275d8}.btn-outline-primary:hover{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-outline-primary.focus,.btn-outline-primary:focus{-webkit-box-shadow:0 0 0 2px rgba(2,117,216,.5);box-shadow:0 0 0 2px rgba(2,117,216,.5)}.btn-outline-primary.disabled,.btn-outline-primary:disabled{color:#0275d8;background-color:transparent}.btn-outline-primary.active,.btn-outline-primary:active,.show>.btn-outline-primary.dropdown-toggle{color:#fff;background-color:#0275d8;border-color:#0275d8}.btn-outline-secondary{color:#ccc;background-image:none;background-color:transparent;border-color:#ccc}.btn-outline-secondary:hover{color:#fff;background-color:#ccc;border-color:#ccc}.btn-outline-secondary.focus,.btn-outline-secondary:focus{-webkit-box-shadow:0 0 0 2px rgba(204,204,204,.5);box-shadow:0 0 0 2px rgba(204,204,204,.5)}.btn-outline-secondary.disabled,.btn-outline-secondary:disabled{color:#ccc;background-color:transparent}.btn-outline-secondary.active,.btn-outline-secondary:active,.show>.btn-outline-secondary.dropdown-toggle{color:#fff;background-color:#ccc;border-color:#ccc}.btn-outline-info{color:#5bc0de;background-image:none;background-color:transparent;border-color:#5bc0de}.btn-outline-info:hover{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-outline-info.focus,.btn-outline-info:focus{-webkit-box-shadow:0 0 0 2px rgba(91,192,222,.5);box-shadow:0 0 0 2px rgba(91,192,222,.5)}.btn-outline-info.disabled,.btn-outline-info:disabled{color:#5bc0de;background-color:transparent}.btn-outline-info.active,.btn-outline-info:active,.show>.btn-outline-info.dropdown-toggle{color:#fff;background-color:#5bc0de;border-color:#5bc0de}.btn-outline-success{color:#5cb85c;background-image:none;background-color:transparent;border-color:#5cb85c}.btn-outline-success:hover{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-outline-success.focus,.btn-outline-success:focus{-webkit-box-shadow:0 0 0 2px rgba(92,184,92,.5);box-shadow:0 0 0 2px rgba(92,184,92,.5)}.btn-outline-success.disabled,.btn-outline-success:disabled{color:#5cb85c;background-color:transparent}.btn-outline-success.active,.btn-outline-success:active,.show>.btn-outline-success.dropdown-toggle{color:#fff;background-color:#5cb85c;border-color:#5cb85c}.btn-outline-warning{color:#f0ad4e;background-image:none;background-color:transparent;border-color:#f0ad4e}.btn-outline-warning:hover{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-outline-warning.focus,.btn-outline-warning:focus{-webkit-box-shadow:0 0 0 2px rgba(240,173,78,.5);box-shadow:0 0 0 2px rgba(240,173,78,.5)}.btn-outline-warning.disabled,.btn-outline-warning:disabled{color:#f0ad4e;background-color:transparent}.btn-outline-warning.active,.btn-outline-warning:active,.show>.btn-outline-warning.dropdown-toggle{color:#fff;background-color:#f0ad4e;border-color:#f0ad4e}.btn-outline-danger{color:#d9534f;background-image:none;background-color:transparent;border-color:#d9534f}.btn-outline-danger:hover{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-outline-danger.focus,.btn-outline-danger:focus{-webkit-box-shadow:0 0 0 2px rgba(217,83,79,.5);box-shadow:0 0 0 2px rgba(217,83,79,.5)}.btn-outline-danger.disabled,.btn-outline-danger:disabled{color:#d9534f;background-color:transparent}.btn-outline-danger.active,.btn-outline-danger:active,.show>.btn-outline-danger.dropdown-toggle{color:#fff;background-color:#d9534f;border-color:#d9534f}.btn-link{font-weight:400;color:#0275d8;border-radius:0}.btn-link,.btn-link.active,.btn-link:active,.btn-link:disabled{background-color:transparent}.btn-link,.btn-link:active,.btn-link:focus{border-color:transparent}.btn-link:hover{border-color:transparent}.btn-link:focus,.btn-link:hover{color:#014c8c;text-decoration:underline;background-color:transparent}.btn-link:disabled{color:#636c72}.btn-link:disabled:focus,.btn-link:disabled:hover{text-decoration:none}.btn-group-lg>.btn,.btn-lg{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.btn-group-sm>.btn,.btn-sm{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.btn-block{display:block;width:100%}.btn-block+.btn-block{margin-top:.5rem}input[type=button].btn-block,input[type=reset].btn-block,input[type=submit].btn-block{width:100%}.fade{opacity:0;-webkit-transition:opacity .15s linear;-o-transition:opacity .15s linear;transition:opacity .15s linear}.fade.show{opacity:1}.collapse{display:none}.collapse.show{display:block}tr.collapse.show{display:table-row}tbody.collapse.show{display:table-row-group}.collapsing{position:relative;height:0;overflow:hidden;-webkit-transition:height .35s ease;-o-transition:height .35s ease;transition:height .35s ease}.dropdown,.dropup{position:relative}.dropdown-toggle::after{display:inline-block;width:0;height:0;margin-left:.3em;vertical-align:middle;content:"";border-top:.3em solid;border-right:.3em solid transparent;border-left:.3em solid transparent}.dropdown-toggle:focus{outline:0}.dropup .dropdown-toggle::after{border-top:0;border-bottom:.3em solid}.dropdown-menu{position:absolute;top:100%;left:0;z-index:1000;display:none;float:left;min-width:10rem;padding:.5rem 0;margin:.125rem 0 0;font-size:1rem;color:#292b2c;text-align:left;list-style:none;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.dropdown-divider{height:1px;margin:.5rem 0;overflow:hidden;background-color:#eceeef}.dropdown-item{display:block;width:100%;padding:3px 1.5rem;clear:both;font-weight:400;color:#292b2c;text-align:inherit;white-space:nowrap;background:0 0;border:0}.dropdown-item:focus,.dropdown-item:hover{color:#1d1e1f;text-decoration:none;background-color:#f7f7f9}.dropdown-item.active,.dropdown-item:active{color:#fff;text-decoration:none;background-color:#0275d8}.dropdown-item.disabled,.dropdown-item:disabled{color:#636c72;cursor:not-allowed;background-color:transparent}.show>.dropdown-menu{display:block}.show>a{outline:0}.dropdown-menu-right{right:0;left:auto}.dropdown-menu-left{right:auto;left:0}.dropdown-header{display:block;padding:.5rem 1.5rem;margin-bottom:0;font-size:.875rem;color:#636c72;white-space:nowrap}.dropdown-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;z-index:990}.dropup .dropdown-menu{top:auto;bottom:100%;margin-bottom:.125rem}.btn-group,.btn-group-vertical{position:relative;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;vertical-align:middle}.btn-group-vertical>.btn,.btn-group>.btn{position:relative;-webkit-box-flex:0;-webkit-flex:0 1 auto;-ms-flex:0 1 auto;flex:0 1 auto}.btn-group-vertical>.btn:hover,.btn-group>.btn:hover{z-index:2}.btn-group-vertical>.btn.active,.btn-group-vertical>.btn:active,.btn-group-vertical>.btn:focus,.btn-group>.btn.active,.btn-group>.btn:active,.btn-group>.btn:focus{z-index:2}.btn-group .btn+.btn,.btn-group .btn+.btn-group,.btn-group .btn-group+.btn,.btn-group .btn-group+.btn-group,.btn-group-vertical .btn+.btn,.btn-group-vertical .btn+.btn-group,.btn-group-vertical .btn-group+.btn,.btn-group-vertical .btn-group+.btn-group{margin-left:-1px}.btn-toolbar{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-pack:start;-webkit-justify-content:flex-start;-ms-flex-pack:start;justify-content:flex-start}.btn-toolbar .input-group{width:auto}.btn-group>.btn:not(:first-child):not(:last-child):not(.dropdown-toggle){border-radius:0}.btn-group>.btn:first-child{margin-left:0}.btn-group>.btn:first-child:not(:last-child):not(.dropdown-toggle){border-bottom-right-radius:0;border-top-right-radius:0}.btn-group>.btn:last-child:not(:first-child),.btn-group>.dropdown-toggle:not(:first-child){border-bottom-left-radius:0;border-top-left-radius:0}.btn-group>.btn-group{float:left}.btn-group>.btn-group:not(:first-child):not(:last-child)>.btn{border-radius:0}.btn-group>.btn-group:first-child:not(:last-child)>.btn:last-child,.btn-group>.btn-group:first-child:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-top-right-radius:0}.btn-group>.btn-group:last-child:not(:first-child)>.btn:first-child{border-bottom-left-radius:0;border-top-left-radius:0}.btn-group .dropdown-toggle:active,.btn-group.open .dropdown-toggle{outline:0}.btn+.dropdown-toggle-split{padding-right:.75rem;padding-left:.75rem}.btn+.dropdown-toggle-split::after{margin-left:0}.btn-group-sm>.btn+.dropdown-toggle-split,.btn-sm+.dropdown-toggle-split{padding-right:.375rem;padding-left:.375rem}.btn-group-lg>.btn+.dropdown-toggle-split,.btn-lg+.dropdown-toggle-split{padding-right:1.125rem;padding-left:1.125rem}.btn-group-vertical{display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;-webkit-box-align:start;-webkit-align-items:flex-start;-ms-flex-align:start;align-items:flex-start;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center}.btn-group-vertical .btn,.btn-group-vertical .btn-group{width:100%}.btn-group-vertical>.btn+.btn,.btn-group-vertical>.btn+.btn-group,.btn-group-vertical>.btn-group+.btn,.btn-group-vertical>.btn-group+.btn-group{margin-top:-1px;margin-left:0}.btn-group-vertical>.btn:not(:first-child):not(:last-child){border-radius:0}.btn-group-vertical>.btn:first-child:not(:last-child){border-bottom-right-radius:0;border-bottom-left-radius:0}.btn-group-vertical>.btn:last-child:not(:first-child){border-top-right-radius:0;border-top-left-radius:0}.btn-group-vertical>.btn-group:not(:first-child):not(:last-child)>.btn{border-radius:0}.btn-group-vertical>.btn-group:first-child:not(:last-child)>.btn:last-child,.btn-group-vertical>.btn-group:first-child:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-bottom-left-radius:0}.btn-group-vertical>.btn-group:last-child:not(:first-child)>.btn:first-child{border-top-right-radius:0;border-top-left-radius:0}[data-toggle=buttons]>.btn input[type=checkbox],[data-toggle=buttons]>.btn input[type=radio],[data-toggle=buttons]>.btn-group>.btn input[type=checkbox],[data-toggle=buttons]>.btn-group>.btn input[type=radio]{position:absolute;clip:rect(0,0,0,0);pointer-events:none}.input-group{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;width:100%}.input-group .form-control{position:relative;z-index:2;-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;width:1%;margin-bottom:0}.input-group .form-control:active,.input-group .form-control:focus,.input-group .form-control:hover{z-index:3}.input-group .form-control,.input-group-addon,.input-group-btn{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center}.input-group .form-control:not(:first-child):not(:last-child),.input-group-addon:not(:first-child):not(:last-child),.input-group-btn:not(:first-child):not(:last-child){border-radius:0}.input-group-addon,.input-group-btn{white-space:nowrap;vertical-align:middle}.input-group-addon{padding:.5rem .75rem;margin-bottom:0;font-size:1rem;font-weight:400;line-height:1.25;color:#464a4c;text-align:center;background-color:#eceeef;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.input-group-addon.form-control-sm,.input-group-sm>.input-group-addon,.input-group-sm>.input-group-btn>.input-group-addon.btn{padding:.25rem .5rem;font-size:.875rem;border-radius:.2rem}.input-group-addon.form-control-lg,.input-group-lg>.input-group-addon,.input-group-lg>.input-group-btn>.input-group-addon.btn{padding:.75rem 1.5rem;font-size:1.25rem;border-radius:.3rem}.input-group-addon input[type=checkbox],.input-group-addon input[type=radio]{margin-top:0}.input-group .form-control:not(:last-child),.input-group-addon:not(:last-child),.input-group-btn:not(:first-child)>.btn-group:not(:last-child)>.btn,.input-group-btn:not(:first-child)>.btn:not(:last-child):not(.dropdown-toggle),.input-group-btn:not(:last-child)>.btn,.input-group-btn:not(:last-child)>.btn-group>.btn,.input-group-btn:not(:last-child)>.dropdown-toggle{border-bottom-right-radius:0;border-top-right-radius:0}.input-group-addon:not(:last-child){border-right:0}.input-group .form-control:not(:first-child),.input-group-addon:not(:first-child),.input-group-btn:not(:first-child)>.btn,.input-group-btn:not(:first-child)>.btn-group>.btn,.input-group-btn:not(:first-child)>.dropdown-toggle,.input-group-btn:not(:last-child)>.btn-group:not(:first-child)>.btn,.input-group-btn:not(:last-child)>.btn:not(:first-child){border-bottom-left-radius:0;border-top-left-radius:0}.form-control+.input-group-addon:not(:first-child){border-left:0}.input-group-btn{position:relative;font-size:0;white-space:nowrap}.input-group-btn>.btn{position:relative;-webkit-box-flex:1;-webkit-flex:1 1 0%;-ms-flex:1 1 0%;flex:1 1 0%}.input-group-btn>.btn+.btn{margin-left:-1px}.input-group-btn>.btn:active,.input-group-btn>.btn:focus,.input-group-btn>.btn:hover{z-index:3}.input-group-btn:not(:last-child)>.btn,.input-group-btn:not(:last-child)>.btn-group{margin-right:-1px}.input-group-btn:not(:first-child)>.btn,.input-group-btn:not(:first-child)>.btn-group{z-index:2;margin-left:-1px}.input-group-btn:not(:first-child)>.btn-group:active,.input-group-btn:not(:first-child)>.btn-group:focus,.input-group-btn:not(:first-child)>.btn-group:hover,.input-group-btn:not(:first-child)>.btn:active,.input-group-btn:not(:first-child)>.btn:focus,.input-group-btn:not(:first-child)>.btn:hover{z-index:3}.custom-control{position:relative;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;min-height:1.5rem;padding-left:1.5rem;margin-right:1rem;cursor:pointer}.custom-control-input{position:absolute;z-index:-1;opacity:0}.custom-control-input:checked~.custom-control-indicator{color:#fff;background-color:#0275d8}.custom-control-input:focus~.custom-control-indicator{-webkit-box-shadow:0 0 0 1px #fff,0 0 0 3px #0275d8;box-shadow:0 0 0 1px #fff,0 0 0 3px #0275d8}.custom-control-input:active~.custom-control-indicator{color:#fff;background-color:#8fcafe}.custom-control-input:disabled~.custom-control-indicator{cursor:not-allowed;background-color:#eceeef}.custom-control-input:disabled~.custom-control-description{color:#636c72;cursor:not-allowed}.custom-control-indicator{position:absolute;top:.25rem;left:0;display:block;width:1rem;height:1rem;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;background-color:#ddd;background-repeat:no-repeat;background-position:center center;-webkit-background-size:50% 50%;background-size:50% 50%}.custom-checkbox .custom-control-indicator{border-radius:.25rem}.custom-checkbox .custom-control-input:checked~.custom-control-indicator{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3E%3Cpath fill='%23fff' d='M6.564.75l-3.59 3.612-1.538-1.55L0 4.26 2.974 7.25 8 2.193z'/%3E%3C/svg%3E")}.custom-checkbox .custom-control-input:indeterminate~.custom-control-indicator{background-color:#0275d8;background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 4 4'%3E%3Cpath stroke='%23fff' d='M0 2h4'/%3E%3C/svg%3E")}.custom-radio .custom-control-indicator{border-radius:50%}.custom-radio .custom-control-input:checked~.custom-control-indicator{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3E%3Ccircle r='3' fill='%23fff'/%3E%3C/svg%3E")}.custom-controls-stacked{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column}.custom-controls-stacked .custom-control{margin-bottom:.25rem}.custom-controls-stacked .custom-control+.custom-control{margin-left:0}.custom-select{display:inline-block;max-width:100%;height:calc(2.25rem + 2px);padding:.375rem 1.75rem .375rem .75rem;line-height:1.25;color:#464a4c;vertical-align:middle;background:#fff url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 4 5'%3E%3Cpath fill='%23333' d='M2 0L0 2h4zm0 5L0 3h4z'/%3E%3C/svg%3E") no-repeat right .75rem center;-webkit-background-size:8px 10px;background-size:8px 10px;border:1px solid rgba(0,0,0,.15);border-radius:.25rem;-moz-appearance:none;-webkit-appearance:none}.custom-select:focus{border-color:#5cb3fd;outline:0}.custom-select:focus::-ms-value{color:#464a4c;background-color:#fff}.custom-select:disabled{color:#636c72;cursor:not-allowed;background-color:#eceeef}.custom-select::-ms-expand{opacity:0}.custom-select-sm{padding-top:.375rem;padding-bottom:.375rem;font-size:75%}.custom-file{position:relative;display:inline-block;max-width:100%;height:2.5rem;margin-bottom:0;cursor:pointer}.custom-file-input{min-width:14rem;max-width:100%;height:2.5rem;margin:0;filter:alpha(opacity=0);opacity:0}.custom-file-control{position:absolute;top:0;right:0;left:0;z-index:5;height:2.5rem;padding:.5rem 1rem;line-height:1.5;color:#464a4c;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;background-color:#fff;border:1px solid rgba(0,0,0,.15);border-radius:.25rem}.custom-file-control:lang(en)::after{content:"Choose file..."}.custom-file-control::before{position:absolute;top:-1px;right:-1px;bottom:-1px;z-index:6;display:block;height:2.5rem;padding:.5rem 1rem;line-height:1.5;color:#464a4c;background-color:#eceeef;border:1px solid rgba(0,0,0,.15);border-radius:0 .25rem .25rem 0}.custom-file-control:lang(en)::before{content:"Browse"}.nav{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;padding-left:0;margin-bottom:0;list-style:none}.nav-link{display:block;padding:.5em 1em}.nav-link:focus,.nav-link:hover{text-decoration:none}.nav-link.disabled{color:#636c72;cursor:not-allowed}.nav-tabs{border-bottom:1px solid #ddd}.nav-tabs .nav-item{margin-bottom:-1px}.nav-tabs .nav-link{border:1px solid transparent;border-top-right-radius:.25rem;border-top-left-radius:.25rem}.nav-tabs .nav-link:focus,.nav-tabs .nav-link:hover{border-color:#eceeef #eceeef #ddd}.nav-tabs .nav-link.disabled{color:#636c72;background-color:transparent;border-color:transparent}.nav-tabs .nav-item.show .nav-link,.nav-tabs .nav-link.active{color:#464a4c;background-color:#fff;border-color:#ddd #ddd #fff}.nav-tabs .dropdown-menu{margin-top:-1px;border-top-right-radius:0;border-top-left-radius:0}.nav-pills .nav-link{border-radius:.25rem}.nav-pills .nav-item.show .nav-link,.nav-pills .nav-link.active{color:#fff;cursor:default;background-color:#0275d8}.nav-fill .nav-item{-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;text-align:center}.nav-justified .nav-item{-webkit-box-flex:1;-webkit-flex:1 1 100%;-ms-flex:1 1 100%;flex:1 1 100%;text-align:center}.tab-content>.tab-pane{display:none}.tab-content>.active{display:block}.navbar{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding:.5rem 1rem}.navbar-brand{display:inline-block;padding-top:.25rem;padding-bottom:.25rem;margin-right:1rem;font-size:1.25rem;line-height:inherit;white-space:nowrap}.navbar-brand:focus,.navbar-brand:hover{text-decoration:none}.navbar-nav{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding-left:0;margin-bottom:0;list-style:none}.navbar-nav .nav-link{padding-right:0;padding-left:0}.navbar-text{display:inline-block;padding-top:.425rem;padding-bottom:.425rem}.navbar-toggler{-webkit-align-self:flex-start;-ms-flex-item-align:start;align-self:flex-start;padding:.25rem .75rem;font-size:1.25rem;line-height:1;background:0 0;border:1px solid transparent;border-radius:.25rem}.navbar-toggler:focus,.navbar-toggler:hover{text-decoration:none}.navbar-toggler-icon{display:inline-block;width:1.5em;height:1.5em;vertical-align:middle;content:"";background:no-repeat center center;-webkit-background-size:100% 100%;background-size:100% 100%}.navbar-toggler-left{position:absolute;left:1rem}.navbar-toggler-right{position:absolute;right:1rem}@media (max-width:575px){.navbar-toggleable .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable>.container{padding-right:0;padding-left:0}}@media (min-width:576px){.navbar-toggleable{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable .navbar-toggler{display:none}}@media (max-width:767px){.navbar-toggleable-sm .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-sm>.container{padding-right:0;padding-left:0}}@media (min-width:768px){.navbar-toggleable-sm{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-sm .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-sm .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-sm>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-sm .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-sm .navbar-toggler{display:none}}@media (max-width:991px){.navbar-toggleable-md .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-md>.container{padding-right:0;padding-left:0}}@media (min-width:992px){.navbar-toggleable-md{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-md .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-md .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-md>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-md .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-md .navbar-toggler{display:none}}@media (max-width:1199px){.navbar-toggleable-lg .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-lg>.container{padding-right:0;padding-left:0}}@media (min-width:1200px){.navbar-toggleable-lg{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-lg .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-lg .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-lg>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-lg .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-lg .navbar-toggler{display:none}}.navbar-toggleable-xl{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-xl .navbar-nav .dropdown-menu{position:static;float:none}.navbar-toggleable-xl>.container{padding-right:0;padding-left:0}.navbar-toggleable-xl .navbar-nav{-webkit-box-orient:horizontal;-webkit-box-direction:normal;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row}.navbar-toggleable-xl .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-toggleable-xl>.container{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-wrap:nowrap;-ms-flex-wrap:nowrap;flex-wrap:nowrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center}.navbar-toggleable-xl .navbar-collapse{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;width:100%}.navbar-toggleable-xl .navbar-toggler{display:none}.navbar-light .navbar-brand,.navbar-light .navbar-toggler{color:rgba(0,0,0,.9)}.navbar-light .navbar-brand:focus,.navbar-light .navbar-brand:hover,.navbar-light .navbar-toggler:focus,.navbar-light .navbar-toggler:hover{color:rgba(0,0,0,.9)}.navbar-light .navbar-nav .nav-link{color:rgba(0,0,0,.5)}.navbar-light .navbar-nav .nav-link:focus,.navbar-light .navbar-nav .nav-link:hover{color:rgba(0,0,0,.7)}.navbar-light .navbar-nav .nav-link.disabled{color:rgba(0,0,0,.3)}.navbar-light .navbar-nav .active>.nav-link,.navbar-light .navbar-nav .nav-link.active,.navbar-light .navbar-nav .nav-link.open,.navbar-light .navbar-nav .open>.nav-link{color:rgba(0,0,0,.9)}.navbar-light .navbar-toggler{border-color:rgba(0,0,0,.1)}.navbar-light .navbar-toggler-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg viewBox='0 0 32 32' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath stroke='rgba(0, 0, 0, 0.5)' stroke-width='2' stroke-linecap='round' stroke-miterlimit='10' d='M4 8h24M4 16h24M4 24h24'/%3E%3C/svg%3E")}.navbar-light .navbar-text{color:rgba(0,0,0,.5)}.navbar-inverse .navbar-brand,.navbar-inverse .navbar-toggler{color:#fff}.navbar-inverse .navbar-brand:focus,.navbar-inverse .navbar-brand:hover,.navbar-inverse .navbar-toggler:focus,.navbar-inverse .navbar-toggler:hover{color:#fff}.navbar-inverse .navbar-nav .nav-link{color:rgba(255,255,255,.5)}.navbar-inverse .navbar-nav .nav-link:focus,.navbar-inverse .navbar-nav .nav-link:hover{color:rgba(255,255,255,.75)}.navbar-inverse .navbar-nav .nav-link.disabled{color:rgba(255,255,255,.25)}.navbar-inverse .navbar-nav .active>.nav-link,.navbar-inverse .navbar-nav .nav-link.active,.navbar-inverse .navbar-nav .nav-link.open,.navbar-inverse .navbar-nav .open>.nav-link{color:#fff}.navbar-inverse .navbar-toggler{border-color:rgba(255,255,255,.1)}.navbar-inverse .navbar-toggler-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg viewBox='0 0 32 32' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath stroke='rgba(255, 255, 255, 0.5)' stroke-width='2' stroke-linecap='round' stroke-miterlimit='10' d='M4 8h24M4 16h24M4 24h24'/%3E%3C/svg%3E")}.navbar-inverse .navbar-text{color:rgba(255,255,255,.5)}.card{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;background-color:#fff;border:1px solid rgba(0,0,0,.125);border-radius:.25rem}.card-block{-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;padding:1.25rem}.card-title{margin-bottom:.75rem}.card-subtitle{margin-top:-.375rem;margin-bottom:0}.card-text:last-child{margin-bottom:0}.card-link:hover{text-decoration:none}.card-link+.card-link{margin-left:1.25rem}.card>.list-group:first-child .list-group-item:first-child{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.card>.list-group:last-child .list-group-item:last-child{border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.card-header{padding:.75rem 1.25rem;margin-bottom:0;background-color:#f7f7f9;border-bottom:1px solid rgba(0,0,0,.125)}.card-header:first-child{border-radius:calc(.25rem - 1px) calc(.25rem - 1px) 0 0}.card-footer{padding:.75rem 1.25rem;background-color:#f7f7f9;border-top:1px solid rgba(0,0,0,.125)}.card-footer:last-child{border-radius:0 0 calc(.25rem - 1px) calc(.25rem - 1px)}.card-header-tabs{margin-right:-.625rem;margin-bottom:-.75rem;margin-left:-.625rem;border-bottom:0}.card-header-pills{margin-right:-.625rem;margin-left:-.625rem}.card-primary{background-color:#0275d8;border-color:#0275d8}.card-primary .card-footer,.card-primary .card-header{background-color:transparent}.card-success{background-color:#5cb85c;border-color:#5cb85c}.card-success .card-footer,.card-success .card-header{background-color:transparent}.card-info{background-color:#5bc0de;border-color:#5bc0de}.card-info .card-footer,.card-info .card-header{background-color:transparent}.card-warning{background-color:#f0ad4e;border-color:#f0ad4e}.card-warning .card-footer,.card-warning .card-header{background-color:transparent}.card-danger{background-color:#d9534f;border-color:#d9534f}.card-danger .card-footer,.card-danger .card-header{background-color:transparent}.card-outline-primary{background-color:transparent;border-color:#0275d8}.card-outline-secondary{background-color:transparent;border-color:#ccc}.card-outline-info{background-color:transparent;border-color:#5bc0de}.card-outline-success{background-color:transparent;border-color:#5cb85c}.card-outline-warning{background-color:transparent;border-color:#f0ad4e}.card-outline-danger{background-color:transparent;border-color:#d9534f}.card-inverse{color:rgba(255,255,255,.65)}.card-inverse .card-footer,.card-inverse .card-header{background-color:transparent;border-color:rgba(255,255,255,.2)}.card-inverse .card-blockquote,.card-inverse .card-footer,.card-inverse .card-header,.card-inverse .card-title{color:#fff}.card-inverse .card-blockquote .blockquote-footer,.card-inverse .card-link,.card-inverse .card-subtitle,.card-inverse .card-text{color:rgba(255,255,255,.65)}.card-inverse .card-link:focus,.card-inverse .card-link:hover{color:#fff}.card-blockquote{padding:0;margin-bottom:0;border-left:0}.card-img{border-radius:calc(.25rem - 1px)}.card-img-overlay{position:absolute;top:0;right:0;bottom:0;left:0;padding:1.25rem}.card-img-top{border-top-right-radius:calc(.25rem - 1px);border-top-left-radius:calc(.25rem - 1px)}.card-img-bottom{border-bottom-right-radius:calc(.25rem - 1px);border-bottom-left-radius:calc(.25rem - 1px)}@media (min-width:576px){.card-deck{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap}.card-deck .card{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-flex:1;-webkit-flex:1 0 0%;-ms-flex:1 0 0%;flex:1 0 0%;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column}.card-deck .card:not(:first-child){margin-left:15px}.card-deck .card:not(:last-child){margin-right:15px}}@media (min-width:576px){.card-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap}.card-group .card{-webkit-box-flex:1;-webkit-flex:1 0 0%;-ms-flex:1 0 0%;flex:1 0 0%}.card-group .card+.card{margin-left:0;border-left:0}.card-group .card:first-child{border-bottom-right-radius:0;border-top-right-radius:0}.card-group .card:first-child .card-img-top{border-top-right-radius:0}.card-group .card:first-child .card-img-bottom{border-bottom-right-radius:0}.card-group .card:last-child{border-bottom-left-radius:0;border-top-left-radius:0}.card-group .card:last-child .card-img-top{border-top-left-radius:0}.card-group .card:last-child .card-img-bottom{border-bottom-left-radius:0}.card-group .card:not(:first-child):not(:last-child){border-radius:0}.card-group .card:not(:first-child):not(:last-child) .card-img-bottom,.card-group .card:not(:first-child):not(:last-child) .card-img-top{border-radius:0}}@media (min-width:576px){.card-columns{-webkit-column-count:3;-moz-column-count:3;column-count:3;-webkit-column-gap:1.25rem;-moz-column-gap:1.25rem;column-gap:1.25rem}.card-columns .card{display:inline-block;width:100%;margin-bottom:.75rem}}.breadcrumb{padding:.75rem 1rem;margin-bottom:1rem;list-style:none;background-color:#eceeef;border-radius:.25rem}.breadcrumb::after{display:block;content:"";clear:both}.breadcrumb-item{float:left}.breadcrumb-item+.breadcrumb-item::before{display:inline-block;padding-right:.5rem;padding-left:.5rem;color:#636c72;content:"/"}.breadcrumb-item+.breadcrumb-item:hover::before{text-decoration:underline}.breadcrumb-item+.breadcrumb-item:hover::before{text-decoration:none}.breadcrumb-item.active{color:#636c72}.pagination{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;padding-left:0;list-style:none;border-radius:.25rem}.page-item:first-child .page-link{margin-left:0;border-bottom-left-radius:.25rem;border-top-left-radius:.25rem}.page-item:last-child .page-link{border-bottom-right-radius:.25rem;border-top-right-radius:.25rem}.page-item.active .page-link{z-index:2;color:#fff;background-color:#0275d8;border-color:#0275d8}.page-item.disabled .page-link{color:#636c72;pointer-events:none;cursor:not-allowed;background-color:#fff;border-color:#ddd}.page-link{position:relative;display:block;padding:.5rem .75rem;margin-left:-1px;line-height:1.25;color:#0275d8;background-color:#fff;border:1px solid #ddd}.page-link:focus,.page-link:hover{color:#014c8c;text-decoration:none;background-color:#eceeef;border-color:#ddd}.pagination-lg .page-link{padding:.75rem 1.5rem;font-size:1.25rem}.pagination-lg .page-item:first-child .page-link{border-bottom-left-radius:.3rem;border-top-left-radius:.3rem}.pagination-lg .page-item:last-child .page-link{border-bottom-right-radius:.3rem;border-top-right-radius:.3rem}.pagination-sm .page-link{padding:.25rem .5rem;font-size:.875rem}.pagination-sm .page-item:first-child .page-link{border-bottom-left-radius:.2rem;border-top-left-radius:.2rem}.pagination-sm .page-item:last-child .page-link{border-bottom-right-radius:.2rem;border-top-right-radius:.2rem}.badge{display:inline-block;padding:.25em .4em;font-size:75%;font-weight:700;line-height:1;color:#fff;text-align:center;white-space:nowrap;vertical-align:baseline;border-radius:.25rem}.badge:empty{display:none}.btn .badge{position:relative;top:-1px}a.badge:focus,a.badge:hover{color:#fff;text-decoration:none;cursor:pointer}.badge-pill{padding-right:.6em;padding-left:.6em;border-radius:10rem}.badge-default{background-color:#636c72}.badge-default[href]:focus,.badge-default[href]:hover{background-color:#4b5257}.badge-primary{background-color:#0275d8}.badge-primary[href]:focus,.badge-primary[href]:hover{background-color:#025aa5}.badge-success{background-color:#5cb85c}.badge-success[href]:focus,.badge-success[href]:hover{background-color:#449d44}.badge-info{background-color:#5bc0de}.badge-info[href]:focus,.badge-info[href]:hover{background-color:#31b0d5}.badge-warning{background-color:#f0ad4e}.badge-warning[href]:focus,.badge-warning[href]:hover{background-color:#ec971f}.badge-danger{background-color:#d9534f}.badge-danger[href]:focus,.badge-danger[href]:hover{background-color:#c9302c}.jumbotron{padding:2rem 1rem;margin-bottom:2rem;background-color:#eceeef;border-radius:.3rem}@media (min-width:576px){.jumbotron{padding:4rem 2rem}}.jumbotron-hr{border-top-color:#d0d5d8}.jumbotron-fluid{padding-right:0;padding-left:0;border-radius:0}.alert{padding:.75rem 1.25rem;margin-bottom:1rem;border:1px solid transparent;border-radius:.25rem}.alert-heading{color:inherit}.alert-link{font-weight:700}.alert-dismissible .close{position:relative;top:-.75rem;right:-1.25rem;padding:.75rem 1.25rem;color:inherit}.alert-success{background-color:#dff0d8;border-color:#d0e9c6;color:#3c763d}.alert-success hr{border-top-color:#c1e2b3}.alert-success .alert-link{color:#2b542c}.alert-info{background-color:#d9edf7;border-color:#bcdff1;color:#31708f}.alert-info hr{border-top-color:#a6d5ec}.alert-info .alert-link{color:#245269}.alert-warning{background-color:#fcf8e3;border-color:#faf2cc;color:#8a6d3b}.alert-warning hr{border-top-color:#f7ecb5}.alert-warning .alert-link{color:#66512c}.alert-danger{background-color:#f2dede;border-color:#ebcccc;color:#a94442}.alert-danger hr{border-top-color:#e4b9b9}.alert-danger .alert-link{color:#843534}@-webkit-keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}@-o-keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}@keyframes progress-bar-stripes{from{background-position:1rem 0}to{background-position:0 0}}.progress{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;overflow:hidden;font-size:.75rem;line-height:1rem;text-align:center;background-color:#eceeef;border-radius:.25rem}.progress-bar{height:1rem;color:#fff;background-color:#0275d8}.progress-bar-striped{background-image:-webkit-linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);background-image:-o-linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);background-image:linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);-webkit-background-size:1rem 1rem;background-size:1rem 1rem}.progress-bar-animated{-webkit-animation:progress-bar-stripes 1s linear infinite;-o-animation:progress-bar-stripes 1s linear infinite;animation:progress-bar-stripes 1s linear infinite}.media{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:start;-webkit-align-items:flex-start;-ms-flex-align:start;align-items:flex-start}.media-body{-webkit-box-flex:1;-webkit-flex:1 1 0%;-ms-flex:1 1 0%;flex:1 1 0%}.list-group{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;padding-left:0;margin-bottom:0}.list-group-item-action{width:100%;color:#464a4c;text-align:inherit}.list-group-item-action .list-group-item-heading{color:#292b2c}.list-group-item-action:focus,.list-group-item-action:hover{color:#464a4c;text-decoration:none;background-color:#f7f7f9}.list-group-item-action:active{color:#292b2c;background-color:#eceeef}.list-group-item{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-flow:row wrap;-ms-flex-flow:row wrap;flex-flow:row wrap;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;padding:.75rem 1.25rem;margin-bottom:-1px;background-color:#fff;border:1px solid rgba(0,0,0,.125)}.list-group-item:first-child{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.list-group-item:last-child{margin-bottom:0;border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.list-group-item:focus,.list-group-item:hover{text-decoration:none}.list-group-item.disabled,.list-group-item:disabled{color:#636c72;cursor:not-allowed;background-color:#fff}.list-group-item.disabled .list-group-item-heading,.list-group-item:disabled .list-group-item-heading{color:inherit}.list-group-item.disabled .list-group-item-text,.list-group-item:disabled .list-group-item-text{color:#636c72}.list-group-item.active{z-index:2;color:#fff;background-color:#0275d8;border-color:#0275d8}.list-group-item.active .list-group-item-heading,.list-group-item.active .list-group-item-heading>.small,.list-group-item.active .list-group-item-heading>small{color:inherit}.list-group-item.active .list-group-item-text{color:#daeeff}.list-group-flush .list-group-item{border-right:0;border-left:0;border-radius:0}.list-group-flush:first-child .list-group-item:first-child{border-top:0}.list-group-flush:last-child .list-group-item:last-child{border-bottom:0}.list-group-item-success{color:#3c763d;background-color:#dff0d8}a.list-group-item-success,button.list-group-item-success{color:#3c763d}a.list-group-item-success .list-group-item-heading,button.list-group-item-success .list-group-item-heading{color:inherit}a.list-group-item-success:focus,a.list-group-item-success:hover,button.list-group-item-success:focus,button.list-group-item-success:hover{color:#3c763d;background-color:#d0e9c6}a.list-group-item-success.active,button.list-group-item-success.active{color:#fff;background-color:#3c763d;border-color:#3c763d}.list-group-item-info{color:#31708f;background-color:#d9edf7}a.list-group-item-info,button.list-group-item-info{color:#31708f}a.list-group-item-info .list-group-item-heading,button.list-group-item-info .list-group-item-heading{color:inherit}a.list-group-item-info:focus,a.list-group-item-info:hover,button.list-group-item-info:focus,button.list-group-item-info:hover{color:#31708f;background-color:#c4e3f3}a.list-group-item-info.active,button.list-group-item-info.active{color:#fff;background-color:#31708f;border-color:#31708f}.list-group-item-warning{color:#8a6d3b;background-color:#fcf8e3}a.list-group-item-warning,button.list-group-item-warning{color:#8a6d3b}a.list-group-item-warning .list-group-item-heading,button.list-group-item-warning .list-group-item-heading{color:inherit}a.list-group-item-warning:focus,a.list-group-item-warning:hover,button.list-group-item-warning:focus,button.list-group-item-warning:hover{color:#8a6d3b;background-color:#faf2cc}a.list-group-item-warning.active,button.list-group-item-warning.active{color:#fff;background-color:#8a6d3b;border-color:#8a6d3b}.list-group-item-danger{color:#a94442;background-color:#f2dede}a.list-group-item-danger,button.list-group-item-danger{color:#a94442}a.list-group-item-danger .list-group-item-heading,button.list-group-item-danger .list-group-item-heading{color:inherit}a.list-group-item-danger:focus,a.list-group-item-danger:hover,button.list-group-item-danger:focus,button.list-group-item-danger:hover{color:#a94442;background-color:#ebcccc}a.list-group-item-danger.active,button.list-group-item-danger.active{color:#fff;background-color:#a94442;border-color:#a94442}.embed-responsive{position:relative;display:block;width:100%;padding:0;overflow:hidden}.embed-responsive::before{display:block;content:""}.embed-responsive .embed-responsive-item,.embed-responsive embed,.embed-responsive iframe,.embed-responsive object,.embed-responsive video{position:absolute;top:0;bottom:0;left:0;width:100%;height:100%;border:0}.embed-responsive-21by9::before{padding-top:42.857143%}.embed-responsive-16by9::before{padding-top:56.25%}.embed-responsive-4by3::before{padding-top:75%}.embed-responsive-1by1::before{padding-top:100%}.close{float:right;font-size:1.5rem;font-weight:700;line-height:1;color:#000;text-shadow:0 1px 0 #fff;opacity:.5}.close:focus,.close:hover{color:#000;text-decoration:none;cursor:pointer;opacity:.75}button.close{padding:0;cursor:pointer;background:0 0;border:0;-webkit-appearance:none}.modal-open{overflow:hidden}.modal{position:fixed;top:0;right:0;bottom:0;left:0;z-index:1050;display:none;overflow:hidden;outline:0}.modal.fade .modal-dialog{-webkit-transition:-webkit-transform .3s ease-out;transition:-webkit-transform .3s ease-out;-o-transition:-o-transform .3s ease-out;transition:transform .3s ease-out;transition:transform .3s ease-out,-webkit-transform .3s ease-out,-o-transform .3s ease-out;-webkit-transform:translate(0,-25%);-o-transform:translate(0,-25%);transform:translate(0,-25%)}.modal.show .modal-dialog{-webkit-transform:translate(0,0);-o-transform:translate(0,0);transform:translate(0,0)}.modal-open .modal{overflow-x:hidden;overflow-y:auto}.modal-dialog{position:relative;width:auto;margin:10px}.modal-content{position:relative;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-ms-flex-direction:column;flex-direction:column;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.2);border-radius:.3rem;outline:0}.modal-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;z-index:1040;background-color:#000}.modal-backdrop.fade{opacity:0}.modal-backdrop.show{opacity:.5}.modal-header{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:justify;-webkit-justify-content:space-between;-ms-flex-pack:justify;justify-content:space-between;padding:15px;border-bottom:1px solid #eceeef}.modal-title{margin-bottom:0;line-height:1.5}.modal-body{position:relative;-webkit-box-flex:1;-webkit-flex:1 1 auto;-ms-flex:1 1 auto;flex:1 1 auto;padding:15px}.modal-footer{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:end;-webkit-justify-content:flex-end;-ms-flex-pack:end;justify-content:flex-end;padding:15px;border-top:1px solid #eceeef}.modal-footer>:not(:first-child){margin-left:.25rem}.modal-footer>:not(:last-child){margin-right:.25rem}.modal-scrollbar-measure{position:absolute;top:-9999px;width:50px;height:50px;overflow:scroll}@media (min-width:576px){.modal-dialog{max-width:500px;margin:30px auto}.modal-sm{max-width:300px}}@media (min-width:992px){.modal-lg{max-width:800px}}.tooltip{position:absolute;z-index:1070;display:block;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-style:normal;font-weight:400;letter-spacing:normal;line-break:auto;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;white-space:normal;word-break:normal;word-spacing:normal;font-size:.875rem;word-wrap:break-word;opacity:0}.tooltip.show{opacity:.9}.tooltip.bs-tether-element-attached-bottom,.tooltip.tooltip-top{padding:5px 0;margin-top:-3px}.tooltip.bs-tether-element-attached-bottom .tooltip-inner::before,.tooltip.tooltip-top .tooltip-inner::before{bottom:0;left:50%;margin-left:-5px;content:"";border-width:5px 5px 0;border-top-color:#000}.tooltip.bs-tether-element-attached-left,.tooltip.tooltip-right{padding:0 5px;margin-left:3px}.tooltip.bs-tether-element-attached-left .tooltip-inner::before,.tooltip.tooltip-right .tooltip-inner::before{top:50%;left:0;margin-top:-5px;content:"";border-width:5px 5px 5px 0;border-right-color:#000}.tooltip.bs-tether-element-attached-top,.tooltip.tooltip-bottom{padding:5px 0;margin-top:3px}.tooltip.bs-tether-element-attached-top .tooltip-inner::before,.tooltip.tooltip-bottom .tooltip-inner::before{top:0;left:50%;margin-left:-5px;content:"";border-width:0 5px 5px;border-bottom-color:#000}.tooltip.bs-tether-element-attached-right,.tooltip.tooltip-left{padding:0 5px;margin-left:-3px}.tooltip.bs-tether-element-attached-right .tooltip-inner::before,.tooltip.tooltip-left .tooltip-inner::before{top:50%;right:0;margin-top:-5px;content:"";border-width:5px 0 5px 5px;border-left-color:#000}.tooltip-inner{max-width:200px;padding:3px 8px;color:#fff;text-align:center;background-color:#000;border-radius:.25rem}.tooltip-inner::before{position:absolute;width:0;height:0;border-color:transparent;border-style:solid}.popover{position:absolute;top:0;left:0;z-index:1060;display:block;max-width:276px;padding:1px;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-style:normal;font-weight:400;letter-spacing:normal;line-break:auto;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;white-space:normal;word-break:normal;word-spacing:normal;font-size:.875rem;word-wrap:break-word;background-color:#fff;-webkit-background-clip:padding-box;background-clip:padding-box;border:1px solid rgba(0,0,0,.2);border-radius:.3rem}.popover.bs-tether-element-attached-bottom,.popover.popover-top{margin-top:-10px}.popover.bs-tether-element-attached-bottom::after,.popover.bs-tether-element-attached-bottom::before,.popover.popover-top::after,.popover.popover-top::before{left:50%;border-bottom-width:0}.popover.bs-tether-element-attached-bottom::before,.popover.popover-top::before{bottom:-11px;margin-left:-11px;border-top-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-bottom::after,.popover.popover-top::after{bottom:-10px;margin-left:-10px;border-top-color:#fff}.popover.bs-tether-element-attached-left,.popover.popover-right{margin-left:10px}.popover.bs-tether-element-attached-left::after,.popover.bs-tether-element-attached-left::before,.popover.popover-right::after,.popover.popover-right::before{top:50%;border-left-width:0}.popover.bs-tether-element-attached-left::before,.popover.popover-right::before{left:-11px;margin-top:-11px;border-right-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-left::after,.popover.popover-right::after{left:-10px;margin-top:-10px;border-right-color:#fff}.popover.bs-tether-element-attached-top,.popover.popover-bottom{margin-top:10px}.popover.bs-tether-element-attached-top::after,.popover.bs-tether-element-attached-top::before,.popover.popover-bottom::after,.popover.popover-bottom::before{left:50%;border-top-width:0}.popover.bs-tether-element-attached-top::before,.popover.popover-bottom::before{top:-11px;margin-left:-11px;border-bottom-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-top::after,.popover.popover-bottom::after{top:-10px;margin-left:-10px;border-bottom-color:#f7f7f7}.popover.bs-tether-element-attached-top .popover-title::before,.popover.popover-bottom .popover-title::before{position:absolute;top:0;left:50%;display:block;width:20px;margin-left:-10px;content:"";border-bottom:1px solid #f7f7f7}.popover.bs-tether-element-attached-right,.popover.popover-left{margin-left:-10px}.popover.bs-tether-element-attached-right::after,.popover.bs-tether-element-attached-right::before,.popover.popover-left::after,.popover.popover-left::before{top:50%;border-right-width:0}.popover.bs-tether-element-attached-right::before,.popover.popover-left::before{right:-11px;margin-top:-11px;border-left-color:rgba(0,0,0,.25)}.popover.bs-tether-element-attached-right::after,.popover.popover-left::after{right:-10px;margin-top:-10px;border-left-color:#fff}.popover-title{padding:8px 14px;margin-bottom:0;font-size:1rem;background-color:#f7f7f7;border-bottom:1px solid #ebebeb;border-top-right-radius:calc(.3rem - 1px);border-top-left-radius:calc(.3rem - 1px)}.popover-title:empty{display:none}.popover-content{padding:9px 14px}.popover::after,.popover::before{position:absolute;display:block;width:0;height:0;border-color:transparent;border-style:solid}.popover::before{content:"";border-width:11px}.popover::after{content:"";border-width:10px}.carousel{position:relative}.carousel-inner{position:relative;width:100%;overflow:hidden}.carousel-item{position:relative;display:none;width:100%}@media (-webkit-transform-3d){.carousel-item{-webkit-transition:-webkit-transform .6s ease-in-out;transition:-webkit-transform .6s ease-in-out;-o-transition:-o-transform .6s ease-in-out;transition:transform .6s ease-in-out;transition:transform .6s ease-in-out,-webkit-transform .6s ease-in-out,-o-transform .6s ease-in-out;-webkit-backface-visibility:hidden;backface-visibility:hidden;-webkit-perspective:1000px;perspective:1000px}}@supports ((-webkit-transform:translate3d(0,0,0)) or (transform:translate3d(0,0,0))){.carousel-item{-webkit-transition:-webkit-transform .6s ease-in-out;transition:-webkit-transform .6s ease-in-out;-o-transition:-o-transform .6s ease-in-out;transition:transform .6s ease-in-out;transition:transform .6s ease-in-out,-webkit-transform .6s ease-in-out,-o-transform .6s ease-in-out;-webkit-backface-visibility:hidden;backface-visibility:hidden;-webkit-perspective:1000px;perspective:1000px}}.carousel-item-next,.carousel-item-prev,.carousel-item.active{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex}.carousel-item-next,.carousel-item-prev{position:absolute;top:0}@media (-webkit-transform-3d){.carousel-item-next.carousel-item-left,.carousel-item-prev.carousel-item-right{-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0)}.active.carousel-item-right,.carousel-item-next{-webkit-transform:translate3d(100%,0,0);transform:translate3d(100%,0,0)}.active.carousel-item-left,.carousel-item-prev{-webkit-transform:translate3d(-100%,0,0);transform:translate3d(-100%,0,0)}}@supports ((-webkit-transform:translate3d(0,0,0)) or (transform:translate3d(0,0,0))){.carousel-item-next.carousel-item-left,.carousel-item-prev.carousel-item-right{-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0)}.active.carousel-item-right,.carousel-item-next{-webkit-transform:translate3d(100%,0,0);transform:translate3d(100%,0,0)}.active.carousel-item-left,.carousel-item-prev{-webkit-transform:translate3d(-100%,0,0);transform:translate3d(-100%,0,0)}}.carousel-control-next,.carousel-control-prev{position:absolute;top:0;bottom:0;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;width:15%;color:#fff;text-align:center;opacity:.5}.carousel-control-next:focus,.carousel-control-next:hover,.carousel-control-prev:focus,.carousel-control-prev:hover{color:#fff;text-decoration:none;outline:0;opacity:.9}.carousel-control-prev{left:0}.carousel-control-next{right:0}.carousel-control-next-icon,.carousel-control-prev-icon{display:inline-block;width:20px;height:20px;background:transparent no-repeat center center;-webkit-background-size:100% 100%;background-size:100% 100%}.carousel-control-prev-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23fff' viewBox='0 0 8 8'%3E%3Cpath d='M4 0l-4 4 4 4 1.5-1.5-2.5-2.5 2.5-2.5-1.5-1.5z'/%3E%3C/svg%3E")}.carousel-control-next-icon{background-image:url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' fill='%23fff' viewBox='0 0 8 8'%3E%3Cpath d='M1.5 0l-1.5 1.5 2.5 2.5-2.5 2.5 1.5 1.5 4-4-4-4z'/%3E%3C/svg%3E")}.carousel-indicators{position:absolute;right:0;bottom:10px;left:0;z-index:15;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-box-pack:center;-webkit-justify-content:center;-ms-flex-pack:center;justify-content:center;padding-left:0;margin-right:15%;margin-left:15%;list-style:none}.carousel-indicators li{position:relative;-webkit-box-flex:1;-webkit-flex:1 0 auto;-ms-flex:1 0 auto;flex:1 0 auto;max-width:30px;height:3px;margin-right:3px;margin-left:3px;text-indent:-999px;cursor:pointer;background-color:rgba(255,255,255,.5)}.carousel-indicators li::before{position:absolute;top:-10px;left:0;display:inline-block;width:100%;height:10px;content:""}.carousel-indicators li::after{position:absolute;bottom:-10px;left:0;display:inline-block;width:100%;height:10px;content:""}.carousel-indicators .active{background-color:#fff}.carousel-caption{position:absolute;right:15%;bottom:20px;left:15%;z-index:10;padding-top:20px;padding-bottom:20px;color:#fff;text-align:center}.align-baseline{vertical-align:baseline!important}.align-top{vertical-align:top!important}.align-middle{vertical-align:middle!important}.align-bottom{vertical-align:bottom!important}.align-text-bottom{vertical-align:text-bottom!important}.align-text-top{vertical-align:text-top!important}.bg-faded{background-color:#f7f7f7}.bg-primary{background-color:#0275d8!important}a.bg-primary:focus,a.bg-primary:hover{background-color:#025aa5!important}.bg-success{background-color:#5cb85c!important}a.bg-success:focus,a.bg-success:hover{background-color:#449d44!important}.bg-info{background-color:#5bc0de!important}a.bg-info:focus,a.bg-info:hover{background-color:#31b0d5!important}.bg-warning{background-color:#f0ad4e!important}a.bg-warning:focus,a.bg-warning:hover{background-color:#ec971f!important}.bg-danger{background-color:#d9534f!important}a.bg-danger:focus,a.bg-danger:hover{background-color:#c9302c!important}.bg-inverse{background-color:#292b2c!important}a.bg-inverse:focus,a.bg-inverse:hover{background-color:#101112!important}.border-0{border:0!important}.border-top-0{border-top:0!important}.border-right-0{border-right:0!important}.border-bottom-0{border-bottom:0!important}.border-left-0{border-left:0!important}.rounded{border-radius:.25rem}.rounded-top{border-top-right-radius:.25rem;border-top-left-radius:.25rem}.rounded-right{border-bottom-right-radius:.25rem;border-top-right-radius:.25rem}.rounded-bottom{border-bottom-right-radius:.25rem;border-bottom-left-radius:.25rem}.rounded-left{border-bottom-left-radius:.25rem;border-top-left-radius:.25rem}.rounded-circle{border-radius:50%}.rounded-0{border-radius:0}.clearfix::after{display:block;content:"";clear:both}.d-none{display:none!important}.d-inline{display:inline!important}.d-inline-block{display:inline-block!important}.d-block{display:block!important}.d-table{display:table!important}.d-table-cell{display:table-cell!important}.d-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}@media (min-width:576px){.d-sm-none{display:none!important}.d-sm-inline{display:inline!important}.d-sm-inline-block{display:inline-block!important}.d-sm-block{display:block!important}.d-sm-table{display:table!important}.d-sm-table-cell{display:table-cell!important}.d-sm-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-sm-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:768px){.d-md-none{display:none!important}.d-md-inline{display:inline!important}.d-md-inline-block{display:inline-block!important}.d-md-block{display:block!important}.d-md-table{display:table!important}.d-md-table-cell{display:table-cell!important}.d-md-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-md-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:992px){.d-lg-none{display:none!important}.d-lg-inline{display:inline!important}.d-lg-inline-block{display:inline-block!important}.d-lg-block{display:block!important}.d-lg-table{display:table!important}.d-lg-table-cell{display:table-cell!important}.d-lg-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-lg-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}@media (min-width:1200px){.d-xl-none{display:none!important}.d-xl-inline{display:inline!important}.d-xl-inline-block{display:inline-block!important}.d-xl-block{display:block!important}.d-xl-table{display:table!important}.d-xl-table-cell{display:table-cell!important}.d-xl-flex{display:-webkit-box!important;display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}.d-xl-inline-flex{display:-webkit-inline-box!important;display:-webkit-inline-flex!important;display:-ms-inline-flexbox!important;display:inline-flex!important}}.flex-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}@media (min-width:576px){.flex-sm-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-sm-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-sm-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-sm-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-sm-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-sm-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-sm-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-sm-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-sm-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-sm-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-sm-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-sm-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-sm-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-sm-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-sm-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-sm-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-sm-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-sm-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-sm-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-sm-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-sm-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-sm-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-sm-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-sm-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-sm-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-sm-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-sm-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-sm-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-sm-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-sm-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-sm-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-sm-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:768px){.flex-md-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-md-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-md-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-md-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-md-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-md-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-md-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-md-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-md-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-md-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-md-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-md-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-md-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-md-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-md-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-md-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-md-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-md-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-md-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-md-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-md-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-md-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-md-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-md-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-md-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-md-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-md-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-md-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-md-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-md-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-md-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-md-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:992px){.flex-lg-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-lg-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-lg-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-lg-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-lg-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-lg-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-lg-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-lg-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-lg-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-lg-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-lg-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-lg-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-lg-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-lg-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-lg-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-lg-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-lg-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-lg-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-lg-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-lg-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-lg-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-lg-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-lg-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-lg-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-lg-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-lg-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-lg-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-lg-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-lg-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-lg-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-lg-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-lg-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}@media (min-width:1200px){.flex-xl-first{-webkit-box-ordinal-group:0;-webkit-order:-1;-ms-flex-order:-1;order:-1}.flex-xl-last{-webkit-box-ordinal-group:2;-webkit-order:1;-ms-flex-order:1;order:1}.flex-xl-unordered{-webkit-box-ordinal-group:1;-webkit-order:0;-ms-flex-order:0;order:0}.flex-xl-row{-webkit-box-orient:horizontal!important;-webkit-box-direction:normal!important;-webkit-flex-direction:row!important;-ms-flex-direction:row!important;flex-direction:row!important}.flex-xl-column{-webkit-box-orient:vertical!important;-webkit-box-direction:normal!important;-webkit-flex-direction:column!important;-ms-flex-direction:column!important;flex-direction:column!important}.flex-xl-row-reverse{-webkit-box-orient:horizontal!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:row-reverse!important;-ms-flex-direction:row-reverse!important;flex-direction:row-reverse!important}.flex-xl-column-reverse{-webkit-box-orient:vertical!important;-webkit-box-direction:reverse!important;-webkit-flex-direction:column-reverse!important;-ms-flex-direction:column-reverse!important;flex-direction:column-reverse!important}.flex-xl-wrap{-webkit-flex-wrap:wrap!important;-ms-flex-wrap:wrap!important;flex-wrap:wrap!important}.flex-xl-nowrap{-webkit-flex-wrap:nowrap!important;-ms-flex-wrap:nowrap!important;flex-wrap:nowrap!important}.flex-xl-wrap-reverse{-webkit-flex-wrap:wrap-reverse!important;-ms-flex-wrap:wrap-reverse!important;flex-wrap:wrap-reverse!important}.justify-content-xl-start{-webkit-box-pack:start!important;-webkit-justify-content:flex-start!important;-ms-flex-pack:start!important;justify-content:flex-start!important}.justify-content-xl-end{-webkit-box-pack:end!important;-webkit-justify-content:flex-end!important;-ms-flex-pack:end!important;justify-content:flex-end!important}.justify-content-xl-center{-webkit-box-pack:center!important;-webkit-justify-content:center!important;-ms-flex-pack:center!important;justify-content:center!important}.justify-content-xl-between{-webkit-box-pack:justify!important;-webkit-justify-content:space-between!important;-ms-flex-pack:justify!important;justify-content:space-between!important}.justify-content-xl-around{-webkit-justify-content:space-around!important;-ms-flex-pack:distribute!important;justify-content:space-around!important}.align-items-xl-start{-webkit-box-align:start!important;-webkit-align-items:flex-start!important;-ms-flex-align:start!important;align-items:flex-start!important}.align-items-xl-end{-webkit-box-align:end!important;-webkit-align-items:flex-end!important;-ms-flex-align:end!important;align-items:flex-end!important}.align-items-xl-center{-webkit-box-align:center!important;-webkit-align-items:center!important;-ms-flex-align:center!important;align-items:center!important}.align-items-xl-baseline{-webkit-box-align:baseline!important;-webkit-align-items:baseline!important;-ms-flex-align:baseline!important;align-items:baseline!important}.align-items-xl-stretch{-webkit-box-align:stretch!important;-webkit-align-items:stretch!important;-ms-flex-align:stretch!important;align-items:stretch!important}.align-content-xl-start{-webkit-align-content:flex-start!important;-ms-flex-line-pack:start!important;align-content:flex-start!important}.align-content-xl-end{-webkit-align-content:flex-end!important;-ms-flex-line-pack:end!important;align-content:flex-end!important}.align-content-xl-center{-webkit-align-content:center!important;-ms-flex-line-pack:center!important;align-content:center!important}.align-content-xl-between{-webkit-align-content:space-between!important;-ms-flex-line-pack:justify!important;align-content:space-between!important}.align-content-xl-around{-webkit-align-content:space-around!important;-ms-flex-line-pack:distribute!important;align-content:space-around!important}.align-content-xl-stretch{-webkit-align-content:stretch!important;-ms-flex-line-pack:stretch!important;align-content:stretch!important}.align-self-xl-auto{-webkit-align-self:auto!important;-ms-flex-item-align:auto!important;-ms-grid-row-align:auto!important;align-self:auto!important}.align-self-xl-start{-webkit-align-self:flex-start!important;-ms-flex-item-align:start!important;align-self:flex-start!important}.align-self-xl-end{-webkit-align-self:flex-end!important;-ms-flex-item-align:end!important;align-self:flex-end!important}.align-self-xl-center{-webkit-align-self:center!important;-ms-flex-item-align:center!important;-ms-grid-row-align:center!important;align-self:center!important}.align-self-xl-baseline{-webkit-align-self:baseline!important;-ms-flex-item-align:baseline!important;align-self:baseline!important}.align-self-xl-stretch{-webkit-align-self:stretch!important;-ms-flex-item-align:stretch!important;-ms-grid-row-align:stretch!important;align-self:stretch!important}}.float-left{float:left!important}.float-right{float:right!important}.float-none{float:none!important}@media (min-width:576px){.float-sm-left{float:left!important}.float-sm-right{float:right!important}.float-sm-none{float:none!important}}@media (min-width:768px){.float-md-left{float:left!important}.float-md-right{float:right!important}.float-md-none{float:none!important}}@media (min-width:992px){.float-lg-left{float:left!important}.float-lg-right{float:right!important}.float-lg-none{float:none!important}}@media (min-width:1200px){.float-xl-left{float:left!important}.float-xl-right{float:right!important}.float-xl-none{float:none!important}}.fixed-top{position:fixed;top:0;right:0;left:0;z-index:1030}.fixed-bottom{position:fixed;right:0;bottom:0;left:0;z-index:1030}.sticky-top{position:-webkit-sticky;position:sticky;top:0;z-index:1030}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.w-25{width:25%!important}.w-50{width:50%!important}.w-75{width:75%!important}.w-100{width:100%!important}.h-25{height:25%!important}.h-50{height:50%!important}.h-75{height:75%!important}.h-100{height:100%!important}.mw-100{max-width:100%!important}.mh-100{max-height:100%!important}.m-0{margin:0 0!important}.mt-0{margin-top:0!important}.mr-0{margin-right:0!important}.mb-0{margin-bottom:0!important}.ml-0{margin-left:0!important}.mx-0{margin-right:0!important;margin-left:0!important}.my-0{margin-top:0!important;margin-bottom:0!important}.m-1{margin:.25rem .25rem!important}.mt-1{margin-top:.25rem!important}.mr-1{margin-right:.25rem!important}.mb-1{margin-bottom:.25rem!important}.ml-1{margin-left:.25rem!important}.mx-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-2{margin:.5rem .5rem!important}.mt-2{margin-top:.5rem!important}.mr-2{margin-right:.5rem!important}.mb-2{margin-bottom:.5rem!important}.ml-2{margin-left:.5rem!important}.mx-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-3{margin:1rem 1rem!important}.mt-3{margin-top:1rem!important}.mr-3{margin-right:1rem!important}.mb-3{margin-bottom:1rem!important}.ml-3{margin-left:1rem!important}.mx-3{margin-right:1rem!important;margin-left:1rem!important}.my-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-4{margin:1.5rem 1.5rem!important}.mt-4{margin-top:1.5rem!important}.mr-4{margin-right:1.5rem!important}.mb-4{margin-bottom:1.5rem!important}.ml-4{margin-left:1.5rem!important}.mx-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-5{margin:3rem 3rem!important}.mt-5{margin-top:3rem!important}.mr-5{margin-right:3rem!important}.mb-5{margin-bottom:3rem!important}.ml-5{margin-left:3rem!important}.mx-5{margin-right:3rem!important;margin-left:3rem!important}.my-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-0{padding:0 0!important}.pt-0{padding-top:0!important}.pr-0{padding-right:0!important}.pb-0{padding-bottom:0!important}.pl-0{padding-left:0!important}.px-0{padding-right:0!important;padding-left:0!important}.py-0{padding-top:0!important;padding-bottom:0!important}.p-1{padding:.25rem .25rem!important}.pt-1{padding-top:.25rem!important}.pr-1{padding-right:.25rem!important}.pb-1{padding-bottom:.25rem!important}.pl-1{padding-left:.25rem!important}.px-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-2{padding:.5rem .5rem!important}.pt-2{padding-top:.5rem!important}.pr-2{padding-right:.5rem!important}.pb-2{padding-bottom:.5rem!important}.pl-2{padding-left:.5rem!important}.px-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-3{padding:1rem 1rem!important}.pt-3{padding-top:1rem!important}.pr-3{padding-right:1rem!important}.pb-3{padding-bottom:1rem!important}.pl-3{padding-left:1rem!important}.px-3{padding-right:1rem!important;padding-left:1rem!important}.py-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-4{padding:1.5rem 1.5rem!important}.pt-4{padding-top:1.5rem!important}.pr-4{padding-right:1.5rem!important}.pb-4{padding-bottom:1.5rem!important}.pl-4{padding-left:1.5rem!important}.px-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-5{padding:3rem 3rem!important}.pt-5{padding-top:3rem!important}.pr-5{padding-right:3rem!important}.pb-5{padding-bottom:3rem!important}.pl-5{padding-left:3rem!important}.px-5{padding-right:3rem!important;padding-left:3rem!important}.py-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-auto{margin:auto!important}.mt-auto{margin-top:auto!important}.mr-auto{margin-right:auto!important}.mb-auto{margin-bottom:auto!important}.ml-auto{margin-left:auto!important}.mx-auto{margin-right:auto!important;margin-left:auto!important}.my-auto{margin-top:auto!important;margin-bottom:auto!important}@media (min-width:576px){.m-sm-0{margin:0 0!important}.mt-sm-0{margin-top:0!important}.mr-sm-0{margin-right:0!important}.mb-sm-0{margin-bottom:0!important}.ml-sm-0{margin-left:0!important}.mx-sm-0{margin-right:0!important;margin-left:0!important}.my-sm-0{margin-top:0!important;margin-bottom:0!important}.m-sm-1{margin:.25rem .25rem!important}.mt-sm-1{margin-top:.25rem!important}.mr-sm-1{margin-right:.25rem!important}.mb-sm-1{margin-bottom:.25rem!important}.ml-sm-1{margin-left:.25rem!important}.mx-sm-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-sm-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-sm-2{margin:.5rem .5rem!important}.mt-sm-2{margin-top:.5rem!important}.mr-sm-2{margin-right:.5rem!important}.mb-sm-2{margin-bottom:.5rem!important}.ml-sm-2{margin-left:.5rem!important}.mx-sm-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-sm-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-sm-3{margin:1rem 1rem!important}.mt-sm-3{margin-top:1rem!important}.mr-sm-3{margin-right:1rem!important}.mb-sm-3{margin-bottom:1rem!important}.ml-sm-3{margin-left:1rem!important}.mx-sm-3{margin-right:1rem!important;margin-left:1rem!important}.my-sm-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-sm-4{margin:1.5rem 1.5rem!important}.mt-sm-4{margin-top:1.5rem!important}.mr-sm-4{margin-right:1.5rem!important}.mb-sm-4{margin-bottom:1.5rem!important}.ml-sm-4{margin-left:1.5rem!important}.mx-sm-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-sm-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-sm-5{margin:3rem 3rem!important}.mt-sm-5{margin-top:3rem!important}.mr-sm-5{margin-right:3rem!important}.mb-sm-5{margin-bottom:3rem!important}.ml-sm-5{margin-left:3rem!important}.mx-sm-5{margin-right:3rem!important;margin-left:3rem!important}.my-sm-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-sm-0{padding:0 0!important}.pt-sm-0{padding-top:0!important}.pr-sm-0{padding-right:0!important}.pb-sm-0{padding-bottom:0!important}.pl-sm-0{padding-left:0!important}.px-sm-0{padding-right:0!important;padding-left:0!important}.py-sm-0{padding-top:0!important;padding-bottom:0!important}.p-sm-1{padding:.25rem .25rem!important}.pt-sm-1{padding-top:.25rem!important}.pr-sm-1{padding-right:.25rem!important}.pb-sm-1{padding-bottom:.25rem!important}.pl-sm-1{padding-left:.25rem!important}.px-sm-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-sm-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-sm-2{padding:.5rem .5rem!important}.pt-sm-2{padding-top:.5rem!important}.pr-sm-2{padding-right:.5rem!important}.pb-sm-2{padding-bottom:.5rem!important}.pl-sm-2{padding-left:.5rem!important}.px-sm-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-sm-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-sm-3{padding:1rem 1rem!important}.pt-sm-3{padding-top:1rem!important}.pr-sm-3{padding-right:1rem!important}.pb-sm-3{padding-bottom:1rem!important}.pl-sm-3{padding-left:1rem!important}.px-sm-3{padding-right:1rem!important;padding-left:1rem!important}.py-sm-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-sm-4{padding:1.5rem 1.5rem!important}.pt-sm-4{padding-top:1.5rem!important}.pr-sm-4{padding-right:1.5rem!important}.pb-sm-4{padding-bottom:1.5rem!important}.pl-sm-4{padding-left:1.5rem!important}.px-sm-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-sm-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-sm-5{padding:3rem 3rem!important}.pt-sm-5{padding-top:3rem!important}.pr-sm-5{padding-right:3rem!important}.pb-sm-5{padding-bottom:3rem!important}.pl-sm-5{padding-left:3rem!important}.px-sm-5{padding-right:3rem!important;padding-left:3rem!important}.py-sm-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-sm-auto{margin:auto!important}.mt-sm-auto{margin-top:auto!important}.mr-sm-auto{margin-right:auto!important}.mb-sm-auto{margin-bottom:auto!important}.ml-sm-auto{margin-left:auto!important}.mx-sm-auto{margin-right:auto!important;margin-left:auto!important}.my-sm-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:768px){.m-md-0{margin:0 0!important}.mt-md-0{margin-top:0!important}.mr-md-0{margin-right:0!important}.mb-md-0{margin-bottom:0!important}.ml-md-0{margin-left:0!important}.mx-md-0{margin-right:0!important;margin-left:0!important}.my-md-0{margin-top:0!important;margin-bottom:0!important}.m-md-1{margin:.25rem .25rem!important}.mt-md-1{margin-top:.25rem!important}.mr-md-1{margin-right:.25rem!important}.mb-md-1{margin-bottom:.25rem!important}.ml-md-1{margin-left:.25rem!important}.mx-md-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-md-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-md-2{margin:.5rem .5rem!important}.mt-md-2{margin-top:.5rem!important}.mr-md-2{margin-right:.5rem!important}.mb-md-2{margin-bottom:.5rem!important}.ml-md-2{margin-left:.5rem!important}.mx-md-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-md-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-md-3{margin:1rem 1rem!important}.mt-md-3{margin-top:1rem!important}.mr-md-3{margin-right:1rem!important}.mb-md-3{margin-bottom:1rem!important}.ml-md-3{margin-left:1rem!important}.mx-md-3{margin-right:1rem!important;margin-left:1rem!important}.my-md-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-md-4{margin:1.5rem 1.5rem!important}.mt-md-4{margin-top:1.5rem!important}.mr-md-4{margin-right:1.5rem!important}.mb-md-4{margin-bottom:1.5rem!important}.ml-md-4{margin-left:1.5rem!important}.mx-md-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-md-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-md-5{margin:3rem 3rem!important}.mt-md-5{margin-top:3rem!important}.mr-md-5{margin-right:3rem!important}.mb-md-5{margin-bottom:3rem!important}.ml-md-5{margin-left:3rem!important}.mx-md-5{margin-right:3rem!important;margin-left:3rem!important}.my-md-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-md-0{padding:0 0!important}.pt-md-0{padding-top:0!important}.pr-md-0{padding-right:0!important}.pb-md-0{padding-bottom:0!important}.pl-md-0{padding-left:0!important}.px-md-0{padding-right:0!important;padding-left:0!important}.py-md-0{padding-top:0!important;padding-bottom:0!important}.p-md-1{padding:.25rem .25rem!important}.pt-md-1{padding-top:.25rem!important}.pr-md-1{padding-right:.25rem!important}.pb-md-1{padding-bottom:.25rem!important}.pl-md-1{padding-left:.25rem!important}.px-md-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-md-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-md-2{padding:.5rem .5rem!important}.pt-md-2{padding-top:.5rem!important}.pr-md-2{padding-right:.5rem!important}.pb-md-2{padding-bottom:.5rem!important}.pl-md-2{padding-left:.5rem!important}.px-md-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-md-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-md-3{padding:1rem 1rem!important}.pt-md-3{padding-top:1rem!important}.pr-md-3{padding-right:1rem!important}.pb-md-3{padding-bottom:1rem!important}.pl-md-3{padding-left:1rem!important}.px-md-3{padding-right:1rem!important;padding-left:1rem!important}.py-md-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-md-4{padding:1.5rem 1.5rem!important}.pt-md-4{padding-top:1.5rem!important}.pr-md-4{padding-right:1.5rem!important}.pb-md-4{padding-bottom:1.5rem!important}.pl-md-4{padding-left:1.5rem!important}.px-md-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-md-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-md-5{padding:3rem 3rem!important}.pt-md-5{padding-top:3rem!important}.pr-md-5{padding-right:3rem!important}.pb-md-5{padding-bottom:3rem!important}.pl-md-5{padding-left:3rem!important}.px-md-5{padding-right:3rem!important;padding-left:3rem!important}.py-md-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-md-auto{margin:auto!important}.mt-md-auto{margin-top:auto!important}.mr-md-auto{margin-right:auto!important}.mb-md-auto{margin-bottom:auto!important}.ml-md-auto{margin-left:auto!important}.mx-md-auto{margin-right:auto!important;margin-left:auto!important}.my-md-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:992px){.m-lg-0{margin:0 0!important}.mt-lg-0{margin-top:0!important}.mr-lg-0{margin-right:0!important}.mb-lg-0{margin-bottom:0!important}.ml-lg-0{margin-left:0!important}.mx-lg-0{margin-right:0!important;margin-left:0!important}.my-lg-0{margin-top:0!important;margin-bottom:0!important}.m-lg-1{margin:.25rem .25rem!important}.mt-lg-1{margin-top:.25rem!important}.mr-lg-1{margin-right:.25rem!important}.mb-lg-1{margin-bottom:.25rem!important}.ml-lg-1{margin-left:.25rem!important}.mx-lg-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-lg-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-lg-2{margin:.5rem .5rem!important}.mt-lg-2{margin-top:.5rem!important}.mr-lg-2{margin-right:.5rem!important}.mb-lg-2{margin-bottom:.5rem!important}.ml-lg-2{margin-left:.5rem!important}.mx-lg-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-lg-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-lg-3{margin:1rem 1rem!important}.mt-lg-3{margin-top:1rem!important}.mr-lg-3{margin-right:1rem!important}.mb-lg-3{margin-bottom:1rem!important}.ml-lg-3{margin-left:1rem!important}.mx-lg-3{margin-right:1rem!important;margin-left:1rem!important}.my-lg-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-lg-4{margin:1.5rem 1.5rem!important}.mt-lg-4{margin-top:1.5rem!important}.mr-lg-4{margin-right:1.5rem!important}.mb-lg-4{margin-bottom:1.5rem!important}.ml-lg-4{margin-left:1.5rem!important}.mx-lg-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-lg-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-lg-5{margin:3rem 3rem!important}.mt-lg-5{margin-top:3rem!important}.mr-lg-5{margin-right:3rem!important}.mb-lg-5{margin-bottom:3rem!important}.ml-lg-5{margin-left:3rem!important}.mx-lg-5{margin-right:3rem!important;margin-left:3rem!important}.my-lg-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-lg-0{padding:0 0!important}.pt-lg-0{padding-top:0!important}.pr-lg-0{padding-right:0!important}.pb-lg-0{padding-bottom:0!important}.pl-lg-0{padding-left:0!important}.px-lg-0{padding-right:0!important;padding-left:0!important}.py-lg-0{padding-top:0!important;padding-bottom:0!important}.p-lg-1{padding:.25rem .25rem!important}.pt-lg-1{padding-top:.25rem!important}.pr-lg-1{padding-right:.25rem!important}.pb-lg-1{padding-bottom:.25rem!important}.pl-lg-1{padding-left:.25rem!important}.px-lg-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-lg-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-lg-2{padding:.5rem .5rem!important}.pt-lg-2{padding-top:.5rem!important}.pr-lg-2{padding-right:.5rem!important}.pb-lg-2{padding-bottom:.5rem!important}.pl-lg-2{padding-left:.5rem!important}.px-lg-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-lg-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-lg-3{padding:1rem 1rem!important}.pt-lg-3{padding-top:1rem!important}.pr-lg-3{padding-right:1rem!important}.pb-lg-3{padding-bottom:1rem!important}.pl-lg-3{padding-left:1rem!important}.px-lg-3{padding-right:1rem!important;padding-left:1rem!important}.py-lg-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-lg-4{padding:1.5rem 1.5rem!important}.pt-lg-4{padding-top:1.5rem!important}.pr-lg-4{padding-right:1.5rem!important}.pb-lg-4{padding-bottom:1.5rem!important}.pl-lg-4{padding-left:1.5rem!important}.px-lg-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-lg-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-lg-5{padding:3rem 3rem!important}.pt-lg-5{padding-top:3rem!important}.pr-lg-5{padding-right:3rem!important}.pb-lg-5{padding-bottom:3rem!important}.pl-lg-5{padding-left:3rem!important}.px-lg-5{padding-right:3rem!important;padding-left:3rem!important}.py-lg-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-lg-auto{margin:auto!important}.mt-lg-auto{margin-top:auto!important}.mr-lg-auto{margin-right:auto!important}.mb-lg-auto{margin-bottom:auto!important}.ml-lg-auto{margin-left:auto!important}.mx-lg-auto{margin-right:auto!important;margin-left:auto!important}.my-lg-auto{margin-top:auto!important;margin-bottom:auto!important}}@media (min-width:1200px){.m-xl-0{margin:0 0!important}.mt-xl-0{margin-top:0!important}.mr-xl-0{margin-right:0!important}.mb-xl-0{margin-bottom:0!important}.ml-xl-0{margin-left:0!important}.mx-xl-0{margin-right:0!important;margin-left:0!important}.my-xl-0{margin-top:0!important;margin-bottom:0!important}.m-xl-1{margin:.25rem .25rem!important}.mt-xl-1{margin-top:.25rem!important}.mr-xl-1{margin-right:.25rem!important}.mb-xl-1{margin-bottom:.25rem!important}.ml-xl-1{margin-left:.25rem!important}.mx-xl-1{margin-right:.25rem!important;margin-left:.25rem!important}.my-xl-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.m-xl-2{margin:.5rem .5rem!important}.mt-xl-2{margin-top:.5rem!important}.mr-xl-2{margin-right:.5rem!important}.mb-xl-2{margin-bottom:.5rem!important}.ml-xl-2{margin-left:.5rem!important}.mx-xl-2{margin-right:.5rem!important;margin-left:.5rem!important}.my-xl-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.m-xl-3{margin:1rem 1rem!important}.mt-xl-3{margin-top:1rem!important}.mr-xl-3{margin-right:1rem!important}.mb-xl-3{margin-bottom:1rem!important}.ml-xl-3{margin-left:1rem!important}.mx-xl-3{margin-right:1rem!important;margin-left:1rem!important}.my-xl-3{margin-top:1rem!important;margin-bottom:1rem!important}.m-xl-4{margin:1.5rem 1.5rem!important}.mt-xl-4{margin-top:1.5rem!important}.mr-xl-4{margin-right:1.5rem!important}.mb-xl-4{margin-bottom:1.5rem!important}.ml-xl-4{margin-left:1.5rem!important}.mx-xl-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.my-xl-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.m-xl-5{margin:3rem 3rem!important}.mt-xl-5{margin-top:3rem!important}.mr-xl-5{margin-right:3rem!important}.mb-xl-5{margin-bottom:3rem!important}.ml-xl-5{margin-left:3rem!important}.mx-xl-5{margin-right:3rem!important;margin-left:3rem!important}.my-xl-5{margin-top:3rem!important;margin-bottom:3rem!important}.p-xl-0{padding:0 0!important}.pt-xl-0{padding-top:0!important}.pr-xl-0{padding-right:0!important}.pb-xl-0{padding-bottom:0!important}.pl-xl-0{padding-left:0!important}.px-xl-0{padding-right:0!important;padding-left:0!important}.py-xl-0{padding-top:0!important;padding-bottom:0!important}.p-xl-1{padding:.25rem .25rem!important}.pt-xl-1{padding-top:.25rem!important}.pr-xl-1{padding-right:.25rem!important}.pb-xl-1{padding-bottom:.25rem!important}.pl-xl-1{padding-left:.25rem!important}.px-xl-1{padding-right:.25rem!important;padding-left:.25rem!important}.py-xl-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.p-xl-2{padding:.5rem .5rem!important}.pt-xl-2{padding-top:.5rem!important}.pr-xl-2{padding-right:.5rem!important}.pb-xl-2{padding-bottom:.5rem!important}.pl-xl-2{padding-left:.5rem!important}.px-xl-2{padding-right:.5rem!important;padding-left:.5rem!important}.py-xl-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.p-xl-3{padding:1rem 1rem!important}.pt-xl-3{padding-top:1rem!important}.pr-xl-3{padding-right:1rem!important}.pb-xl-3{padding-bottom:1rem!important}.pl-xl-3{padding-left:1rem!important}.px-xl-3{padding-right:1rem!important;padding-left:1rem!important}.py-xl-3{padding-top:1rem!important;padding-bottom:1rem!important}.p-xl-4{padding:1.5rem 1.5rem!important}.pt-xl-4{padding-top:1.5rem!important}.pr-xl-4{padding-right:1.5rem!important}.pb-xl-4{padding-bottom:1.5rem!important}.pl-xl-4{padding-left:1.5rem!important}.px-xl-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.py-xl-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.p-xl-5{padding:3rem 3rem!important}.pt-xl-5{padding-top:3rem!important}.pr-xl-5{padding-right:3rem!important}.pb-xl-5{padding-bottom:3rem!important}.pl-xl-5{padding-left:3rem!important}.px-xl-5{padding-right:3rem!important;padding-left:3rem!important}.py-xl-5{padding-top:3rem!important;padding-bottom:3rem!important}.m-xl-auto{margin:auto!important}.mt-xl-auto{margin-top:auto!important}.mr-xl-auto{margin-right:auto!important}.mb-xl-auto{margin-bottom:auto!important}.ml-xl-auto{margin-left:auto!important}.mx-xl-auto{margin-right:auto!important;margin-left:auto!important}.my-xl-auto{margin-top:auto!important;margin-bottom:auto!important}}.text-justify{text-align:justify!important}.text-nowrap{white-space:nowrap!important}.text-truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.text-left{text-align:left!important}.text-right{text-align:right!important}.text-center{text-align:center!important}@media (min-width:576px){.text-sm-left{text-align:left!important}.text-sm-right{text-align:right!important}.text-sm-center{text-align:center!important}}@media (min-width:768px){.text-md-left{text-align:left!important}.text-md-right{text-align:right!important}.text-md-center{text-align:center!important}}@media (min-width:992px){.text-lg-left{text-align:left!important}.text-lg-right{text-align:right!important}.text-lg-center{text-align:center!important}}@media (min-width:1200px){.text-xl-left{text-align:left!important}.text-xl-right{text-align:right!important}.text-xl-center{text-align:center!important}}.text-lowercase{text-transform:lowercase!important}.text-uppercase{text-transform:uppercase!important}.text-capitalize{text-transform:capitalize!important}.font-weight-normal{font-weight:400}.font-weight-bold{font-weight:700}.font-italic{font-style:italic}.text-white{color:#fff!important}.text-muted{color:#636c72!important}a.text-muted:focus,a.text-muted:hover{color:#4b5257!important}.text-primary{color:#0275d8!important}a.text-primary:focus,a.text-primary:hover{color:#025aa5!important}.text-success{color:#5cb85c!important}a.text-success:focus,a.text-success:hover{color:#449d44!important}.text-info{color:#5bc0de!important}a.text-info:focus,a.text-info:hover{color:#31b0d5!important}.text-warning{color:#f0ad4e!important}a.text-warning:focus,a.text-warning:hover{color:#ec971f!important}.text-danger{color:#d9534f!important}a.text-danger:focus,a.text-danger:hover{color:#c9302c!important}.text-gray-dark{color:#292b2c!important}a.text-gray-dark:focus,a.text-gray-dark:hover{color:#101112!important}.text-hide{font:0/0 a;color:transparent;text-shadow:none;background-color:transparent;border:0}.invisible{visibility:hidden!important}.hidden-xs-up{display:none!important}@media (max-width:575px){.hidden-xs-down{display:none!important}}@media (min-width:576px){.hidden-sm-up{display:none!important}}@media (max-width:767px){.hidden-sm-down{display:none!important}}@media (min-width:768px){.hidden-md-up{display:none!important}}@media (max-width:991px){.hidden-md-down{display:none!important}}@media (min-width:992px){.hidden-lg-up{display:none!important}}@media (max-width:1199px){.hidden-lg-down{display:none!important}}@media (min-width:1200px){.hidden-xl-up{display:none!important}}.hidden-xl-down{display:none!important}.visible-print-block{display:none!important}@media print{.visible-print-block{display:block!important}}.visible-print-inline{display:none!important}@media print{.visible-print-inline{display:inline!important}}.visible-print-inline-block{display:none!important}@media print{.visible-print-inline-block{display:inline-block!important}}@media print{.hidden-print{display:none!important}}/*# sourceMappingURL=bootstrap.min.css.map */ \ No newline at end of file diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/external.png b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/external.png new file mode 100644 index 0000000..7e1a5f0 Binary files /dev/null and b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/external.png differ diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.css b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.css new file mode 100644 index 0000000..4303138 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.css @@ -0,0 +1 @@ +table.dataTable{width:100%;margin:0 auto;clear:both;border-collapse:separate;border-spacing:0}table.dataTable thead th,table.dataTable tfoot th{font-weight:bold}table.dataTable thead th,table.dataTable thead td{padding:10px 18px;border-bottom:1px solid #111}table.dataTable thead th:active,table.dataTable thead td:active{outline:none}table.dataTable tfoot th,table.dataTable tfoot td{padding:10px 18px 6px 18px;border-top:1px solid #111}table.dataTable thead .sorting,table.dataTable thead .sorting_asc,table.dataTable thead .sorting_desc,table.dataTable thead .sorting_asc_disabled,table.dataTable thead .sorting_desc_disabled{cursor:pointer;*cursor:hand;background-repeat:no-repeat;background-position:center right}table.dataTable thead .sorting{background-image:url("sort_both.png")}table.dataTable thead .sorting_asc{background-image:url("sort_asc.png")}table.dataTable thead .sorting_desc{background-image:url("sort_desc.png")}table.dataTable thead .sorting_asc_disabled{background-image:url("sort_asc_disabled.png")}table.dataTable thead .sorting_desc_disabled{background-image:url("sort_desc_disabled.png")}table.dataTable tbody tr{background-color:#ffffff}table.dataTable tbody tr.selected{background-color:#B0BED9}table.dataTable tbody th,table.dataTable tbody td{padding:8px 10px}table.dataTable.row-border tbody th,table.dataTable.row-border tbody td,table.dataTable.display tbody th,table.dataTable.display tbody td{border-top:1px solid #ddd}table.dataTable.row-border tbody tr:first-child th,table.dataTable.row-border tbody tr:first-child td,table.dataTable.display tbody tr:first-child th,table.dataTable.display tbody tr:first-child td{border-top:none}table.dataTable.cell-border tbody th,table.dataTable.cell-border tbody td{border-top:1px solid #ddd;border-right:1px solid #ddd}table.dataTable.cell-border tbody tr th:first-child,table.dataTable.cell-border tbody tr td:first-child{border-left:1px solid #ddd}table.dataTable.cell-border tbody tr:first-child th,table.dataTable.cell-border tbody tr:first-child td{border-top:none}table.dataTable.stripe tbody tr.odd,table.dataTable.display tbody tr.odd{background-color:#f9f9f9}table.dataTable.stripe tbody tr.odd.selected,table.dataTable.display tbody tr.odd.selected{background-color:#acbad4}table.dataTable.hover tbody tr:hover,table.dataTable.display tbody tr:hover{background-color:#f6f6f6}table.dataTable.hover tbody tr:hover.selected,table.dataTable.display tbody tr:hover.selected{background-color:#aab7d1}table.dataTable.order-column tbody tr>.sorting_1,table.dataTable.order-column tbody tr>.sorting_2,table.dataTable.order-column tbody tr>.sorting_3,table.dataTable.display tbody tr>.sorting_1,table.dataTable.display tbody tr>.sorting_2,table.dataTable.display tbody tr>.sorting_3{background-color:#fafafa}table.dataTable.order-column tbody tr.selected>.sorting_1,table.dataTable.order-column tbody tr.selected>.sorting_2,table.dataTable.order-column tbody tr.selected>.sorting_3,table.dataTable.display tbody tr.selected>.sorting_1,table.dataTable.display tbody tr.selected>.sorting_2,table.dataTable.display tbody tr.selected>.sorting_3{background-color:#acbad5}table.dataTable.display tbody tr.odd>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd>.sorting_1{background-color:#f1f1f1}table.dataTable.display tbody tr.odd>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd>.sorting_2{background-color:#f3f3f3}table.dataTable.display tbody tr.odd>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd>.sorting_3{background-color:whitesmoke}table.dataTable.display tbody tr.odd.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_1{background-color:#a6b4cd}table.dataTable.display tbody tr.odd.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_2{background-color:#a8b5cf}table.dataTable.display tbody tr.odd.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_3{background-color:#a9b7d1}table.dataTable.display tbody tr.even>.sorting_1,table.dataTable.order-column.stripe tbody tr.even>.sorting_1{background-color:#fafafa}table.dataTable.display tbody tr.even>.sorting_2,table.dataTable.order-column.stripe tbody tr.even>.sorting_2{background-color:#fcfcfc}table.dataTable.display tbody tr.even>.sorting_3,table.dataTable.order-column.stripe tbody tr.even>.sorting_3{background-color:#fefefe}table.dataTable.display tbody tr.even.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_1{background-color:#acbad5}table.dataTable.display tbody tr.even.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_2{background-color:#aebcd6}table.dataTable.display tbody tr.even.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_3{background-color:#afbdd8}table.dataTable.display tbody tr:hover>.sorting_1,table.dataTable.order-column.hover tbody tr:hover>.sorting_1{background-color:#eaeaea}table.dataTable.display tbody tr:hover>.sorting_2,table.dataTable.order-column.hover tbody tr:hover>.sorting_2{background-color:#ececec}table.dataTable.display tbody tr:hover>.sorting_3,table.dataTable.order-column.hover tbody tr:hover>.sorting_3{background-color:#efefef}table.dataTable.display tbody tr:hover.selected>.sorting_1,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_1{background-color:#a2aec7}table.dataTable.display tbody tr:hover.selected>.sorting_2,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_2{background-color:#a3b0c9}table.dataTable.display tbody tr:hover.selected>.sorting_3,table.dataTable.order-column.hover tbody tr:hover.selected>.sorting_3{background-color:#a5b2cb}table.dataTable.no-footer{border-bottom:1px solid #111}table.dataTable.nowrap th,table.dataTable.nowrap td{white-space:nowrap}table.dataTable.compact thead th,table.dataTable.compact thead td{padding:4px 17px 4px 4px}table.dataTable.compact tfoot th,table.dataTable.compact tfoot td{padding:4px}table.dataTable.compact tbody th,table.dataTable.compact tbody td{padding:4px}table.dataTable th.dt-left,table.dataTable td.dt-left{text-align:left}table.dataTable th.dt-center,table.dataTable td.dt-center,table.dataTable td.dataTables_empty{text-align:center}table.dataTable th.dt-right,table.dataTable td.dt-right{text-align:right}table.dataTable th.dt-justify,table.dataTable td.dt-justify{text-align:justify}table.dataTable th.dt-nowrap,table.dataTable td.dt-nowrap{white-space:nowrap}table.dataTable thead th.dt-head-left,table.dataTable thead td.dt-head-left,table.dataTable tfoot th.dt-head-left,table.dataTable tfoot td.dt-head-left{text-align:left}table.dataTable thead th.dt-head-center,table.dataTable thead td.dt-head-center,table.dataTable tfoot th.dt-head-center,table.dataTable tfoot td.dt-head-center{text-align:center}table.dataTable thead th.dt-head-right,table.dataTable thead td.dt-head-right,table.dataTable tfoot th.dt-head-right,table.dataTable tfoot td.dt-head-right{text-align:right}table.dataTable thead th.dt-head-justify,table.dataTable thead td.dt-head-justify,table.dataTable tfoot th.dt-head-justify,table.dataTable tfoot td.dt-head-justify{text-align:justify}table.dataTable thead th.dt-head-nowrap,table.dataTable thead td.dt-head-nowrap,table.dataTable tfoot th.dt-head-nowrap,table.dataTable tfoot td.dt-head-nowrap{white-space:nowrap}table.dataTable tbody th.dt-body-left,table.dataTable tbody td.dt-body-left{text-align:left}table.dataTable tbody th.dt-body-center,table.dataTable tbody td.dt-body-center{text-align:center}table.dataTable tbody th.dt-body-right,table.dataTable tbody td.dt-body-right{text-align:right}table.dataTable tbody th.dt-body-justify,table.dataTable tbody td.dt-body-justify{text-align:justify}table.dataTable tbody th.dt-body-nowrap,table.dataTable tbody td.dt-body-nowrap{white-space:nowrap}table.dataTable,table.dataTable th,table.dataTable td{box-sizing:content-box}.dataTables_wrapper{position:relative;clear:both;*zoom:1;zoom:1}.dataTables_wrapper .dataTables_length{float:left}.dataTables_wrapper .dataTables_filter{float:right;text-align:right}.dataTables_wrapper .dataTables_filter input{margin-left:0.5em}.dataTables_wrapper .dataTables_info{clear:both;float:left;padding-top:0.755em}.dataTables_wrapper .dataTables_paginate{float:right;text-align:right;padding-top:0.25em}.dataTables_wrapper .dataTables_paginate .paginate_button{box-sizing:border-box;display:inline-block;min-width:1.5em;padding:0.5em 1em;margin-left:2px;text-align:center;text-decoration:none !important;cursor:pointer;*cursor:hand;color:#333 !important;border:1px solid transparent;border-radius:2px}.dataTables_wrapper .dataTables_paginate .paginate_button.current,.dataTables_wrapper .dataTables_paginate .paginate_button.current:hover{color:#333 !important;border:1px solid #979797;background-color:white;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #fff), color-stop(100%, #dcdcdc));background:-webkit-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-moz-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-ms-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:-o-linear-gradient(top, #fff 0%, #dcdcdc 100%);background:linear-gradient(to bottom, #fff 0%, #dcdcdc 100%)}.dataTables_wrapper .dataTables_paginate .paginate_button.disabled,.dataTables_wrapper .dataTables_paginate .paginate_button.disabled:hover,.dataTables_wrapper .dataTables_paginate .paginate_button.disabled:active{cursor:default;color:#666 !important;border:1px solid transparent;background:transparent;box-shadow:none}.dataTables_wrapper .dataTables_paginate .paginate_button:hover{color:white !important;border:1px solid #111;background-color:#585858;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #585858), color-stop(100%, #111));background:-webkit-linear-gradient(top, #585858 0%, #111 100%);background:-moz-linear-gradient(top, #585858 0%, #111 100%);background:-ms-linear-gradient(top, #585858 0%, #111 100%);background:-o-linear-gradient(top, #585858 0%, #111 100%);background:linear-gradient(to bottom, #585858 0%, #111 100%)}.dataTables_wrapper .dataTables_paginate .paginate_button:active{outline:none;background-color:#2b2b2b;background:-webkit-gradient(linear, left top, left bottom, color-stop(0%, #2b2b2b), color-stop(100%, #0c0c0c));background:-webkit-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-moz-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-ms-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:-o-linear-gradient(top, #2b2b2b 0%, #0c0c0c 100%);background:linear-gradient(to bottom, #2b2b2b 0%, #0c0c0c 100%);box-shadow:inset 0 0 3px #111}.dataTables_wrapper .dataTables_paginate .ellipsis{padding:0 1em}.dataTables_wrapper .dataTables_processing{position:absolute;top:50%;left:50%;width:100%;height:40px;margin-left:-50%;margin-top:-25px;padding-top:20px;text-align:center;font-size:1.2em;background-color:white;background:-webkit-gradient(linear, left top, right top, color-stop(0%, rgba(255,255,255,0)), color-stop(25%, rgba(255,255,255,0.9)), color-stop(75%, rgba(255,255,255,0.9)), color-stop(100%, rgba(255,255,255,0)));background:-webkit-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-moz-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-ms-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:-o-linear-gradient(left, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%);background:linear-gradient(to right, rgba(255,255,255,0) 0%, rgba(255,255,255,0.9) 25%, rgba(255,255,255,0.9) 75%, rgba(255,255,255,0) 100%)}.dataTables_wrapper .dataTables_length,.dataTables_wrapper .dataTables_filter,.dataTables_wrapper .dataTables_info,.dataTables_wrapper .dataTables_processing,.dataTables_wrapper .dataTables_paginate{color:#333}.dataTables_wrapper .dataTables_scroll{clear:both}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody{*margin-top:-1px;-webkit-overflow-scrolling:touch}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>th,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>td,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>th,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>td{vertical-align:middle}.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>th>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>thead>tr>td>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>th>div.dataTables_sizing,.dataTables_wrapper .dataTables_scroll div.dataTables_scrollBody>table>tbody>tr>td>div.dataTables_sizing{height:0;overflow:hidden;margin:0 !important;padding:0 !important}.dataTables_wrapper.no-footer .dataTables_scrollBody{border-bottom:1px solid #111}.dataTables_wrapper.no-footer div.dataTables_scrollHead table.dataTable,.dataTables_wrapper.no-footer div.dataTables_scrollBody>table{border-bottom:none}.dataTables_wrapper:after{visibility:hidden;display:block;content:"";clear:both;height:0}@media screen and (max-width: 767px){.dataTables_wrapper .dataTables_info,.dataTables_wrapper .dataTables_paginate{float:none;text-align:center}.dataTables_wrapper .dataTables_paginate{margin-top:0.5em}}@media screen and (max-width: 640px){.dataTables_wrapper .dataTables_length,.dataTables_wrapper .dataTables_filter{float:none;text-align:center}.dataTables_wrapper .dataTables_filter{margin-top:0.5em}} diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.js b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.js new file mode 100644 index 0000000..07af1c3 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.dataTables.min.js @@ -0,0 +1,166 @@ +/*! + DataTables 1.10.19 + ©2008-2018 SpryMedia Ltd - datatables.net/license +*/ +(function(h){"function"===typeof define&&define.amd?define(["jquery"],function(E){return h(E,window,document)}):"object"===typeof exports?module.exports=function(E,H){E||(E=window);H||(H="undefined"!==typeof window?require("jquery"):require("jquery")(E));return h(H,E,E.document)}:h(jQuery,window,document)})(function(h,E,H,k){function Z(a){var b,c,d={};h.each(a,function(e){if((b=e.match(/^([^A-Z]+?)([A-Z])/))&&-1!=="a aa ai ao as b fn i m o s ".indexOf(b[1]+" "))c=e.replace(b[0],b[2].toLowerCase()), +d[c]=e,"o"===b[1]&&Z(a[e])});a._hungarianMap=d}function J(a,b,c){a._hungarianMap||Z(a);var d;h.each(b,function(e){d=a._hungarianMap[e];if(d!==k&&(c||b[d]===k))"o"===d.charAt(0)?(b[d]||(b[d]={}),h.extend(!0,b[d],b[e]),J(a[d],b[d],c)):b[d]=b[e]})}function Ca(a){var b=n.defaults.oLanguage,c=b.sDecimal;c&&Da(c);if(a){var d=a.sZeroRecords;!a.sEmptyTable&&(d&&"No data available in table"===b.sEmptyTable)&&F(a,a,"sZeroRecords","sEmptyTable");!a.sLoadingRecords&&(d&&"Loading..."===b.sLoadingRecords)&&F(a, +a,"sZeroRecords","sLoadingRecords");a.sInfoThousands&&(a.sThousands=a.sInfoThousands);(a=a.sDecimal)&&c!==a&&Da(a)}}function fb(a){A(a,"ordering","bSort");A(a,"orderMulti","bSortMulti");A(a,"orderClasses","bSortClasses");A(a,"orderCellsTop","bSortCellsTop");A(a,"order","aaSorting");A(a,"orderFixed","aaSortingFixed");A(a,"paging","bPaginate");A(a,"pagingType","sPaginationType");A(a,"pageLength","iDisplayLength");A(a,"searching","bFilter");"boolean"===typeof a.sScrollX&&(a.sScrollX=a.sScrollX?"100%": +"");"boolean"===typeof a.scrollX&&(a.scrollX=a.scrollX?"100%":"");if(a=a.aoSearchCols)for(var b=0,c=a.length;b").css({position:"fixed",top:0,left:-1*h(E).scrollLeft(),height:1,width:1, +overflow:"hidden"}).append(h("
    ").css({position:"absolute",top:1,left:1,width:100,overflow:"scroll"}).append(h("
    ").css({width:"100%",height:10}))).appendTo("body"),d=c.children(),e=d.children();b.barWidth=d[0].offsetWidth-d[0].clientWidth;b.bScrollOversize=100===e[0].offsetWidth&&100!==d[0].clientWidth;b.bScrollbarLeft=1!==Math.round(e.offset().left);b.bBounding=c[0].getBoundingClientRect().width?!0:!1;c.remove()}h.extend(a.oBrowser,n.__browser);a.oScroll.iBarWidth=n.__browser.barWidth} +function ib(a,b,c,d,e,f){var g,j=!1;c!==k&&(g=c,j=!0);for(;d!==e;)a.hasOwnProperty(d)&&(g=j?b(g,a[d],d,a):a[d],j=!0,d+=f);return g}function Ea(a,b){var c=n.defaults.column,d=a.aoColumns.length,c=h.extend({},n.models.oColumn,c,{nTh:b?b:H.createElement("th"),sTitle:c.sTitle?c.sTitle:b?b.innerHTML:"",aDataSort:c.aDataSort?c.aDataSort:[d],mData:c.mData?c.mData:d,idx:d});a.aoColumns.push(c);c=a.aoPreSearchCols;c[d]=h.extend({},n.models.oSearch,c[d]);ka(a,d,h(b).data())}function ka(a,b,c){var b=a.aoColumns[b], +d=a.oClasses,e=h(b.nTh);if(!b.sWidthOrig){b.sWidthOrig=e.attr("width")||null;var f=(e.attr("style")||"").match(/width:\s*(\d+[pxem%]+)/);f&&(b.sWidthOrig=f[1])}c!==k&&null!==c&&(gb(c),J(n.defaults.column,c),c.mDataProp!==k&&!c.mData&&(c.mData=c.mDataProp),c.sType&&(b._sManualType=c.sType),c.className&&!c.sClass&&(c.sClass=c.className),c.sClass&&e.addClass(c.sClass),h.extend(b,c),F(b,c,"sWidth","sWidthOrig"),c.iDataSort!==k&&(b.aDataSort=[c.iDataSort]),F(b,c,"aDataSort"));var g=b.mData,j=S(g),i=b.mRender? +S(b.mRender):null,c=function(a){return"string"===typeof a&&-1!==a.indexOf("@")};b._bAttrSrc=h.isPlainObject(g)&&(c(g.sort)||c(g.type)||c(g.filter));b._setter=null;b.fnGetData=function(a,b,c){var d=j(a,b,k,c);return i&&b?i(d,b,a,c):d};b.fnSetData=function(a,b,c){return N(g)(a,b,c)};"number"!==typeof g&&(a._rowReadObject=!0);a.oFeatures.bSort||(b.bSortable=!1,e.addClass(d.sSortableNone));a=-1!==h.inArray("asc",b.asSorting);c=-1!==h.inArray("desc",b.asSorting);!b.bSortable||!a&&!c?(b.sSortingClass=d.sSortableNone, +b.sSortingClassJUI=""):a&&!c?(b.sSortingClass=d.sSortableAsc,b.sSortingClassJUI=d.sSortJUIAscAllowed):!a&&c?(b.sSortingClass=d.sSortableDesc,b.sSortingClassJUI=d.sSortJUIDescAllowed):(b.sSortingClass=d.sSortable,b.sSortingClassJUI=d.sSortJUI)}function $(a){if(!1!==a.oFeatures.bAutoWidth){var b=a.aoColumns;Fa(a);for(var c=0,d=b.length;cq[f])d(l.length+q[f],m);else if("string"=== +typeof q[f]){j=0;for(i=l.length;jb&&a[e]--; -1!=d&&c===k&&a.splice(d, +1)}function da(a,b,c,d){var e=a.aoData[b],f,g=function(c,d){for(;c.childNodes.length;)c.removeChild(c.firstChild);c.innerHTML=B(a,b,d,"display")};if("dom"===c||(!c||"auto"===c)&&"dom"===e.src)e._aData=Ia(a,e,d,d===k?k:e._aData).data;else{var j=e.anCells;if(j)if(d!==k)g(j[d],d);else{c=0;for(f=j.length;c").appendTo(g));b=0;for(c=l.length;btr").attr("role","row");h(g).find(">tr>th, >tr>td").addClass(m.sHeaderTH);h(j).find(">tr>th, >tr>td").addClass(m.sFooterTH);if(null!==j){a=a.aoFooter[0];b=0;for(c=a.length;b=a.fnRecordsDisplay()?0:g,a.iInitDisplayStart=-1);var g=a._iDisplayStart,m=a.fnDisplayEnd();if(a.bDeferLoading)a.bDeferLoading=!1,a.iDraw++,C(a,!1);else if(j){if(!a.bDestroying&&!mb(a))return}else a.iDraw++;if(0!==i.length){f=j?a.aoData.length:m;for(j=j?0:g;j",{"class":e?d[0]:""}).append(h("",{valign:"top",colSpan:V(a),"class":a.oClasses.sRowEmpty}).html(c))[0];r(a,"aoHeaderCallback","header",[h(a.nTHead).children("tr")[0],Ka(a),g,m,i]);r(a,"aoFooterCallback","footer",[h(a.nTFoot).children("tr")[0],Ka(a),g,m,i]);d=h(a.nTBody);d.children().detach(); +d.append(h(b));r(a,"aoDrawCallback","draw",[a]);a.bSorted=!1;a.bFiltered=!1;a.bDrawing=!1}}function T(a,b){var c=a.oFeatures,d=c.bFilter;c.bSort&&nb(a);d?ga(a,a.oPreviousSearch):a.aiDisplay=a.aiDisplayMaster.slice();!0!==b&&(a._iDisplayStart=0);a._drawHold=b;P(a);a._drawHold=!1}function ob(a){var b=a.oClasses,c=h(a.nTable),c=h("
    ").insertBefore(c),d=a.oFeatures,e=h("
    ",{id:a.sTableId+"_wrapper","class":b.sWrapper+(a.nTFoot?"":" "+b.sNoFooter)});a.nHolding=c[0];a.nTableWrapper=e[0];a.nTableReinsertBefore= +a.nTable.nextSibling;for(var f=a.sDom.split(""),g,j,i,m,l,q,k=0;k")[0];m=f[k+1];if("'"==m||'"'==m){l="";for(q=2;f[k+q]!=m;)l+=f[k+q],q++;"H"==l?l=b.sJUIHeader:"F"==l&&(l=b.sJUIFooter);-1!=l.indexOf(".")?(m=l.split("."),i.id=m[0].substr(1,m[0].length-1),i.className=m[1]):"#"==l.charAt(0)?i.id=l.substr(1,l.length-1):i.className=l;k+=q}e.append(i);e=h(i)}else if(">"==j)e=e.parent();else if("l"==j&&d.bPaginate&&d.bLengthChange)g=pb(a);else if("f"==j&& +d.bFilter)g=qb(a);else if("r"==j&&d.bProcessing)g=rb(a);else if("t"==j)g=sb(a);else if("i"==j&&d.bInfo)g=tb(a);else if("p"==j&&d.bPaginate)g=ub(a);else if(0!==n.ext.feature.length){i=n.ext.feature;q=0;for(m=i.length;q',j=d.sSearch,j=j.match(/_INPUT_/)?j.replace("_INPUT_", +g):j+g,b=h("
    ",{id:!f.f?c+"_filter":null,"class":b.sFilter}).append(h("
    ").addClass(b.sLength);a.aanFeatures.l||(i[0].id=c+"_length");i.children().append(a.oLanguage.sLengthMenu.replace("_MENU_",e[0].outerHTML));h("select",i).val(a._iDisplayLength).on("change.DT",function(){Ra(a,h(this).val());P(a)});h(a.nTable).on("length.dt.DT",function(b,c,d){a=== +c&&h("select",i).val(d)});return i[0]}function ub(a){var b=a.sPaginationType,c=n.ext.pager[b],d="function"===typeof c,e=function(a){P(a)},b=h("
    ").addClass(a.oClasses.sPaging+b)[0],f=a.aanFeatures;d||c.fnInit(a,b,e);f.p||(b.id=a.sTableId+"_paginate",a.aoDrawCallback.push({fn:function(a){if(d){var b=a._iDisplayStart,i=a._iDisplayLength,h=a.fnRecordsDisplay(),l=-1===i,b=l?0:Math.ceil(b/i),i=l?1:Math.ceil(h/i),h=c(b,i),k,l=0;for(k=f.p.length;lf&&(d=0)):"first"==b?d=0:"previous"==b?(d=0<=e?d-e:0,0>d&&(d=0)):"next"==b?d+e",{id:!a.aanFeatures.r?a.sTableId+"_processing":null,"class":a.oClasses.sProcessing}).html(a.oLanguage.sProcessing).insertBefore(a.nTable)[0]} +function C(a,b){a.oFeatures.bProcessing&&h(a.aanFeatures.r).css("display",b?"block":"none");r(a,null,"processing",[a,b])}function sb(a){var b=h(a.nTable);b.attr("role","grid");var c=a.oScroll;if(""===c.sX&&""===c.sY)return a.nTable;var d=c.sX,e=c.sY,f=a.oClasses,g=b.children("caption"),j=g.length?g[0]._captionSide:null,i=h(b[0].cloneNode(!1)),m=h(b[0].cloneNode(!1)),l=b.children("tfoot");l.length||(l=null);i=h("
    ",{"class":f.sScrollWrapper}).append(h("
    ",{"class":f.sScrollHead}).css({overflow:"hidden", +position:"relative",border:0,width:d?!d?null:v(d):"100%"}).append(h("
    ",{"class":f.sScrollHeadInner}).css({"box-sizing":"content-box",width:c.sXInner||"100%"}).append(i.removeAttr("id").css("margin-left",0).append("top"===j?g:null).append(b.children("thead"))))).append(h("
    ",{"class":f.sScrollBody}).css({position:"relative",overflow:"auto",width:!d?null:v(d)}).append(b));l&&i.append(h("
    ",{"class":f.sScrollFoot}).css({overflow:"hidden",border:0,width:d?!d?null:v(d):"100%"}).append(h("
    ", +{"class":f.sScrollFootInner}).append(m.removeAttr("id").css("margin-left",0).append("bottom"===j?g:null).append(b.children("tfoot")))));var b=i.children(),k=b[0],f=b[1],t=l?b[2]:null;if(d)h(f).on("scroll.DT",function(){var a=this.scrollLeft;k.scrollLeft=a;l&&(t.scrollLeft=a)});h(f).css(e&&c.bCollapse?"max-height":"height",e);a.nScrollHead=k;a.nScrollBody=f;a.nScrollFoot=t;a.aoDrawCallback.push({fn:la,sName:"scrolling"});return i[0]}function la(a){var b=a.oScroll,c=b.sX,d=b.sXInner,e=b.sY,b=b.iBarWidth, +f=h(a.nScrollHead),g=f[0].style,j=f.children("div"),i=j[0].style,m=j.children("table"),j=a.nScrollBody,l=h(j),q=j.style,t=h(a.nScrollFoot).children("div"),n=t.children("table"),o=h(a.nTHead),p=h(a.nTable),s=p[0],r=s.style,u=a.nTFoot?h(a.nTFoot):null,x=a.oBrowser,U=x.bScrollOversize,Xb=D(a.aoColumns,"nTh"),Q,L,R,w,Ua=[],y=[],z=[],A=[],B,C=function(a){a=a.style;a.paddingTop="0";a.paddingBottom="0";a.borderTopWidth="0";a.borderBottomWidth="0";a.height=0};L=j.scrollHeight>j.clientHeight;if(a.scrollBarVis!== +L&&a.scrollBarVis!==k)a.scrollBarVis=L,$(a);else{a.scrollBarVis=L;p.children("thead, tfoot").remove();u&&(R=u.clone().prependTo(p),Q=u.find("tr"),R=R.find("tr"));w=o.clone().prependTo(p);o=o.find("tr");L=w.find("tr");w.find("th, td").removeAttr("tabindex");c||(q.width="100%",f[0].style.width="100%");h.each(ra(a,w),function(b,c){B=aa(a,b);c.style.width=a.aoColumns[B].sWidth});u&&I(function(a){a.style.width=""},R);f=p.outerWidth();if(""===c){r.width="100%";if(U&&(p.find("tbody").height()>j.offsetHeight|| +"scroll"==l.css("overflow-y")))r.width=v(p.outerWidth()-b);f=p.outerWidth()}else""!==d&&(r.width=v(d),f=p.outerWidth());I(C,L);I(function(a){z.push(a.innerHTML);Ua.push(v(h(a).css("width")))},L);I(function(a,b){if(h.inArray(a,Xb)!==-1)a.style.width=Ua[b]},o);h(L).height(0);u&&(I(C,R),I(function(a){A.push(a.innerHTML);y.push(v(h(a).css("width")))},R),I(function(a,b){a.style.width=y[b]},Q),h(R).height(0));I(function(a,b){a.innerHTML='
    '+z[b]+"
    ";a.childNodes[0].style.height= +"0";a.childNodes[0].style.overflow="hidden";a.style.width=Ua[b]},L);u&&I(function(a,b){a.innerHTML='
    '+A[b]+"
    ";a.childNodes[0].style.height="0";a.childNodes[0].style.overflow="hidden";a.style.width=y[b]},R);if(p.outerWidth()j.offsetHeight||"scroll"==l.css("overflow-y")?f+b:f;if(U&&(j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=v(Q-b);(""===c||""!==d)&&K(a,1,"Possible column misalignment",6)}else Q="100%";q.width=v(Q); +g.width=v(Q);u&&(a.nScrollFoot.style.width=v(Q));!e&&U&&(q.height=v(s.offsetHeight+b));c=p.outerWidth();m[0].style.width=v(c);i.width=v(c);d=p.height()>j.clientHeight||"scroll"==l.css("overflow-y");e="padding"+(x.bScrollbarLeft?"Left":"Right");i[e]=d?b+"px":"0px";u&&(n[0].style.width=v(c),t[0].style.width=v(c),t[0].style[e]=d?b+"px":"0px");p.children("colgroup").insertBefore(p.children("thead"));l.scroll();if((a.bSorted||a.bFiltered)&&!a._drawHold)j.scrollTop=0}}function I(a,b,c){for(var d=0,e=0, +f=b.length,g,j;e").appendTo(j.find("tbody"));j.find("thead, tfoot").remove();j.append(h(a.nTHead).clone()).append(h(a.nTFoot).clone());j.find("tfoot th, tfoot td").css("width","");m=ra(a,j.find("thead")[0]);for(n=0;n").css({width:o.sWidthOrig,margin:0,padding:0,border:0,height:1}));if(a.aoData.length)for(n=0;n").css(f||e?{position:"absolute",top:0,left:0,height:1,right:0,overflow:"hidden"}:{}).append(j).appendTo(k);f&&g?j.width(g):f?(j.css("width","auto"),j.removeAttr("width"),j.width()").css("width",v(a)).appendTo(b||H.body),d=c[0].offsetWidth;c.remove();return d}function Gb(a, +b){var c=Hb(a,b);if(0>c)return null;var d=a.aoData[c];return!d.nTr?h("").html(B(a,c,b,"display"))[0]:d.anCells[b]}function Hb(a,b){for(var c,d=-1,e=-1,f=0,g=a.aoData.length;fd&&(d=c.length,e=f);return e}function v(a){return null===a?"0px":"number"==typeof a?0>a?"0px":a+"px":a.match(/\d$/)?a+"px":a}function X(a){var b,c,d=[],e=a.aoColumns,f,g,j,i;b=a.aaSortingFixed;c=h.isPlainObject(b);var m=[];f=function(a){a.length&& +!h.isArray(a[0])?m.push(a):h.merge(m,a)};h.isArray(b)&&f(b);c&&b.pre&&f(b.pre);f(a.aaSorting);c&&b.post&&f(b.post);for(a=0;ae?1:0,0!==c)return"asc"===j.dir?c:-c;c=d[a];e=d[b];return ce?1:0}):i.sort(function(a,b){var c,g,j,i,k=h.length,n=f[a]._aSortData,o=f[b]._aSortData;for(j=0;jg?1:0})}a.bSorted=!0}function Jb(a){for(var b,c,d=a.aoColumns,e=X(a),a=a.oLanguage.oAria,f=0,g=d.length;f/g,"");var i=c.nTh;i.removeAttribute("aria-sort");c.bSortable&&(0e?e+1:3));e=0;for(f=d.length;ee?e+1:3))}a.aLastSort=d}function Ib(a,b){var c=a.aoColumns[b],d=n.ext.order[c.sSortDataType],e;d&&(e=d.call(a.oInstance,a,b,ba(a,b)));for(var f,g=n.ext.type.order[c.sType+"-pre"],j=0,i=a.aoData.length;j=f.length?[0,c[1]]:c)}));b.search!==k&&h.extend(a.oPreviousSearch,Cb(b.search));if(b.columns){d=0;for(e=b.columns.length;d=c&&(b=c-d);b-=b%d;if(-1===d||0>b)b=0;a._iDisplayStart=b}function Na(a,b){var c=a.renderer,d=n.ext.renderer[b];return h.isPlainObject(c)&&c[b]?d[c[b]]||d._:"string"=== +typeof c?d[c]||d._:d._}function y(a){return a.oFeatures.bServerSide?"ssp":a.ajax||a.sAjaxSource?"ajax":"dom"}function ia(a,b){var c=[],c=Lb.numbers_length,d=Math.floor(c/2);b<=c?c=Y(0,b):a<=d?(c=Y(0,c-2),c.push("ellipsis"),c.push(b-1)):(a>=b-1-d?c=Y(b-(c-2),b):(c=Y(a-d+2,a+d-1),c.push("ellipsis"),c.push(b-1)),c.splice(0,0,"ellipsis"),c.splice(0,0,0));c.DT_el="span";return c}function Da(a){h.each({num:function(b){return za(b,a)},"num-fmt":function(b){return za(b,a,Ya)},"html-num":function(b){return za(b, +a,Aa)},"html-num-fmt":function(b){return za(b,a,Aa,Ya)}},function(b,c){x.type.order[b+a+"-pre"]=c;b.match(/^html\-/)&&(x.type.search[b+a]=x.type.search.html)})}function Mb(a){return function(){var b=[ya(this[n.ext.iApiIndex])].concat(Array.prototype.slice.call(arguments));return n.ext.internal[a].apply(this,b)}}var n=function(a){this.$=function(a,b){return this.api(!0).$(a,b)};this._=function(a,b){return this.api(!0).rows(a,b).data()};this.api=function(a){return a?new s(ya(this[x.iApiIndex])):new s(this)}; +this.fnAddData=function(a,b){var c=this.api(!0),d=h.isArray(a)&&(h.isArray(a[0])||h.isPlainObject(a[0]))?c.rows.add(a):c.row.add(a);(b===k||b)&&c.draw();return d.flatten().toArray()};this.fnAdjustColumnSizing=function(a){var b=this.api(!0).columns.adjust(),c=b.settings()[0],d=c.oScroll;a===k||a?b.draw(!1):(""!==d.sX||""!==d.sY)&&la(c)};this.fnClearTable=function(a){var b=this.api(!0).clear();(a===k||a)&&b.draw()};this.fnClose=function(a){this.api(!0).row(a).child.hide()};this.fnDeleteRow=function(a, +b,c){var d=this.api(!0),a=d.rows(a),e=a.settings()[0],h=e.aoData[a[0][0]];a.remove();b&&b.call(this,e,h);(c===k||c)&&d.draw();return h};this.fnDestroy=function(a){this.api(!0).destroy(a)};this.fnDraw=function(a){this.api(!0).draw(a)};this.fnFilter=function(a,b,c,d,e,h){e=this.api(!0);null===b||b===k?e.search(a,c,d,h):e.column(b).search(a,c,d,h);e.draw()};this.fnGetData=function(a,b){var c=this.api(!0);if(a!==k){var d=a.nodeName?a.nodeName.toLowerCase():"";return b!==k||"td"==d||"th"==d?c.cell(a,b).data(): +c.row(a).data()||null}return c.data().toArray()};this.fnGetNodes=function(a){var b=this.api(!0);return a!==k?b.row(a).node():b.rows().nodes().flatten().toArray()};this.fnGetPosition=function(a){var b=this.api(!0),c=a.nodeName.toUpperCase();return"TR"==c?b.row(a).index():"TD"==c||"TH"==c?(a=b.cell(a).index(),[a.row,a.columnVisible,a.column]):null};this.fnIsOpen=function(a){return this.api(!0).row(a).child.isShown()};this.fnOpen=function(a,b,c){return this.api(!0).row(a).child(b,c).show().child()[0]}; +this.fnPageChange=function(a,b){var c=this.api(!0).page(a);(b===k||b)&&c.draw(!1)};this.fnSetColumnVis=function(a,b,c){a=this.api(!0).column(a).visible(b);(c===k||c)&&a.columns.adjust().draw()};this.fnSettings=function(){return ya(this[x.iApiIndex])};this.fnSort=function(a){this.api(!0).order(a).draw()};this.fnSortListener=function(a,b,c){this.api(!0).order.listener(a,b,c)};this.fnUpdate=function(a,b,c,d,e){var h=this.api(!0);c===k||null===c?h.row(b).data(a):h.cell(b,c).data(a);(e===k||e)&&h.columns.adjust(); +(d===k||d)&&h.draw();return 0};this.fnVersionCheck=x.fnVersionCheck;var b=this,c=a===k,d=this.length;c&&(a={});this.oApi=this.internal=x.internal;for(var e in n.ext.internal)e&&(this[e]=Mb(e));this.each(function(){var e={},g=1").appendTo(q)); +p.nTHead=b[0];b=q.children("tbody");b.length===0&&(b=h("").appendTo(q));p.nTBody=b[0];b=q.children("tfoot");if(b.length===0&&a.length>0&&(p.oScroll.sX!==""||p.oScroll.sY!==""))b=h("").appendTo(q);if(b.length===0||b.children().length===0)q.addClass(u.sNoFooter);else if(b.length>0){p.nTFoot=b[0];ea(p.aoFooter,p.nTFoot)}if(g.aaData)for(j=0;j/g,Zb=/^\d{2,4}[\.\/\-]\d{1,2}[\.\/\-]\d{1,2}([T ]{1}\d{1,2}[:\.]\d{2}([\.:]\d{2})?)?$/,$b=RegExp("(\\/|\\.|\\*|\\+|\\?|\\||\\(|\\)|\\[|\\]|\\{|\\}|\\\\|\\$|\\^|\\-)","g"),Ya=/[',$£€¥%\u2009\u202F\u20BD\u20a9\u20BArfkɃΞ]/gi,M=function(a){return!a||!0===a||"-"===a?!0:!1},Ob=function(a){var b=parseInt(a,10);return!isNaN(b)&& +isFinite(a)?b:null},Pb=function(a,b){Za[b]||(Za[b]=RegExp(Qa(b),"g"));return"string"===typeof a&&"."!==b?a.replace(/\./g,"").replace(Za[b],"."):a},$a=function(a,b,c){var d="string"===typeof a;if(M(a))return!0;b&&d&&(a=Pb(a,b));c&&d&&(a=a.replace(Ya,""));return!isNaN(parseFloat(a))&&isFinite(a)},Qb=function(a,b,c){return M(a)?!0:!(M(a)||"string"===typeof a)?null:$a(a.replace(Aa,""),b,c)?!0:null},D=function(a,b,c){var d=[],e=0,f=a.length;if(c!==k)for(;ea.length)){b=a.slice().sort();for(var c=b[0],d=1,e=b.length;d")[0],Wb=va.textContent!==k,Yb= +/<.*?>/g,Oa=n.util.throttle,Sb=[],w=Array.prototype,ac=function(a){var b,c,d=n.settings,e=h.map(d,function(a){return a.nTable});if(a){if(a.nTable&&a.oApi)return[a];if(a.nodeName&&"table"===a.nodeName.toLowerCase())return b=h.inArray(a,e),-1!==b?[d[b]]:null;if(a&&"function"===typeof a.settings)return a.settings().toArray();"string"===typeof a?c=h(a):a instanceof h&&(c=a)}else return[];if(c)return c.map(function(){b=h.inArray(this,e);return-1!==b?d[b]:null}).toArray()};s=function(a,b){if(!(this instanceof +s))return new s(a,b);var c=[],d=function(a){(a=ac(a))&&(c=c.concat(a))};if(h.isArray(a))for(var e=0,f=a.length;ea?new s(b[a],this[a]):null},filter:function(a){var b=[];if(w.filter)b=w.filter.call(this,a,this);else for(var c=0,d=this.length;c").addClass(b),h("td",c).addClass(b).html(a)[0].colSpan=V(d),e.push(c[0]))};f(a,b);c._details&&c._details.detach();c._details=h(e); +c._detailsShow&&c._details.insertAfter(c.nTr)}return this});o(["row().child.show()","row().child().show()"],function(){Ub(this,!0);return this});o(["row().child.hide()","row().child().hide()"],function(){Ub(this,!1);return this});o(["row().child.remove()","row().child().remove()"],function(){db(this);return this});o("row().child.isShown()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]]._detailsShow||!1:!1});var bc=/^([^:]+):(name|visIdx|visible)$/,Vb=function(a,b, +c,d,e){for(var c=[],d=0,f=e.length;d=0?b:g.length+b];if(typeof a==="function"){var e=Ba(c,f);return h.map(g,function(b,f){return a(f,Vb(c,f,0,0,e),i[f])?f:null})}var k=typeof a==="string"?a.match(bc): +"";if(k)switch(k[2]){case "visIdx":case "visible":b=parseInt(k[1],10);if(b<0){var n=h.map(g,function(a,b){return a.bVisible?b:null});return[n[n.length+b]]}return[aa(c,b)];case "name":return h.map(j,function(a,b){return a===k[1]?b:null});default:return[]}if(a.nodeName&&a._DT_CellIndex)return[a._DT_CellIndex.column];b=h(i).filter(a).map(function(){return h.inArray(this,i)}).toArray();if(b.length||!a.nodeName)return b;b=h(a).closest("*[data-dt-column]");return b.length?[b.data("dt-column")]:[]},c,f)}, +1);c.selector.cols=a;c.selector.opts=b;return c});u("columns().header()","column().header()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTh},1)});u("columns().footer()","column().footer()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTf},1)});u("columns().data()","column().data()",function(){return this.iterator("column-rows",Vb,1)});u("columns().dataSrc()","column().dataSrc()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].mData}, +1)});u("columns().cache()","column().cache()",function(a){return this.iterator("column-rows",function(b,c,d,e,f){return ja(b.aoData,f,"search"===a?"_aFilterData":"_aSortData",c)},1)});u("columns().nodes()","column().nodes()",function(){return this.iterator("column-rows",function(a,b,c,d,e){return ja(a.aoData,e,"anCells",b)},1)});u("columns().visible()","column().visible()",function(a,b){var c=this.iterator("column",function(b,c){if(a===k)return b.aoColumns[c].bVisible;var f=b.aoColumns,g=f[c],j=b.aoData, +i,m,l;if(a!==k&&g.bVisible!==a){if(a){var n=h.inArray(!0,D(f,"bVisible"),c+1);i=0;for(m=j.length;id;return!0};n.isDataTable= +n.fnIsDataTable=function(a){var b=h(a).get(0),c=!1;if(a instanceof n.Api)return!0;h.each(n.settings,function(a,e){var f=e.nScrollHead?h("table",e.nScrollHead)[0]:null,g=e.nScrollFoot?h("table",e.nScrollFoot)[0]:null;if(e.nTable===b||f===b||g===b)c=!0});return c};n.tables=n.fnTables=function(a){var b=!1;h.isPlainObject(a)&&(b=a.api,a=a.visible);var c=h.map(n.settings,function(b){if(!a||a&&h(b.nTable).is(":visible"))return b.nTable});return b?new s(c):c};n.camelToHungarian=J;o("$()",function(a,b){var c= +this.rows(b).nodes(),c=h(c);return h([].concat(c.filter(a).toArray(),c.find(a).toArray()))});h.each(["on","one","off"],function(a,b){o(b+"()",function(){var a=Array.prototype.slice.call(arguments);a[0]=h.map(a[0].split(/\s/),function(a){return!a.match(/\.dt\b/)?a+".dt":a}).join(" ");var d=h(this.tables().nodes());d[b].apply(d,a);return this})});o("clear()",function(){return this.iterator("table",function(a){oa(a)})});o("settings()",function(){return new s(this.context,this.context)});o("init()",function(){var a= +this.context;return a.length?a[0].oInit:null});o("data()",function(){return this.iterator("table",function(a){return D(a.aoData,"_aData")}).flatten()});o("destroy()",function(a){a=a||!1;return this.iterator("table",function(b){var c=b.nTableWrapper.parentNode,d=b.oClasses,e=b.nTable,f=b.nTBody,g=b.nTHead,j=b.nTFoot,i=h(e),f=h(f),k=h(b.nTableWrapper),l=h.map(b.aoData,function(a){return a.nTr}),o;b.bDestroying=!0;r(b,"aoDestroyCallback","destroy",[b]);a||(new s(b)).columns().visible(!0);k.off(".DT").find(":not(tbody *)").off(".DT"); +h(E).off(".DT-"+b.sInstance);e!=g.parentNode&&(i.children("thead").detach(),i.append(g));j&&e!=j.parentNode&&(i.children("tfoot").detach(),i.append(j));b.aaSorting=[];b.aaSortingFixed=[];wa(b);h(l).removeClass(b.asStripeClasses.join(" "));h("th, td",g).removeClass(d.sSortable+" "+d.sSortableAsc+" "+d.sSortableDesc+" "+d.sSortableNone);f.children().detach();f.append(l);g=a?"remove":"detach";i[g]();k[g]();!a&&c&&(c.insertBefore(e,b.nTableReinsertBefore),i.css("width",b.sDestroyWidth).removeClass(d.sTable), +(o=b.asDestroyStripes.length)&&f.children().each(function(a){h(this).addClass(b.asDestroyStripes[a%o])}));c=h.inArray(b,n.settings);-1!==c&&n.settings.splice(c,1)})});h.each(["column","row","cell"],function(a,b){o(b+"s().every()",function(a){var d=this.selector.opts,e=this;return this.iterator(b,function(f,g,h,i,m){a.call(e[b](g,"cell"===b?h:d,"cell"===b?d:k),g,h,i,m)})})});o("i18n()",function(a,b,c){var d=this.context[0],a=S(a)(d.oLanguage);a===k&&(a=b);c!==k&&h.isPlainObject(a)&&(a=a[c]!==k?a[c]: +a._);return a.replace("%d",c)});n.version="1.10.19";n.settings=[];n.models={};n.models.oSearch={bCaseInsensitive:!0,sSearch:"",bRegex:!1,bSmart:!0};n.models.oRow={nTr:null,anCells:null,_aData:[],_aSortData:null,_aFilterData:null,_sFilterRow:null,_sRowStripe:"",src:null,idx:-1};n.models.oColumn={idx:null,aDataSort:null,asSorting:null,bSearchable:null,bSortable:null,bVisible:null,_sManualType:null,_bAttrSrc:!1,fnCreatedCell:null,fnGetData:null,fnSetData:null,mData:null,mRender:null,nTh:null,nTf:null, +sClass:null,sContentPadding:null,sDefaultContent:null,sName:null,sSortDataType:"std",sSortingClass:null,sSortingClassJUI:null,sTitle:null,sType:null,sWidth:null,sWidthOrig:null};n.defaults={aaData:null,aaSorting:[[0,"asc"]],aaSortingFixed:[],ajax:null,aLengthMenu:[10,25,50,100],aoColumns:null,aoColumnDefs:null,aoSearchCols:[],asStripeClasses:null,bAutoWidth:!0,bDeferRender:!1,bDestroy:!1,bFilter:!0,bInfo:!0,bLengthChange:!0,bPaginate:!0,bProcessing:!1,bRetrieve:!1,bScrollCollapse:!1,bServerSide:!1, +bSort:!0,bSortMulti:!0,bSortCellsTop:!1,bSortClasses:!0,bStateSave:!1,fnCreatedRow:null,fnDrawCallback:null,fnFooterCallback:null,fnFormatNumber:function(a){return a.toString().replace(/\B(?=(\d{3})+(?!\d))/g,this.oLanguage.sThousands)},fnHeaderCallback:null,fnInfoCallback:null,fnInitComplete:null,fnPreDrawCallback:null,fnRowCallback:null,fnServerData:null,fnServerParams:null,fnStateLoadCallback:function(a){try{return JSON.parse((-1===a.iStateDuration?sessionStorage:localStorage).getItem("DataTables_"+ +a.sInstance+"_"+location.pathname))}catch(b){}},fnStateLoadParams:null,fnStateLoaded:null,fnStateSaveCallback:function(a,b){try{(-1===a.iStateDuration?sessionStorage:localStorage).setItem("DataTables_"+a.sInstance+"_"+location.pathname,JSON.stringify(b))}catch(c){}},fnStateSaveParams:null,iStateDuration:7200,iDeferLoading:null,iDisplayLength:10,iDisplayStart:0,iTabIndex:0,oClasses:{},oLanguage:{oAria:{sSortAscending:": activate to sort column ascending",sSortDescending:": activate to sort column descending"}, +oPaginate:{sFirst:"First",sLast:"Last",sNext:"Next",sPrevious:"Previous"},sEmptyTable:"No data available in table",sInfo:"Showing _START_ to _END_ of _TOTAL_ entries",sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({}, +n.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",renderer:null,rowId:"DT_RowId"};Z(n.defaults);n.defaults.column={aDataSort:null,iDataSort:-1,asSorting:["asc","desc"],bSearchable:!0,bSortable:!0,bVisible:!0,fnCreatedCell:null,mData:null,mRender:null,sCellType:"td",sClass:"",sContentPadding:"",sDefaultContent:null,sName:"",sSortDataType:"std",sTitle:null,sType:null,sWidth:null}; +Z(n.defaults.column);n.models.oSettings={oFeatures:{bAutoWidth:null,bDeferRender:null,bFilter:null,bInfo:null,bLengthChange:null,bPaginate:null,bProcessing:null,bServerSide:null,bSort:null,bSortMulti:null,bSortClasses:null,bStateSave:null},oScroll:{bCollapse:null,iBarWidth:0,sX:null,sXInner:null,sY:null},oLanguage:{fnInfoCallback:null},oBrowser:{bScrollOversize:!1,bScrollbarLeft:!1,bBounding:!1,barWidth:0},ajax:null,aanFeatures:[],aoData:[],aiDisplay:[],aiDisplayMaster:[],aIds:{},aoColumns:[],aoHeader:[], +aoFooter:[],oPreviousSearch:{},aoPreSearchCols:[],aaSorting:null,aaSortingFixed:[],asStripeClasses:null,asDestroyStripes:[],sDestroyWidth:0,aoRowCallback:[],aoHeaderCallback:[],aoFooterCallback:[],aoDrawCallback:[],aoRowCreatedCallback:[],aoPreDrawCallback:[],aoInitComplete:[],aoStateSaveParams:[],aoStateLoadParams:[],aoStateLoaded:[],sTableId:"",nTable:null,nTHead:null,nTFoot:null,nTBody:null,nTableWrapper:null,bDeferLoading:!1,bInitialised:!1,aoOpenRows:[],sDom:null,searchDelay:null,sPaginationType:"two_button", +iStateDuration:0,aoStateSave:[],aoStateLoad:[],oSavedState:null,oLoadedState:null,sAjaxSource:null,sAjaxDataProp:null,bAjaxDataGet:!0,jqXHR:null,json:k,oAjaxData:k,fnServerData:null,aoServerParams:[],sServerMethod:null,fnFormatNumber:null,aLengthMenu:null,iDraw:0,bDrawing:!1,iDrawError:-1,_iDisplayLength:10,_iDisplayStart:0,_iRecordsTotal:0,_iRecordsDisplay:0,oClasses:{},bFiltered:!1,bSorted:!1,bSortCellsTop:null,oInit:null,aoDestroyCallback:[],fnRecordsTotal:function(){return"ssp"==y(this)?1*this._iRecordsTotal: +this.aiDisplayMaster.length},fnRecordsDisplay:function(){return"ssp"==y(this)?1*this._iRecordsDisplay:this.aiDisplay.length},fnDisplayEnd:function(){var a=this._iDisplayLength,b=this._iDisplayStart,c=b+a,d=this.aiDisplay.length,e=this.oFeatures,f=e.bPaginate;return e.bServerSide?!1===f||-1===a?b+d:Math.min(b+a,this._iRecordsDisplay):!f||c>d||-1===a?d:c},oInstance:null,sInstance:null,iTabIndex:0,nScrollHead:null,nScrollFoot:null,aLastSort:[],oPlugins:{},rowIdFn:null,rowId:null};n.ext=x={buttons:{}, +classes:{},builder:"-source-",errMode:"alert",feature:[],search:[],selector:{cell:[],column:[],row:[]},internal:{},legacy:{ajax:null},pager:{},renderer:{pageButton:{},header:{}},order:{},type:{detect:[],search:{},order:{}},_unique:0,fnVersionCheck:n.fnVersionCheck,iApiIndex:0,oJUIClasses:{},sVersion:n.version};h.extend(x,{afnFiltering:x.search,aTypes:x.type.detect,ofnSearch:x.type.search,oSort:x.type.order,afnSortData:x.order,aoFeatures:x.feature,oApi:x.internal,oStdClasses:x.classes,oPagination:x.pager}); +h.extend(n.ext.classes,{sTable:"dataTable",sNoFooter:"no-footer",sPageButton:"paginate_button",sPageButtonActive:"current",sPageButtonDisabled:"disabled",sStripeOdd:"odd",sStripeEven:"even",sRowEmpty:"dataTables_empty",sWrapper:"dataTables_wrapper",sFilter:"dataTables_filter",sInfo:"dataTables_info",sPaging:"dataTables_paginate paging_",sLength:"dataTables_length",sProcessing:"dataTables_processing",sSortAsc:"sorting_asc",sSortDesc:"sorting_desc",sSortable:"sorting",sSortableAsc:"sorting_asc_disabled", +sSortableDesc:"sorting_desc_disabled",sSortableNone:"sorting_disabled",sSortColumn:"sorting_",sFilterInput:"",sLengthSelect:"",sScrollWrapper:"dataTables_scroll",sScrollHead:"dataTables_scrollHead",sScrollHeadInner:"dataTables_scrollHeadInner",sScrollBody:"dataTables_scrollBody",sScrollFoot:"dataTables_scrollFoot",sScrollFootInner:"dataTables_scrollFootInner",sHeaderTH:"",sFooterTH:"",sSortJUIAsc:"",sSortJUIDesc:"",sSortJUI:"",sSortJUIAscAllowed:"",sSortJUIDescAllowed:"",sSortJUIWrapper:"",sSortIcon:"", +sJUIHeader:"",sJUIFooter:""});var Lb=n.ext.pager;h.extend(Lb,{simple:function(){return["previous","next"]},full:function(){return["first","previous","next","last"]},numbers:function(a,b){return[ia(a,b)]},simple_numbers:function(a,b){return["previous",ia(a,b),"next"]},full_numbers:function(a,b){return["first","previous",ia(a,b),"next","last"]},first_last_numbers:function(a,b){return["first",ia(a,b),"last"]},_numbers:ia,numbers_length:7});h.extend(!0,n.ext.renderer,{pageButton:{_:function(a,b,c,d,e, +f){var g=a.oClasses,j=a.oLanguage.oPaginate,i=a.oLanguage.oAria.paginate||{},m,l,n=0,o=function(b,d){var k,s,u,r,v=function(b){Ta(a,b.data.action,true)};k=0;for(s=d.length;k").appendTo(b);o(u,r)}else{m=null;l="";switch(r){case "ellipsis":b.append('…');break;case "first":m=j.sFirst;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "previous":m=j.sPrevious;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "next":m= +j.sNext;l=r+(e",{"class":g.sPageButton+" "+l,"aria-controls":a.sTableId,"aria-label":i[r],"data-dt-idx":n,tabindex:a.iTabIndex,id:c===0&&typeof r==="string"?a.sTableId+"_"+r:null}).html(m).appendTo(b);Wa(u,{action:r},v);n++}}}},s;try{s=h(b).find(H.activeElement).data("dt-idx")}catch(u){}o(h(b).empty(),d);s!==k&&h(b).find("[data-dt-idx="+ +s+"]").focus()}}});h.extend(n.ext.type.detect,[function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c)?"num"+c:null},function(a){if(a&&!(a instanceof Date)&&!Zb.test(a))return null;var b=Date.parse(a);return null!==b&&!isNaN(b)||M(a)?"date":null},function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c,!0)?"num-fmt"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c)?"html-num"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c,!0)?"html-num-fmt"+c:null},function(a){return M(a)|| +"string"===typeof a&&-1!==a.indexOf("<")?"html":null}]);h.extend(n.ext.type.search,{html:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," ").replace(Aa,""):""},string:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," "):a}});var za=function(a,b,c,d){if(0!==a&&(!a||"-"===a))return-Infinity;b&&(a=Pb(a,b));a.replace&&(c&&(a=a.replace(c,"")),d&&(a=a.replace(d,"")));return 1*a};h.extend(x.type.order,{"date-pre":function(a){a=Date.parse(a);return isNaN(a)?-Infinity:a},"html-pre":function(a){return M(a)? +"":a.replace?a.replace(/<.*?>/g,"").toLowerCase():a+""},"string-pre":function(a){return M(a)?"":"string"===typeof a?a.toLowerCase():!a.toString?"":a.toString()},"string-asc":function(a,b){return ab?1:0},"string-desc":function(a,b){return ab?-1:0}});Da("");h.extend(!0,n.ext.renderer,{header:{_:function(a,b,c,d){h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(c.sSortingClass+" "+d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc: +c.sSortingClass)}})},jqueryui:function(a,b,c,d){h("
    ").addClass(d.sSortJUIWrapper).append(b.contents()).append(h("").addClass(d.sSortIcon+" "+c.sSortingClassJUI)).appendTo(b);h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass);b.find("span."+d.sSortIcon).removeClass(d.sSortJUIAsc+" "+d.sSortJUIDesc+" "+d.sSortJUI+" "+d.sSortJUIAscAllowed+" "+d.sSortJUIDescAllowed).addClass(h[e]== +"asc"?d.sSortJUIAsc:h[e]=="desc"?d.sSortJUIDesc:c.sSortingClassJUI)}})}}});var eb=function(a){return"string"===typeof a?a.replace(//g,">").replace(/"/g,"""):a};n.render={number:function(a,b,c,d,e){return{display:function(f){if("number"!==typeof f&&"string"!==typeof f)return f;var g=0>f?"-":"",h=parseFloat(f);if(isNaN(h))return eb(f);h=h.toFixed(c);f=Math.abs(h);h=parseInt(f,10);f=c?b+(f-h).toFixed(c).substring(2):"";return g+(d||"")+h.toString().replace(/\B(?=(\d{3})+(?!\d))/g, +a)+f+(e||"")}}},text:function(){return{display:eb,filter:eb}}};h.extend(n.ext.internal,{_fnExternApiFunc:Mb,_fnBuildAjax:sa,_fnAjaxUpdate:mb,_fnAjaxParameters:vb,_fnAjaxUpdateDraw:wb,_fnAjaxDataSrc:ta,_fnAddColumn:Ea,_fnColumnOptions:ka,_fnAdjustColumnSizing:$,_fnVisibleToColumnIndex:aa,_fnColumnIndexToVisible:ba,_fnVisbleColumns:V,_fnGetColumns:ma,_fnColumnTypes:Ga,_fnApplyColumnDefs:jb,_fnHungarianMap:Z,_fnCamelToHungarian:J,_fnLanguageCompat:Ca,_fnBrowserDetect:hb,_fnAddData:O,_fnAddTr:na,_fnNodeToDataIndex:function(a, +b){return b._DT_RowIndex!==k?b._DT_RowIndex:null},_fnNodeToColumnIndex:function(a,b,c){return h.inArray(c,a.aoData[b].anCells)},_fnGetCellData:B,_fnSetCellData:kb,_fnSplitObjNotation:Ja,_fnGetObjectDataFn:S,_fnSetObjectDataFn:N,_fnGetDataMaster:Ka,_fnClearTable:oa,_fnDeleteIndex:pa,_fnInvalidate:da,_fnGetRowElements:Ia,_fnCreateTr:Ha,_fnBuildHead:lb,_fnDrawHead:fa,_fnDraw:P,_fnReDraw:T,_fnAddOptionsHtml:ob,_fnDetectHeader:ea,_fnGetUniqueThs:ra,_fnFeatureHtmlFilter:qb,_fnFilterComplete:ga,_fnFilterCustom:zb, +_fnFilterColumn:yb,_fnFilter:xb,_fnFilterCreateSearch:Pa,_fnEscapeRegex:Qa,_fnFilterData:Ab,_fnFeatureHtmlInfo:tb,_fnUpdateInfo:Db,_fnInfoMacros:Eb,_fnInitialise:ha,_fnInitComplete:ua,_fnLengthChange:Ra,_fnFeatureHtmlLength:pb,_fnFeatureHtmlPaginate:ub,_fnPageChange:Ta,_fnFeatureHtmlProcessing:rb,_fnProcessingDisplay:C,_fnFeatureHtmlTable:sb,_fnScrollDraw:la,_fnApplyToChildren:I,_fnCalculateColumnWidths:Fa,_fnThrottle:Oa,_fnConvertToWidth:Fb,_fnGetWidestNode:Gb,_fnGetMaxLenString:Hb,_fnStringToCss:v, +_fnSortFlatten:X,_fnSort:nb,_fnSortAria:Jb,_fnSortListener:Va,_fnSortAttachListener:Ma,_fnSortingClasses:wa,_fnSortData:Ib,_fnSaveState:xa,_fnLoadState:Kb,_fnSettingsFromNode:ya,_fnLog:K,_fnMap:F,_fnBindAction:Wa,_fnCallbackReg:z,_fnCallbackFire:r,_fnLengthOverflow:Sa,_fnRenderer:Na,_fnDataSource:y,_fnRowAttributes:La,_fnExtend:Xa,_fnCalculateEnd:function(){}});h.fn.dataTable=n;n.$=h;h.fn.dataTableSettings=n.settings;h.fn.dataTableExt=n.ext;h.fn.DataTable=function(a){return h(this).dataTable(a).api()}; +h.each(n,function(a,b){h.fn.DataTable[a]=b});return h.fn.dataTable}); diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.min.js b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.min.js new file mode 100644 index 0000000..4d9b3a2 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/default/static/jquery.min.js @@ -0,0 +1,2 @@ +/*! jQuery v3.3.1 | (c) JS Foundation and other contributors | jquery.org/license */ +!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(e,t){"use strict";var n=[],r=e.document,i=Object.getPrototypeOf,o=n.slice,a=n.concat,s=n.push,u=n.indexOf,l={},c=l.toString,f=l.hasOwnProperty,p=f.toString,d=p.call(Object),h={},g=function e(t){return"function"==typeof t&&"number"!=typeof t.nodeType},y=function e(t){return null!=t&&t===t.window},v={type:!0,src:!0,noModule:!0};function m(e,t,n){var i,o=(t=t||r).createElement("script");if(o.text=e,n)for(i in v)n[i]&&(o[i]=n[i]);t.head.appendChild(o).parentNode.removeChild(o)}function x(e){return null==e?e+"":"object"==typeof e||"function"==typeof e?l[c.call(e)]||"object":typeof e}var b="3.3.1",w=function(e,t){return new w.fn.init(e,t)},T=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g;w.fn=w.prototype={jquery:"3.3.1",constructor:w,length:0,toArray:function(){return o.call(this)},get:function(e){return null==e?o.call(this):e<0?this[e+this.length]:this[e]},pushStack:function(e){var t=w.merge(this.constructor(),e);return t.prevObject=this,t},each:function(e){return w.each(this,e)},map:function(e){return this.pushStack(w.map(this,function(t,n){return e.call(t,n,t)}))},slice:function(){return this.pushStack(o.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(e){var t=this.length,n=+e+(e<0?t:0);return this.pushStack(n>=0&&n0&&t-1 in e)}var E=function(e){var t,n,r,i,o,a,s,u,l,c,f,p,d,h,g,y,v,m,x,b="sizzle"+1*new Date,w=e.document,T=0,C=0,E=ae(),k=ae(),S=ae(),D=function(e,t){return e===t&&(f=!0),0},N={}.hasOwnProperty,A=[],j=A.pop,q=A.push,L=A.push,H=A.slice,O=function(e,t){for(var n=0,r=e.length;n+~]|"+M+")"+M+"*"),z=new RegExp("="+M+"*([^\\]'\"]*?)"+M+"*\\]","g"),X=new RegExp(W),U=new RegExp("^"+R+"$"),V={ID:new RegExp("^#("+R+")"),CLASS:new RegExp("^\\.("+R+")"),TAG:new RegExp("^("+R+"|[*])"),ATTR:new RegExp("^"+I),PSEUDO:new RegExp("^"+W),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+M+"*(even|odd|(([+-]|)(\\d*)n|)"+M+"*(?:([+-]|)"+M+"*(\\d+)|))"+M+"*\\)|)","i"),bool:new RegExp("^(?:"+P+")$","i"),needsContext:new RegExp("^"+M+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+M+"*((?:-\\d)?\\d*)"+M+"*\\)|)(?=[^-]|$)","i")},G=/^(?:input|select|textarea|button)$/i,Y=/^h\d$/i,Q=/^[^{]+\{\s*\[native \w/,J=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,K=/[+~]/,Z=new RegExp("\\\\([\\da-f]{1,6}"+M+"?|("+M+")|.)","ig"),ee=function(e,t,n){var r="0x"+t-65536;return r!==r||n?t:r<0?String.fromCharCode(r+65536):String.fromCharCode(r>>10|55296,1023&r|56320)},te=/([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g,ne=function(e,t){return t?"\0"===e?"\ufffd":e.slice(0,-1)+"\\"+e.charCodeAt(e.length-1).toString(16)+" ":"\\"+e},re=function(){p()},ie=me(function(e){return!0===e.disabled&&("form"in e||"label"in e)},{dir:"parentNode",next:"legend"});try{L.apply(A=H.call(w.childNodes),w.childNodes),A[w.childNodes.length].nodeType}catch(e){L={apply:A.length?function(e,t){q.apply(e,H.call(t))}:function(e,t){var n=e.length,r=0;while(e[n++]=t[r++]);e.length=n-1}}}function oe(e,t,r,i){var o,s,l,c,f,h,v,m=t&&t.ownerDocument,T=t?t.nodeType:9;if(r=r||[],"string"!=typeof e||!e||1!==T&&9!==T&&11!==T)return r;if(!i&&((t?t.ownerDocument||t:w)!==d&&p(t),t=t||d,g)){if(11!==T&&(f=J.exec(e)))if(o=f[1]){if(9===T){if(!(l=t.getElementById(o)))return r;if(l.id===o)return r.push(l),r}else if(m&&(l=m.getElementById(o))&&x(t,l)&&l.id===o)return r.push(l),r}else{if(f[2])return L.apply(r,t.getElementsByTagName(e)),r;if((o=f[3])&&n.getElementsByClassName&&t.getElementsByClassName)return L.apply(r,t.getElementsByClassName(o)),r}if(n.qsa&&!S[e+" "]&&(!y||!y.test(e))){if(1!==T)m=t,v=e;else if("object"!==t.nodeName.toLowerCase()){(c=t.getAttribute("id"))?c=c.replace(te,ne):t.setAttribute("id",c=b),s=(h=a(e)).length;while(s--)h[s]="#"+c+" "+ve(h[s]);v=h.join(","),m=K.test(e)&&ge(t.parentNode)||t}if(v)try{return L.apply(r,m.querySelectorAll(v)),r}catch(e){}finally{c===b&&t.removeAttribute("id")}}}return u(e.replace(B,"$1"),t,r,i)}function ae(){var e=[];function t(n,i){return e.push(n+" ")>r.cacheLength&&delete t[e.shift()],t[n+" "]=i}return t}function se(e){return e[b]=!0,e}function ue(e){var t=d.createElement("fieldset");try{return!!e(t)}catch(e){return!1}finally{t.parentNode&&t.parentNode.removeChild(t),t=null}}function le(e,t){var n=e.split("|"),i=n.length;while(i--)r.attrHandle[n[i]]=t}function ce(e,t){var n=t&&e,r=n&&1===e.nodeType&&1===t.nodeType&&e.sourceIndex-t.sourceIndex;if(r)return r;if(n)while(n=n.nextSibling)if(n===t)return-1;return e?1:-1}function fe(e){return function(t){return"input"===t.nodeName.toLowerCase()&&t.type===e}}function pe(e){return function(t){var n=t.nodeName.toLowerCase();return("input"===n||"button"===n)&&t.type===e}}function de(e){return function(t){return"form"in t?t.parentNode&&!1===t.disabled?"label"in t?"label"in t.parentNode?t.parentNode.disabled===e:t.disabled===e:t.isDisabled===e||t.isDisabled!==!e&&ie(t)===e:t.disabled===e:"label"in t&&t.disabled===e}}function he(e){return se(function(t){return t=+t,se(function(n,r){var i,o=e([],n.length,t),a=o.length;while(a--)n[i=o[a]]&&(n[i]=!(r[i]=n[i]))})})}function ge(e){return e&&"undefined"!=typeof e.getElementsByTagName&&e}n=oe.support={},o=oe.isXML=function(e){var t=e&&(e.ownerDocument||e).documentElement;return!!t&&"HTML"!==t.nodeName},p=oe.setDocument=function(e){var t,i,a=e?e.ownerDocument||e:w;return a!==d&&9===a.nodeType&&a.documentElement?(d=a,h=d.documentElement,g=!o(d),w!==d&&(i=d.defaultView)&&i.top!==i&&(i.addEventListener?i.addEventListener("unload",re,!1):i.attachEvent&&i.attachEvent("onunload",re)),n.attributes=ue(function(e){return e.className="i",!e.getAttribute("className")}),n.getElementsByTagName=ue(function(e){return e.appendChild(d.createComment("")),!e.getElementsByTagName("*").length}),n.getElementsByClassName=Q.test(d.getElementsByClassName),n.getById=ue(function(e){return h.appendChild(e).id=b,!d.getElementsByName||!d.getElementsByName(b).length}),n.getById?(r.filter.ID=function(e){var t=e.replace(Z,ee);return function(e){return e.getAttribute("id")===t}},r.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&g){var n=t.getElementById(e);return n?[n]:[]}}):(r.filter.ID=function(e){var t=e.replace(Z,ee);return function(e){var n="undefined"!=typeof e.getAttributeNode&&e.getAttributeNode("id");return n&&n.value===t}},r.find.ID=function(e,t){if("undefined"!=typeof t.getElementById&&g){var n,r,i,o=t.getElementById(e);if(o){if((n=o.getAttributeNode("id"))&&n.value===e)return[o];i=t.getElementsByName(e),r=0;while(o=i[r++])if((n=o.getAttributeNode("id"))&&n.value===e)return[o]}return[]}}),r.find.TAG=n.getElementsByTagName?function(e,t){return"undefined"!=typeof t.getElementsByTagName?t.getElementsByTagName(e):n.qsa?t.querySelectorAll(e):void 0}:function(e,t){var n,r=[],i=0,o=t.getElementsByTagName(e);if("*"===e){while(n=o[i++])1===n.nodeType&&r.push(n);return r}return o},r.find.CLASS=n.getElementsByClassName&&function(e,t){if("undefined"!=typeof t.getElementsByClassName&&g)return t.getElementsByClassName(e)},v=[],y=[],(n.qsa=Q.test(d.querySelectorAll))&&(ue(function(e){h.appendChild(e).innerHTML="",e.querySelectorAll("[msallowcapture^='']").length&&y.push("[*^$]="+M+"*(?:''|\"\")"),e.querySelectorAll("[selected]").length||y.push("\\["+M+"*(?:value|"+P+")"),e.querySelectorAll("[id~="+b+"-]").length||y.push("~="),e.querySelectorAll(":checked").length||y.push(":checked"),e.querySelectorAll("a#"+b+"+*").length||y.push(".#.+[+~]")}),ue(function(e){e.innerHTML="";var t=d.createElement("input");t.setAttribute("type","hidden"),e.appendChild(t).setAttribute("name","D"),e.querySelectorAll("[name=d]").length&&y.push("name"+M+"*[*^$|!~]?="),2!==e.querySelectorAll(":enabled").length&&y.push(":enabled",":disabled"),h.appendChild(e).disabled=!0,2!==e.querySelectorAll(":disabled").length&&y.push(":enabled",":disabled"),e.querySelectorAll("*,:x"),y.push(",.*:")})),(n.matchesSelector=Q.test(m=h.matches||h.webkitMatchesSelector||h.mozMatchesSelector||h.oMatchesSelector||h.msMatchesSelector))&&ue(function(e){n.disconnectedMatch=m.call(e,"*"),m.call(e,"[s!='']:x"),v.push("!=",W)}),y=y.length&&new RegExp(y.join("|")),v=v.length&&new RegExp(v.join("|")),t=Q.test(h.compareDocumentPosition),x=t||Q.test(h.contains)?function(e,t){var n=9===e.nodeType?e.documentElement:e,r=t&&t.parentNode;return e===r||!(!r||1!==r.nodeType||!(n.contains?n.contains(r):e.compareDocumentPosition&&16&e.compareDocumentPosition(r)))}:function(e,t){if(t)while(t=t.parentNode)if(t===e)return!0;return!1},D=t?function(e,t){if(e===t)return f=!0,0;var r=!e.compareDocumentPosition-!t.compareDocumentPosition;return r||(1&(r=(e.ownerDocument||e)===(t.ownerDocument||t)?e.compareDocumentPosition(t):1)||!n.sortDetached&&t.compareDocumentPosition(e)===r?e===d||e.ownerDocument===w&&x(w,e)?-1:t===d||t.ownerDocument===w&&x(w,t)?1:c?O(c,e)-O(c,t):0:4&r?-1:1)}:function(e,t){if(e===t)return f=!0,0;var n,r=0,i=e.parentNode,o=t.parentNode,a=[e],s=[t];if(!i||!o)return e===d?-1:t===d?1:i?-1:o?1:c?O(c,e)-O(c,t):0;if(i===o)return ce(e,t);n=e;while(n=n.parentNode)a.unshift(n);n=t;while(n=n.parentNode)s.unshift(n);while(a[r]===s[r])r++;return r?ce(a[r],s[r]):a[r]===w?-1:s[r]===w?1:0},d):d},oe.matches=function(e,t){return oe(e,null,null,t)},oe.matchesSelector=function(e,t){if((e.ownerDocument||e)!==d&&p(e),t=t.replace(z,"='$1']"),n.matchesSelector&&g&&!S[t+" "]&&(!v||!v.test(t))&&(!y||!y.test(t)))try{var r=m.call(e,t);if(r||n.disconnectedMatch||e.document&&11!==e.document.nodeType)return r}catch(e){}return oe(t,d,null,[e]).length>0},oe.contains=function(e,t){return(e.ownerDocument||e)!==d&&p(e),x(e,t)},oe.attr=function(e,t){(e.ownerDocument||e)!==d&&p(e);var i=r.attrHandle[t.toLowerCase()],o=i&&N.call(r.attrHandle,t.toLowerCase())?i(e,t,!g):void 0;return void 0!==o?o:n.attributes||!g?e.getAttribute(t):(o=e.getAttributeNode(t))&&o.specified?o.value:null},oe.escape=function(e){return(e+"").replace(te,ne)},oe.error=function(e){throw new Error("Syntax error, unrecognized expression: "+e)},oe.uniqueSort=function(e){var t,r=[],i=0,o=0;if(f=!n.detectDuplicates,c=!n.sortStable&&e.slice(0),e.sort(D),f){while(t=e[o++])t===e[o]&&(i=r.push(o));while(i--)e.splice(r[i],1)}return c=null,e},i=oe.getText=function(e){var t,n="",r=0,o=e.nodeType;if(o){if(1===o||9===o||11===o){if("string"==typeof e.textContent)return e.textContent;for(e=e.firstChild;e;e=e.nextSibling)n+=i(e)}else if(3===o||4===o)return e.nodeValue}else while(t=e[r++])n+=i(t);return n},(r=oe.selectors={cacheLength:50,createPseudo:se,match:V,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(e){return e[1]=e[1].replace(Z,ee),e[3]=(e[3]||e[4]||e[5]||"").replace(Z,ee),"~="===e[2]&&(e[3]=" "+e[3]+" "),e.slice(0,4)},CHILD:function(e){return e[1]=e[1].toLowerCase(),"nth"===e[1].slice(0,3)?(e[3]||oe.error(e[0]),e[4]=+(e[4]?e[5]+(e[6]||1):2*("even"===e[3]||"odd"===e[3])),e[5]=+(e[7]+e[8]||"odd"===e[3])):e[3]&&oe.error(e[0]),e},PSEUDO:function(e){var t,n=!e[6]&&e[2];return V.CHILD.test(e[0])?null:(e[3]?e[2]=e[4]||e[5]||"":n&&X.test(n)&&(t=a(n,!0))&&(t=n.indexOf(")",n.length-t)-n.length)&&(e[0]=e[0].slice(0,t),e[2]=n.slice(0,t)),e.slice(0,3))}},filter:{TAG:function(e){var t=e.replace(Z,ee).toLowerCase();return"*"===e?function(){return!0}:function(e){return e.nodeName&&e.nodeName.toLowerCase()===t}},CLASS:function(e){var t=E[e+" "];return t||(t=new RegExp("(^|"+M+")"+e+"("+M+"|$)"))&&E(e,function(e){return t.test("string"==typeof e.className&&e.className||"undefined"!=typeof e.getAttribute&&e.getAttribute("class")||"")})},ATTR:function(e,t,n){return function(r){var i=oe.attr(r,e);return null==i?"!="===t:!t||(i+="","="===t?i===n:"!="===t?i!==n:"^="===t?n&&0===i.indexOf(n):"*="===t?n&&i.indexOf(n)>-1:"$="===t?n&&i.slice(-n.length)===n:"~="===t?(" "+i.replace($," ")+" ").indexOf(n)>-1:"|="===t&&(i===n||i.slice(0,n.length+1)===n+"-"))}},CHILD:function(e,t,n,r,i){var o="nth"!==e.slice(0,3),a="last"!==e.slice(-4),s="of-type"===t;return 1===r&&0===i?function(e){return!!e.parentNode}:function(t,n,u){var l,c,f,p,d,h,g=o!==a?"nextSibling":"previousSibling",y=t.parentNode,v=s&&t.nodeName.toLowerCase(),m=!u&&!s,x=!1;if(y){if(o){while(g){p=t;while(p=p[g])if(s?p.nodeName.toLowerCase()===v:1===p.nodeType)return!1;h=g="only"===e&&!h&&"nextSibling"}return!0}if(h=[a?y.firstChild:y.lastChild],a&&m){x=(d=(l=(c=(f=(p=y)[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]||[])[0]===T&&l[1])&&l[2],p=d&&y.childNodes[d];while(p=++d&&p&&p[g]||(x=d=0)||h.pop())if(1===p.nodeType&&++x&&p===t){c[e]=[T,d,x];break}}else if(m&&(x=d=(l=(c=(f=(p=t)[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]||[])[0]===T&&l[1]),!1===x)while(p=++d&&p&&p[g]||(x=d=0)||h.pop())if((s?p.nodeName.toLowerCase()===v:1===p.nodeType)&&++x&&(m&&((c=(f=p[b]||(p[b]={}))[p.uniqueID]||(f[p.uniqueID]={}))[e]=[T,x]),p===t))break;return(x-=i)===r||x%r==0&&x/r>=0}}},PSEUDO:function(e,t){var n,i=r.pseudos[e]||r.setFilters[e.toLowerCase()]||oe.error("unsupported pseudo: "+e);return i[b]?i(t):i.length>1?(n=[e,e,"",t],r.setFilters.hasOwnProperty(e.toLowerCase())?se(function(e,n){var r,o=i(e,t),a=o.length;while(a--)e[r=O(e,o[a])]=!(n[r]=o[a])}):function(e){return i(e,0,n)}):i}},pseudos:{not:se(function(e){var t=[],n=[],r=s(e.replace(B,"$1"));return r[b]?se(function(e,t,n,i){var o,a=r(e,null,i,[]),s=e.length;while(s--)(o=a[s])&&(e[s]=!(t[s]=o))}):function(e,i,o){return t[0]=e,r(t,null,o,n),t[0]=null,!n.pop()}}),has:se(function(e){return function(t){return oe(e,t).length>0}}),contains:se(function(e){return e=e.replace(Z,ee),function(t){return(t.textContent||t.innerText||i(t)).indexOf(e)>-1}}),lang:se(function(e){return U.test(e||"")||oe.error("unsupported lang: "+e),e=e.replace(Z,ee).toLowerCase(),function(t){var n;do{if(n=g?t.lang:t.getAttribute("xml:lang")||t.getAttribute("lang"))return(n=n.toLowerCase())===e||0===n.indexOf(e+"-")}while((t=t.parentNode)&&1===t.nodeType);return!1}}),target:function(t){var n=e.location&&e.location.hash;return n&&n.slice(1)===t.id},root:function(e){return e===h},focus:function(e){return e===d.activeElement&&(!d.hasFocus||d.hasFocus())&&!!(e.type||e.href||~e.tabIndex)},enabled:de(!1),disabled:de(!0),checked:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&!!e.checked||"option"===t&&!!e.selected},selected:function(e){return e.parentNode&&e.parentNode.selectedIndex,!0===e.selected},empty:function(e){for(e=e.firstChild;e;e=e.nextSibling)if(e.nodeType<6)return!1;return!0},parent:function(e){return!r.pseudos.empty(e)},header:function(e){return Y.test(e.nodeName)},input:function(e){return G.test(e.nodeName)},button:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&"button"===e.type||"button"===t},text:function(e){var t;return"input"===e.nodeName.toLowerCase()&&"text"===e.type&&(null==(t=e.getAttribute("type"))||"text"===t.toLowerCase())},first:he(function(){return[0]}),last:he(function(e,t){return[t-1]}),eq:he(function(e,t,n){return[n<0?n+t:n]}),even:he(function(e,t){for(var n=0;n=0;)e.push(r);return e}),gt:he(function(e,t,n){for(var r=n<0?n+t:n;++r1?function(t,n,r){var i=e.length;while(i--)if(!e[i](t,n,r))return!1;return!0}:e[0]}function be(e,t,n){for(var r=0,i=t.length;r-1&&(o[l]=!(a[l]=f))}}else v=we(v===a?v.splice(h,v.length):v),i?i(null,a,v,u):L.apply(a,v)})}function Ce(e){for(var t,n,i,o=e.length,a=r.relative[e[0].type],s=a||r.relative[" "],u=a?1:0,c=me(function(e){return e===t},s,!0),f=me(function(e){return O(t,e)>-1},s,!0),p=[function(e,n,r){var i=!a&&(r||n!==l)||((t=n).nodeType?c(e,n,r):f(e,n,r));return t=null,i}];u1&&xe(p),u>1&&ve(e.slice(0,u-1).concat({value:" "===e[u-2].type?"*":""})).replace(B,"$1"),n,u0,i=e.length>0,o=function(o,a,s,u,c){var f,h,y,v=0,m="0",x=o&&[],b=[],w=l,C=o||i&&r.find.TAG("*",c),E=T+=null==w?1:Math.random()||.1,k=C.length;for(c&&(l=a===d||a||c);m!==k&&null!=(f=C[m]);m++){if(i&&f){h=0,a||f.ownerDocument===d||(p(f),s=!g);while(y=e[h++])if(y(f,a||d,s)){u.push(f);break}c&&(T=E)}n&&((f=!y&&f)&&v--,o&&x.push(f))}if(v+=m,n&&m!==v){h=0;while(y=t[h++])y(x,b,a,s);if(o){if(v>0)while(m--)x[m]||b[m]||(b[m]=j.call(u));b=we(b)}L.apply(u,b),c&&!o&&b.length>0&&v+t.length>1&&oe.uniqueSort(u)}return c&&(T=E,l=w),x};return n?se(o):o}return s=oe.compile=function(e,t){var n,r=[],i=[],o=S[e+" "];if(!o){t||(t=a(e)),n=t.length;while(n--)(o=Ce(t[n]))[b]?r.push(o):i.push(o);(o=S(e,Ee(i,r))).selector=e}return o},u=oe.select=function(e,t,n,i){var o,u,l,c,f,p="function"==typeof e&&e,d=!i&&a(e=p.selector||e);if(n=n||[],1===d.length){if((u=d[0]=d[0].slice(0)).length>2&&"ID"===(l=u[0]).type&&9===t.nodeType&&g&&r.relative[u[1].type]){if(!(t=(r.find.ID(l.matches[0].replace(Z,ee),t)||[])[0]))return n;p&&(t=t.parentNode),e=e.slice(u.shift().value.length)}o=V.needsContext.test(e)?0:u.length;while(o--){if(l=u[o],r.relative[c=l.type])break;if((f=r.find[c])&&(i=f(l.matches[0].replace(Z,ee),K.test(u[0].type)&&ge(t.parentNode)||t))){if(u.splice(o,1),!(e=i.length&&ve(u)))return L.apply(n,i),n;break}}}return(p||s(e,d))(i,t,!g,n,!t||K.test(e)&&ge(t.parentNode)||t),n},n.sortStable=b.split("").sort(D).join("")===b,n.detectDuplicates=!!f,p(),n.sortDetached=ue(function(e){return 1&e.compareDocumentPosition(d.createElement("fieldset"))}),ue(function(e){return e.innerHTML="","#"===e.firstChild.getAttribute("href")})||le("type|href|height|width",function(e,t,n){if(!n)return e.getAttribute(t,"type"===t.toLowerCase()?1:2)}),n.attributes&&ue(function(e){return e.innerHTML="",e.firstChild.setAttribute("value",""),""===e.firstChild.getAttribute("value")})||le("value",function(e,t,n){if(!n&&"input"===e.nodeName.toLowerCase())return e.defaultValue}),ue(function(e){return null==e.getAttribute("disabled")})||le(P,function(e,t,n){var r;if(!n)return!0===e[t]?t.toLowerCase():(r=e.getAttributeNode(t))&&r.specified?r.value:null}),oe}(e);w.find=E,w.expr=E.selectors,w.expr[":"]=w.expr.pseudos,w.uniqueSort=w.unique=E.uniqueSort,w.text=E.getText,w.isXMLDoc=E.isXML,w.contains=E.contains,w.escapeSelector=E.escape;var k=function(e,t,n){var r=[],i=void 0!==n;while((e=e[t])&&9!==e.nodeType)if(1===e.nodeType){if(i&&w(e).is(n))break;r.push(e)}return r},S=function(e,t){for(var n=[];e;e=e.nextSibling)1===e.nodeType&&e!==t&&n.push(e);return n},D=w.expr.match.needsContext;function N(e,t){return e.nodeName&&e.nodeName.toLowerCase()===t.toLowerCase()}var A=/^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i;function j(e,t,n){return g(t)?w.grep(e,function(e,r){return!!t.call(e,r,e)!==n}):t.nodeType?w.grep(e,function(e){return e===t!==n}):"string"!=typeof t?w.grep(e,function(e){return u.call(t,e)>-1!==n}):w.filter(t,e,n)}w.filter=function(e,t,n){var r=t[0];return n&&(e=":not("+e+")"),1===t.length&&1===r.nodeType?w.find.matchesSelector(r,e)?[r]:[]:w.find.matches(e,w.grep(t,function(e){return 1===e.nodeType}))},w.fn.extend({find:function(e){var t,n,r=this.length,i=this;if("string"!=typeof e)return this.pushStack(w(e).filter(function(){for(t=0;t1?w.uniqueSort(n):n},filter:function(e){return this.pushStack(j(this,e||[],!1))},not:function(e){return this.pushStack(j(this,e||[],!0))},is:function(e){return!!j(this,"string"==typeof e&&D.test(e)?w(e):e||[],!1).length}});var q,L=/^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/;(w.fn.init=function(e,t,n){var i,o;if(!e)return this;if(n=n||q,"string"==typeof e){if(!(i="<"===e[0]&&">"===e[e.length-1]&&e.length>=3?[null,e,null]:L.exec(e))||!i[1]&&t)return!t||t.jquery?(t||n).find(e):this.constructor(t).find(e);if(i[1]){if(t=t instanceof w?t[0]:t,w.merge(this,w.parseHTML(i[1],t&&t.nodeType?t.ownerDocument||t:r,!0)),A.test(i[1])&&w.isPlainObject(t))for(i in t)g(this[i])?this[i](t[i]):this.attr(i,t[i]);return this}return(o=r.getElementById(i[2]))&&(this[0]=o,this.length=1),this}return e.nodeType?(this[0]=e,this.length=1,this):g(e)?void 0!==n.ready?n.ready(e):e(w):w.makeArray(e,this)}).prototype=w.fn,q=w(r);var H=/^(?:parents|prev(?:Until|All))/,O={children:!0,contents:!0,next:!0,prev:!0};w.fn.extend({has:function(e){var t=w(e,this),n=t.length;return this.filter(function(){for(var e=0;e-1:1===n.nodeType&&w.find.matchesSelector(n,e))){o.push(n);break}return this.pushStack(o.length>1?w.uniqueSort(o):o)},index:function(e){return e?"string"==typeof e?u.call(w(e),this[0]):u.call(this,e.jquery?e[0]:e):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(e,t){return this.pushStack(w.uniqueSort(w.merge(this.get(),w(e,t))))},addBack:function(e){return this.add(null==e?this.prevObject:this.prevObject.filter(e))}});function P(e,t){while((e=e[t])&&1!==e.nodeType);return e}w.each({parent:function(e){var t=e.parentNode;return t&&11!==t.nodeType?t:null},parents:function(e){return k(e,"parentNode")},parentsUntil:function(e,t,n){return k(e,"parentNode",n)},next:function(e){return P(e,"nextSibling")},prev:function(e){return P(e,"previousSibling")},nextAll:function(e){return k(e,"nextSibling")},prevAll:function(e){return k(e,"previousSibling")},nextUntil:function(e,t,n){return k(e,"nextSibling",n)},prevUntil:function(e,t,n){return k(e,"previousSibling",n)},siblings:function(e){return S((e.parentNode||{}).firstChild,e)},children:function(e){return S(e.firstChild)},contents:function(e){return N(e,"iframe")?e.contentDocument:(N(e,"template")&&(e=e.content||e),w.merge([],e.childNodes))}},function(e,t){w.fn[e]=function(n,r){var i=w.map(this,t,n);return"Until"!==e.slice(-5)&&(r=n),r&&"string"==typeof r&&(i=w.filter(r,i)),this.length>1&&(O[e]||w.uniqueSort(i),H.test(e)&&i.reverse()),this.pushStack(i)}});var M=/[^\x20\t\r\n\f]+/g;function R(e){var t={};return w.each(e.match(M)||[],function(e,n){t[n]=!0}),t}w.Callbacks=function(e){e="string"==typeof e?R(e):w.extend({},e);var t,n,r,i,o=[],a=[],s=-1,u=function(){for(i=i||e.once,r=t=!0;a.length;s=-1){n=a.shift();while(++s-1)o.splice(n,1),n<=s&&s--}),this},has:function(e){return e?w.inArray(e,o)>-1:o.length>0},empty:function(){return o&&(o=[]),this},disable:function(){return i=a=[],o=n="",this},disabled:function(){return!o},lock:function(){return i=a=[],n||t||(o=n=""),this},locked:function(){return!!i},fireWith:function(e,n){return i||(n=[e,(n=n||[]).slice?n.slice():n],a.push(n),t||u()),this},fire:function(){return l.fireWith(this,arguments),this},fired:function(){return!!r}};return l};function I(e){return e}function W(e){throw e}function $(e,t,n,r){var i;try{e&&g(i=e.promise)?i.call(e).done(t).fail(n):e&&g(i=e.then)?i.call(e,t,n):t.apply(void 0,[e].slice(r))}catch(e){n.apply(void 0,[e])}}w.extend({Deferred:function(t){var n=[["notify","progress",w.Callbacks("memory"),w.Callbacks("memory"),2],["resolve","done",w.Callbacks("once memory"),w.Callbacks("once memory"),0,"resolved"],["reject","fail",w.Callbacks("once memory"),w.Callbacks("once memory"),1,"rejected"]],r="pending",i={state:function(){return r},always:function(){return o.done(arguments).fail(arguments),this},"catch":function(e){return i.then(null,e)},pipe:function(){var e=arguments;return w.Deferred(function(t){w.each(n,function(n,r){var i=g(e[r[4]])&&e[r[4]];o[r[1]](function(){var e=i&&i.apply(this,arguments);e&&g(e.promise)?e.promise().progress(t.notify).done(t.resolve).fail(t.reject):t[r[0]+"With"](this,i?[e]:arguments)})}),e=null}).promise()},then:function(t,r,i){var o=0;function a(t,n,r,i){return function(){var s=this,u=arguments,l=function(){var e,l;if(!(t=o&&(r!==W&&(s=void 0,u=[e]),n.rejectWith(s,u))}};t?c():(w.Deferred.getStackHook&&(c.stackTrace=w.Deferred.getStackHook()),e.setTimeout(c))}}return w.Deferred(function(e){n[0][3].add(a(0,e,g(i)?i:I,e.notifyWith)),n[1][3].add(a(0,e,g(t)?t:I)),n[2][3].add(a(0,e,g(r)?r:W))}).promise()},promise:function(e){return null!=e?w.extend(e,i):i}},o={};return w.each(n,function(e,t){var a=t[2],s=t[5];i[t[1]]=a.add,s&&a.add(function(){r=s},n[3-e][2].disable,n[3-e][3].disable,n[0][2].lock,n[0][3].lock),a.add(t[3].fire),o[t[0]]=function(){return o[t[0]+"With"](this===o?void 0:this,arguments),this},o[t[0]+"With"]=a.fireWith}),i.promise(o),t&&t.call(o,o),o},when:function(e){var t=arguments.length,n=t,r=Array(n),i=o.call(arguments),a=w.Deferred(),s=function(e){return function(n){r[e]=this,i[e]=arguments.length>1?o.call(arguments):n,--t||a.resolveWith(r,i)}};if(t<=1&&($(e,a.done(s(n)).resolve,a.reject,!t),"pending"===a.state()||g(i[n]&&i[n].then)))return a.then();while(n--)$(i[n],s(n),a.reject);return a.promise()}});var B=/^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/;w.Deferred.exceptionHook=function(t,n){e.console&&e.console.warn&&t&&B.test(t.name)&&e.console.warn("jQuery.Deferred exception: "+t.message,t.stack,n)},w.readyException=function(t){e.setTimeout(function(){throw t})};var F=w.Deferred();w.fn.ready=function(e){return F.then(e)["catch"](function(e){w.readyException(e)}),this},w.extend({isReady:!1,readyWait:1,ready:function(e){(!0===e?--w.readyWait:w.isReady)||(w.isReady=!0,!0!==e&&--w.readyWait>0||F.resolveWith(r,[w]))}}),w.ready.then=F.then;function _(){r.removeEventListener("DOMContentLoaded",_),e.removeEventListener("load",_),w.ready()}"complete"===r.readyState||"loading"!==r.readyState&&!r.documentElement.doScroll?e.setTimeout(w.ready):(r.addEventListener("DOMContentLoaded",_),e.addEventListener("load",_));var z=function(e,t,n,r,i,o,a){var s=0,u=e.length,l=null==n;if("object"===x(n)){i=!0;for(s in n)z(e,t,s,n[s],!0,o,a)}else if(void 0!==r&&(i=!0,g(r)||(a=!0),l&&(a?(t.call(e,r),t=null):(l=t,t=function(e,t,n){return l.call(w(e),n)})),t))for(;s1,null,!0)},removeData:function(e){return this.each(function(){K.remove(this,e)})}}),w.extend({queue:function(e,t,n){var r;if(e)return t=(t||"fx")+"queue",r=J.get(e,t),n&&(!r||Array.isArray(n)?r=J.access(e,t,w.makeArray(n)):r.push(n)),r||[]},dequeue:function(e,t){t=t||"fx";var n=w.queue(e,t),r=n.length,i=n.shift(),o=w._queueHooks(e,t),a=function(){w.dequeue(e,t)};"inprogress"===i&&(i=n.shift(),r--),i&&("fx"===t&&n.unshift("inprogress"),delete o.stop,i.call(e,a,o)),!r&&o&&o.empty.fire()},_queueHooks:function(e,t){var n=t+"queueHooks";return J.get(e,n)||J.access(e,n,{empty:w.Callbacks("once memory").add(function(){J.remove(e,[t+"queue",n])})})}}),w.fn.extend({queue:function(e,t){var n=2;return"string"!=typeof e&&(t=e,e="fx",n--),arguments.length\x20\t\r\n\f]+)/i,he=/^$|^module$|\/(?:java|ecma)script/i,ge={option:[1,""],thead:[1,"","
    "],col:[2,"","
    "],tr:[2,"","
    "],td:[3,"","
    "],_default:[0,"",""]};ge.optgroup=ge.option,ge.tbody=ge.tfoot=ge.colgroup=ge.caption=ge.thead,ge.th=ge.td;function ye(e,t){var n;return n="undefined"!=typeof e.getElementsByTagName?e.getElementsByTagName(t||"*"):"undefined"!=typeof e.querySelectorAll?e.querySelectorAll(t||"*"):[],void 0===t||t&&N(e,t)?w.merge([e],n):n}function ve(e,t){for(var n=0,r=e.length;n-1)i&&i.push(o);else if(l=w.contains(o.ownerDocument,o),a=ye(f.appendChild(o),"script"),l&&ve(a),n){c=0;while(o=a[c++])he.test(o.type||"")&&n.push(o)}return f}!function(){var e=r.createDocumentFragment().appendChild(r.createElement("div")),t=r.createElement("input");t.setAttribute("type","radio"),t.setAttribute("checked","checked"),t.setAttribute("name","t"),e.appendChild(t),h.checkClone=e.cloneNode(!0).cloneNode(!0).lastChild.checked,e.innerHTML="",h.noCloneChecked=!!e.cloneNode(!0).lastChild.defaultValue}();var be=r.documentElement,we=/^key/,Te=/^(?:mouse|pointer|contextmenu|drag|drop)|click/,Ce=/^([^.]*)(?:\.(.+)|)/;function Ee(){return!0}function ke(){return!1}function Se(){try{return r.activeElement}catch(e){}}function De(e,t,n,r,i,o){var a,s;if("object"==typeof t){"string"!=typeof n&&(r=r||n,n=void 0);for(s in t)De(e,s,n,r,t[s],o);return e}if(null==r&&null==i?(i=n,r=n=void 0):null==i&&("string"==typeof n?(i=r,r=void 0):(i=r,r=n,n=void 0)),!1===i)i=ke;else if(!i)return e;return 1===o&&(a=i,(i=function(e){return w().off(e),a.apply(this,arguments)}).guid=a.guid||(a.guid=w.guid++)),e.each(function(){w.event.add(this,t,i,r,n)})}w.event={global:{},add:function(e,t,n,r,i){var o,a,s,u,l,c,f,p,d,h,g,y=J.get(e);if(y){n.handler&&(n=(o=n).handler,i=o.selector),i&&w.find.matchesSelector(be,i),n.guid||(n.guid=w.guid++),(u=y.events)||(u=y.events={}),(a=y.handle)||(a=y.handle=function(t){return"undefined"!=typeof w&&w.event.triggered!==t.type?w.event.dispatch.apply(e,arguments):void 0}),l=(t=(t||"").match(M)||[""]).length;while(l--)d=g=(s=Ce.exec(t[l])||[])[1],h=(s[2]||"").split(".").sort(),d&&(f=w.event.special[d]||{},d=(i?f.delegateType:f.bindType)||d,f=w.event.special[d]||{},c=w.extend({type:d,origType:g,data:r,handler:n,guid:n.guid,selector:i,needsContext:i&&w.expr.match.needsContext.test(i),namespace:h.join(".")},o),(p=u[d])||((p=u[d]=[]).delegateCount=0,f.setup&&!1!==f.setup.call(e,r,h,a)||e.addEventListener&&e.addEventListener(d,a)),f.add&&(f.add.call(e,c),c.handler.guid||(c.handler.guid=n.guid)),i?p.splice(p.delegateCount++,0,c):p.push(c),w.event.global[d]=!0)}},remove:function(e,t,n,r,i){var o,a,s,u,l,c,f,p,d,h,g,y=J.hasData(e)&&J.get(e);if(y&&(u=y.events)){l=(t=(t||"").match(M)||[""]).length;while(l--)if(s=Ce.exec(t[l])||[],d=g=s[1],h=(s[2]||"").split(".").sort(),d){f=w.event.special[d]||{},p=u[d=(r?f.delegateType:f.bindType)||d]||[],s=s[2]&&new RegExp("(^|\\.)"+h.join("\\.(?:.*\\.|)")+"(\\.|$)"),a=o=p.length;while(o--)c=p[o],!i&&g!==c.origType||n&&n.guid!==c.guid||s&&!s.test(c.namespace)||r&&r!==c.selector&&("**"!==r||!c.selector)||(p.splice(o,1),c.selector&&p.delegateCount--,f.remove&&f.remove.call(e,c));a&&!p.length&&(f.teardown&&!1!==f.teardown.call(e,h,y.handle)||w.removeEvent(e,d,y.handle),delete u[d])}else for(d in u)w.event.remove(e,d+t[l],n,r,!0);w.isEmptyObject(u)&&J.remove(e,"handle events")}},dispatch:function(e){var t=w.event.fix(e),n,r,i,o,a,s,u=new Array(arguments.length),l=(J.get(this,"events")||{})[t.type]||[],c=w.event.special[t.type]||{};for(u[0]=t,n=1;n=1))for(;l!==this;l=l.parentNode||this)if(1===l.nodeType&&("click"!==e.type||!0!==l.disabled)){for(o=[],a={},n=0;n-1:w.find(i,this,null,[l]).length),a[i]&&o.push(r);o.length&&s.push({elem:l,handlers:o})}return l=this,u\x20\t\r\n\f]*)[^>]*)\/>/gi,Ae=/\s*$/g;function Le(e,t){return N(e,"table")&&N(11!==t.nodeType?t:t.firstChild,"tr")?w(e).children("tbody")[0]||e:e}function He(e){return e.type=(null!==e.getAttribute("type"))+"/"+e.type,e}function Oe(e){return"true/"===(e.type||"").slice(0,5)?e.type=e.type.slice(5):e.removeAttribute("type"),e}function Pe(e,t){var n,r,i,o,a,s,u,l;if(1===t.nodeType){if(J.hasData(e)&&(o=J.access(e),a=J.set(t,o),l=o.events)){delete a.handle,a.events={};for(i in l)for(n=0,r=l[i].length;n1&&"string"==typeof y&&!h.checkClone&&je.test(y))return e.each(function(i){var o=e.eq(i);v&&(t[0]=y.call(this,i,o.html())),Re(o,t,n,r)});if(p&&(i=xe(t,e[0].ownerDocument,!1,e,r),o=i.firstChild,1===i.childNodes.length&&(i=o),o||r)){for(u=(s=w.map(ye(i,"script"),He)).length;f")},clone:function(e,t,n){var r,i,o,a,s=e.cloneNode(!0),u=w.contains(e.ownerDocument,e);if(!(h.noCloneChecked||1!==e.nodeType&&11!==e.nodeType||w.isXMLDoc(e)))for(a=ye(s),r=0,i=(o=ye(e)).length;r0&&ve(a,!u&&ye(e,"script")),s},cleanData:function(e){for(var t,n,r,i=w.event.special,o=0;void 0!==(n=e[o]);o++)if(Y(n)){if(t=n[J.expando]){if(t.events)for(r in t.events)i[r]?w.event.remove(n,r):w.removeEvent(n,r,t.handle);n[J.expando]=void 0}n[K.expando]&&(n[K.expando]=void 0)}}}),w.fn.extend({detach:function(e){return Ie(this,e,!0)},remove:function(e){return Ie(this,e)},text:function(e){return z(this,function(e){return void 0===e?w.text(this):this.empty().each(function(){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||(this.textContent=e)})},null,e,arguments.length)},append:function(){return Re(this,arguments,function(e){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||Le(this,e).appendChild(e)})},prepend:function(){return Re(this,arguments,function(e){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var t=Le(this,e);t.insertBefore(e,t.firstChild)}})},before:function(){return Re(this,arguments,function(e){this.parentNode&&this.parentNode.insertBefore(e,this)})},after:function(){return Re(this,arguments,function(e){this.parentNode&&this.parentNode.insertBefore(e,this.nextSibling)})},empty:function(){for(var e,t=0;null!=(e=this[t]);t++)1===e.nodeType&&(w.cleanData(ye(e,!1)),e.textContent="");return this},clone:function(e,t){return e=null!=e&&e,t=null==t?e:t,this.map(function(){return w.clone(this,e,t)})},html:function(e){return z(this,function(e){var t=this[0]||{},n=0,r=this.length;if(void 0===e&&1===t.nodeType)return t.innerHTML;if("string"==typeof e&&!Ae.test(e)&&!ge[(de.exec(e)||["",""])[1].toLowerCase()]){e=w.htmlPrefilter(e);try{for(;n=0&&(u+=Math.max(0,Math.ceil(e["offset"+t[0].toUpperCase()+t.slice(1)]-o-u-s-.5))),u}function et(e,t,n){var r=$e(e),i=Fe(e,t,r),o="border-box"===w.css(e,"boxSizing",!1,r),a=o;if(We.test(i)){if(!n)return i;i="auto"}return a=a&&(h.boxSizingReliable()||i===e.style[t]),("auto"===i||!parseFloat(i)&&"inline"===w.css(e,"display",!1,r))&&(i=e["offset"+t[0].toUpperCase()+t.slice(1)],a=!0),(i=parseFloat(i)||0)+Ze(e,t,n||(o?"border":"content"),a,r,i)+"px"}w.extend({cssHooks:{opacity:{get:function(e,t){if(t){var n=Fe(e,"opacity");return""===n?"1":n}}}},cssNumber:{animationIterationCount:!0,columnCount:!0,fillOpacity:!0,flexGrow:!0,flexShrink:!0,fontWeight:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{},style:function(e,t,n,r){if(e&&3!==e.nodeType&&8!==e.nodeType&&e.style){var i,o,a,s=G(t),u=Xe.test(t),l=e.style;if(u||(t=Je(s)),a=w.cssHooks[t]||w.cssHooks[s],void 0===n)return a&&"get"in a&&void 0!==(i=a.get(e,!1,r))?i:l[t];"string"==(o=typeof n)&&(i=ie.exec(n))&&i[1]&&(n=ue(e,t,i),o="number"),null!=n&&n===n&&("number"===o&&(n+=i&&i[3]||(w.cssNumber[s]?"":"px")),h.clearCloneStyle||""!==n||0!==t.indexOf("background")||(l[t]="inherit"),a&&"set"in a&&void 0===(n=a.set(e,n,r))||(u?l.setProperty(t,n):l[t]=n))}},css:function(e,t,n,r){var i,o,a,s=G(t);return Xe.test(t)||(t=Je(s)),(a=w.cssHooks[t]||w.cssHooks[s])&&"get"in a&&(i=a.get(e,!0,n)),void 0===i&&(i=Fe(e,t,r)),"normal"===i&&t in Ve&&(i=Ve[t]),""===n||n?(o=parseFloat(i),!0===n||isFinite(o)?o||0:i):i}}),w.each(["height","width"],function(e,t){w.cssHooks[t]={get:function(e,n,r){if(n)return!ze.test(w.css(e,"display"))||e.getClientRects().length&&e.getBoundingClientRect().width?et(e,t,r):se(e,Ue,function(){return et(e,t,r)})},set:function(e,n,r){var i,o=$e(e),a="border-box"===w.css(e,"boxSizing",!1,o),s=r&&Ze(e,t,r,a,o);return a&&h.scrollboxSize()===o.position&&(s-=Math.ceil(e["offset"+t[0].toUpperCase()+t.slice(1)]-parseFloat(o[t])-Ze(e,t,"border",!1,o)-.5)),s&&(i=ie.exec(n))&&"px"!==(i[3]||"px")&&(e.style[t]=n,n=w.css(e,t)),Ke(e,n,s)}}}),w.cssHooks.marginLeft=_e(h.reliableMarginLeft,function(e,t){if(t)return(parseFloat(Fe(e,"marginLeft"))||e.getBoundingClientRect().left-se(e,{marginLeft:0},function(){return e.getBoundingClientRect().left}))+"px"}),w.each({margin:"",padding:"",border:"Width"},function(e,t){w.cssHooks[e+t]={expand:function(n){for(var r=0,i={},o="string"==typeof n?n.split(" "):[n];r<4;r++)i[e+oe[r]+t]=o[r]||o[r-2]||o[0];return i}},"margin"!==e&&(w.cssHooks[e+t].set=Ke)}),w.fn.extend({css:function(e,t){return z(this,function(e,t,n){var r,i,o={},a=0;if(Array.isArray(t)){for(r=$e(e),i=t.length;a1)}});function tt(e,t,n,r,i){return new tt.prototype.init(e,t,n,r,i)}w.Tween=tt,tt.prototype={constructor:tt,init:function(e,t,n,r,i,o){this.elem=e,this.prop=n,this.easing=i||w.easing._default,this.options=t,this.start=this.now=this.cur(),this.end=r,this.unit=o||(w.cssNumber[n]?"":"px")},cur:function(){var e=tt.propHooks[this.prop];return e&&e.get?e.get(this):tt.propHooks._default.get(this)},run:function(e){var t,n=tt.propHooks[this.prop];return this.options.duration?this.pos=t=w.easing[this.easing](e,this.options.duration*e,0,1,this.options.duration):this.pos=t=e,this.now=(this.end-this.start)*t+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),n&&n.set?n.set(this):tt.propHooks._default.set(this),this}},tt.prototype.init.prototype=tt.prototype,tt.propHooks={_default:{get:function(e){var t;return 1!==e.elem.nodeType||null!=e.elem[e.prop]&&null==e.elem.style[e.prop]?e.elem[e.prop]:(t=w.css(e.elem,e.prop,""))&&"auto"!==t?t:0},set:function(e){w.fx.step[e.prop]?w.fx.step[e.prop](e):1!==e.elem.nodeType||null==e.elem.style[w.cssProps[e.prop]]&&!w.cssHooks[e.prop]?e.elem[e.prop]=e.now:w.style(e.elem,e.prop,e.now+e.unit)}}},tt.propHooks.scrollTop=tt.propHooks.scrollLeft={set:function(e){e.elem.nodeType&&e.elem.parentNode&&(e.elem[e.prop]=e.now)}},w.easing={linear:function(e){return e},swing:function(e){return.5-Math.cos(e*Math.PI)/2},_default:"swing"},w.fx=tt.prototype.init,w.fx.step={};var nt,rt,it=/^(?:toggle|show|hide)$/,ot=/queueHooks$/;function at(){rt&&(!1===r.hidden&&e.requestAnimationFrame?e.requestAnimationFrame(at):e.setTimeout(at,w.fx.interval),w.fx.tick())}function st(){return e.setTimeout(function(){nt=void 0}),nt=Date.now()}function ut(e,t){var n,r=0,i={height:e};for(t=t?1:0;r<4;r+=2-t)i["margin"+(n=oe[r])]=i["padding"+n]=e;return t&&(i.opacity=i.width=e),i}function lt(e,t,n){for(var r,i=(pt.tweeners[t]||[]).concat(pt.tweeners["*"]),o=0,a=i.length;o1)},removeAttr:function(e){return this.each(function(){w.removeAttr(this,e)})}}),w.extend({attr:function(e,t,n){var r,i,o=e.nodeType;if(3!==o&&8!==o&&2!==o)return"undefined"==typeof e.getAttribute?w.prop(e,t,n):(1===o&&w.isXMLDoc(e)||(i=w.attrHooks[t.toLowerCase()]||(w.expr.match.bool.test(t)?dt:void 0)),void 0!==n?null===n?void w.removeAttr(e,t):i&&"set"in i&&void 0!==(r=i.set(e,n,t))?r:(e.setAttribute(t,n+""),n):i&&"get"in i&&null!==(r=i.get(e,t))?r:null==(r=w.find.attr(e,t))?void 0:r)},attrHooks:{type:{set:function(e,t){if(!h.radioValue&&"radio"===t&&N(e,"input")){var n=e.value;return e.setAttribute("type",t),n&&(e.value=n),t}}}},removeAttr:function(e,t){var n,r=0,i=t&&t.match(M);if(i&&1===e.nodeType)while(n=i[r++])e.removeAttribute(n)}}),dt={set:function(e,t,n){return!1===t?w.removeAttr(e,n):e.setAttribute(n,n),n}},w.each(w.expr.match.bool.source.match(/\w+/g),function(e,t){var n=ht[t]||w.find.attr;ht[t]=function(e,t,r){var i,o,a=t.toLowerCase();return r||(o=ht[a],ht[a]=i,i=null!=n(e,t,r)?a:null,ht[a]=o),i}});var gt=/^(?:input|select|textarea|button)$/i,yt=/^(?:a|area)$/i;w.fn.extend({prop:function(e,t){return z(this,w.prop,e,t,arguments.length>1)},removeProp:function(e){return this.each(function(){delete this[w.propFix[e]||e]})}}),w.extend({prop:function(e,t,n){var r,i,o=e.nodeType;if(3!==o&&8!==o&&2!==o)return 1===o&&w.isXMLDoc(e)||(t=w.propFix[t]||t,i=w.propHooks[t]),void 0!==n?i&&"set"in i&&void 0!==(r=i.set(e,n,t))?r:e[t]=n:i&&"get"in i&&null!==(r=i.get(e,t))?r:e[t]},propHooks:{tabIndex:{get:function(e){var t=w.find.attr(e,"tabindex");return t?parseInt(t,10):gt.test(e.nodeName)||yt.test(e.nodeName)&&e.href?0:-1}}},propFix:{"for":"htmlFor","class":"className"}}),h.optSelected||(w.propHooks.selected={get:function(e){var t=e.parentNode;return t&&t.parentNode&&t.parentNode.selectedIndex,null},set:function(e){var t=e.parentNode;t&&(t.selectedIndex,t.parentNode&&t.parentNode.selectedIndex)}}),w.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){w.propFix[this.toLowerCase()]=this});function vt(e){return(e.match(M)||[]).join(" ")}function mt(e){return e.getAttribute&&e.getAttribute("class")||""}function xt(e){return Array.isArray(e)?e:"string"==typeof e?e.match(M)||[]:[]}w.fn.extend({addClass:function(e){var t,n,r,i,o,a,s,u=0;if(g(e))return this.each(function(t){w(this).addClass(e.call(this,t,mt(this)))});if((t=xt(e)).length)while(n=this[u++])if(i=mt(n),r=1===n.nodeType&&" "+vt(i)+" "){a=0;while(o=t[a++])r.indexOf(" "+o+" ")<0&&(r+=o+" ");i!==(s=vt(r))&&n.setAttribute("class",s)}return this},removeClass:function(e){var t,n,r,i,o,a,s,u=0;if(g(e))return this.each(function(t){w(this).removeClass(e.call(this,t,mt(this)))});if(!arguments.length)return this.attr("class","");if((t=xt(e)).length)while(n=this[u++])if(i=mt(n),r=1===n.nodeType&&" "+vt(i)+" "){a=0;while(o=t[a++])while(r.indexOf(" "+o+" ")>-1)r=r.replace(" "+o+" "," ");i!==(s=vt(r))&&n.setAttribute("class",s)}return this},toggleClass:function(e,t){var n=typeof e,r="string"===n||Array.isArray(e);return"boolean"==typeof t&&r?t?this.addClass(e):this.removeClass(e):g(e)?this.each(function(n){w(this).toggleClass(e.call(this,n,mt(this),t),t)}):this.each(function(){var t,i,o,a;if(r){i=0,o=w(this),a=xt(e);while(t=a[i++])o.hasClass(t)?o.removeClass(t):o.addClass(t)}else void 0!==e&&"boolean"!==n||((t=mt(this))&&J.set(this,"__className__",t),this.setAttribute&&this.setAttribute("class",t||!1===e?"":J.get(this,"__className__")||""))})},hasClass:function(e){var t,n,r=0;t=" "+e+" ";while(n=this[r++])if(1===n.nodeType&&(" "+vt(mt(n))+" ").indexOf(t)>-1)return!0;return!1}});var bt=/\r/g;w.fn.extend({val:function(e){var t,n,r,i=this[0];{if(arguments.length)return r=g(e),this.each(function(n){var i;1===this.nodeType&&(null==(i=r?e.call(this,n,w(this).val()):e)?i="":"number"==typeof i?i+="":Array.isArray(i)&&(i=w.map(i,function(e){return null==e?"":e+""})),(t=w.valHooks[this.type]||w.valHooks[this.nodeName.toLowerCase()])&&"set"in t&&void 0!==t.set(this,i,"value")||(this.value=i))});if(i)return(t=w.valHooks[i.type]||w.valHooks[i.nodeName.toLowerCase()])&&"get"in t&&void 0!==(n=t.get(i,"value"))?n:"string"==typeof(n=i.value)?n.replace(bt,""):null==n?"":n}}}),w.extend({valHooks:{option:{get:function(e){var t=w.find.attr(e,"value");return null!=t?t:vt(w.text(e))}},select:{get:function(e){var t,n,r,i=e.options,o=e.selectedIndex,a="select-one"===e.type,s=a?null:[],u=a?o+1:i.length;for(r=o<0?u:a?o:0;r-1)&&(n=!0);return n||(e.selectedIndex=-1),o}}}}),w.each(["radio","checkbox"],function(){w.valHooks[this]={set:function(e,t){if(Array.isArray(t))return e.checked=w.inArray(w(e).val(),t)>-1}},h.checkOn||(w.valHooks[this].get=function(e){return null===e.getAttribute("value")?"on":e.value})}),h.focusin="onfocusin"in e;var wt=/^(?:focusinfocus|focusoutblur)$/,Tt=function(e){e.stopPropagation()};w.extend(w.event,{trigger:function(t,n,i,o){var a,s,u,l,c,p,d,h,v=[i||r],m=f.call(t,"type")?t.type:t,x=f.call(t,"namespace")?t.namespace.split("."):[];if(s=h=u=i=i||r,3!==i.nodeType&&8!==i.nodeType&&!wt.test(m+w.event.triggered)&&(m.indexOf(".")>-1&&(m=(x=m.split(".")).shift(),x.sort()),c=m.indexOf(":")<0&&"on"+m,t=t[w.expando]?t:new w.Event(m,"object"==typeof t&&t),t.isTrigger=o?2:3,t.namespace=x.join("."),t.rnamespace=t.namespace?new RegExp("(^|\\.)"+x.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,t.result=void 0,t.target||(t.target=i),n=null==n?[t]:w.makeArray(n,[t]),d=w.event.special[m]||{},o||!d.trigger||!1!==d.trigger.apply(i,n))){if(!o&&!d.noBubble&&!y(i)){for(l=d.delegateType||m,wt.test(l+m)||(s=s.parentNode);s;s=s.parentNode)v.push(s),u=s;u===(i.ownerDocument||r)&&v.push(u.defaultView||u.parentWindow||e)}a=0;while((s=v[a++])&&!t.isPropagationStopped())h=s,t.type=a>1?l:d.bindType||m,(p=(J.get(s,"events")||{})[t.type]&&J.get(s,"handle"))&&p.apply(s,n),(p=c&&s[c])&&p.apply&&Y(s)&&(t.result=p.apply(s,n),!1===t.result&&t.preventDefault());return t.type=m,o||t.isDefaultPrevented()||d._default&&!1!==d._default.apply(v.pop(),n)||!Y(i)||c&&g(i[m])&&!y(i)&&((u=i[c])&&(i[c]=null),w.event.triggered=m,t.isPropagationStopped()&&h.addEventListener(m,Tt),i[m](),t.isPropagationStopped()&&h.removeEventListener(m,Tt),w.event.triggered=void 0,u&&(i[c]=u)),t.result}},simulate:function(e,t,n){var r=w.extend(new w.Event,n,{type:e,isSimulated:!0});w.event.trigger(r,null,t)}}),w.fn.extend({trigger:function(e,t){return this.each(function(){w.event.trigger(e,t,this)})},triggerHandler:function(e,t){var n=this[0];if(n)return w.event.trigger(e,t,n,!0)}}),h.focusin||w.each({focus:"focusin",blur:"focusout"},function(e,t){var n=function(e){w.event.simulate(t,e.target,w.event.fix(e))};w.event.special[t]={setup:function(){var r=this.ownerDocument||this,i=J.access(r,t);i||r.addEventListener(e,n,!0),J.access(r,t,(i||0)+1)},teardown:function(){var r=this.ownerDocument||this,i=J.access(r,t)-1;i?J.access(r,t,i):(r.removeEventListener(e,n,!0),J.remove(r,t))}}});var Ct=e.location,Et=Date.now(),kt=/\?/;w.parseXML=function(t){var n;if(!t||"string"!=typeof t)return null;try{n=(new e.DOMParser).parseFromString(t,"text/xml")}catch(e){n=void 0}return n&&!n.getElementsByTagName("parsererror").length||w.error("Invalid XML: "+t),n};var St=/\[\]$/,Dt=/\r?\n/g,Nt=/^(?:submit|button|image|reset|file)$/i,At=/^(?:input|select|textarea|keygen)/i;function jt(e,t,n,r){var i;if(Array.isArray(t))w.each(t,function(t,i){n||St.test(e)?r(e,i):jt(e+"["+("object"==typeof i&&null!=i?t:"")+"]",i,n,r)});else if(n||"object"!==x(t))r(e,t);else for(i in t)jt(e+"["+i+"]",t[i],n,r)}w.param=function(e,t){var n,r=[],i=function(e,t){var n=g(t)?t():t;r[r.length]=encodeURIComponent(e)+"="+encodeURIComponent(null==n?"":n)};if(Array.isArray(e)||e.jquery&&!w.isPlainObject(e))w.each(e,function(){i(this.name,this.value)});else for(n in e)jt(n,e[n],t,i);return r.join("&")},w.fn.extend({serialize:function(){return w.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var e=w.prop(this,"elements");return e?w.makeArray(e):this}).filter(function(){var e=this.type;return this.name&&!w(this).is(":disabled")&&At.test(this.nodeName)&&!Nt.test(e)&&(this.checked||!pe.test(e))}).map(function(e,t){var n=w(this).val();return null==n?null:Array.isArray(n)?w.map(n,function(e){return{name:t.name,value:e.replace(Dt,"\r\n")}}):{name:t.name,value:n.replace(Dt,"\r\n")}}).get()}});var qt=/%20/g,Lt=/#.*$/,Ht=/([?&])_=[^&]*/,Ot=/^(.*?):[ \t]*([^\r\n]*)$/gm,Pt=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Mt=/^(?:GET|HEAD)$/,Rt=/^\/\//,It={},Wt={},$t="*/".concat("*"),Bt=r.createElement("a");Bt.href=Ct.href;function Ft(e){return function(t,n){"string"!=typeof t&&(n=t,t="*");var r,i=0,o=t.toLowerCase().match(M)||[];if(g(n))while(r=o[i++])"+"===r[0]?(r=r.slice(1)||"*",(e[r]=e[r]||[]).unshift(n)):(e[r]=e[r]||[]).push(n)}}function _t(e,t,n,r){var i={},o=e===Wt;function a(s){var u;return i[s]=!0,w.each(e[s]||[],function(e,s){var l=s(t,n,r);return"string"!=typeof l||o||i[l]?o?!(u=l):void 0:(t.dataTypes.unshift(l),a(l),!1)}),u}return a(t.dataTypes[0])||!i["*"]&&a("*")}function zt(e,t){var n,r,i=w.ajaxSettings.flatOptions||{};for(n in t)void 0!==t[n]&&((i[n]?e:r||(r={}))[n]=t[n]);return r&&w.extend(!0,e,r),e}function Xt(e,t,n){var r,i,o,a,s=e.contents,u=e.dataTypes;while("*"===u[0])u.shift(),void 0===r&&(r=e.mimeType||t.getResponseHeader("Content-Type"));if(r)for(i in s)if(s[i]&&s[i].test(r)){u.unshift(i);break}if(u[0]in n)o=u[0];else{for(i in n){if(!u[0]||e.converters[i+" "+u[0]]){o=i;break}a||(a=i)}o=o||a}if(o)return o!==u[0]&&u.unshift(o),n[o]}function Ut(e,t,n,r){var i,o,a,s,u,l={},c=e.dataTypes.slice();if(c[1])for(a in e.converters)l[a.toLowerCase()]=e.converters[a];o=c.shift();while(o)if(e.responseFields[o]&&(n[e.responseFields[o]]=t),!u&&r&&e.dataFilter&&(t=e.dataFilter(t,e.dataType)),u=o,o=c.shift())if("*"===o)o=u;else if("*"!==u&&u!==o){if(!(a=l[u+" "+o]||l["* "+o]))for(i in l)if((s=i.split(" "))[1]===o&&(a=l[u+" "+s[0]]||l["* "+s[0]])){!0===a?a=l[i]:!0!==l[i]&&(o=s[0],c.unshift(s[1]));break}if(!0!==a)if(a&&e["throws"])t=a(t);else try{t=a(t)}catch(e){return{state:"parsererror",error:a?e:"No conversion from "+u+" to "+o}}}return{state:"success",data:t}}w.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:Ct.href,type:"GET",isLocal:Pt.test(Ct.protocol),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":$t,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/\bxml\b/,html:/\bhtml/,json:/\bjson\b/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":JSON.parse,"text xml":w.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(e,t){return t?zt(zt(e,w.ajaxSettings),t):zt(w.ajaxSettings,e)},ajaxPrefilter:Ft(It),ajaxTransport:Ft(Wt),ajax:function(t,n){"object"==typeof t&&(n=t,t=void 0),n=n||{};var i,o,a,s,u,l,c,f,p,d,h=w.ajaxSetup({},n),g=h.context||h,y=h.context&&(g.nodeType||g.jquery)?w(g):w.event,v=w.Deferred(),m=w.Callbacks("once memory"),x=h.statusCode||{},b={},T={},C="canceled",E={readyState:0,getResponseHeader:function(e){var t;if(c){if(!s){s={};while(t=Ot.exec(a))s[t[1].toLowerCase()]=t[2]}t=s[e.toLowerCase()]}return null==t?null:t},getAllResponseHeaders:function(){return c?a:null},setRequestHeader:function(e,t){return null==c&&(e=T[e.toLowerCase()]=T[e.toLowerCase()]||e,b[e]=t),this},overrideMimeType:function(e){return null==c&&(h.mimeType=e),this},statusCode:function(e){var t;if(e)if(c)E.always(e[E.status]);else for(t in e)x[t]=[x[t],e[t]];return this},abort:function(e){var t=e||C;return i&&i.abort(t),k(0,t),this}};if(v.promise(E),h.url=((t||h.url||Ct.href)+"").replace(Rt,Ct.protocol+"//"),h.type=n.method||n.type||h.method||h.type,h.dataTypes=(h.dataType||"*").toLowerCase().match(M)||[""],null==h.crossDomain){l=r.createElement("a");try{l.href=h.url,l.href=l.href,h.crossDomain=Bt.protocol+"//"+Bt.host!=l.protocol+"//"+l.host}catch(e){h.crossDomain=!0}}if(h.data&&h.processData&&"string"!=typeof h.data&&(h.data=w.param(h.data,h.traditional)),_t(It,h,n,E),c)return E;(f=w.event&&h.global)&&0==w.active++&&w.event.trigger("ajaxStart"),h.type=h.type.toUpperCase(),h.hasContent=!Mt.test(h.type),o=h.url.replace(Lt,""),h.hasContent?h.data&&h.processData&&0===(h.contentType||"").indexOf("application/x-www-form-urlencoded")&&(h.data=h.data.replace(qt,"+")):(d=h.url.slice(o.length),h.data&&(h.processData||"string"==typeof h.data)&&(o+=(kt.test(o)?"&":"?")+h.data,delete h.data),!1===h.cache&&(o=o.replace(Ht,"$1"),d=(kt.test(o)?"&":"?")+"_="+Et+++d),h.url=o+d),h.ifModified&&(w.lastModified[o]&&E.setRequestHeader("If-Modified-Since",w.lastModified[o]),w.etag[o]&&E.setRequestHeader("If-None-Match",w.etag[o])),(h.data&&h.hasContent&&!1!==h.contentType||n.contentType)&&E.setRequestHeader("Content-Type",h.contentType),E.setRequestHeader("Accept",h.dataTypes[0]&&h.accepts[h.dataTypes[0]]?h.accepts[h.dataTypes[0]]+("*"!==h.dataTypes[0]?", "+$t+"; q=0.01":""):h.accepts["*"]);for(p in h.headers)E.setRequestHeader(p,h.headers[p]);if(h.beforeSend&&(!1===h.beforeSend.call(g,E,h)||c))return E.abort();if(C="abort",m.add(h.complete),E.done(h.success),E.fail(h.error),i=_t(Wt,h,n,E)){if(E.readyState=1,f&&y.trigger("ajaxSend",[E,h]),c)return E;h.async&&h.timeout>0&&(u=e.setTimeout(function(){E.abort("timeout")},h.timeout));try{c=!1,i.send(b,k)}catch(e){if(c)throw e;k(-1,e)}}else k(-1,"No Transport");function k(t,n,r,s){var l,p,d,b,T,C=n;c||(c=!0,u&&e.clearTimeout(u),i=void 0,a=s||"",E.readyState=t>0?4:0,l=t>=200&&t<300||304===t,r&&(b=Xt(h,E,r)),b=Ut(h,b,E,l),l?(h.ifModified&&((T=E.getResponseHeader("Last-Modified"))&&(w.lastModified[o]=T),(T=E.getResponseHeader("etag"))&&(w.etag[o]=T)),204===t||"HEAD"===h.type?C="nocontent":304===t?C="notmodified":(C=b.state,p=b.data,l=!(d=b.error))):(d=C,!t&&C||(C="error",t<0&&(t=0))),E.status=t,E.statusText=(n||C)+"",l?v.resolveWith(g,[p,C,E]):v.rejectWith(g,[E,C,d]),E.statusCode(x),x=void 0,f&&y.trigger(l?"ajaxSuccess":"ajaxError",[E,h,l?p:d]),m.fireWith(g,[E,C]),f&&(y.trigger("ajaxComplete",[E,h]),--w.active||w.event.trigger("ajaxStop")))}return E},getJSON:function(e,t,n){return w.get(e,t,n,"json")},getScript:function(e,t){return w.get(e,void 0,t,"script")}}),w.each(["get","post"],function(e,t){w[t]=function(e,n,r,i){return g(n)&&(i=i||r,r=n,n=void 0),w.ajax(w.extend({url:e,type:t,dataType:i,data:n,success:r},w.isPlainObject(e)&&e))}}),w._evalUrl=function(e){return w.ajax({url:e,type:"GET",dataType:"script",cache:!0,async:!1,global:!1,"throws":!0})},w.fn.extend({wrapAll:function(e){var t;return this[0]&&(g(e)&&(e=e.call(this[0])),t=w(e,this[0].ownerDocument).eq(0).clone(!0),this[0].parentNode&&t.insertBefore(this[0]),t.map(function(){var e=this;while(e.firstElementChild)e=e.firstElementChild;return e}).append(this)),this},wrapInner:function(e){return g(e)?this.each(function(t){w(this).wrapInner(e.call(this,t))}):this.each(function(){var t=w(this),n=t.contents();n.length?n.wrapAll(e):t.append(e)})},wrap:function(e){var t=g(e);return this.each(function(n){w(this).wrapAll(t?e.call(this,n):e)})},unwrap:function(e){return this.parent(e).not("body").each(function(){w(this).replaceWith(this.childNodes)}),this}}),w.expr.pseudos.hidden=function(e){return!w.expr.pseudos.visible(e)},w.expr.pseudos.visible=function(e){return!!(e.offsetWidth||e.offsetHeight||e.getClientRects().length)},w.ajaxSettings.xhr=function(){try{return new e.XMLHttpRequest}catch(e){}};var Vt={0:200,1223:204},Gt=w.ajaxSettings.xhr();h.cors=!!Gt&&"withCredentials"in Gt,h.ajax=Gt=!!Gt,w.ajaxTransport(function(t){var n,r;if(h.cors||Gt&&!t.crossDomain)return{send:function(i,o){var a,s=t.xhr();if(s.open(t.type,t.url,t.async,t.username,t.password),t.xhrFields)for(a in t.xhrFields)s[a]=t.xhrFields[a];t.mimeType&&s.overrideMimeType&&s.overrideMimeType(t.mimeType),t.crossDomain||i["X-Requested-With"]||(i["X-Requested-With"]="XMLHttpRequest");for(a in i)s.setRequestHeader(a,i[a]);n=function(e){return function(){n&&(n=r=s.onload=s.onerror=s.onabort=s.ontimeout=s.onreadystatechange=null,"abort"===e?s.abort():"error"===e?"number"!=typeof s.status?o(0,"error"):o(s.status,s.statusText):o(Vt[s.status]||s.status,s.statusText,"text"!==(s.responseType||"text")||"string"!=typeof s.responseText?{binary:s.response}:{text:s.responseText},s.getAllResponseHeaders()))}},s.onload=n(),r=s.onerror=s.ontimeout=n("error"),void 0!==s.onabort?s.onabort=r:s.onreadystatechange=function(){4===s.readyState&&e.setTimeout(function(){n&&r()})},n=n("abort");try{s.send(t.hasContent&&t.data||null)}catch(e){if(n)throw e}},abort:function(){n&&n()}}}),w.ajaxPrefilter(function(e){e.crossDomain&&(e.contents.script=!1)}),w.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/\b(?:java|ecma)script\b/},converters:{"text script":function(e){return w.globalEval(e),e}}}),w.ajaxPrefilter("script",function(e){void 0===e.cache&&(e.cache=!1),e.crossDomain&&(e.type="GET")}),w.ajaxTransport("script",function(e){if(e.crossDomain){var t,n;return{send:function(i,o){t=w(" + + + + +
    +
    +
    +
    + + Logo + ArchiveBox + +
    +
    + Documentation   |   + Source   |   + Website +
    +
    +
    +
    + + + + + + + + + + $rows +
    BookmarkedSnapshot ($num_links)FilesOriginal URL
    + + + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/legacy/main_index_row.html b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/legacy/main_index_row.html new file mode 100644 index 0000000..9112eac --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/themes/legacy/main_index_row.html @@ -0,0 +1,16 @@ + + $bookmarked_date + + + + $title + $tags + + + + 📄 + $num_outputs + + + $url + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/util.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/util.py new file mode 100644 index 0000000..5530ab4 --- /dev/null +++ b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/util.py @@ -0,0 +1,318 @@ +__package__ = 'archivebox' + +import re +import requests +import json as pyjson + +from typing import List, Optional, Any +from pathlib import Path +from inspect import signature +from functools import wraps +from hashlib import sha256 +from urllib.parse import urlparse, quote, unquote +from html import escape, unescape +from datetime import datetime +from dateparser import parse as dateparser +from requests.exceptions import RequestException, ReadTimeout + +from .vendor.base32_crockford import encode as base32_encode # type: ignore +from w3lib.encoding import html_body_declared_encoding, http_content_type_encoding + +try: + import chardet + detect_encoding = lambda rawdata: chardet.detect(rawdata)["encoding"] +except ImportError: + detect_encoding = lambda rawdata: "utf-8" + +### Parsing Helpers + +# All of these are (str) -> str +# shortcuts to: https://docs.python.org/3/library/urllib.parse.html#url-parsing +scheme = lambda url: urlparse(url).scheme.lower() +without_scheme = lambda url: urlparse(url)._replace(scheme='').geturl().strip('//') +without_query = lambda url: urlparse(url)._replace(query='').geturl().strip('//') +without_fragment = lambda url: urlparse(url)._replace(fragment='').geturl().strip('//') +without_path = lambda url: urlparse(url)._replace(path='', fragment='', query='').geturl().strip('//') +path = lambda url: urlparse(url).path +basename = lambda url: urlparse(url).path.rsplit('/', 1)[-1] +domain = lambda url: urlparse(url).netloc +query = lambda url: urlparse(url).query +fragment = lambda url: urlparse(url).fragment +extension = lambda url: basename(url).rsplit('.', 1)[-1].lower() if '.' in basename(url) else '' +base_url = lambda url: without_scheme(url) # uniq base url used to dedupe links + +without_www = lambda url: url.replace('://www.', '://', 1) +without_trailing_slash = lambda url: url[:-1] if url[-1] == '/' else url.replace('/?', '?') +hashurl = lambda url: base32_encode(int(sha256(base_url(url).encode('utf-8')).hexdigest(), 16))[:20] + +urlencode = lambda s: s and quote(s, encoding='utf-8', errors='replace') +urldecode = lambda s: s and unquote(s) +htmlencode = lambda s: s and escape(s, quote=True) +htmldecode = lambda s: s and unescape(s) + +short_ts = lambda ts: str(parse_date(ts).timestamp()).split('.')[0] +ts_to_date = lambda ts: ts and parse_date(ts).strftime('%Y-%m-%d %H:%M') +ts_to_iso = lambda ts: ts and parse_date(ts).isoformat() + + +URL_REGEX = re.compile( + r'http[s]?://' # start matching from allowed schemes + r'(?:[a-zA-Z]|[0-9]' # followed by allowed alphanum characters + r'|[$-_@.&+]|[!*\(\),]' # or allowed symbols + r'|(?:%[0-9a-fA-F][0-9a-fA-F]))' # or allowed unicode bytes + r'[^\]\[\(\)<>"\'\s]+', # stop parsing at these symbols + re.IGNORECASE, +) + +COLOR_REGEX = re.compile(r'\[(?P\d+)(;(?P\d+)(;(?P\d+))?)?m') + +def is_static_file(url: str): + # TODO: the proper way is with MIME type detection + ext, not only extension + from .config import STATICFILE_EXTENSIONS + return extension(url).lower() in STATICFILE_EXTENSIONS + + +def enforce_types(func): + """ + Enforce function arg and kwarg types at runtime using its python3 type hints + """ + # TODO: check return type as well + + @wraps(func) + def typechecked_function(*args, **kwargs): + sig = signature(func) + + def check_argument_type(arg_key, arg_val): + try: + annotation = sig.parameters[arg_key].annotation + except KeyError: + annotation = None + + if annotation is not None and annotation.__class__ is type: + if not isinstance(arg_val, annotation): + raise TypeError( + '{}(..., {}: {}) got unexpected {} argument {}={}'.format( + func.__name__, + arg_key, + annotation.__name__, + type(arg_val).__name__, + arg_key, + str(arg_val)[:64], + ) + ) + + # check args + for arg_val, arg_key in zip(args, sig.parameters): + check_argument_type(arg_key, arg_val) + + # check kwargs + for arg_key, arg_val in kwargs.items(): + check_argument_type(arg_key, arg_val) + + return func(*args, **kwargs) + + return typechecked_function + + +def docstring(text: Optional[str]): + """attach the given docstring to the decorated function""" + def decorator(func): + if text: + func.__doc__ = text + return func + return decorator + + +@enforce_types +def str_between(string: str, start: str, end: str=None) -> str: + """(12345, , ) -> 12345""" + + content = string.split(start, 1)[-1] + if end is not None: + content = content.rsplit(end, 1)[0] + + return content + + +@enforce_types +def parse_date(date: Any) -> Optional[datetime]: + """Parse unix timestamps, iso format, and human-readable strings""" + + if date is None: + return None + + if isinstance(date, datetime): + return date + + if isinstance(date, (float, int)): + date = str(date) + + if isinstance(date, str): + return dateparser(date) + + raise ValueError('Tried to parse invalid date! {}'.format(date)) + + +@enforce_types +def download_url(url: str, timeout: int=None) -> str: + """Download the contents of a remote url and return the text""" + from .config import TIMEOUT, CHECK_SSL_VALIDITY, WGET_USER_AGENT + timeout = timeout or TIMEOUT + response = requests.get( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + ) + + content_type = response.headers.get('Content-Type', '') + encoding = http_content_type_encoding(content_type) or html_body_declared_encoding(response.text) + + if encoding is not None: + response.encoding = encoding + + return response.text + +@enforce_types +def get_headers(url: str, timeout: int=None) -> str: + """Download the contents of a remote url and return the headers""" + from .config import TIMEOUT, CHECK_SSL_VALIDITY, WGET_USER_AGENT + timeout = timeout or TIMEOUT + + try: + response = requests.head( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + allow_redirects=True, + ) + if response.status_code >= 400: + raise RequestException + except ReadTimeout: + raise + except RequestException: + response = requests.get( + url, + headers={'User-Agent': WGET_USER_AGENT}, + verify=CHECK_SSL_VALIDITY, + timeout=timeout, + stream=True + ) + + return pyjson.dumps(dict(response.headers), indent=4) + + +@enforce_types +def chrome_args(**options) -> List[str]: + """helper to build up a chrome shell command with arguments""" + + from .config import CHROME_OPTIONS + + options = {**CHROME_OPTIONS, **options} + + cmd_args = [options['CHROME_BINARY']] + + if options['CHROME_HEADLESS']: + cmd_args += ('--headless',) + + if not options['CHROME_SANDBOX']: + # assume this means we are running inside a docker container + # in docker, GPU support is limited, sandboxing is unecessary, + # and SHM is limited to 64MB by default (which is too low to be usable). + cmd_args += ( + '--no-sandbox', + '--disable-gpu', + '--disable-dev-shm-usage', + '--disable-software-rasterizer', + ) + + + if not options['CHECK_SSL_VALIDITY']: + cmd_args += ('--disable-web-security', '--ignore-certificate-errors') + + if options['CHROME_USER_AGENT']: + cmd_args += ('--user-agent={}'.format(options['CHROME_USER_AGENT']),) + + if options['RESOLUTION']: + cmd_args += ('--window-size={}'.format(options['RESOLUTION']),) + + if options['TIMEOUT']: + cmd_args += ('--timeout={}'.format((options['TIMEOUT']) * 1000),) + + if options['CHROME_USER_DATA_DIR']: + cmd_args.append('--user-data-dir={}'.format(options['CHROME_USER_DATA_DIR'])) + + return cmd_args + + +def ansi_to_html(text): + """ + Based on: https://stackoverflow.com/questions/19212665/python-converting-ansi-color-codes-to-html + """ + from .config import COLOR_DICT + + TEMPLATE = '
    ' + text = text.replace('[m', '    ') + + def single_sub(match): + argsdict = match.groupdict() + if argsdict['arg_3'] is None: + if argsdict['arg_2'] is None: + _, color = 0, argsdict['arg_1'] + else: + _, color = argsdict['arg_1'], argsdict['arg_2'] + else: + _, color = argsdict['arg_3'], argsdict['arg_2'] + + return TEMPLATE.format(COLOR_DICT[color][0]) + + return COLOR_REGEX.sub(single_sub, text) + + +class AttributeDict(dict): + """Helper to allow accessing dict values via Example.key or Example['key']""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # Recursively convert nested dicts to AttributeDicts (optional): + # for key, val in self.items(): + # if isinstance(val, dict) and type(val) is not AttributeDict: + # self[key] = AttributeDict(val) + + def __getattr__(self, attr: str) -> Any: + return dict.__getitem__(self, attr) + + def __setattr__(self, attr: str, value: Any) -> None: + return dict.__setitem__(self, attr, value) + + +class ExtendedEncoder(pyjson.JSONEncoder): + """ + Extended json serializer that supports serializing several model + fields and objects + """ + + def default(self, obj): + cls_name = obj.__class__.__name__ + + if hasattr(obj, '_asdict'): + return obj._asdict() + + elif isinstance(obj, bytes): + return obj.decode() + + elif isinstance(obj, datetime): + return obj.isoformat() + + elif isinstance(obj, Exception): + return '{}: {}'.format(obj.__class__.__name__, obj) + + elif isinstance(obj, Path): + return str(obj) + + elif cls_name in ('dict_items', 'dict_keys', 'dict_values'): + return tuple(obj) + + return pyjson.JSONEncoder.default(self, obj) + diff --git a/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/vendor/__init__.py b/archivebox-0.5.3/debian/archivebox/usr/lib/python3/dist-packages/archivebox/vendor/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/archivebox-0.5.3/debian/archivebox/usr/share/doc/archivebox/changelog.Debian.gz b/archivebox-0.5.3/debian/archivebox/usr/share/doc/archivebox/changelog.Debian.gz new file mode 100644 index 0000000..8f8bf75 Binary files /dev/null and b/archivebox-0.5.3/debian/archivebox/usr/share/doc/archivebox/changelog.Debian.gz differ diff --git a/archivebox-0.5.3/debian/changelog b/archivebox-0.5.3/debian/changelog new file mode 100644 index 0000000..073c0a4 --- /dev/null +++ b/archivebox-0.5.3/debian/changelog @@ -0,0 +1,5 @@ +archivebox (0.5.3-1) focal; urgency=low + + * source package automatically created by stdeb 0.10.0 + + -- Nick Sweeting Fri, 08 Jan 2021 08:14:46 -0500 diff --git a/archivebox-0.5.3/debian/compat b/archivebox-0.5.3/debian/compat new file mode 100644 index 0000000..ec63514 --- /dev/null +++ b/archivebox-0.5.3/debian/compat @@ -0,0 +1 @@ +9 diff --git a/archivebox-0.5.3/debian/control b/archivebox-0.5.3/debian/control new file mode 100644 index 0000000..5e6bb06 --- /dev/null +++ b/archivebox-0.5.3/debian/control @@ -0,0 +1,34 @@ +Source: archivebox +Maintainer: Nick Sweeting +Section: python +Priority: optional +Build-Depends: python3-setuptools, python3-all, debhelper (>= 9), dh-python, python3-pip, python3-setuptools, python3-wheel, python3-stdeb +Standards-Version: 3.9.1 +Homepage: https://github.com/ArchiveBox/ArchiveBox +X-Python-Version: >= 3.7 + +Package: archivebox +Architecture: all +Depends: ${misc:Depends}, ${python3:Depends}, nodejs, chromium-browser, wget, curl, git, ffmpeg, youtube-dl, python3-atomicwrites, python3-croniter, python3-crontab, python3-dateparser, python3-django, python3-django-extensions, python3-django-jsonfield, python3-mypy-extensions, python3-requests, python3-w3lib, ripgrep +Description: The self-hosted internet archive. +
    + +

    ArchiveBox
    The open-source self-hosted web archive.

    + . + ▶️ Quickstart | + Demo | + Github | + Documentation | + Info & Motivation | + Community | + Roadmap + . + 
    + "Your own personal internet archive" (网站存档 / 爬虫)
+
    + . + + . + + + diff --git a/archivebox-0.5.3/debian/files b/archivebox-0.5.3/debian/files new file mode 100644 index 0000000..8e2e065 --- /dev/null +++ b/archivebox-0.5.3/debian/files @@ -0,0 +1,2 @@ +archivebox_0.5.3-1_all.deb python optional +archivebox_0.5.3-1_amd64.buildinfo python optional diff --git a/archivebox-0.5.3/debian/rules b/archivebox-0.5.3/debian/rules new file mode 100755 index 0000000..e5863d4 --- /dev/null +++ b/archivebox-0.5.3/debian/rules @@ -0,0 +1,21 @@ +#!/usr/bin/make -f + +# This file was automatically generated by stdeb 0.10.0 at +# Fri, 08 Jan 2021 08:14:46 -0500 + +%: + dh $@ --with python3 --buildsystem=python_distutils + +override_dh_auto_clean: + python3 setup.py clean -a + find . -name \*.pyc -exec rm {} \; + +override_dh_auto_build: + python3 setup.py build --force + +override_dh_auto_install: + python3 setup.py install --force --root=debian/archivebox --no-compile -O0 --install-layout=deb --prefix=/usr + +override_dh_python2: + dh_python2 --no-guessing-versions + diff --git a/archivebox-0.5.3/debian/source/format b/archivebox-0.5.3/debian/source/format new file mode 100644 index 0000000..163aaf8 --- /dev/null +++ b/archivebox-0.5.3/debian/source/format @@ -0,0 +1 @@ +3.0 (quilt) diff --git a/archivebox-0.5.3/debian/source/options b/archivebox-0.5.3/debian/source/options new file mode 100644 index 0000000..bcc4bbb --- /dev/null +++ b/archivebox-0.5.3/debian/source/options @@ -0,0 +1 @@ +extend-diff-ignore="\.egg-info$" \ No newline at end of file diff --git a/archivebox-0.5.3/setup.cfg b/archivebox-0.5.3/setup.cfg new file mode 100644 index 0000000..8bfd5a1 --- /dev/null +++ b/archivebox-0.5.3/setup.cfg @@ -0,0 +1,4 @@ +[egg_info] +tag_build = +tag_date = 0 + diff --git a/archivebox-0.5.3/setup.py b/archivebox-0.5.3/setup.py new file mode 100755 index 0000000..0754823 --- /dev/null +++ b/archivebox-0.5.3/setup.py @@ -0,0 +1,123 @@ +import json +import setuptools + +from pathlib import Path + + +PKG_NAME = "archivebox" +DESCRIPTION = "The self-hosted internet archive." +LICENSE = "MIT" +AUTHOR = "Nick Sweeting" +AUTHOR_EMAIL="git@nicksweeting.com" +REPO_URL = "https://github.com/ArchiveBox/ArchiveBox" +PROJECT_URLS = { + "Source": f"{REPO_URL}", + "Documentation": f"{REPO_URL}/wiki", + "Bug Tracker": f"{REPO_URL}/issues", + "Changelog": f"{REPO_URL}/wiki/Changelog", + "Roadmap": f"{REPO_URL}/wiki/Roadmap", + "Community": f"{REPO_URL}/wiki/Web-Archiving-Community", + "Donate": f"{REPO_URL}/wiki/Donations", +} + +ROOT_DIR = Path(__file__).parent.resolve() +PACKAGE_DIR = ROOT_DIR / PKG_NAME + +README = (PACKAGE_DIR / "README.md").read_text(encoding='utf-8', errors='ignore') +VERSION = json.loads((PACKAGE_DIR / "package.json").read_text().strip())['version'] + +# To see when setup.py gets called (uncomment for debugging): +# import sys +# print(PACKAGE_DIR, f" (v{VERSION})") +# print('>', sys.executable, *sys.argv) + + +setuptools.setup( + name=PKG_NAME, + version=VERSION, + license=LICENSE, + author=AUTHOR, + author_email=AUTHOR_EMAIL, + description=DESCRIPTION, + long_description=README, + long_description_content_type="text/markdown", + url=REPO_URL, + project_urls=PROJECT_URLS, + python_requires=">=3.7", + setup_requires=[ + "wheel", + ], + install_requires=[ + # only add things here that have corresponding apt python3-packages available + # anything added here also needs to be added to our package dependencies in + # stdeb.cfg (apt), archivebox.rb (brew), Dockerfile, etc. + # if there is no apt python3-package equivalent, then vendor it instead in + # ./archivebox/vendor/ + "requests==2.24.0", + "atomicwrites==1.4.0", + "mypy-extensions==0.4.3", + "django==3.1.3", + "django-extensions==3.0.3", + "dateparser", + "ipython", + "youtube-dl", + "python-crontab==2.5.1", + "croniter==0.3.34", + "w3lib==1.22.0", + ], + extras_require={ + 'dev': [ + "setuptools", + "twine", + "wheel", + "flake8", + "ipdb", + "mypy", + "django-stubs", + "sphinx", + "sphinx-rtd-theme", + "recommonmark", + "pytest", + "bottle", + "stdeb", + ], + }, + packages=[PKG_NAME], + include_package_data=True, # see MANIFEST.in + entry_points={ + "console_scripts": [ + f"{PKG_NAME} = {PKG_NAME}.cli:main", + ], + }, + classifiers=[ + "License :: OSI Approved :: MIT License", + "Natural Language :: English", + "Operating System :: OS Independent", + "Development Status :: 4 - Beta", + + "Topic :: Utilities", + "Topic :: System :: Archiving", + "Topic :: System :: Archiving :: Backup", + "Topic :: System :: Recovery Tools", + "Topic :: Sociology :: History", + "Topic :: Internet :: WWW/HTTP", + "Topic :: Internet :: WWW/HTTP :: Indexing/Search", + "Topic :: Internet :: WWW/HTTP :: WSGI :: Application", + "Topic :: Software Development :: Libraries :: Python Modules", + + "Intended Audience :: Developers", + "Intended Audience :: Education", + "Intended Audience :: End Users/Desktop", + "Intended Audience :: Information Technology", + "Intended Audience :: Legal Industry", + "Intended Audience :: System Administrators", + + "Environment :: Console", + "Environment :: Web Environment", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", + "Framework :: Django", + "Typing :: Typed", + ], +) diff --git a/archivebox_0.5.3-1.debian.tar.xz b/archivebox_0.5.3-1.debian.tar.xz new file mode 100644 index 0000000..bb5e4ca Binary files /dev/null and b/archivebox_0.5.3-1.debian.tar.xz differ diff --git a/archivebox_0.5.3-1.dsc b/archivebox_0.5.3-1.dsc new file mode 100644 index 0000000..42dfee6 --- /dev/null +++ b/archivebox_0.5.3-1.dsc @@ -0,0 +1,40 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA512 + +Format: 3.0 (quilt) +Source: archivebox +Binary: archivebox +Architecture: all +Version: 0.5.3-1 +Maintainer: Nick Sweeting +Homepage: https://github.com/ArchiveBox/ArchiveBox +Standards-Version: 3.9.1 +Build-Depends: python3-setuptools, python3-all, debhelper (>= 9), dh-python, python3-pip, python3-wheel, python3-stdeb +Package-List: + archivebox deb python optional arch=all +Checksums-Sha1: + a350c46e25ac3d3b30fab0efbd0226e5c0fe28a3 258683 archivebox_0.5.3.orig.tar.gz + d1952afed11f3a21e5e3794779edcd3ff6207df5 1556 archivebox_0.5.3-1.debian.tar.xz +Checksums-Sha256: + 50b2c976296734d6e6b54975826c2ada9cd13f2c15dd585a2a94553530cf1541 258683 archivebox_0.5.3.orig.tar.gz + 9bdd60a9601e9dc7768789a4bb865b24b0cb0583887ff4f1f4106fdcae8d8cd7 1556 archivebox_0.5.3-1.debian.tar.xz +Files: + a69bba33bee581722227d9fecdb78f75 258683 archivebox_0.5.3.orig.tar.gz + 77de8c5ee4b0158d7d35b162e1e04417 1556 archivebox_0.5.3-1.debian.tar.xz + +-----BEGIN PGP SIGNATURE----- + +iQIzBAEBCgAdFiEEfVaV07YYhyZHhh1Rw4E3p8FnWYgFAl/4Ws4ACgkQw4E3p8Fn +WYhR1w/+Ib5tG2bcImEI6wQF9+8CC3zYY69XGlfiFp3NOFFZeKBlDugd5h5Wz2TZ +o7GF2N/tbaTWjSpTYTTLzEu54a1aavOX6dgBUBpMkFpCVycUvodQyetegu4ts4NK +CVNB22bUR9lgYIptYKCDza03lMVL7GXCWlafJ96Xs/iDMn8hVquvbEYREzrab2BA +WmQZlmWi9dwU+LJygct5mkZ+CQANiSMaTsD+E2pZL9BEQ5j5hrh2TiVFgL7TqPmV +n4aVV9f2m6uSmpPidajtb857T8O1s7Wt+jn1YYM2Hg1O/A6FaZUX4Osi6fVh5YvY +Qa/eOy3NMv+60UP00ouWcyf1CQY3aYrM4ROA+CfP6EFY85qcGrrGKPzyKR9airfc +lmJ3CEA39Dq+FrxqLTFEIv+DvHpE/LdkVvd5MTj35sIJ1hy06Rwcz3Squhiaslxq +4ZDKQot/Qp1lhD68KBOlrV2POH3QqX6duglbpFP3eGUi56zFB7fosdHQIooiBZp4 +7cqdQq71I0iDLJW/9hRBIAnrHwwJS3s5ZMz3+vjxMojjK0yN9JwZwoYG9xoflYlH +LdM2O1v7uye4ljZhpAcCaM0qdjLy/YrtdfDUS36N0jHV2Rjo7847EC82WLk5MNCx +ivbRUG6qHdkAAQQmA+V0BmNz0EjLAFHyuXr6N8ejwUpNucmms5g= +=5Si8 +-----END PGP SIGNATURE----- diff --git a/archivebox_0.5.3-1_all.deb b/archivebox_0.5.3-1_all.deb new file mode 100644 index 0000000..dce55fb Binary files /dev/null and b/archivebox_0.5.3-1_all.deb differ diff --git a/archivebox_0.5.3-1_amd64.buildinfo b/archivebox_0.5.3-1_amd64.buildinfo new file mode 100644 index 0000000..1819dd9 --- /dev/null +++ b/archivebox_0.5.3-1_amd64.buildinfo @@ -0,0 +1,200 @@ +Format: 1.0 +Source: archivebox +Binary: archivebox +Architecture: all +Version: 0.5.3-1 +Checksums-Md5: + a33259a861e3555310abd95e0247cd7d 194244 archivebox_0.5.3-1_all.deb +Checksums-Sha1: + 87f8e8b136545f3ea0e0bd46503a06ddee4295b2 194244 archivebox_0.5.3-1_all.deb +Checksums-Sha256: + 176d539e4139cf2e618b12ce299791cfcdd861a3004a758952dea998dfd09ef3 194244 archivebox_0.5.3-1_all.deb +Build-Origin: Ubuntu +Build-Architecture: amd64 +Build-Date: Fri, 08 Jan 2021 08:14:53 -0500 +Build-Tainted-By: + merged-usr-via-symlinks + usr-local-has-configs + usr-local-has-libraries + usr-local-has-programs +Installed-Build-Depends: + autoconf (= 2.69-11.1), + automake (= 1:1.16.1-4ubuntu6), + autopoint (= 0.19.8.1-10build1), + autotools-dev (= 20180224.1), + base-files (= 11ubuntu5.2), + base-passwd (= 3.5.47), + bash (= 5.0-6ubuntu1.1), + binutils (= 2.34-6ubuntu1), + binutils-common (= 2.34-6ubuntu1), + binutils-x86-64-linux-gnu (= 2.34-6ubuntu1), + bsdmainutils (= 11.1.2ubuntu3), + bsdutils (= 1:2.34-0.1ubuntu9.1), + build-essential (= 12.8ubuntu1.1), + bzip2 (= 1.0.8-2), + ca-certificates (= 20201027ubuntu0.20.04.1), + coreutils (= 8.30-3ubuntu2), + cpp (= 4:9.3.0-1ubuntu2), + cpp-9 (= 9.3.0-17ubuntu1~20.04), + dash (= 0.5.10.2-6), + debconf (= 1.5.73), + debhelper (= 12.10ubuntu1), + debianutils (= 4.9.1), + dh-autoreconf (= 19), + dh-python (= 4.20191017ubuntu7), + dh-strip-nondeterminism (= 1.7.0-1), + diffutils (= 1:3.7-3), + dpkg (= 1.19.7ubuntu3), + dpkg-dev (= 1.19.7ubuntu3), + dwz (= 0.13-5), + file (= 1:5.38-4), + findutils (= 4.7.0-1ubuntu1), + g++ (= 4:9.3.0-1ubuntu2), + g++-9 (= 9.3.0-17ubuntu1~20.04), + gcc (= 4:9.3.0-1ubuntu2), + gcc-10-base (= 10.2.0-5ubuntu1~20.04), + gcc-9 (= 9.3.0-17ubuntu1~20.04), + gcc-9-base (= 9.3.0-17ubuntu1~20.04), + gettext (= 0.19.8.1-10build1), + gettext-base (= 0.19.8.1-10build1), + grep (= 3.4-1), + groff-base (= 1.22.4-4build1), + gzip (= 1.10-0ubuntu4), + hostname (= 3.23), + init-system-helpers (= 1.57), + install-info (= 6.7.0.dfsg.2-5), + intltool-debian (= 0.35.0+20060710.5), + libacl1 (= 2.2.53-6), + libarchive-zip-perl (= 1.67-2), + libasan5 (= 9.3.0-17ubuntu1~20.04), + libatomic1 (= 10.2.0-5ubuntu1~20.04), + libattr1 (= 1:2.4.48-5), + libaudit-common (= 1:2.8.5-2ubuntu6), + libaudit1 (= 1:2.8.5-2ubuntu6), + libbinutils (= 2.34-6ubuntu1), + libblkid1 (= 2.34-0.1ubuntu9.1), + libbsd0 (= 0.10.0-1), + libbz2-1.0 (= 1.0.8-2), + libc-bin (= 2.31-0ubuntu9.1), + libc-dev-bin (= 2.31-0ubuntu9.1), + libc6 (= 2.31-0ubuntu9.1), + libc6-dev (= 2.31-0ubuntu9.1), + libcap-ng0 (= 0.7.9-2.1build1), + libcc1-0 (= 10.2.0-5ubuntu1~20.04), + libcroco3 (= 0.6.13-1), + libcrypt-dev (= 1:4.4.10-10ubuntu4), + libcrypt1 (= 1:4.4.10-10ubuntu4), + libctf-nobfd0 (= 2.34-6ubuntu1), + libctf0 (= 2.34-6ubuntu1), + libdb5.3 (= 5.3.28+dfsg1-0.6ubuntu2), + libdebconfclient0 (= 0.251ubuntu1), + libdebhelper-perl (= 12.10ubuntu1), + libdpkg-perl (= 1.19.7ubuntu3), + libelf1 (= 0.176-1.1build1), + libexpat1 (= 2.2.9-1build1), + libffi7 (= 3.3-4), + libfile-stripnondeterminism-perl (= 1.7.0-1), + libgcc-9-dev (= 9.3.0-17ubuntu1~20.04), + libgcc-s1 (= 10.2.0-5ubuntu1~20.04), + libgcrypt20 (= 1.8.5-5ubuntu1), + libgdbm-compat4 (= 1.18.1-5), + libgdbm6 (= 1.18.1-5), + libglib2.0-0 (= 2.64.3-1~ubuntu20.04.1), + libgmp10 (= 2:6.2.0+dfsg-4), + libgomp1 (= 10.2.0-5ubuntu1~20.04), + libgpg-error0 (= 1.37-1), + libicu66 (= 66.1-2ubuntu2), + libisl22 (= 0.22.1-1), + libitm1 (= 10.2.0-5ubuntu1~20.04), + liblsan0 (= 10.2.0-5ubuntu1~20.04), + liblz4-1 (= 1.9.2-2), + liblzma5 (= 5.2.4-1ubuntu1), + libmagic-mgc (= 1:5.38-4), + libmagic1 (= 1:5.38-4), + libmount1 (= 2.34-0.1ubuntu9.1), + libmpc3 (= 1.1.0-1), + libmpdec2 (= 2.4.2-3), + libmpfr6 (= 4.0.2-1), + libncursesw6 (= 6.2-0ubuntu2), + libpam-modules (= 1.3.1-5ubuntu4.1), + libpam-modules-bin (= 1.3.1-5ubuntu4.1), + libpam-runtime (= 1.3.1-5ubuntu4.1), + libpam0g (= 1.3.1-5ubuntu4.1), + libpcre2-8-0 (= 10.34-7), + libpcre3 (= 2:8.39-12build1), + libperl5.30 (= 5.30.0-9ubuntu0.2), + libpipeline1 (= 1.5.2-2build1), + libpython3-stdlib (= 3.8.2-0ubuntu2), + libpython3.8-minimal (= 3.8.5-1~20.04), + libpython3.8-stdlib (= 3.8.5-1~20.04), + libquadmath0 (= 10.2.0-5ubuntu1~20.04), + libreadline8 (= 8.0-4), + libseccomp2 (= 2.4.3-1ubuntu3.20.04.3), + libselinux1 (= 3.0-1build2), + libsigsegv2 (= 2.12-2), + libsmartcols1 (= 2.34-0.1ubuntu9.1), + libsqlite3-0 (= 3.31.1-4ubuntu0.2), + libssl1.1 (= 1.1.1f-1ubuntu2.1), + libstdc++-9-dev (= 9.3.0-17ubuntu1~20.04), + libstdc++6 (= 10.2.0-5ubuntu1~20.04), + libsub-override-perl (= 0.09-2), + libsystemd0 (= 245.4-4ubuntu3.3), + libtinfo6 (= 6.2-0ubuntu2), + libtool (= 2.4.6-14), + libtsan0 (= 10.2.0-5ubuntu1~20.04), + libubsan1 (= 10.2.0-5ubuntu1~20.04), + libuchardet0 (= 0.0.6-3build1), + libudev1 (= 245.4-4ubuntu3.3), + libunistring2 (= 0.9.10-2), + libuuid1 (= 2.34-0.1ubuntu9.1), + libxml2 (= 2.9.10+dfsg-5), + libzstd1 (= 1.4.4+dfsg-3), + linux-libc-dev (= 5.4.0-59.65), + login (= 1:4.8.1-1ubuntu5.20.04), + lsb-base (= 11.1.0ubuntu2), + m4 (= 1.4.18-4), + make (= 4.2.1-1.2), + man-db (= 2.9.1-1), + mawk (= 1.3.4.20200120-2), + mime-support (= 3.64ubuntu1), + ncurses-base (= 6.2-0ubuntu2), + ncurses-bin (= 6.2-0ubuntu2), + openssl (= 1.1.1f-1ubuntu2.1), + patch (= 2.7.6-6), + perl (= 5.30.0-9ubuntu0.2), + perl-base (= 5.30.0-9ubuntu0.2), + perl-modules-5.30 (= 5.30.0-9ubuntu0.2), + po-debconf (= 1.0.21), + python-pip-whl (= 20.0.2-5ubuntu1.1), + python3 (= 3.8.2-0ubuntu2), + python3-all (= 3.8.2-0ubuntu2), + python3-certifi (= 2019.11.28-1), + python3-chardet (= 3.0.4-4build1), + python3-distutils (= 3.8.5-1~20.04.1), + python3-idna (= 2.8-1), + python3-lib2to3 (= 3.8.5-1~20.04.1), + python3-minimal (= 3.8.2-0ubuntu2), + python3-pip (= 20.0.2-5ubuntu1.1), + python3-pkg-resources (= 45.2.0-1), + python3-requests (= 2.22.0-2ubuntu1), + python3-setuptools (= 45.2.0-1), + python3-six (= 1.14.0-2), + python3-stdeb (= 0.8.5-3), + python3-urllib3 (= 1.25.8-2ubuntu0.1), + python3-wheel (= 0.34.2-1), + python3.8 (= 3.8.5-1~20.04), + python3.8-minimal (= 3.8.5-1~20.04), + readline-common (= 8.0-4), + sed (= 4.7-1), + sensible-utils (= 0.0.12+nmu1), + sysvinit-utils (= 2.96-2.1ubuntu1), + tar (= 1.30+dfsg-7), + tzdata (= 2020d-0ubuntu0.20.04), + util-linux (= 2.34-0.1ubuntu9.1), + xz-utils (= 5.2.4-1ubuntu1), + zlib1g (= 1:1.2.11.dfsg-2ubuntu1.2) +Environment: + DEB_BUILD_OPTIONS="parallel=24" + LANG="en_US.UTF-8" + LC_ALL="en_US.UTF-8" + SOURCE_DATE_EPOCH="1610111686" diff --git a/archivebox_0.5.3-1_amd64.changes b/archivebox_0.5.3-1_amd64.changes new file mode 100644 index 0000000..07f9365 --- /dev/null +++ b/archivebox_0.5.3-1_amd64.changes @@ -0,0 +1,25 @@ +Format: 1.8 +Date: Fri, 08 Jan 2021 08:14:46 -0500 +Source: archivebox +Binary: archivebox +Architecture: all +Version: 0.5.3-1 +Distribution: focal +Urgency: low +Maintainer: Nick Sweeting +Changed-By: Nick Sweeting +Description: + archivebox - The self-hosted internet archive. +Changes: + archivebox (0.5.3-1) focal; urgency=low + . + * source package automatically created by stdeb 0.10.0 +Checksums-Sha1: + 87f8e8b136545f3ea0e0bd46503a06ddee4295b2 194244 archivebox_0.5.3-1_all.deb + 37ee2c2f698b6ae41cc226f723541391c75e1e64 6265 archivebox_0.5.3-1_amd64.buildinfo +Checksums-Sha256: + 176d539e4139cf2e618b12ce299791cfcdd861a3004a758952dea998dfd09ef3 194244 archivebox_0.5.3-1_all.deb + 7229749267a5d2bbcfe9451320ce8f301aab6747343213d1e80d1c691b5e119f 6265 archivebox_0.5.3-1_amd64.buildinfo +Files: + a33259a861e3555310abd95e0247cd7d 194244 python optional archivebox_0.5.3-1_all.deb + 8ffc941fcedb835850be69ccddf65885 6265 python optional archivebox_0.5.3-1_amd64.buildinfo diff --git a/archivebox_0.5.3-1_source.archivebox-ppa.upload b/archivebox_0.5.3-1_source.archivebox-ppa.upload new file mode 100644 index 0000000..4d55b4b --- /dev/null +++ b/archivebox_0.5.3-1_source.archivebox-ppa.upload @@ -0,0 +1,5 @@ +Successfully uploaded archivebox_0.5.3-1.dsc to ppa.launchpad.net for archivebox-ppa. +Successfully uploaded archivebox_0.5.3.orig.tar.gz to ppa.launchpad.net for archivebox-ppa. +Successfully uploaded archivebox_0.5.3-1.debian.tar.xz to ppa.launchpad.net for archivebox-ppa. +Successfully uploaded archivebox_0.5.3-1_source.buildinfo to ppa.launchpad.net for archivebox-ppa. +Successfully uploaded archivebox_0.5.3-1_source.changes to ppa.launchpad.net for archivebox-ppa. diff --git a/archivebox_0.5.3-1_source.buildinfo b/archivebox_0.5.3-1_source.buildinfo new file mode 100644 index 0000000..658cd4c --- /dev/null +++ b/archivebox_0.5.3-1_source.buildinfo @@ -0,0 +1,220 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA512 + +Format: 1.0 +Source: archivebox +Binary: archivebox +Architecture: source +Version: 0.5.3-1 +Checksums-Md5: + a8c145339737010beb02a81925cf942a 1822 archivebox_0.5.3-1.dsc +Checksums-Sha1: + ebc9cb5985ca3b9749af86cf3f79a0250888c651 1822 archivebox_0.5.3-1.dsc +Checksums-Sha256: + fd34ba74d913d39b90b9e487a504ed094c7371272c41e902c818091619d58ca0 1822 archivebox_0.5.3-1.dsc +Build-Origin: Ubuntu +Build-Architecture: amd64 +Build-Date: Fri, 08 Jan 2021 08:14:48 -0500 +Build-Tainted-By: + merged-usr-via-symlinks + usr-local-has-configs + usr-local-has-libraries + usr-local-has-programs +Installed-Build-Depends: + autoconf (= 2.69-11.1), + automake (= 1:1.16.1-4ubuntu6), + autopoint (= 0.19.8.1-10build1), + autotools-dev (= 20180224.1), + base-files (= 11ubuntu5.2), + base-passwd (= 3.5.47), + bash (= 5.0-6ubuntu1.1), + binutils (= 2.34-6ubuntu1), + binutils-common (= 2.34-6ubuntu1), + binutils-x86-64-linux-gnu (= 2.34-6ubuntu1), + bsdmainutils (= 11.1.2ubuntu3), + bsdutils (= 1:2.34-0.1ubuntu9.1), + build-essential (= 12.8ubuntu1.1), + bzip2 (= 1.0.8-2), + ca-certificates (= 20201027ubuntu0.20.04.1), + coreutils (= 8.30-3ubuntu2), + cpp (= 4:9.3.0-1ubuntu2), + cpp-9 (= 9.3.0-17ubuntu1~20.04), + dash (= 0.5.10.2-6), + debconf (= 1.5.73), + debhelper (= 12.10ubuntu1), + debianutils (= 4.9.1), + dh-autoreconf (= 19), + dh-python (= 4.20191017ubuntu7), + dh-strip-nondeterminism (= 1.7.0-1), + diffutils (= 1:3.7-3), + dpkg (= 1.19.7ubuntu3), + dpkg-dev (= 1.19.7ubuntu3), + dwz (= 0.13-5), + file (= 1:5.38-4), + findutils (= 4.7.0-1ubuntu1), + g++ (= 4:9.3.0-1ubuntu2), + g++-9 (= 9.3.0-17ubuntu1~20.04), + gcc (= 4:9.3.0-1ubuntu2), + gcc-10-base (= 10.2.0-5ubuntu1~20.04), + gcc-9 (= 9.3.0-17ubuntu1~20.04), + gcc-9-base (= 9.3.0-17ubuntu1~20.04), + gettext (= 0.19.8.1-10build1), + gettext-base (= 0.19.8.1-10build1), + grep (= 3.4-1), + groff-base (= 1.22.4-4build1), + gzip (= 1.10-0ubuntu4), + hostname (= 3.23), + init-system-helpers (= 1.57), + install-info (= 6.7.0.dfsg.2-5), + intltool-debian (= 0.35.0+20060710.5), + libacl1 (= 2.2.53-6), + libarchive-zip-perl (= 1.67-2), + libasan5 (= 9.3.0-17ubuntu1~20.04), + libatomic1 (= 10.2.0-5ubuntu1~20.04), + libattr1 (= 1:2.4.48-5), + libaudit-common (= 1:2.8.5-2ubuntu6), + libaudit1 (= 1:2.8.5-2ubuntu6), + libbinutils (= 2.34-6ubuntu1), + libblkid1 (= 2.34-0.1ubuntu9.1), + libbsd0 (= 0.10.0-1), + libbz2-1.0 (= 1.0.8-2), + libc-bin (= 2.31-0ubuntu9.1), + libc-dev-bin (= 2.31-0ubuntu9.1), + libc6 (= 2.31-0ubuntu9.1), + libc6-dev (= 2.31-0ubuntu9.1), + libcap-ng0 (= 0.7.9-2.1build1), + libcc1-0 (= 10.2.0-5ubuntu1~20.04), + libcroco3 (= 0.6.13-1), + libcrypt-dev (= 1:4.4.10-10ubuntu4), + libcrypt1 (= 1:4.4.10-10ubuntu4), + libctf-nobfd0 (= 2.34-6ubuntu1), + libctf0 (= 2.34-6ubuntu1), + libdb5.3 (= 5.3.28+dfsg1-0.6ubuntu2), + libdebconfclient0 (= 0.251ubuntu1), + libdebhelper-perl (= 12.10ubuntu1), + libdpkg-perl (= 1.19.7ubuntu3), + libelf1 (= 0.176-1.1build1), + libexpat1 (= 2.2.9-1build1), + libffi7 (= 3.3-4), + libfile-stripnondeterminism-perl (= 1.7.0-1), + libgcc-9-dev (= 9.3.0-17ubuntu1~20.04), + libgcc-s1 (= 10.2.0-5ubuntu1~20.04), + libgcrypt20 (= 1.8.5-5ubuntu1), + libgdbm-compat4 (= 1.18.1-5), + libgdbm6 (= 1.18.1-5), + libglib2.0-0 (= 2.64.3-1~ubuntu20.04.1), + libgmp10 (= 2:6.2.0+dfsg-4), + libgomp1 (= 10.2.0-5ubuntu1~20.04), + libgpg-error0 (= 1.37-1), + libicu66 (= 66.1-2ubuntu2), + libisl22 (= 0.22.1-1), + libitm1 (= 10.2.0-5ubuntu1~20.04), + liblsan0 (= 10.2.0-5ubuntu1~20.04), + liblz4-1 (= 1.9.2-2), + liblzma5 (= 5.2.4-1ubuntu1), + libmagic-mgc (= 1:5.38-4), + libmagic1 (= 1:5.38-4), + libmount1 (= 2.34-0.1ubuntu9.1), + libmpc3 (= 1.1.0-1), + libmpdec2 (= 2.4.2-3), + libmpfr6 (= 4.0.2-1), + libncursesw6 (= 6.2-0ubuntu2), + libpam-modules (= 1.3.1-5ubuntu4.1), + libpam-modules-bin (= 1.3.1-5ubuntu4.1), + libpam-runtime (= 1.3.1-5ubuntu4.1), + libpam0g (= 1.3.1-5ubuntu4.1), + libpcre2-8-0 (= 10.34-7), + libpcre3 (= 2:8.39-12build1), + libperl5.30 (= 5.30.0-9ubuntu0.2), + libpipeline1 (= 1.5.2-2build1), + libpython3-stdlib (= 3.8.2-0ubuntu2), + libpython3.8-minimal (= 3.8.5-1~20.04), + libpython3.8-stdlib (= 3.8.5-1~20.04), + libquadmath0 (= 10.2.0-5ubuntu1~20.04), + libreadline8 (= 8.0-4), + libseccomp2 (= 2.4.3-1ubuntu3.20.04.3), + libselinux1 (= 3.0-1build2), + libsigsegv2 (= 2.12-2), + libsmartcols1 (= 2.34-0.1ubuntu9.1), + libsqlite3-0 (= 3.31.1-4ubuntu0.2), + libssl1.1 (= 1.1.1f-1ubuntu2.1), + libstdc++-9-dev (= 9.3.0-17ubuntu1~20.04), + libstdc++6 (= 10.2.0-5ubuntu1~20.04), + libsub-override-perl (= 0.09-2), + libsystemd0 (= 245.4-4ubuntu3.3), + libtinfo6 (= 6.2-0ubuntu2), + libtool (= 2.4.6-14), + libtsan0 (= 10.2.0-5ubuntu1~20.04), + libubsan1 (= 10.2.0-5ubuntu1~20.04), + libuchardet0 (= 0.0.6-3build1), + libudev1 (= 245.4-4ubuntu3.3), + libunistring2 (= 0.9.10-2), + libuuid1 (= 2.34-0.1ubuntu9.1), + libxml2 (= 2.9.10+dfsg-5), + libzstd1 (= 1.4.4+dfsg-3), + linux-libc-dev (= 5.4.0-59.65), + login (= 1:4.8.1-1ubuntu5.20.04), + lsb-base (= 11.1.0ubuntu2), + m4 (= 1.4.18-4), + make (= 4.2.1-1.2), + man-db (= 2.9.1-1), + mawk (= 1.3.4.20200120-2), + mime-support (= 3.64ubuntu1), + ncurses-base (= 6.2-0ubuntu2), + ncurses-bin (= 6.2-0ubuntu2), + openssl (= 1.1.1f-1ubuntu2.1), + patch (= 2.7.6-6), + perl (= 5.30.0-9ubuntu0.2), + perl-base (= 5.30.0-9ubuntu0.2), + perl-modules-5.30 (= 5.30.0-9ubuntu0.2), + po-debconf (= 1.0.21), + python-pip-whl (= 20.0.2-5ubuntu1.1), + python3 (= 3.8.2-0ubuntu2), + python3-all (= 3.8.2-0ubuntu2), + python3-certifi (= 2019.11.28-1), + python3-chardet (= 3.0.4-4build1), + python3-distutils (= 3.8.5-1~20.04.1), + python3-idna (= 2.8-1), + python3-lib2to3 (= 3.8.5-1~20.04.1), + python3-minimal (= 3.8.2-0ubuntu2), + python3-pip (= 20.0.2-5ubuntu1.1), + python3-pkg-resources (= 45.2.0-1), + python3-requests (= 2.22.0-2ubuntu1), + python3-setuptools (= 45.2.0-1), + python3-six (= 1.14.0-2), + python3-stdeb (= 0.8.5-3), + python3-urllib3 (= 1.25.8-2ubuntu0.1), + python3-wheel (= 0.34.2-1), + python3.8 (= 3.8.5-1~20.04), + python3.8-minimal (= 3.8.5-1~20.04), + readline-common (= 8.0-4), + sed (= 4.7-1), + sensible-utils (= 0.0.12+nmu1), + sysvinit-utils (= 2.96-2.1ubuntu1), + tar (= 1.30+dfsg-7), + tzdata (= 2020d-0ubuntu0.20.04), + util-linux (= 2.34-0.1ubuntu9.1), + xz-utils (= 5.2.4-1ubuntu1), + zlib1g (= 1:1.2.11.dfsg-2ubuntu1.2) +Environment: + DEB_BUILD_OPTIONS="parallel=24" + LANG="en_US.UTF-8" + LC_ALL="en_US.UTF-8" + SOURCE_DATE_EPOCH="1610111686" + +-----BEGIN PGP SIGNATURE----- + +iQIzBAEBCgAdFiEEfVaV07YYhyZHhh1Rw4E3p8FnWYgFAl/4Wu8ACgkQw4E3p8Fn +WYgIYxAAlaCfOwNwvM1NUT8Xl5qStpSm6CRQ1+zx0KD+vwQ+QfmYCrlRhvVwrpiW +qiAqeyMwROqhHVqSRE55pAmQgOlBKPII3JR4GyL9g5a35IKW6CzTpoq3U5UzxicI +xpMGNkavTvxX7zYWuG4wgYuEMHHvvKR1MyN2ExDlQvej3XSzr8eP0Nqs9hCVS3kP +dlsREb0VXucB2GBvG5kcDPfQ7kZIX2iaLjh0lK9Ij8S3JqQU2GBOH1akz88n2CMM +IHi646ZNhsQ+NGQPskSOqS+8bgiJenXJ024xnsI9JFIaN6BOOLqleBj/EP2oMugo +JFYk9Mxoyo2ja7aRq+XBoQLTbzozusPUoEWaID8f7/E1pp7F42qRYag8qc3E/qTJ +PpVkkTyg8NvwrzI4jHdqieOschQI3NkvghJOL7yoRcmiRJAPdAbpJt58g9l24F/X +A++4BM4HGN8KdWpdk/PUQkk7RWSlEhx9+ITTM46Vt1BqRQr/Yc94ntVpSwrZIU21 +S5DHXckRgslQLpewT5H0756T76P6I3kdU/5f6iMuLU8empBo+MBJXMcHFz+zR5Fu +i60pEos2GUWKA4B16E14PdRmZ85nJVWtLQVO+5x5i5oxOk8RC3nPhKto+wkTBNRd +OGZ610CpkJ0oAKUZguLkR6Oe/DXp62aQptGXyp1QMPHOngn7Q88= +=smXO +-----END PGP SIGNATURE----- diff --git a/archivebox_0.5.3-1_source.changes b/archivebox_0.5.3-1_source.changes new file mode 100644 index 0000000..a98d8b7 --- /dev/null +++ b/archivebox_0.5.3-1_source.changes @@ -0,0 +1,48 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA512 + +Format: 1.8 +Date: Fri, 08 Jan 2021 08:14:46 -0500 +Source: archivebox +Architecture: source +Version: 0.5.3-1 +Distribution: focal +Urgency: low +Maintainer: Nick Sweeting +Changed-By: Nick Sweeting +Changes: + archivebox (0.5.3-1) focal; urgency=low + . + * source package automatically created by stdeb 0.10.0 +Checksums-Sha1: + ebc9cb5985ca3b9749af86cf3f79a0250888c651 1822 archivebox_0.5.3-1.dsc + a350c46e25ac3d3b30fab0efbd0226e5c0fe28a3 258683 archivebox_0.5.3.orig.tar.gz + d1952afed11f3a21e5e3794779edcd3ff6207df5 1556 archivebox_0.5.3-1.debian.tar.xz + 47aea2f95e98f3984a78a743a6b133baef352efc 7133 archivebox_0.5.3-1_source.buildinfo +Checksums-Sha256: + fd34ba74d913d39b90b9e487a504ed094c7371272c41e902c818091619d58ca0 1822 archivebox_0.5.3-1.dsc + 50b2c976296734d6e6b54975826c2ada9cd13f2c15dd585a2a94553530cf1541 258683 archivebox_0.5.3.orig.tar.gz + 9bdd60a9601e9dc7768789a4bb865b24b0cb0583887ff4f1f4106fdcae8d8cd7 1556 archivebox_0.5.3-1.debian.tar.xz + d4b6e02dcc5a15ee1ef94353c243cdb57849e459e47af9e2523d891cb00c43e6 7133 archivebox_0.5.3-1_source.buildinfo +Files: + a8c145339737010beb02a81925cf942a 1822 python optional archivebox_0.5.3-1.dsc + a69bba33bee581722227d9fecdb78f75 258683 python optional archivebox_0.5.3.orig.tar.gz + 77de8c5ee4b0158d7d35b162e1e04417 1556 python optional archivebox_0.5.3-1.debian.tar.xz + 67962d47cc8564035390506015844779 7133 python optional archivebox_0.5.3-1_source.buildinfo + +-----BEGIN PGP SIGNATURE----- + +iQIzBAEBCgAdFiEEfVaV07YYhyZHhh1Rw4E3p8FnWYgFAl/4Wu8ACgkQw4E3p8Fn +WYidQg//ZMUbaHnc3grDV0A+RcK6qqZ3apap6Tp01clcmc3QoMdALtBur2aMW1VV +SJ2XjP+onoTJtzj5JP8UZGmvqzeJTyk/GOollQ7Wunv52E0/7S4UJfckLkm+YtKb +ni1mBnEiNasjTy6OmMhe1mX7ArkxnmJKHb7Taz1KEVZrKjQDrlk9Ct9FaRSO604s +uJJn0oHAfCqzyahWLiqifmvHJ9ECM+b6qMJ/Z8Gs/GQ3Z5SvGIel3T8nu4D+R/1O +yW4osGPdEsEqm9/L8HRsDzVpOpBnn4/7t11TBca6wZMOMxDjI0FKgR86EcrOb/IV +mkKSHN35yTxFoeq3EjlVdwDif0A3K1xExTKluZH2bk/KUXt8bW06h5d2IbByQZes +fvq8hIQ0VRgU3hroZl32d0wk+/tG5YDlfjNTgvwk4zjsUwXisds02ICPqPNd06da +QdwyBIYenNcmCcd4+Gka0phzzOBGxTP4q5UqkY01l16otdW+tQEc8xL4mdQYlOsr +ac7GUkhzhO6Mo1tf3hDxe4bpQ/YRqD4WeZPCIoCKGW1PUFxS2Epx5fjG78fMLxMO +b0SNokBvURle95MaqBAaJBOFp+HTiFcpI+dPvBoLRHaKUrBfwUH/eIl5QaqwxQ7h +sNFLObmoXj2TVo4SAaDvYTCBioDUbbRXzDDlpnRFWU7yXvqnmNI= +=VV8m +-----END PGP SIGNATURE----- diff --git a/archivebox_0.5.3.orig.tar.gz b/archivebox_0.5.3.orig.tar.gz new file mode 100644 index 0000000..3421ca5 Binary files /dev/null and b/archivebox_0.5.3.orig.tar.gz differ