Micrawler - tempas.L3S.de/Micrawler (demo)
A micro archive is a snapshot of a fixed (evolving) set of URLs that are representative for some object or entity (at a given time). Hence, such an archive can be used to describe and / or derive information about its subject at the time of the crawl.
Examples are:
-
A person, represented by his/her personal website, social media profiles, etc.
-
A software version, represented by a collections of all URLs describing the software (e.g., official website, documentation, source code, etc.) at a specific time, to be cited in a scientific publication where it is used.
E.g., SINGULAR: http://tempas.l3s.de/micrawler/?spec=http://swmath.org/software/866
Micro archives are described by textual crawl specifications in Micrawler with lines of the following form:
-
The first line of a spec can start with an
@followed by a timestamp to anchor it to this time. This is considered for already crawled specs and will be automatically added by Micrawler. E.g.,@20171025120000(25-10-2017 12:00:00). -
URLs either alone, one per line, or preceded by a label in the form
label: http://URL. Once a crawl is completed, URL lines are enriched by Micrawler with additional meta data in space separated format, e.g.Label: http://URL meta1 meta2 ... -
Properties to add additional information to a spec in the form
key=value. Any property is allowed and can be employed by one of the services described in the following section. Properties at the beginning of the specification (after the@timestmap) are assigned to the micro archive, properties following a URL are assigned to the corresponding resource. Reserved properties aretitle=...andtype=...to describe the micro archive. -
Comments are indicated by a
#at the beginning of a line.
Micrawler is a web application backed by a set of server modules, which are not part of this project.
These modules define how and where the resources to be archived are processed by means of the following extension points, which can be configured in the micrawler-config.js file:
-
Spec proxy: A service to extract / derive / generate the specification of the micro archive from a given seed URL (e.g., a Wikipedia article to refer to an entity). As a fallback simply all linked URL from a given page may be extracted.
-
Crawl queue: A logic or service that provides the queue of URLs to be crawled from the crawl specification. In the basic case, these are exactly the URLs specified in the spec.
-
Archiving/crawl service: The service(s) to perform the actual archiving, e.g., every URL may be send to the same Web archiving on demand service or certain URLs may be treated as special cases, such as software repositories.
-
Archive meta service: Given a single URL with the timestamp of the micro archive, this service should provide the corresponding meta data about the archived capture, like exact time of the snapshot, status code, etc. (this should correspond to the used archiving/crawl service)
-
Analyzers: These can be defined per
typeof a crawl specification to treat micro archives of different types differently. An analyzer is given the specification enriched with timestamp and archive metadata, from which it can extract/mine additional meta information, provided in a single JSON object. -
Viewer: The URL to render/replay the archived resource.
-
Persistence provider: Given the final enriched specification with timestamp and metadata, the persistence provider should store it permanently and return a persistent URL to the micro archive.
Please note: All services in the micrawler-config.js file in this repo and currently deployed to the prototype on tempas.L3S.de/Micrawler are only for demonstration purposes and should not be used in production. Especially the permalinks as generated by our current persistence provider are not guaranteed to be actually persisted. As archiving service we currently use the great Save Page Now service by the Internet Archive.
The MIT License (MIT)
Copyright (c) 2017 Helge Holzmann
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.