Search engine for the distributed web
Lens is an opt-in search engine and data collection tool to aid content discovery of the distributed web. It exposes a simple, minimal API for intelligently indexing and querying content on IPFS.
Initially integrated with Temporal, Lens will allow users to optionally have the data they upload be searched and indexed and be awarded with RTC for participating in the data collection process. Users can then search for content using a simple-to-use API.
Searching through Lens will be facilitated through Temporal web. Optionally, we will have a service independent from Temporal which users can submit content to have it be indexed. This however, is not compensated with RTC. In order to receive the RTC, you must participate through Lens indexing within the Temporal web interface.
Lens exposes a simple API via gRPC. The definitions are in
RTradeLtd/grpc.
The Lens API, summarized, currently consists of three core RPCs:
service LensV2 {
rpc Index(IndexReq) returns (IndexResp) {}
rpc Search(SearchReq) returns (SearchResp) {}
rpc Remove(RemoveReq) returns (RemoveResp) {}
}Golang bindings for the Lens API can be found in
RTradeLtd/grpc.
Only IPFS CIDs are supported, and must be either images, text files, or pdfs. We attempt to determine the content type via mime type sniffing, and use that to determine whether or not we can analyze the content.
Please see the following table for supported content types that we can index.
Note if the type is listed as <type>/* it means that any "sub type" of that
mime type is supported.
| Mime Type | Support Level | Tested Types |
|---|---|---|
text/* |
Beta | text/plain, text/html |
image/* |
Beta | image/jpeg |
application/pdf |
Beta | application/pdf |
The recommended way to deploy a Lens instance is via the
rtradetech/lens
Docker image.
$> docker pull rtradetech/lens:latestA docker-compose configuration
is available that also starts up other prerequisites:
$> wget -O lens.yml https://raw.githubusercontent.com/RTradeLtd/Lens/master/lens.yml
$> LENS=latest BASE=/my/dir docker-compose -f lens.yml upThis project requires:
To fetch the codebase, use go get:
$> go get github.com/RTradeLtd/LensA rudimentary Makefile target make dep
is available for installing the required dependencies.