Skip to content

Home

Jump to bottom
Sowmya Balasubramanian edited this page Oct 30, 2020 · 19 revisions

This page gives an overview of simple lookup service

For installation instructions, click here: sLS Installation

Introduction

The Simple Lookup Service(sLS) is a REST/JSON based lookup service written in Java.

It registers collections of key-value pairs referred as "records". The records will remain in the lookup service for a stipulated amount of time, known as "lease" time. Clients can renew the lease before it expires. If a client does not renew the lease, the record is automatically removed from the loookup service. A client can also explicitly delete a record.

In addition, a lookup service may also subscribe to another lookup service. When a lookup service subscribes to another lookup service, from that point onwards, the former will automatically receive new record registrations or updates to existing records from the latter.

The Simple Lookup Service can be deployed as a single centralized node or in a distributed fashion.

Motivation

The sLS project was started as a replacement for the XML based perfSONAR Lookup Service.The perfSONAR community developed and deployed a lookup service that registers all perfSONAR services, and some other non-perfSONAR services (Eg: OSCARS friendly names). Over the past few years, as the perfSONAR community steadily grew, it increased the number of toolkit deployments. This, in turn, increased the number of services registering with the LS and the number of queries to the LS. The perfSONAR LS deployment found it difficult to deal with this increase in load. This resulted in some serious performance and reliability issues. A redesign effort was initiated to address these issues.

Requirements and Scope

Since, sLS was meant to be a replacement for the perfSONAR LS, the requirements for sLS were driven by the needs of the perfSONAR community. As a result, although the requirements themselves are generic enough, some of the numbers in the requirements specification was determined by the needs of the perfSONAR community.

The requirements can be summarized as follows:

  • Scalability - The lookup service should support be able to support at least 10,000 service records without significant performance degradation. The maximum limit is 100,000.
  • Query Time - The average time to respond to a query should be 1 second. The maximum time bound on the lookup service should be 5 seconds.
  • Registration Time - The maximum time that it should take a service to be registered and become globally visible is 1 hour.
  • Availability - The LS can be deployed in a manner which will provide 99.9% uptime.
  • Security - The LS should allow records to contain authentication and/or authorization credentials that can be consumed by the querying client. It is NOT a requirement that the lookup service be responsible for verifying or enforcing security credentials.
  • Queries - The lookup service should be able to apply a set of filters and return records that match these filters. Additionally, the sLS also addresses the following key issues:
  • generic API
  • easy to use
  • stability

Design Overview

Data Representation

As described in the "Overview" section, the data is represented as "records". Records are nothing but a collection of key-value pairs. The keys are strings and values are list of one or more strings. A sample record may look like this:

{   "type": "service",
    "service-type": ["service-accesspoint": ["http://somehost.some.domain:port/service/bwctl"]("bwctl"],),
    "uri": "lookup/service/uniquerecordid"
    "expires": "2013-05-06T23:10:14.577Z"
}

Simple Lookup Service defines a set of records. Since, sLS is designed to be very general, it can therefore register any type of record, including user-defined ones.

Details about the predefined records and guidelines to define new record types can be found in the Records document.

API

Most of the web services are based on either SOAP/XML or REST/JSON. Each technology has its own set of advantages and disadvantages. REST/JSON based APIs are usually simple and easy to use. Also, it is easy and intuitive to express key-value pairs in JSON. Hence, REST/JSON was the obvious choice for sLS.

The API itself can be divided into 3 main sections:

  1. Record Management - Register/edit a service using simple HTTP calls(POST, DELETE, GET). Example:
curl -v -H "Content-Type: application/json" -X POST -d '{"type":[Ping Responder"]("service"],"service-locator":["somehost.es.net"],"service-name":["ESnet),"service-type":[ "ping"]}' http://somehost.some.domain:8090/lookup/records
  1. Query - Query the lookup service for records using HTTP GET Example:
curl -v -H "Content-Type: application/js" -X GET http://somehost.some.domain:8090/lookup/records/?type=service
  1. Subscribe - Get new registrations and updates to records on a separate channel. To learn more about the API, please refer to APISpec

Backend Database

The sLS data is represented as simple key-value pairs and the API is in JSON. Our approach was to keep the coupling between the data and the database loose and at the same time optimize performance by choosing a database that can inherently support key-value pairs and thus avoid translating data from one format to another. We originally used MongoDB as the backend database.

Starting from 3.0, lookup service uses elasticsearch for its backend.

Architecture

The Simple Lookup Service(sLS) node consists of a simple webservice (a REST/JSON based API) and a backend (elasticsearch from 3.0 onwards).

Clients

The Simple Lookup Service is simple and easy to use that it can be used directly by command line clients like curl. In addition, we intend to provide client libraries in Perl, Python and Java. Developers can use these client libraries to develop web applications or use with third party software to interact with sLS.

Details about the clients can be found in the ClientAPI doc.

Clone this wiki locally