Sowmya Balasubramanian edited this page Dec 16, 2016 · 14 revisions

This page gives an overview of simple lookup service

For installation instructions, click here: sLS Installation

Introduction

The Simple Lookup Service(sLS) is a REST/JSON based lookup service written in Java.

It registers collections of key-value pairs referred as "records". The records will remain in the lookup service for a stipulated amount of time, known as "lease" time. Clients can renew the lease before it expires. If a client does not renew the lease, the record is automatically removed from the loookup service. A client can also explicitly delete a record.

In addition, a lookup service may also subscribe to another lookup service. When a lookup service subscribes to another lookup service, from that point onwards, the former will automatically receive new record registrations or updates to existing records from the latter.

The Simple Lookup Service can be deployed as a single centralized node or in a distributed fashion.

Motivation

The sLS project was started as a replacement for the XML based perfSONAR Lookup Service.The perfSONAR community developed and deployed a lookup service that registers all perfSONAR services, and some other non-perfSONAR services (Eg: OSCARS friendly names). Over the past few years, as the perfSONAR community steadily grew, it increased the number of toolkit deployments. This, in turn, increased the number of services registering with the LS and the number of queries to the LS. The perfSONAR LS deployment found it difficult to deal with this increase in load. This resulted in some serious performance and reliability issues. A redesign effort was initiated to address these issues.

Requirements and Scope

Since, sLS was meant to be a replacement for the perfSONAR LS, the requirements for sLS were driven by the needs of the perfSONAR community. As a result, although the requirements themselves are generic enough, some of the numbers in the requirements specification was determined by the needs of the perfSONAR community.

The requirements can be summarized as follows:

  • Scalability - The lookup service should support be able to support at least 10,000 service records without significant performance degradation. The maximum limit is 100,000.
  • Query Time - The average time to respond to a query should be 1 second. The maximum time bound on the lookup service should be 5 seconds.
  • Registration Time - The maximum time that it should take a service to be registered and become globally visible is 1 hour.
  • Availability - The LS can be deployed in a manner which will provide 99.9% uptime.
  • Security - The LS should allow records to contain authentication and/or authorization credentials that can be consumed by the querying client. It is NOT a requirement that the lookup service be responsible for verifying or enforcing security credentials.
  • Queries - The lookup service should be able to apply a set of filters and return records that match these filters. Additionally, the sLS also addresses the following key issues:
  • generic API
  • easy to use
  • stability

Design Overview

Data Representation

As described in the "Overview" section, the data is represented as "records". Records are nothing but a collection of key-value pairs. The keys are strings and values are list of one or more strings. A sample record may look like this:

{   "type": "service",
    "service-type": ["service-accesspoint": ["http://somehost.some.domain:port/service/bwctl"]("bwctl"],),
    "uri": "lookup/service/uniquerecordid"
    "expires": "2013-05-06T23:10:14.577Z"
}

Simple Lookup Service defines a set of records. Since, sLS is designed to be very general, it can therefore register any type of record, including user-defined ones.

Details about the predefined records and guidelines to define new record types can be found in the Records document.

API

Most of the web services are based on either SOAP/XML or REST/JSON. Each technology has its own set of advantages and disadvantages. REST/JSON based APIs are usually simple and easy to use. Also, it is easy and intuitive to express key-value pairs in JSON. Hence, REST/JSON was the obvious choice for sLS.

The API itself can be divided into 3 main sections:

  1. Record Management - Register/edit a service using simple HTTP calls(POST, DELETE, GET). Example:
curl -v -H "Content-Type: application/json" -X POST -d '{"type":[Ping Responder"]("service"],"service-locator":["somehost.es.net"],"service-name":["ESnet),"service-type":[ "ping"]}' http://somehost.some.domain:8090/lookup/records
  1. Query - Query the lookup service for records using HTTP GET Example:
curl -v -H "Content-Type: application/js" -X GET http://somehost.some.domain:8090/lookup/records/?type=service
  1. Subscribe - Get new registrations and updates to records on a separate channel. To learn more about the API, please refer to APISpec

Backend Database

The sLS data is represented as simple key-value pairs and the API is in JSON. Our approach was to keep the coupling between the data and the database loose and at the same time optimize performance by choosing a database that can inherently support key-value pairs and thus avoid translating data from one format to another. Since, MongoDB met the requirements, we decided to use MongoDB as the backend database for sLS.

Architecture

The Simple Lookup Service(sLS) node is the basic building block of sLS architecture. As shown in figure, the sLS node consists of API and the database. sLS Building Block

This sLS node is the source of truth. This is where clients will register/edit records. The sLS architecture will contain one or more nodes of this type.

sLS Cache/Aggregator

This node will aggregate data from all the core/source nodes and will help in optimizing query performance. This node does not accept registrations or edits to the records.The sLS architecture may contain zero or more nodes of this type. sLS Architecture

Clients

The Simple Lookup Service is simple and easy to use that it can be used directly by command line clients like curl. In addition, we intend to provide client libraries in Perl, Python and Java. Developers can use these client libraries to develop web applications or use with third party software to interact with sLS.

Details about the clients can be found in the ClientAPI doc.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.