SiteWhere is an industrial strength open-source application enablement platform for the Internet of Things (IoT). It provides a multi-tenant microservice-based infrastructure that includes device/asset management, data ingestion, big-data storage, and integration through a modern, scalable architecture. SiteWhere provides REST APIs for all syste…
Java Groovy
Clone or download
Permalink
Failed to load latest commit information.
gradle/wrapper Upgrade gradle version. Jan 8, 2018
service-asset-management Remove 'force' flag for deletes. #630 Jul 16, 2018
service-batch-operations Remove 'force' flag for deletes. #630 Jul 16, 2018
service-command-delivery Refactor gRPC model into separate libraries. Jul 17, 2018
service-device-management Remove references to deleted flag since it's no longer used. #630 Jul 18, 2018
service-device-registration Refactor gRPC model into separate libraries. Jul 17, 2018
service-device-state Refactor gRPC model into separate libraries. Jul 17, 2018
service-event-management Refactor gRPC model into separate libraries. Jul 17, 2018
service-event-search Implement event search configuration/parser/Solr provider. Jul 17, 2018
service-event-sources Refactor gRPC model into separate libraries. Jul 17, 2018
service-inbound-processing Refactor gRPC model into separate libraries. Jul 17, 2018
service-instance-management Refactor gRPC model into separate libraries. Jul 17, 2018
service-label-generation Reduce logging for Apache Curator and Spring Web. Jun 21, 2018
service-outbound-connectors Implement event search configuration/parser/Solr provider. Jul 17, 2018
service-rule-processing Refactor gRPC model into separate libraries. Jul 17, 2018
service-schedule-management Refactor gRPC model into separate libraries. Jul 17, 2018
service-streaming-media Move stream management support into streaming-media microservice. Jul 12, 2018
service-tenant-management Remove uniqueness constraint on tenant auth token. #632 Jul 18, 2018
service-user-management Remove references to deleted flag since it's no longer used. #630 Jul 18, 2018
service-web-rest Allow device assignment info to be returned in device state query. Jul 18, 2018
sitewhere-cassandra Refactoring Cassandra device event support. Jul 12, 2018
sitewhere-client Update measurements APIs to use single measurement instead of map. Jul 3, 2018
sitewhere-communication Refactor outbound connector support. Jul 13, 2018
sitewhere-configuration Refactor outbound connector support. Jul 13, 2018
sitewhere-core-api Remove references to deleted flag since it's no longer used. #630 Jul 18, 2018
sitewhere-core-lifecycle Add undelivered commands topic and refactor router internals. Jun 29, 2018
sitewhere-core Remove references to deleted flag since it's no longer used. #630 Jul 18, 2018
sitewhere-grpc-asset-management Remove 'force' flag for deletes. #630 Jul 16, 2018
sitewhere-grpc-batch-management Remove 'force' flag for deletes. #630 Jul 16, 2018
sitewhere-grpc-client Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-grpc-device-management Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-grpc-device-state Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-grpc-event-management Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-grpc-label-generation Refactor auth context handling for server API implementations. Jun 5, 2018
sitewhere-grpc-model Remove references to deleted flag since it's no longer used. #630 Jul 18, 2018
sitewhere-grpc-schedule-management Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-grpc-tenant-management Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-grpc-user-management Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-hbase Remove references to deleted flag since it's no longer used. #630 Jul 18, 2018
sitewhere-influxdb Fix invalid date conversion in gRPC. Update datatype in protos. May 24, 2018
sitewhere-jdk Reduce dependencies for smaller footprint. May 10, 2018
sitewhere-microservice Refactor gRPC model into separate libraries. Jul 17, 2018
sitewhere-mongodb Remove 'force' flag for deletes. #630 Jul 16, 2018
sitewhere-solr Implement event search configuration/parser/Solr provider. Jul 17, 2018
sitewhere-spark Update measurements APIs to use single measurement instead of map. Jul 3, 2018
.gitignore Remove Eclipse project file. This is generated by buildship plugin. Jul 5, 2016
.travis.yml Added utility script for TravisCI to publish Javadoc to website. Jan 30, 2016
CODE_OF_CONDUCT.md Added code of conduct Feb 9, 2018
HEADER Add header verifier to Gradle build. Start fixing headers. #535 Oct 15, 2017
LICENSE.txt Migrating to 2.0 architecture. Sep 5, 2017
README.md Updated version and some of the wording regarding early access. Jul 18, 2018
build.gradle Update version to RC1. Jun 23, 2018
gradle.properties Add build support for pushing to Docker Hub. Feb 8, 2018
gradlew Add GRPC library. Upgrade gradle wrapper dependency. #535 Sep 7, 2017
gradlew.bat Add GRPC library. Upgrade gradle wrapper dependency. #535 Sep 7, 2017
package-lock.json More support for Apache Cassandra event management. #582 Mar 21, 2018
settings.gradle Implement event search configuration/parser/Solr provider. Jul 17, 2018

README.md

Build Status

SiteWhere


NOTE: SiteWhere 2.0 RC1 is an early release candidate of the new architecture and is considered an "beta" quality preview. This release is not intended for production use! Some features will change before the 2.0 GA release.

SiteWhere is an industrial-strength open source IoT Application Enablement Platform that facilitates the ingestion, storage, processing, and integration of device data at massive scale. The platform is based on a modern microservices architecture and has been designed from the ground up for reliable, high throughput, low latency processing and dynamic scalability. SiteWhere takes advantage of proven technologies such as Apache Kafka and Docker in order to scale efficiently to the loads expected in large IoT projects. Rather than using a monolithic architecture, SiteWhere embraces a completely distributed approach using microservices to allow scaling at the component level so that the system may be tailored to the customer use case.

SiteWhere Administration

The SiteWhere microservices are built with a framework approach using clearly defined APIs so that new technologies can easily be integrated as the IoT ecosystem evolves. The remainder of this document covers the core technologies used by SiteWhere and how they fit together to build a comprehensive system.

Microservices

SiteWhere 2.0 introduces a much different architectural approach than was used in the 1.x platform. While the core APIs are mostly unchanged, the system implementation has moved from a monolithic approach to one based on microservices.

SiteWhere Architecture

This approach provides a number of advantages over the previous architecture.

Separation of Concerns

Each microservice is a completely self-contained entity that has its own unique configuration schema, internal components, data persistence, and interactions with the event processing pipeline. SiteWhere microservices are built on top of a custom microservice framework and run as separate Spring Boot processes, each contained in its own Docker image.

Separating the system logic into microservices allows the interactions between various areas of the system to be more clearly defined. This transition has already resulted in a more understandable and maintainable system and should continue to pay dividends as more features are added.

Scale What You Need. Leave Out What You Don't

The microservice architecture allows individual functional areas of the system to be scaled independently or left out completely. In use cases where REST processing tends to be a bottleneck, multiple REST microservices can be run concurrently to handle the load. Conversely, services such as presence management that may not be required can be left out so that processing power can be dedicated to other aspects of the system.

Instance Management

The 2.0 architecture introduces the concept of a SiteWhere instance, which allows the distributed system to act as a cohesive unit with some aspects addressed at the global level. All of the microservices for a single SiteWhere instance must be running on the same Docker infrastucture, though the system can be spread across tens or hundreds of machines using technologies such as Docker Swarm or Kubernetes.

Configuration Management with Apache ZooKeeper

SiteWhere 2.0 moves system configuration from the filesystem into Apache ZooKeeper to allow for a centralized, coordinated approach to configuration management. ZooKeeper contains a hierarchical structure which represents the configuration for a SiteWhere instance and all of the microservices that are used to realize it.

Each microservice has a direct connection to ZooKeeper and uses the hierarchy to determine information such as the instance it belongs to and the configuration it should use. Microservices listen for changes to the configuration data and react dynamically to updates. No configuration is stored locally within the microservice, which prevents problems with keeping services in sync as system configuration is updated.

Event Processing with Apache Kafka

The event processing pipeline in SiteWhere 2.0 has been completely redesigned and uses Apache Kafka to provide a resilient, high-performance mechanism for progressively processing device event data. Each microservice plugs into key points in the event processing pipeline, reading data from well-known inbound topics, processing data, then sending data to well-known outbound topics. External entites that are interested in data at any point in the pipeline can act as consumers of the SiteWhere topics to use the data as it move through the system.

In the SiteWhere 1.x architecture, the pipeline for outbound processing used a blocking approach which meant that any single outbound processor could block the outbound pipeline. In SiteWhere 2.0, each outbound consumer is a true Kafka consumer with its own offset marker into the event stream. This mechanism allows for outbound processors to process data at their own pace without slowing down other processors.

Using Kafka also has other advantages that are leveraged by SiteWhere. Since all data for the distributed log is stored on disk, it is possible to "replay" the event stream based on previously gathered data. This is extremely valuable for aspects such as debugging processing logic or load testing the system.

Inter-Microservice Communication with GRPC

While device event data generally flows in a pipeline from microservice to microservice on Kafka topics, there are some operations that need to occur directly between microservices. For instance, device management and event management persistence are each contained in separate microservices, so as new events come in to the system, the inbound processing microservice has to connect with the event persistence microservice to store the events. SiteWhere 2.0 uses GRPC to establish a long-lived connection between microservices that need to communicate with each other. Since GRPC uses persistent HTTP2 connections, the overhead for interactions is greatly reduced, allowing for decoupling without a significant performance penalty.

The entire SiteWhere data model has been captured in Google Protocol Buffers format so that it can be used within GRPC services. All of the SiteWhere APIs are now exposed directly as GRPC services as well, allowing for high-performance, low-latency access to what was previously only accessible via REST. The REST APIs are still made available via the Web/REST microservice, but they use the GRPC APIs underneath to provide a consistent approach to accessing data.

Since the number of instances of a given microservice can change over time as the service is scaled up or down, SiteWhere automatically handles the process of connecting/disconnecting the GRPC pipes between microservices. Each outbound GRPC client is demulitplexed across the pool of services that can satisfy the requests, allowing the requests to be processed in parallel.

Distributed Multitenancy

The SiteWhere 1.x approach to multitenancy was to use a separate "tenant engine" for each tenant. The engine supported all tenant-specific tasks such as data persistence, event processing, etc. Since SiteWhere 2.0 has moved to a microservices architecture, the multitenant model has been distributed as well. SiteWhere supports two types of microservices: global and multitenant.

Global Microservices

Global microservices do not handle tenant-specific tasks. These services handle aspects such as instance-wide user management and tenant management that are not specific to individual system tenants. The Web/REST microservice that supports the administrative application and REST services is a global service, since supporting a separate web container for each tenant would be cumbersome and would break existing SiteWhere 1.x applications. There is also a global instance management microservice that monitors various aspects of the entire instance and reports updates to the individual microservces via Kafka.

Multitenant Microservices

Most of the SiteWhere 2.0 services are multitenant microservices which delegate traffic to tenant engines that do the actual processing. For instance, the inbound processing microservice actually consists of many inbound processing tenant engines, each of which is configured separately and can be started/stopped/reconfigured without affecting other tenant engines.

The new approach to tenant engines changes the dynamics of SiteWhere event processing. It is now possible to stop a single tenant engine without the need for stopping tenant engines running in other microservices. For instance, inbound processing for a tenant can be stopped and reconfigured while the rest of the tenant pipeline continues processing. Since new events can be allowed to stack up in Kafka, the tenant engine can be stopped, reconfigured, and restarted, then resume where it left off with no data loss.

Release Documentation

More documentation for this "early access" release can be found here.


Copyright (c) 2009-2018 SiteWhere LLC. All rights reserved.