Skip to content

Home

Jump to bottom
Michael Tecourt edited this page Sep 15, 2017 · 60 revisions

Cereebro

Getting Started with Cereebro

The most effective way to use Cereebro is by coupling it with Spring Cloud and Netflix OSS : Get started with Cereebro, Spring Cloud Netflix and the Eureka Server

Key concepts

Cereebro is made of two parts :

  • Cereebro Snitch : an "agent" installed on every application, that snitches on the component and its relationship with others
  • Cereebro Server : reads the information provided by all the snitches in the system, and aggregates them to reconstitute the whole component graph

Core classes

  • Component : The base unit of a system : application, database, cache, file... for the moment only the minimal component attributes are managed (name + type)
  • ComponentRelationships : a Component and its relationships with others (Dependency or Consumer components)

On the client side (Snitch) :

  • RelationshipDetector : analyses the internals of an application to detect relationships with other components
  • Snitch : reveals information on a Component and its relationships by using RelationshipDetectors
  • SystemFragment : a portion of a given system, revealed from a Snitch's point of view

On the server side :

  • SnitchRegistry : holds a registry of Snitch intances
  • System : The big picture, consisting of all Components and their relationships
  • SystemResolver : browses a SnitchRegistry to build the dependency graph of a System

Cereebro Snitch

A Cereebro Snitch consists of two things :

  • a unique URI that can be accessed somehow
  • providing a SystemFragment, either as a Java object or JSON

Each of your application should be enhanced with the Cereebro Snitch so it can publish information about its relationships.

We provide Spring Boot and Spring Cloud starters to speed things up.
Out of the box cereebro-snitch-spring-boot-starter adds the following features to your application :

  • detect automatically :
    • SQL databases (MySQL, Oracle, DB2, MSSQL...)
    • Apache Cassandra databases
    • Redis clusters
    • MongoDB databases
    • Elasticsearch connections
    • Neo4j databases
    • RabbitMQ connections
    • OAuth 2.0 authorization server (token info / user info)
    • Applications downstream of a Zuul proxy
    • LDAP servers
  • detect beans annotated with Cereebro annotations : @DependencyHint, @ConsumerHint or @RelationshipHints
  • detect relationships declared in application properties
  • provide an actuator Snitch endpoint that will reveal your applications relationships

When using Spring Cloud Netflix, which we highly recommend for micro-service architectures, we add :

  • detect @FeignClient annotated beans as dependencies
  • detect dependencies to a Eureka Server
  • populate Eureka metadata with the Snitch endpoint URI and JSON SystemFragment

You could implement your own Snitch without Spring if you wanted to, as the cereebro-core modules does not depend on Spring. It would probably make you rewrite most of the relationship detection features though, as ours rely on the Spring Context and Spring Boot's conditional features.

Automatic relationship detection

The RelationshipDetector interface is in charge of analyzing some kind of application relationship.

Cereebro provides the following implementations by default :

Note that configuration properties should be used as a last resort for dependencies.
You should try to document dependencies by annotating beans and configuration classes with Cereebro annotations, because everything your application depends on should be materialized by some bean.
Consumers are not known by the application most of the times, but if these consumers cary the Cereebro Snitch, the Cereebro server will be able to reconstitute the relationship from the other side.
One valid case for relying on properties is to document a consumer outside the scope of the system.

To detect new types of components, you can provide your own implementations of RelationshipDetector and declare them as Spring beans. For annotation based detection, you can extend AnnotationRelationshipDetector<T>.
The Cereebro Snitch endpoint aggregates the results of all relationship detector instances in the Spring context.

Detectors can be customized or deactivated with properties, see the configuration options for more info.

We welcome contributions to the cereebro-snitch module for technologies supported by Spring Boot and Spring Cloud.

Using Cereebro annotations

Some relationships cannot be detected automatically by us. For example, you may consume an HTTP API using tools like Spring's RestTemplate, in which case there is no way we can guess the name of the component your application depends on.

We have create Cereebro annotations to document your Spring @Configuration classes, @Bean factory methods or @Component classes. :

Most of the time, you should be only documenting dependencies as they are always "materialized" by some bean or configuration. Consumers can be retrieved the other way around : if component A is a consumer of component B, then component B is a dependency of component A. The Cereebro server will resolve the dependencies of both components, and figure out the consumer relationship.

Example on a configuration class factory method :

@Bean
@DependencyHint(name = "twitter-api", type = "application/http")
public TwitterApiClient client(RestOperations restTemplate) {
    return new TwitterApiClientImpl(restTemplate);
}

Example on a configuration class that brings multiple relationships :

@Configuration
@Import({ ExternalConfigurationClassThatConfiguresManyRelationships.class })
@RelationshipHints(
    dependencies = { 
        @DependencyHint(name = "some-web-service", type = "application/http"), 
        @DependencyHint(name = "some-database", type = "database/whatever")
    }, 
    consumers = { 
        @ConsumerHint(name = "mobile-application", type = "application/mobile" ) 
    })
public class ExternalImportConfiguration {
    // ...
}

Example on a Spring @Component class (@Service, @Repository, @Controller...) :

@Component
@DependencyHint(name = "google-maps-api", type = "application/http")
public class TrackingService {
    // ...
}

If you write a client library for your API, you can put the annotation directly on your interface, so that any bean implementing your interface will be picked up by our detectors :

/** Note : you only need cereebro-core on the classpath */
@DependencyHint(name = "my-api", type = "application/http")
public interface MyApiClient {
    // ...
}

Snitch Endpoint

The Snitch Endpoint is a Spring Boot actuator endpoint that publishes the relationships information detected.

It can be accessed by default at /cereebro/snitch. This path is relative to the actuator, see the Spring Boot documentation for more.

Typical output looks like this JSON SystemFragment :

{  
   "componentRelationships":[  
      {  
         "component":{  
            "name":"my-application",
            "type":"application/http"
         },
         "dependencies":[  
            {
               "component":{  
                  "name":"some_mysql_schema",
                  "type":"database/relational"
               }
            },
            {  
               "component":{  
                  "name":"some-api",
                  "type":"application/http"
               }
            }
         ],
         "consumers":[  
            {  
               "component":{  
                  "name":"android-app",
                  "type":"application/mobile/android"
               }
            }
         ]
      }
  ]
}

Using Spring Cloud Netflix and Eureka

When using Spring Cloud Netflix, Cereebro will populate Eureka metadata upon registration with the Eureka Server.

cereebro-eureka-component-diagram

The following metadata keys are set by the EurekaMetadataPopulator :

  • io.cereebro.snitch.uri : The Snitch absolute URI
  • io.cereebro.snitch.system-fragment-json : String containing a SystemFragment serialized as JSON

The Cereebro Server processes the system fragment directly, the URI is here to identify the Snitch (and also "just in case").

By default, EurekaMetadataPopulator uses the same logic as spring-cloud-netflix to to determine the hostname and port, see EurekaInstanceConfigBean#getHostname(boolean).

Like they do for the health and info endpoints, we allow you to override the declared Snitch URI with two mutually exclusive properties (EurekaInstanceSnitchProperties) :

# The absolute path of the Snitch endpoint
#   This setting will take precedence over the relative path if both are set.
#   Use this if the actuator runs on a different port than the application server,
#   or if you want to register an https url.
cereebro.snitch.eureka.endpoint-url=

# The relative path of the Snitch endpoint
#   This path is relative to the hostname determined using EurekaInstanceConfigBean.
#   Use this setting if the actuator runs on the same port as the application server,
#   but either the application or the actuator (or both) have a specific context-path.
cereebro.snitch.eureka.endpoint-url-path=

You can switch off Eureka metadata population by setting cereebro.snitch.eureka.enabled to false.

Command line / Installed applications

For command line or installed applications, you can either log the SystemFragment on application startup or write it to a file (see the application properties section). You could run the application with one of these options during tests, or just once manually.

You can then make these files available to your Cereebro Server, and use them to complete your component graph with cereebro.server.snitch.resources=classpath:xxx.json (see Cereebro Server properties for more info).

Application properties

CereebroProperties :

# The host application component name
# Defaults to the value of ${spring.application.name) (preferred), 
# or a random string if none is set
cereebro.application.component.name=
# The host application component type
cereebro.application.component.type=application/http

# Dependency components, will be added to the ones detected programmatically
cereebro.application.dependencies[0].component.name
cereebro.application.dependencies[0].component.type

# Consumer components
cereebro.application.consumers[0].component.name
cereebro.application.consumers[0].component.type

# Log the SystemFragment on startup
cereebro.snitch.log-on-startup=false

# Write the SystemFragment to a file on startup
cereebro.snitch.file.enabled=false
# File location, temporary file by default
# (the location of the temp file will be logged)
cereebro.snitch.file.location=

########################
# Detectors
########################
# Rabbit MQ
cereebro.snitch.detect.rabbit.enabled=true

# Cassandra
cereebro.snitch.detect.cassandra.enabled=true

# Eureka
cereebro.snitch.detect.eureka.enabled=true
cereebro.snitch.detect.eureka.default-name=eureka-server

# JDBC DataSource
cereebro.snitch.detect.jdbc.enabled=true
cereebro.snitch.detect.jdbc.default-name=default_db

# Mongo DB
cereebro.snitch.detect.mongo.enabled=true

# Neo4j
cereebro.snitch.detect.neo4j.enabled=true
cereebro.snitch.detect.neo4j.default-name=default

# OAuth 2.0
cereebro.snitch.detect.oauth2.authorization-server.enabled=true
cereebro.snitch.detect.oauth2.authorization-server.default-name=oauth2-authorization-server

# Redis
cereebro.snitch.detect.redis.enabled=true
cereebro.snitch.detect.redis.default-name=default

# Zuul routes
cereebro.snitch.detect.zuul.enabled=true

# LDAP
cereebro.snitch.detect.ldap.enabled=true
cereebro.snitch.detect.ldap.default-name=default

Note that when using yaml, consumers and dependencies should be defined as arrays :

cereebro:
  application:
    
    dependencies:
      - 
        component:
          name: xxx
          type: yyy
    
    consumers:
      - 
        component:
          name: aaa
          type: bbb

CereebroSnitchMvcEndpoint properties :

# Relative path of the actuator endpoint
endpoints.cereebro.path=/cereebro/snitch

# Enable the actuator endpoint
endpoints.cereebro.enabled=true

# Whether the actuator endpoint is considered sensitive
# (See Spring Boot docs for more information)
endpoints.cereebro.sensitive=true

Cereebro Server

The Cereebro Server is an aggregator of snitched information.
It must know the location of every Snitch in the system through its SnitchRegistry.

Out of the box cereebro-server-spring-boot-starter allows you to target a fixed set of JSON resources (on the file system, classpath, HTTP...) through configuration properties, for example :

cereebro:
  server:
    system:
      name: your system name
      snitch:
        resources: 
          - 'classpath:/snitches/some-fragment-of-my-system.json'
          - 'file:///somewhere/other-part-of-my-system.json'

You should only use a static list of Snitch resources to document applications that cannot be analyzed dynamically, for example installed applications (mobile, desktop...) or short lived processes (batch, tasks...).

Using Spring Cloud Netflix and Eureka

The most powerful way to use Cereebro is along with Spring Cloud, Netflix OSS and the Eureka Server. The Eureka Server is a perfect Snitch registry because it should know every application in your system.

See how to get started.

Spring Boot features

cereebro-spring-boot-component-diagram

If you are not using Spring Cloud, you could target manually each application's Snitch endpoint from your Eureka server :

cereebro:
  server:
    system:
      name: your system name
      snitch:
        resources: 
          - 'http://first-application/cereebro/snitch'
          - 'http://second-application/cereebro/snitch'
          - 'http://third-application/cereebro/snitch'

This approach is somewhat limited, especially in a micro-service architecture where components come by the dozen.
Services may also have various security mechanisms to secure their Snitch endpoint, which we don't handle at this time (Spring's UrlResource does not handle curl style HTTP Basic credentials http://user:password@hostname).

Other sources

The Cereebro Server browses SnitchRegistry instances to access each Snitch.
If you find a different way to discover information about components and their relationships, you can implement your own Snitch and declare a new Spring Bean implementing SnitchRegistry in your Cereebro Server application.

This is basically what our own starters do : by making your Cereebro Server a Eureka Client, we activate a configuration class that declares a DiscoveryClientSnitchRegistry bean.

For example, Zipkin takes another angle on how to discover relationships between micro-services at runtime (tracing). Zipkin's API could very well be adapted as a Cereebro Snitch.

We may support Consul in the near feature (the only problem being that it has poor metadata support).

Customizing the UI

The Cereebro Server offers a basic UI at /cereebro/system.
This same resource can also be consumed as a JSON/HTTP API by adding a header (Accept: application/json) or appending .json at the end of the path.

You can build your UI by consuming this resource and creating your own view, either on the Cereebro Server using Spring MVC, or from a separate web application of yours.

From the Cereebro Server you can indeed use the Java API directly.
To retrieve the System object provided by the HTTP API, you can get a SystemService instance injected from the Spring Context.

Caching

By default, the system is resolved everytime the page is opened.
For large production systems, caching can be added in standard Spring fashion. See the Spring Boot documentation to configure a cache manager using your favorite tool.
DefaultSystemService#get is annotated with @CacheResult, meaning that the resolved System will be cached if a CacheManager is available in the Spring Context. The cache is named io.cereebro.system.

Application properties

CereebroServerProperties :

# Display name of your system
cereebro.server.system.name=cereebro

# Relative path of the system resource (JSON/HTML)
cereebro.server.system.path=/cereebro/system

# Fixed list of Spring org.springframework.core.io.Resource that produce SystemFragment JSON
# Use comma separated strings for properties files, or an array for yaml
cereebro.server.system.snitch.resources=

Versions

See the whole change log.

Mapping with Spring versions :

Cereebro Spring Boot Spring Cloud
0.0.1 1.4.5.RELEASE Camden.SR6
0.0.2 1.4.5.RELEASE Camden.SR6
0.0.3 1.4.5.RELEASE Camden.SR6
0.0.4 1.4.5.RELEASE Camden.SR6
1.0.0 1.4.5.RELEASE Camden.SR6
1.1.0 1.4.5.RELEASE Camden.SR6
1.2.0 1.5.4.RELEASE Dalston.SR1
1.2.1 1.5.4.RELEASE Dalston.SR1
1.2.2 1.5.6.RELEASE Dalston.SR2
1.2.3 1.5.6.RELEASE Dalston.SR2
Clone this wiki locally