Skip to content

Setup Guide

Jump to bottom
Pranav Prakash edited this page Aug 20, 2016 · 6 revisions

All the generic guidelines for setting up the octo a.i. environment. Very soon, we will be offering docker based setup so that most of the process is streamlined. But for now, this is how to get hands on.

Basic Setup

System

A standard *ix system with admin privilages. Tested and verified on

  • OS X 10.11.4 (El Capitan)
  • Ubuntu

Kong

Version 0.6.1

Kafka

Version: 0.8.2.2

We have found based on personal experience that other version of Kafka causes unstability. However, we are also aware that few developers have tried the latest kafka versions and it was working. You can try it on your own and update it here.

Cassandra

Version : 2.2.4

Redis

Version : 3.0.5

Zookeeper

Version : 3.4.7

Ruby

Preferred way: rvm
Version : 2.1.5

Repos

Before proceeding any further, make sure all the above mentioned systems are up and running. A good way to make sure everything is fine is to check their log files for anything suspicious.

Config first

  • Clone the config repo
  • The important files are config.yml and kong_config.yml.
  • Find more details about Configuration Files. This guide has specific and all details about configuration files, environment variables and everything you need to get octo-matic up and running.

At this point, it is crucial to understand how different applications use this config. Each of the application has their own config directory. This directory contains a config.yml file which would contain configs very specific to that repo. All other common things are contained in this config repo. In order to make things work you would have to symlink this config directory to the config directory inside apps. If you do not understand this, you can still move forward. The steps are mentioned for each repo. Just make sure you have the correct configs here.

Gems Second

  • Clone the gems repo
  • Build and install all the gems by running the provided build.sh
  • Symlink the config to gems config by following
ln -fs /path/to/octo/config /path/to/octo/gems/octo-core/lib/octocore/config/config

NOTE:

  • Although we have pushed the gems to rubygems.org, but the database migration step is still dependent on running the rake task from the gems repo. So, this step becomes mandatory.

Initialising Database

This is a one time job. You should perform this before going further. Alongwith the octocore gem comes a utility called octocore-admin.

octocore-admin is essentially just a convenient wrapper around rake octo:migrate rake task. This rake task is defined in the octocore gem. In most cases, you should use the octocore-admin to perform migrations and use rake task only if you understand what you are doing.

octocore-admin

The general usage of octocore-admin is as follows

$ octocore-admin task /path/to/config

task: 				One of init, migrate, reset
/path/to/config		The path to config directory
  • Create Databases

Simply creates the namespaces in cassandra. If you perform this, also perform migrate

$ octocore-admin init /path/to/config
  • Migrate Databases

It can synchronize the cassandra tables to the model specifications. It is non destructive in nature and does not perform db deletion. New and old models are synchronized. Preserves redis cache and kong details.

$ octocore-admin migrate /path/to/config
loading from file: lib/octocore/config/config.yml
loading from file: lib/octocore/config/search/index/user.yml
loading from file: lib/octocore/config/config/config.yml
loading from file: lib/octocore/config/config/kong_config.yml
loading from file: lib/octocore/config/search/index/user.yml
Synchronized schema for Octo::ContactUs
Synchronized schema for Octo::Enterprise
Synchronized schema for Octo::ApiEvent
Synchronized schema for Octo::ApiHit
Synchronized schema for Octo::ApiTrack
Synchronized schema for Octo::AppInit
Synchronized schema for Octo::AppLogin
Synchronized schema for Octo::AppLogout
Synchronized schema for Octo::Authorization
Synchronized schema for Octo::Category
Synchronized schema for Octo::CategoryBaseline
Synchronized schema for Octo::CategoryHit
Synchronized schema for Octo::CategoryTrend
Synchronized schema for Octo::Conversions
Synchronized schema for Octo::Ctr
Synchronized schema for Octo::DimensionChoice
Synchronized schema for Octo::EngagementTime
Synchronized schema for Octo::FunnelData
Synchronized schema for Octo::Funnel
Synchronized schema for Octo::GcmNotification
Synchronized schema for Octo::NewsfeedHit
Synchronized schema for Octo::NotificationHit
Synchronized schema for Octo::Page
Synchronized schema for Octo::PageView
Synchronized schema for Octo::PageloadTime
Synchronized schema for Octo::Product
Synchronized schema for Octo::ProductBaseline
Synchronized schema for Octo::ProductHit
Synchronized schema for Octo::ProductPageView
Synchronized schema for Octo::ProductTrend
Synchronized schema for Octo::PushKey
Synchronized schema for Octo::Rules
Synchronized schema for Octo::Segment
Synchronized schema for Octo::SegmentData
Synchronized schema for Octo::Tag
Synchronized schema for Octo::TagBaseline
Synchronized schema for Octo::TagHit
Synchronized schema for Octo::TagTrend
Synchronized schema for Octo::Template
Synchronized schema for Octo::Plan
Synchronized schema for Octo::Subscriber
Synchronized schema for Octo::User
Synchronized schema for Octo::PushToken
Synchronized schema for Octo::UserBrowserDetails
Synchronized schema for Octo::UserLocationHistory
Synchronized schema for Octo::UserPersona
Synchronized schema for Octo::UserPhoneDetails
Synchronized schema for Octo::UserProfileDetails
Synchronized schema for Octo::UserTimeline
  • Reset Database

Erases the database, and creates a fresh new one. It's a power reset. Nothing is going to come back after you do this. Clears redis cache and kong details as well.

$ octocore-admin reset /path/to/config

NOTE:

  • Some ubuntu users have reported rake octo:migrate taking long time. In that case, please retry it a few times. It is known to work.

APIHandler

  • This repo handles all the API calls and pushes them to kafka
  • Clone APIHandler repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/apihandler/config/config

Creating Users of Octo-platform

  • Use the provided bin/create_kong_config.rb file in this repo alongwith the config directory. Review the contents of kong_yaml config file before.
$ bin/create_kong_config.rb ~/workspace/octo/current/config/
Loading from Config file: /Users/pranav/workspace/octo/current/config//config.yml
Loading from Config file: /Users/pranav/workspace/octo/current/config//kong_config.yml
I, [2016-08-09T17:24:30.009903 #64072]  INFO -- Octo: Octo booting up.
I, [2016-08-09T17:24:30.029038 #64072]  INFO -- Octo: I'm connected now.
I, [2016-08-09T17:24:30.039348 #64072]  INFO -- Octo: Setting callbacks.
I, [2016-08-09T17:24:30.045275 #64072]  INFO -- Octo: {"data"=>[{"upstream_url"=>"http://192.168.99.100:9001/update_push_token", "strip_request_path"=>true, "request_path"=>"/update_push_token", "id"=>"91ad4f1e-2834-4bdd-a42a-de61e1e872c6", "created_at"=>1468495646000, "preserve_host"=>false, "name"=>"UpdatePushToken"}, {"upstream_url"=>"http://192.168.99.100:9001/events", "strip_request_path"=>true, "request_path"=>"/events", "id"=>"c13febf5-757b-46e0-8430-c56256c766a7", "created_at"=>1468495646000, "preserve_host"=>false, "name"=>"events"}], "total"=>2}
I, [2016-08-09T17:24:30.056998 #64072]  INFO -- Octo: Attempting to create new enterprise with name: Jaobng
I, [2016-08-09T17:24:30.391932 #64072]  INFO -- Octo: Created templates for Enterprise: Jaobng
I, [2016-08-09T17:24:30.394817 #64072]  INFO -- Octo: Created segents for Enterprise: Jaobng
I, [2016-08-09T17:24:30.484397 #64072]  INFO -- Octo: Attempting to create new enterprise with name: Octo
I, [2016-08-09T17:24:30.491406 #64072]  INFO -- Octo: Created templates for Enterprise: Octo
I, [2016-08-09T17:24:30.496170 #64072]  INFO -- Octo: Created segents for Enterprise: Octo
I, [2016-08-09T17:24:30.577414 #64072]  INFO -- Octo: Adding Plugin key-auth for api events
I, [2016-08-09T17:24:30.583918 #64072]  INFO -- Octo: Adding Plugin key-auth for api UpdatePushToken
I, [2016-08-09T17:24:30.589688 #64072]  INFO -- Octo: Adding Plugin cors for api events
I, [2016-08-09T17:24:30.596629 #64072]  INFO -- Octo: Adding Plugin cors for api UpdatePushToken
I, [2016-08-09T17:24:30.602885 #64072]  INFO -- Octo: Adding Plugin rate-limiting for api events for consumer: 643c19e5-a913-49d7-bc4d-ed9c9c8183d8
  • Follow the instructions in provided README to run APIHandler.

Adding new users later on

For adding new users, you need to

  • Create their entry in the kong_config.yaml file.
  • Run the create_kong_setup.rb utility. This is an additive process. So, it will only add new enterprises/users and leave the existing ones untouched.

API Keys and credentials

  • The API Credentials are auto-generated. The only place they are exposed is in the dashboard. The dashboard also provides a JS snippet to be embedded into the web pages. For production, this is the recommended approach.
  • On development environments, one can use a utility like kong-dashboard to see the API keys and other kong details. This may prove helpful in debugging if you encounter issues related to kong.

APIConsumer

  • This repo handles all the kafka messages pushed by APIHandler and takes actions and triggers various hooks
  • Clone APIConsumer repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/apiconsumer/config/config
  • Follow the README instructions to get it working.

Recurring jobs

  • This repo is the hearbeat. It triggers various time based jobs.
  • Clone the Recurring Jobs repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/recurring-jobs/lib/config/config
  • Follow the instructions on provided README to start scheduler and workers.

Enterprise Dashboard

  • This is where business comes into picture. This is our end product that businesses use.
  • Clone the Enterprise Dashboard repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/enterprise-dashboard/config/config
  • As usual, follow the README

Newsfeed Webservice

Analytics Webservice

How to verify setup?

Connectivity

At this stage your development environment should be ready to serve the bare minimum of Octomatic platform.

You can use the fakestream command to create fake stream of data. fakestream comes bundled with octocore gem. It can be started like below

$ fakestream /path/to/config

Once you start fakestream, you would see a series of response tokens generated from the apihandler. This establishes the working proof of

  • API Key Setup
  • Kong setup
  • API Handler up and recieving

To know more about this refer to https://github.com/octoai/gems/blob/master/octo-core/bin/fakestream

Callbacks

Further if you open redis-cli and look for keys. You should see a huge list of all kinds of keys that look like below.

$ redis-cli
127.0.0.1:6379> keys *
1) "resque:schedules"
2) "resque:persisted_schedules"
3) "resque:schedules_changed"
....

It is again a working proof of

  • API Consumer receiving messages
  • Event callbacks and triggers happening

Data

Further you can also do cqlsh and see that the data is actually being written into the system.

$ cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.2.4 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> use octo_development ;
cqlsh:octo_development> DESCRIBE users ;

CREATE TABLE octo_development.users (
    enterprise_id uuid,
    id bigint,
    created_at timestamp,
    updated_at timestamp,
    PRIMARY KEY (enterprise_id, id)
) WITH CLUSTERING ORDER BY (id ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

Actually placing it in your web app

  • Clone Javascript-client https://github.com/octoai/javascript-client
  • Follow its readme to create its production bundles
  • Copy those production bundles at a location like /assets/js/. If you have a different location, you may need to edit the below script accordingly.
  • Use the following Snippet in the HTML page where you want Octo-matic to track things for you.
<!-- Octo-matic Script -->
<script type="application/javascript">
  var OCTO_API_KEY = "YOUR_OCTO_API_KEY";
  var octo_opts = {
    cookie_name: '_oid'
    base_url: 'analytics.mydomain.com'
  };

function loadScript(t,e){var o=document.createElement("script")
o.type="text/javascript",o.readyState?o.onreadystatechange=function(){("loaded"==o.readyState||"complete"==o.readyState)&&(o.onreadystatechange=null,e())}:o.onload=function(){e()},o.src=t,document.body.appendChild(o)}"undefined"==typeof jQuery?loadScript("/assets/js/octo-minified-0.1.js",function(){jQuery.octo.apikey=OCTO_API_KEY,jQuery.octo.init(octo_opts)}):loadScript("/assets/js/octo-minified-no-jquery-0.1.js",function(){jQuery.octo.apikey=OCTO_API_KEY,jQuery.octo.init(octo_opts)})


</script>
<!-- Octo-matic ends -->
Clone this wiki locally