Data Analytics Log Manager
Data Analytics Log Manager helps collect log data and provides tools to analyze this data.
It provides a lot of flexibility on the type of log data. A single log entry can contain multiple arbitrary key value pairs (Let's say
"application" => "Test App",
"event" => "Clicked on Wiki" and so on). The analytics tools are powerful as well as easy to use. We aim to make complex analytics easy for not only programmers but also other users such as researchers, managers, designers etc. who may not have prior exposure to programming.
Data Analytics Log Manager provides the following:
An API for receiving log data from multiple applications and storing them in a database (currently PostgreSQL).
Tools to analyse this data. It is possible to filter the data, create a parent-child relationship (by let's say grouping the data by username and visualizing logs for various usernames as child tables), add synthetic/computed data (let's say the number of times the user performed a particular action) etc.
More information about the project is available on wiki.
You may find it useful if you have an application for which you would like to visualize how your users are using it and find patterns hidden in the data.
Existing Web Analytics tools, like Google Analytics, do not provide logging data at the individual-user data, and so are not usable for certain kinds of analytics. Many of the projects would like to capture detailed logs of actions users take in browser-based activities. The application will act as a shared tool for logging the data, transforming it and using CODAP for visualization.
Project Setup, Docker Compose option
- Clone the Log Manager repo. Optionally give the clone directory a more friendly name.
git clone https://github.com/concord-consortium/Data-Analytics-Log-Manager cc-log-manager
- From the root directory of the cloned repo, run:
- By default, the UI should be available to a browser at
Project Setup, Docker Fig option
This project includes a Dockerfile and a Fig (now Docker Compose) configuration that allows you to use containers based on prebuilt Rails and Postgres images, rather than having to install software locally on your machine. (However, production deploys use Heroku rather than Docker images you build.)
- Install and run Docker. Mac users will need to install and run [boot2docker] (http://boot2docker.io/) as well. Mac users can install these via Homebrew.
- Install fig. Note: Docker Compose is an updated version of Fig, so the steps below should work much the same with Compose. However, these instructions were developed using Fig.
fig buildin the root of this project. This will pull the required images, run
bundle install, etc.
- To start the server and database containers, run
fig upin the root of the project
- Initialize the database:
fig run web bin/rake db:create db:migrate
- If you are using boot2docker, you will need to find the IP address of the server, like this:
http://<boot2docker ip>:3000/, if appropriate) to run the app
The Fig configuration mounts the project folder as app root in the container. Thus, there is no special step to get your changes into the container (and running the app does not write to the
web container's filesystem; logs and caches go into the mounted project folder, and database changes persist in the
db container). Of course, you may need to
fig restart web after making changes. However, gems are stored in the container, and there is a special step required when updating the Gemfile:
bundle install in the Docker container without rebuilding
The default Rails image (
rails:onbuild) freezes the bundle for deployment. If you try to change the
Gemfile and run
fig run web bundle install
Gemfile.lock and install the new gems into the
web container, Bundler will tell you to try again with the
--no-deployment flag. It's all a lie. Here's how to run
fig run web /bin/bash -c "bundle config frozen 0 && bundle install && bundle config frozen 1"
docker ps -aand make a note of the name of the container that just stopped (each
fig run webhappens in a new container that "forks" the base
docker commit dataanalyticslogmanager_web_run_1 dataanalyticslogmanager_web(where
dataanalyticslogmanager_web_run_1is assumed to be the name of the recently run container, and
dataanalyticslogmanager_webthe image it uses). This step is necessary so that the next time Fig starts the web container, it uses the saved state of the
fig run webstep.
- You can now
docker rm dataanalyticslogmanager_web_run_1
If Fig gets confused
In order to get Fig to work correctly, I had to rename the project directory to
data_analytics_log_manager; Fig got confused by the mixed case. This may have been fixed by now.
Project Setup, local install (no Docker)
Install the following:
- Rails 4.1+
- PostgreSQL 9.2+
Clone the repository:
git clone https://github.com/apeeyush/Data-Analytics-Log-Manager.git
Run the following commands:
cd Data-Analytics-Log-Manager/ cp config/database.sample.yml config/database.yml bundle install npm install
Setup the database:
config/database.yml file to allow the Rails application to connect to PostgreSQL.
Either update the
password with your existing user's value or create a new user in PostgreSQL having
log_manager. Now, create database
log_manager_development and grant this user all privileges on
log_manager_development database or make it a superuser.
Run migrations on database:
Run application on localhost:
To run rails console:
A user can use Log Manager only after confirming the account. By default, in development emails will be placed in files in tmp/mails.
If you would like to configure a more complex setup, set the following environment variables (see config/initializers/actionmailer.rb):
# Required ENV["MAILER_DOMAIN_NAME"] # your server's domain name ENV["MAILER_SMTP_HOST"] # the SMTP host to send mail through ENV["MAILER_SMTP_USER_NAME"] # the SMTP username you want to use ENV["MAILER_SMTP_PASSWORD"] # the SMTP password you want to use #Optional ENV["MAILER_DELIVERY_METHOD"] # defaults to :file, for production set it to "smtp" ENV["MAILER_SMTP_PORT"] # defaults to 587 ENV["MAILER_SMTP_AUTHENTICATION_METHOD"] # defaults to :login ENV["MAILER_SMTP_STARTTLS_AUTO"] # defaults to true
See the Getting Started Guide for Developers for more information about the technical working of Data Analytics Log Manager.
The style guide for the project is available here.
Feel free to fork the repository and hack on it! Implemented some feature? Well, just send a pull request. Facing problems setting things up or thought of a feature that may be useful for everybody? Contact us or open github issues for questions, bugs, feature requests etc.
Not sure how to start? Email us and let us know you're interested, and what you can do, and we'll figure out something you can help with.
Data Analytics Log Manager is released under the MIT License.