Skip to content
BigConnect core engine
Branch: master
Clone or download
flaviusburca
Latest commit 3b57646 May 17, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
common Update ES plugin config May 17, 2019
docs Refactor readme May 14, 2019
licenses Licensing requirements Jan 4, 2019
plugins Remove obsolete modules May 17, 2019
shell Bump version Mar 19, 2019
test Bump version Mar 19, 2019
.gitignore Update ES plugin config May 17, 2019
.travis.yml Install maven manually May 14, 2019
LICENSE.txt Update license info Jan 8, 2019
NOTICE.txt Create NOTICE.txt Jan 4, 2019
README.md Refactor readme May 14, 2019
pom.xml Remove obsolete modules May 17, 2019

README.md

Welcome to BigConnect! Build Status

bigconnect.io

BigConnect provides an open and extensible platform built on Big Data technologies to understand and work with unstructured data lakes.

It’s an information-agnostic system where all data that flows into the system is transformed, mapped, enriched and then stored in a logical way using a semantic model of concepts, attributes and relationships.

It provides an extensible, unified visual interface with tools for data discovery and analysis, collaboration, alerting and information management.

Main features:

  • Dynamic Data Model The core foundation of BigConnectT is the dynamic, semantic data model. It represents the way you store, correlate and query all information and it’s a conceptual data model that expresses information in a factual way: objects, relations and properties
  • Data ingestion Any kind of information can be ingested: databases, office documents, text files, HTML files, images, audio and video files.
  • Enterprise Search Built on top of ElasticSearch, it features enterprise-grade search features
  • Enrichment Distributed enrichment pipeline based on pluggable Data Workers allow information to be altered anytime, using rules, algorithms and external systems
  • Collaboration Work inside spaces, share invite users to join and share information with others
  • Security Security is enforced both at the UI and datastore level
  • Scalability Featuring a pluggable storage layer, RocksDB and Apache Accumulo implementations are available to achieve massive scalability
  • SQL support Query data using SQL queries
  • Cypher support Run Cypher queries on top of the property graph data model

Demo

Check our YouTube channel for demos

Powerful UI

A beautiful and extensible user interface built with FlightJS and React to help you get the most of BigConnect. Everything in BigConnect UI happens in the context of a Space. When a user is first logged in, a Default space is created for him, using his username. The dashboard and the analysis tools are specific to a certain Space.

Dashboard

The visual console includes a dashboard where users are being taken upon login. Various widgets can be added or removed from the dashboard to bring to focus any relevant information. New widgets can be developed using the available extension points on docs.bigconnect.io

Search

The UI provides a Google-like search experience, enriched with filters, sorting, refinements and spatial queries. The search field supports the Lucene query syntax so you can use special things like wildcard (*), question mark (?), keyword matching, phrase matching, grouping, boolean queries (AND, OR), fuzzy queries (~), proximity matching (~4), range searches (amount:[500 TO 800]) and boosts (^).

Searches can be saved and shared with everybody and can be later used on the space dashboard or inside analysis tools. Sometimes BigConnect refers to a saved search as a "dataset".

Data loading (in development)

Data ingestion is a visual process. Choose a datasource, provide connection details, preview the data and map it to the ontology in a visual way and you are ready to go !

Data ingestion can be done using the UI or using scalable ETL and streaming tools. We currently provide plugins for Pentaho Data Integration and Apache NiFi (as part of the Enterprise offering)

Analysis tools

Data can be analyzed using the various tools available in the platform. BigConnect provides an extensible tools mechanism, where developers can create their own tools. This process is desribed on docs.bigconnect.io We do provide some tools out of the box to get you started, as described below.

Charts

This tool allows you to chart a dataset (or saved search) using the Plotly charting library. Charts can be saved for later use, and can be placed on the space dashboard.

Graph

This tool allows you to view and manipulate data and it currently has a lot of features:

  • Create entities and relations
  • Add datasets to quickly visualize them
  • Find hidden paths between two entities
  • Add related items for an entity
  • Layout the graph using various algorithms: Cola, Euler, Cose etc.
  • Adjust vertex size based on Page Rank, Degree Centrality, Closeness Centrality, Betweenness etc.
Map

This tool allows you to view entities that have geospatial information associated, and provides things like clustering, heatmaps and adding new map layers.

Machine Learning

This tool currently allows you use Jupyter notebooks and H2O to apply machine learning algoritms on a saved dataset. BigConnect includes its own Python runtime and special APIs to load BigConnect data from Python or H2O.

Administration

The admin console allows to customize the data model (ontology), users, roles, UI extensions and various configuration options extraction. The console is extensible, so new items can be easily added as plugins using the provided extension points.

You can’t perform that action at this time.