Navigation Menu

Skip to content
This repository has been archived by the owner on Mar 7, 2018. It is now read-only.

Commit

Permalink
Merge pull request #74 from CatalystCode/c-w-patch-1
Browse files Browse the repository at this point in the history
Remove outdated Kafka entry
  • Loading branch information
c-w committed Aug 9, 2017
2 parents 788c62c + d37b65a commit 7f4c899
Showing 1 changed file with 1 addition and 76 deletions.
77 changes: 1 addition & 76 deletions README.md
Expand Up @@ -13,82 +13,7 @@ This project contains a Spark Streaming job that ingests data into the Fortis sy
3. Narrow down the stream of events based on user-defined geo-areas, target keywords and blacklisted terms.
4. Perform trend detection and aggregate the metrics that back Project Fortis.

At the end of the ingestion pipeline, we publish the events to Kafka from where any downstream processors or aggregators
can consume the data. The schema of the data in Kafka is as follows:

```json
{
"title": "FortisEvent",
"type": "object",
"properties": {
"language": {
"type": "string"
},
"locations": {
"description": "The ids of all places mentioned in the event",
"type": "array",
"items": {
"description": "A Who's-On-First id",
"type": "string"
}
},
"sentiments": {
"type": "array",
"items": {
"description": "Neutral sentiment is 0.6, 0 is most negative, 1 is most positive.",
"type": "number",
"minimum": 0,
"maximum": 1
}
},
"keywords": {
"type": "array",
"items": {
"type": "string"
}
},
"entities": {
"type": "array",
"items": {
"type": "string"
}
},
"summary": {
"type": "string"
},
"id": {
"type": "string"
},
"createdAtEpoch": {
"type": "number"
},
"body": {
"type": "string"
},
"title": {
"type": "string"
},
"publisher": {
"type": "string"
},
"sourceUrl": {
"type": "string"
},
"sharedLocations": {
"description": "The ids of all places explicitly tagged in the event",
"type": "array",
"items": {
"description": "A Who's-On-First id",
"type": "string"
}
}
},
"required": [
"id",
"createdAtEpoch"
]
}
```
At the end of the ingestion pipeline, we publish the events and various aggregations to Cassandra.


## Development setup ##
Expand Down

0 comments on commit 7f4c899

Please sign in to comment.