Persister
Kushal Kant Goyal edited this page Sep 21, 2015
·
5 revisions
Navigation
- Home
- [What is AIDR?](AIDR Overview)
- The science behind AIDR
- [Operator's manual](AIDR Operator's Manual)
- [Public API documentation](API documentation)
System administrators
Developers
- High-level overview
- Common
- DB Manager
- DTO standards
- Database schema
- Manager
- Manager API
- Collector
- Collector API
- Reconnect strategy
- Collector Tester
- Output
- Output API
- Output Buffered
- Output Streaming
- Output Tester
- Persister
- Persister API
- Persister Tester
- Tagger
- Tagger Concepts
- Tagger API
- Tagger Tester
- Trainer
- Trainer API
- PyBossa Trainer
Design and standards
- User needs and personas
- Roadmap
- Design concepts
- Stream processing
- Per collection start or stop
- Standards
- Logging standards
- Coding standards
- Naming conventions
- Testing
- Release checklist
QCRI-specific
Credits
Clone this wiki locally
Name: aidr-persister
Code: https://github.com/Qatar-Computing-Research-Institute/CrisisComputing/tree/master/aidr-persister
Overview
The aidr-persister module stores (persists) the items that have been collected and/or tagged into files. The items are persisted in JSON format in zip-compressed files.
The aidr-persister module also allow users to perform filtering and conversion operations on these files.
Filtering operations are the same ones as implemented in aidr-output.
Conversion operations are basically to allow three types of file format:
- One JSON array containing all items.
- One JSON object on a line per item.
- One comma-separated-value (CSV) line per item.
The aidr-persister module has [no explicit per-collection start/stop](Per collection start or stop).
Technologies
- Google GSON 2.2.4 (JSON processor)
- JEDIS 2.4.2 (REDIS API library)
- SuperCSV (CSV library)
- Jersey 2+ (for JAX-RX 2.0)
- FasterXML (JSON processor)