Permalink
Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
397 lines (348 sloc) 21.1 KB
############################################################################
# Software Roadmap #
############################################################################
This document describes how features will be developed over time.
Code will be developed following the concept of "software stack".
Reference: http://en.wikipedia.org/wiki/Solution_stack
Definition - What does Software Stack mean?
"A software stack is a group of programs that work in tandem to produce a
result or achieve a common goal."
( http://www.techopedia.com/definition/27268/software-stack )
Libraries and functions are based on a stack system where the lower layer is "Layer 0"
and the highest layer is "Layer 6" consisting of a total of 7 software layers.
Layer 0 (Lower level calls dealing directly with OS or Hardware)
|--> Layer 1 (layer 1 functions depend on layer 0 functions)
|--> Layer 2 (layer 2 functions depend on layer 1 functions)
|--> Layer 3 (layer 3 functions depend on layer 2 functions)
|--> Layer 4 (layer 4 functions depend on layer 3 functions)
|--> Layer 5 (layer 5 functions depend on layer 4 functions)
|--> Layer 6 (Higher level calls and complex wrappers)
Each layer is divided in four classes divided as follow:
1) Internal: low level operations and not exposed at higher levels or wrappers
2) Data: functions dealing with data analysis and manipulation (ETL)
3) Network: functions dealing with network operations
4) Utility: components aggregating functions to execute complex tasks
We are planning to develop around 150 components to cover the most basic features
needed for current operations in information security.
The following table outlines the expected distribution of components divided
by level and by class.
|---------|----------|------|---------|---------|-------|
| Planned Component in each layer by class |
|---------|----------|------|---------|---------|-------|
| Layer | Internal | Data | Network | Utility | Total |
|---------|----------|------|---------|---------|-------|
| Layer 0 | 14 | 2 | 0 | 2 | 18 |
| Layer 1 | 7 | 2 | 2 | 9 | 20 |
| Layer 2 | 10 | 1 | 1 | 5 | 17 |
| Layer 3 | 3 | 8 | 5 | 3 | 19 |
| Layer 4 | 2 | 1 | 11 | 9 | 23 |
| Layer 5 | 0 | 5 | 15 | 1 | 21 |
| Layer 6 | 0 | 0 | 1 | 14 | 15 |
|---------|----------|------|---------|---------|-------|
| Total | 36 | 19 | 35 | 43 | 133 |
|---------|----------|------|---------|---------|-------|
To further clarify how the code base is developed we think is beneficial to
provide a one-line description of each planned component.
############################################################################
# Layer 0 - Component Details #
############################################################################
|-------------------------------------------------------------|
| Layer 0 |
|----------------|------------|---------------|---------------|
| Internal | Data | Network | Utility |
|----------------|------------|---------------|---------------|
| OS-like | String | | logging |
| sys-like | Binary | | Py-call-table |
| Memory-space | | | |
| Queue | | | |
| OS-privileges | | | |
| OS-permissions | | | |
| Process-fork | | | |
| Process-spawn | | | |
| Process-clone | | | |
| Stat-umask | | | |
| Sysvar | | | |
| Endian | | | |
| Variables | | | |
| Error | | | |
|----------------|------------|---------------|---------------|
Internal
- OS-like: replicate most of functions present in python "os" module
- sys-like: replicate most of functions present in python "sys" module
- Memory-space: create ram disk for in memory storage
- Queue: create and access generic queue objects
- OS-privileges: work with system calls related to privileges
- OS-permissions: work with system calls related to permissions
- Process-fork: implement process fork system calls
- Process-spawn: implement process spawn system calls
- Process-clone: implement process clone system calls
- Stat-umask: replicate functions present in python stats module plus special cases
- Sysvar: read/write system data and statistics
- Endian: deal with endianness detection and uniformity
- Variables: read/write system and user defined variables
- Error: describe and access internal errors and system messages
Data
- String: functions to deal with data in the form of text string
- Binary: functions to deal with data in the form of binary string
Utility
- logging: message logging and application checkpointing
- Py-call-table: namespace, list of functions, variables and object types
############################################################################
# Layer 1 - Component Details #
############################################################################
|---------------------------------------------------------------------|
| Layer 1 |
|----------------|------------------|--------------|------------------|
| Internal | Data | Network | Utility |
|----------------|------------------|--------------|------------------|
| File-simple | Unicode-patterns | IP-address | Module-loader |
| File-special | Data-entropy | Socket | Number |
| Memory-limits | | | Regex |
| Memory-secheap | | | File-magic |
| Traceback | | | String-whitelist |
| Process | | | String-blacklist |
| Directory | | | Binary-whitelist |
| | | | Binary-blacklist |
| | | | Sys-entropy |
|----------------|------------------|--------------|------------------|
Internal
- File-simple: functions to deal with standard file objects
- File-special: functions to deal with special file objects
- Memory-limits: manage and control memory object limitations
- Memory-secheap: manage and control memory object allocation
- Traceback: intercept programming and interpreter unexpected errors
- Process: create/modify/delete processes
- Directory: work with directory objects
Data
- Unicode-patterns: functions and objects to translate any character in valid unicode object
- Data-entropy: analyze and test entropy in data streams and objects
Network
- IP-address: handling of IPv4 and IPv6 addresses and network
- Socket: create and access network sockets in user space
Utility
- Module-loader: load python modules
- Number: handling numeric operations
- Regex: creation and use of regular expressions
- File-magic: access and use file magic number objects
- String-whitelist: white listing filtering of string objects
- String-blacklist: black listing filtering of string objects
- Binary-whitelist: white listing filtering in binary objects
- Binary-blacklist: black listing filtering in binary objects
- Sys-entropy: measure and monitor file system entropy
############################################################################
# Layer 2 - Component Details #
############################################################################
|-------------------------------------------------------------------|
| Layer 2 |
|----------------|-----------|----------------|---------------------|
| Internal | Data | Network | Utility |
|----------------|-----------|----------------|---------------------|
| Sequence | Unicode | IP-address-ops | Config |
| Process-sync | | | Math |
| Process-async | | | Crypto-hash |
| Socket-ops | | | Random-hash |
| Tester | | | Memory-obj-checksum |
| zlib | | | |
| zip | | | |
| Tar-gnu | | | |
| GZip | | | |
| Random-num-gen | | | |
|----------------|-----------|----------------|---------------------|
Internal
- Sequence: operations on any bite sequence (sort,shuffle,etc..)
- Process-sync: creation of processes using synchronous operations
- Process-async: creation of processes using asynchronous operations
- Socket-ops: socket primitives for data operations
- Tester: code testing and data verification
- zlib: zlib operations
- zip: zip operations
- Tar-gnu: tar operations in archives created with gnu tar
- GZip: gzip operations
- Random-num-gen: function for PRNG
Data
- Unicode: handling unicode objects
Network
- IP-address-ops: handling operations with objects involving IP addresses
Utility
- Config: creation and usage of global configuration file
- Math: function for handling mathematical operations
- Crypto-hash: functions for handling cryptographic hashes
- Random-hash: functions for handling random hashes
- Memory-obj-checksum: check integrity and uniqueness of memory objects
############################################################################
# Layer 3 - Component Details #
############################################################################
|----------------------------------------------------------------------|
| Layer 3 |
|---------------------|-----------------|---------------|--------------|
| Internal | Data | Network | Utility |
|---------------------|-----------------|---------------|--------------|
| Process-exec | MIME | SSL-ASN1 | File-entropy |
| Process-multi-sync | String-patterns | SSL-ciphers | Fuzzy-hash |
| Process-multi-async | Binary-patterns | Network-core | Random-test |
| | JSON | Net-errcode | |
| | YAML | DNS | |
| | CSV | | |
| | On-disk-dicts | | |
| | On-disk-lists | | |
|---------------------|-----------------|---------------|--------------|
Internal
- Process-exec: single process execution routines
- Process-multi-sync: operations of multiple processes using synchronous logic
- Process-multy-async: operations of multiple processes using asynchronous logic
Data
- MIME: functions for handling MIME objects
- String-patterns: string patterns reference, analysis and modification
- Binary-patterns: binary patterns reference, analysis and modification
- JSON: JSON operations
- YAML: YAML operations
- CSV: CSV operations
- On-disk-dicts: operations with on disk dictionary
- On-disk-lists: operations with on disk lists
Network
- SSL-ASN1: ASN1 database and functions for handling ASN1 objects
- SSL-ciphers: cipher database and functions for cipher operations
- Network-core: wrapper around core socket functionalities and functions for network operations
- Net-errcode: network errors database and functions for error detection and correlations
- DNS: DNS protocol operations
Utility
- File-entropy: entropy calculation and detection of standard file objects
- Fuzzy-hash: handling fuzzy hashing
- Random-test: testing random number quality and properties
############################################################################
# Layer 4 - Component Details #
############################################################################
|-------------------------------------------------------------------------|
| Layer 4 |
|----------------------|--------------|-------------------|---------------|
| Internal | Data | Network | Utility |
|----------------------|--------------|-------------------|---------------|
| Proc-multi-sync-mng | KyotoCabinet | FTP | Image-pattern |
| Proc-multi-async-mng | | POP3 | Image-parser |
| | | SMTP | Exif |
| | | IMAP4 | MS-OLE |
| | | HTTP | MS-PE |
| | | cURL-like | CryptoKey |
| | | WhoIs | String-attack |
| | | Secpy-ssl | Useragent |
| | | SSH-patterns | Web-cookie |
| | | Protocol-headers | |
| | | Useragent-pattern | |
|----------------------|--------------|-------------------|---------------|
Internal
- Proc-multi-sync-mng: management of multiple processes using synchronous logic
- Proc-multi-async-mng: management of multiple processes using asynchronous logic
Data
- KyotoCabinet: libraries and functions for handling KyotoCabinet objects
Network
- FTP: FTP protocol operations and client functionalities
- POP3: POP3 protocol operations and client functionalities
- SMTP: SMTP protocol operations and client functionalities
- IMAP4: IMAP4 protocol operations and client functionalities
- HTTP: HTTP protocol operations and client functionalities
- cURL-like: advanced functionalities for protocols compatibles with URI addresses
- WhoIs: WHOIS protocol operations and client functionalities
- Secpy-ssl: porting of standard openssl libraries with improved security controls
- SSH-patterns: database of SSH protocol patterns and functions for standard protocol functionalities
- Protocol-headers: database of common protocol headers and functions for passive protocol analysis
- Useragent-pattern: database of common useragents and functions for useragent string creation
Utility
- Image-pattern: database of binary patterns of common image formats
- Image-parser: parsing of generic image objects and standard image artifacts
- Exif: handling read/write operations of exif metadata objects
- MS-OLE: handling Microsoft OLE objects
- MS-PE: handling Microsoft executable files and PE object analysis
- CryptoKey: functions for handling cryptographic keys
- String-attack: fuzzy string exploitation and string attacks patterns
- Useragent: handling useragent operations and passive useragent analysis
- Web-cookie: handling web cookies
############################################################################
# Layer 5 - Component Details #
############################################################################
|-------------------------------------------------------|
| Layer 5 |
|------------|------------|--------------|--------------|
| Internal | Data | Network | Utility |
|------------|------------|--------------|--------------|
| | Image-jpeg | POP3-server | Authenticode |
| | Image-gif | STMP-server | |
| | Image-bmp | FTP-server | |
| | Image-png | SSH | |
| | Image-tiff | FTPES | |
| | | POP3S | |
| | | SMTPS | |
| | | IMAP4S | |
| | | HTTPS | |
| | | SSL-cURL | |
| | | HTTP-server | |
| | | IMAP4-server | |
| | | DNSsec | |
| | | DNS-cache | |
| | | Virus-rules | |
|------------|------------|--------------|--------------|
Data
- Image-jpeg: handling of JPEG file formats and specific artifacts
- Image-gif: handling of GIF file formats and specific artifacts
- Image-bmp: handling of BMP file formats and specific artifacts
- Image-png: handling of PNG file formats and specific artifacts
- Image-tiff: handling of TIFF file formats and specific artifacts
Network
- POP3-server: POP3 server operations and for client testing
- STMP-server: SMTP server operations and for client testing
- FTP-server: FTP server operations and for client testing
- SSH: SSH protocol objects and client operations
- FTPES: SSL protected FTP client operations
- POP3S: SSL protected POP3 client operations
- SMTPS: SSL protected SMTP client operations
- IMAP4S: SSL protected IMAP4 client operations
- HTTPS: SSL protected HTTP client operations
- SSL-cURL: SSL protected advanced functionalities for protocols compatibles with URI addresses
- HTTP-server: HTTP server operations and for client testing
- IMAP4-server: IMAP4 server operations and for client testing
- DNSsec: DNSSEC protocol objects and client operations
- DNS-cache: local DNS cache for passive and active operations
- Virus-rules: database of well-known patterns of malicious data
Utility
- Authenticode: files with Microsoft Authenticode signatures
############################################################################
# Layer 6 - Component Details #
############################################################################
|----------------------------------------------------------|
| Layer 6 |
|------------|--------|-----------|------------------------|
| Internal | Data | Network | Utility |
|------------|--------|-----------|------------------------|
| | | DNS-proxy | FTP-protocol-analyzer |
| | | | POP3-protocol-analyzer |
| | | | SMTP-protocol-analyzer |
| | | | IMAP-protocol-analyzer |
| | | | HTTP-protocol-analyzer |
| | | | DNS-analyzer |
| | | | jpeg-analyzer |
| | | | gif-analyzer |
| | | | SSL-analyzer |
| | | | SSH-analyzer |
| | | | Socket-gif-hack |
| | | | PCI-DSS-analyzer |
| | | | Modules-Repository |
| | | | Hash-PyPI-modules |
| | | | Image-entropy |
|------------|--------|-----------|------------------------|
Network
- DNS-proxy: DNS proxy with passive and active filtering capabilities
Utility
- FTP-protocol-analyzer: full stack analyzer of FTP protocol
- POP3-protocol-analyzer: full stack analyzer of POP3 protocol
- SMTP-protocol-analyzer: full stack analyzer of SMTP protocol
- IMAP-protocol-analyzer: full stack analyzer of IMAP protocol
- HTTP-protocol-analyzer: full stack analyzer of HTTP protocol
- SSL-analyzer: full stack analyzer of SSL protocol
- SSH-analyzer: full stack analyzer of SSH protocol
- jpeg-analyzer: static analyzer of JPEG files with full artifact analysis
- gif-analyzer: static analyzer of GIF files with full artifact analysis
- Socket-gif-hack: websockets using animated GIF files
- PCI-DSS-analyzer: full stack analyzer with internal PCI-DSS reference database
- Modules-Repository: handling modules verification and secure versioning
- Hash-PyPI-modules: database of file PYPI hashes and functions to access it
- Image-entropy: entropy calculation and detection of entropy in image files
############################################################################