A package for capturing and analyzing network flow data and intraflow data, for network research, forensics, and security monitoring.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
analysis updating model scripts to be compliant with JSON changes Jan 17, 2018
data data tranlation, and SELECT can now have multiple comma-separated fields Mar 11, 2016
doc updated library users document Jul 5, 2018
install modified install targets for libjoy and install/uninstall scripts Mar 28, 2018
resources Move more files for testing under test/ directory Sep 5, 2017
rpm Delete obsolete files for building RPM. Oct 30, 2017
sleuth_pkg Updates fips 140 example policy using NIST SP800-52 revision 2 May 1, 2018
src clean up list printing function ENUMS Aug 9, 2018
test Sleuth FP resources Jan 25, 2018
win-joy Merge pull request #185 from cisco/windows-daemon Apr 16, 2018
windows update version to 3.0.0-dev and submitted bug fixes reported by testers Aug 9, 2018
.gitignore Ignore joy-*.tar created during pkg build process. Sep 14, 2017
.travis.yml Adding initial travis-ci file for continuous integration Nov 7, 2017
CHANGELOG.md updated build_pkg to accept version on command line Jul 20, 2018
COPYING updating License and Readme files Jul 25, 2017
Doxyfile cleaned up code, added doxygen tags, marked functions static that are… Oct 18, 2016
LICENSE initial commit Jan 8, 2016
Makefile fix make test error, update win_types version Jul 20, 2018
README.md update readme to match 3.0.0 release Aug 20, 2018
VERSION update version to 3.0.0-dev and submitted bug fixes reported by testers Aug 9, 2018
build_pkg removed no longer needed patching instructions Jul 20, 2018
compact_bd.txt added support for a compact byte distribution. Given a mapping file f… Mar 11, 2016
config added additonal default library dependacy paths for other operating s… May 2, 2018
examples.sh adding tls fingerprint inferences, and updating --where so that it ca… Jan 26, 2018
internal.net initial commit Jan 8, 2016
json trival typo Aug 30, 2017
sleuth Move tls_sec to before select and filter Jan 26, 2018

README.md

                       _              
                      (_) ___  _   _
                      | |/ _ \| | | |
                      | | (_) | |_| |
                     _/ |\___/ \__, |
                    |__/       |___/

        A package for capturing and analyzing network
     flow data and intraflow data, for network research,
          forensics, and security monitoring.

Build Status

Overview

Joy is a BSD-licensed libpcap-based software package for extracting data features from live network traffic or packet capture (pcap) files, using a flow-oriented model similar to that of IPFIX or Netflow, and then representing these data features in JSON. It also contains analysis tools that can be applied to these data files. Joy can be used to explore data at scale, especially security and threat-relevant data.

JSON is used in order to make the output easily consumable by data analysis tools. While the JSON output files are somewhat verbose, they are reasonably small, and they respond well to compression.

Joy can be configured to obtain intraflow data, that is, data and information about events that occur within a network flow, including:

  • the sequence of lengths and arrival times of IP packets, up to some configurable number of packets.

  • the empirical probability distribution of the bytes within the data portion of a flow, and the entropy derived from that value,

  • the sequence of lengths and arrival times of TLS records,

  • other non-encrypted TLS data, such as the list of offered ciphersuites, the selected ciphersuite, the length of the clientKeyExchange field, and the server certificate strings,

  • DNS names, addresses, and TTLs,

  • HTTP header elements and the first eight bytes of the HTTP body, and

  • the name of the process associated with the flow, for flows originate or terminate on the host on which pcap is running.

Joy is intended for use in security research, forensics, and for the monitoring of (small scale) networks to detect vulnerabilities, threats and other unauthorized or unwanted behavior. Researchers, administrators, penetration testers, and security operations teams can put this information to good use, for the protection of the networks being monitored, and in the case of vulnerabilities, for the benefit of the broader community through improved defensive posture. As with any network monitoring tool, Joy could potentially be misused; do not use it on any network of which you are not the owner or the administrator.

Flow, in positive psychology, is a state in which a person performing an activity is fully immersed in a feeling of energized focus, deep involvement, and joy. This second meaning inspired the choice of name for this software package.

Joy is alpha/beta software; we hope that you use it and benefit from it, but do understand that it is not suitable for production use.

Relation to Cisco ETA

Joy has helped support the research that paved the way for Cisco’s Encrypted Traffic Analytics (ETA), but it is not directly integrated into any of the Cisco products or services that implement ETA. The classifiers in Joy were trained on a small dataset several years ago, and do not represent the classification methods or performance of ETA. The intent of this feature is to allow network researchers to quickly train and deploy their own classifiers on a subset of the data features that Joy produces. For more information on training your own classifier, see saltUI/README or reach out to joy-users@cisco.com.

Credits

This package was written by David McGrew, Blake Anderson, Philip Perricone and Bill Hudson {mcgrew,blaander,phperric,bhudson}@cisco.com of Cisco Systems Advanced Security Research Group (ASRG) and Security and Trust Organization (STO).

Release 2.0

We are pleased to announce the 2.0 release of the package, which has these features:

  • The JSON schema has been updated to be better organized, more readable, and more searchable (by putting searchable keywords as the JSON names),
  • The new sleuth tool replaces query/joyq, and brings new functionality such as —fingerprint,
  • Much improved documentation, which covers the joy and sleuth tools, examples, and the JSON schema (see using-joy)

We hope that you find it useful!

  • the Joy Team

Release 3.0.0

We are pleased to announce the 3.0.0 release of the package, which has these features:

  • Modified JOY infrastructure code to be thread safe.
    • Allowed support multiple work threads for packet processing.
    • Each worker thread uses own output file.
    • Removed global variables for Config.
    • Modified code infrastructure to use Config Structure.
  • Modified the Makefile system to build the JOY infrastructure as a static and shared library.
  • Implemented an API for utilizing the JOY Library (joy_api.[hc]).
  • Implemented a Vector Packet Processing integration scheme to utilize VPP native infrastructure when building that integration.
  • Created 2 API test programs, joy_api_test.c and joy_api_test2.c.
  • Modified existing test programs to link against static JOY library instead of re-compiling the infrastructure code.
  • Modified versioning to use Common Security Module (CSM) conventions.
  • Modified build_pkg to accept package version on the command line.
  • Cleaned up coverity errors and warnings.
  • Various bug fixes.

Quick Start

Joy has been successfully run and tested on Linux (Debian, Ubuntu, CentOS, and Raspbian), Mac OS X and Windows. The system has been built with gcc and GNU make, but it should work with other development environments as well.

Go to the Wiki for a guide on building: Build Instructions

License

See License of Joy

See Copying for licenses of external libraries