Feature Engineering as Composable Functions
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
core/src
examples/src
plugin
project
scalding/src
test/src
tools
.gitignore
.travis.yml
CHANGELOG.markdown
LICENSE
README.markdown
TROUBLESHOOTING.markdown
USERGUIDE.markdown
formatting.md
sbt
scalariform.properties
version.sbt

README.markdown

Coppersmith - Feature Generation, as Functions

Build Status Gitter chat

  1. a person who makes artifacts from copper.

  2. data is malleable; fold and hammer it into various shapes that are more attractive to analysts and data scientists.

coppersmith is a library to enable the joining, aggregation, and synthesis of "features", streams of facts about entities derived from "analytical records".

This library was originally written by a squad within the Analytics & Information group at Commonwealth Bank, looking to improve the task of authoring and maintaining features for use in predictive analytics and machine learning.

Our working hypothesis was that for all the complexity of the business domain and the size of the data sets involved, fundamentally the logic used in feature generation can be described as simple functions and those functions should be able to be composed. The framework now called coppersmith grew out of our efforts to improve the lives of feature authors.

Quick Start

Add the following dependency to your SBT build configuration (typically build.sbt)

libraryDependencies += "au.com.cba.omnia" %% "coppersmith-scalding" % "<coppersmith-version>"

, where <coppersmith-version> is replaced with the version number of coppersmith you want to use (click the preceding link to find the latest version).

Documentation

We have a richly detailed user guide, which we consider a good introduction to coppersmith. PR's to the user guide as you become familiar with the library are especially encouraged!!!

There is also a troubleshooting guide available and a GitHub Pages site which provides additional information including latest version and usage information.

Generated Code

Classes and objects from the commbank.coppersmith.generated, commbank.coppersmith.scalding.generated and commbank.coppersmith.lift.generated packages are generated at build time with MultiwayJoinGenerator. The generated files can be found under the target/scala-2.11/src_managed/main/ directory of the core, scalding and test subprojects respectively.

Versioning

The change log lists all backwards-incompatible changes to the library (i.e. changes which might break existing client code). Any such changes require bumping the second number in the version.

Contributors