Minutes

Mon 11/15/2010 (Joe, Aleks, Kee-Siong, Florian, Eugene, Caleb)

• Joe's thoughts on package management (Package Manager Notes) >> will use RPM and port it to Mac OS
  • thinking through DB migrations
  • polishing approximate counting
• Florian replaced Cephes library by own implementation for computing Student-t distribution (nice!)
  • is working on at Naive-Bayes interface and SQL only code
  • will look at logistic regression plpgsql code by Steven & Luke
• Eugene to look at the Decision Tree
• Kee Siong will talk to Jeff and Caleb about SVEC road map and dev plan.
• Caleb made github accessible from internal network (Yay!)
  • will be reviewing code as soon as reviews are created.
• Aleks to schedule initial reviews using Fisheye/Crucible
  • finishing up k-means clustering (multi-user support, doc)
  • need to "materialize" RelEng resource anytime now
  • start discussing cooperation with GP QA

Mon 11/8/2010 (Joe, Aleks, Kee-Siong, Florian, Eugene, Gavin)

• Development Status page
• Joe to pursue pkg mgr experiments
• Joe to ponder: python is for what? loops, 2nd order, testing harness.
• Gavin points out: API levels, documentation, unit testing (discussion postponed until we have the initial code).
• Aleks to coordinate setup of Fisheye/Crucible/Github environment for the team.

Mon 10/25/2010 (Joe, Aleks, Kee-Siong, Florian, Eugene, Gavin)

• Status
  • Joe: CountMin mods for Frequent Values in the works, challenge is parallelization
  • Florian to begin port/checkin of linear regression
  • Kee-Siong to begin port/checkin of sparse vectors, linking to lapack
  • Eugene to begin port/checkin of decision trees (and to send out a technical ref to the algorithm)
  • Gavin to send out a preliminary package mgr design doc
  • Aleks to begin port/checking of k-means
  • General note: as a group we need to gain experience/design pointers on writing "vanilla SQL" with external drivers in lieu of PL/PGSQL.
• Test harness to be postponed until more experience gathered
• Code review/QA process:
  • Gavin has 4 hours/week to do code reviews (once he returns)
  • When Gavin in San Mateo (week of 11/22) he will drive import of MadLib into GP QA process
• Questions remain on deploying PL/C to GP customers
• Brief discussion of API, including the Oracle way of passing parameters via table insertion. Some sense that we don't like that but it merits more discussion and experience
• GitHub seems week on issue tracking, etc. We need a solution to that before we release to public.

Mon 10/18/2010 (Gavin,Aleks,Steven)

• talk to Caleb about the internal functions vs madlib code changes
• methods for ver 1.0:
  • approximate counting (in works - Joe)
  • k-means (in works - Aleks)
  • random sampling (in works - Aleks)
  • linear regression - shared code GP/madlib (in queue)
  • naive-bayes - shared code GP/madlib (in queue)
  • sparse vector - shared code GP/madlib (in queue)
  • logistic regression (in queue)
• ongoing code review (critical!)
  • all of the above methods should be reviewed by at least two of Aleks/Gavin/Caleb
  • review for performance, code standards, code structure, file/directory structure, tests, documentation
• code review software (github plugin ?)
• package manager ver1.0 specs (Gavin)

Tues 10/5/2010

• Joe to drive todo list
  • build tools (Gavin drives)
  • directories (Joe drives)
  • API (Aleks drives)
• Aleks to set up meeting for team
• Discussion on API:
  • keep working through examples to understand tradeoffs
  • try to boil out distinguishing features: e.g. via type signatures of the code, e.g. via computational patterns like iteration/recursion, etc.
  • eventually we need a set of Guiding Principles for the next person writing a routine: help them figure out which style to use.

Tues 9/20/2010

• Attendees: Joe, Steven, Gavin, Aleks
• Purpose: Initial discussion on the high-level and technical aspects of the MAD lib project.

Project vision:

Goal: The general goal is to create a public forum for collaborating on in-database analytics. For Greenplum, there is a subsidiary goal: to promote the concept of Greenplum as a platform for MAD analytics.

Phase I:

• initial set of in-database analytical functions/recipes (approximate counting, k-means clustering, logistic regression, linear regression, random sampling, maybe sparse vectors)
• standards and structure defined: directory structure, tests and documentation, naming standards, etc.
• simple package mgmt: install/uninstall/versioning capabilities
• limited audience: some GP folks, some faculty and students, some business users

Phase II:

• including community contributions
• increasing audience to more academic and business users
• marketing activities to solicit feedback and new requirements
• implementing more complex package management
• extending to other platforms
• defining a MAD lib benchmark (performance and scalability)
• ready for broader Greenplum marketing promotion of MAD lib

Action items - leading to a Technical Design Document:

1. Defining package management -(hacking PGXS to serve both PostgreSQL and GP platforms for now)- based on an existing solution like python distutils or RubyGems. I think as a group we're relatively agnostic, though it'd be good to stick with a well-supported language toolchain for Postgres (hence Perl/Python are nice choices, Ruby/R secondary choices.)

2. Defining directory structure, installation targets/schema.

3. Defining coding standards ( C: Postgres style, SQL ?, other langs ? )

4. Defining documentation: HTML and PDF targets.

5. Defining testing approach.

6. Defining the MAD lib ‘API’

Help from Gavin:

• Dig up / record existing ideas on package management in Postgres
• Give an overview of coding style, documentation style, testing style, open source style that I’m going to recommend for the project
• Basic outline of what I think we should do on package management
• Overview of PGXS and proposal for directory layout