Time-stamp: <2014-04-07 16:28:57 tony>
but we are rebuilding it.
Here’s a general fast start approach for using this.
- Get access to a Common Lisp implementation on your platform.
Make sure you have BLAS, LAPACK, and their corresponding
development environments on your system.
For example, on a recent debian (testing), one may need:
libffi6 libblas liblapack libgsl libRmath
and the corresponding -dev packages.
- if needed, install quicklisp (http://www.quicklisp.org)
- if wanted, install git (http://www.git-scm.org/)
- Use git to fetch common-lisp-statistics from the repository:
git://github.com/blindglobe/common-lisp-stat.git
and put it into your quicklisp local-projects directory
(you will need to put a few more projects there as well),
OR make sure you have an internet connection and go to step 5 and
fetch a not-so-bleeding-edge version from QuickLisp.
We suggest something like:
mkdir ~/quicklisp/
mkdir ~/quicklisp/local-projects
cd ~/quicklisp/local-projects
git clone git://github.com/blindglobe/common-lisp-stat.git
git clone git://github.com/blindglobe/lisp-matrix.git(with the 5th and on lines representing stuff undergoing rapid change, so possibly out of date on QUICKLISP).
You might need to grab XARRAY as well, and if behind a firewall, might need to use HTTP or similar alternative transport.
- Start up the Common Lisp implementation, and:
(ql:register-local-projects)
(ql:quickload :antik)
(ql:quickload :cls)TODO: Unfortunately, it looks like ANTIK needs to be preloaded, or it gets confused. I don’t yet know why.
- get coffee, tea, water, beer, or a glass of wine, review the Common Lisp Statistics mailing list, chat with friends, etc, until it is done compiling and loading.
- Report success/failure back to the news group ( https://groups.google.com/forum/?fromgroups#!forum/lisp-stat ) or directly to Tony Rossini mail, i.e. ( mailto:blindglobe@gmail.com ) if you don’t want to be public about it.
- Start trying it out. The examples/ directory is a good place to start evaluating the files, one at a time. Preferrably in order!
More details can be fond in the subsequent sections.
Current requirements are basically that you have a Common Lisp implementation such as SBCL or CCL (though CLISP might work) and BLAS/LAPACK on your system.
SBCL and CCL are in active use by developers. We are interested in finding out details regarding successes with other distributions and would be pleased to add such information here.
Currently, LispCabinet would be the simplest way to get started, with either SBCL or CCL as the Lisp Implementation. Need experiences from others using
http://lispcabinet.sourceforge.net/
Most distributions contain SBCL or CLISP. The instructions for getting and installing CCL are relatively straightforward.
Both CCL and SBCL have been remarked to work. CCL seems to be preferred on this platform?
Currently this is a manual operation, which will change in a later version.
Edit the file external/cl-blapack/load-blapack-libs.lisp
Search for the following 3 parameters
- gfortran-lib
- blas-lib
- lapack-lib
You need to check that your dynamic library path has been properly set up in the shell, by setting up your .bashrc (or equivalent shell init file, or equivalent environment variable settings).
i.e. compilation environment and tools, if needed, on Microsoft Windows?
For windows, we recommend you use cygwin to get straightforward access. I’ll document the steps if there is a demand.
Get lapack-dev, blas-dev from your Linux distribution.
i.e., something like:
sudo apt-get install libblas
sudo apt-get install liblapackFor Linux, if installed in a weird place, you need to make sure the loader looks for it.
set LD_LIBRARY_PATH=$LD_LIBRARY_PATH:???and need to replace the ???
SBCL is known to work (0.58+, 1.1.1+)
CCL is thought to work
(CMUCL and CLISP may work, but not tested)
For Mac OSX set
export DYLD_FALLBACK-LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/opt/local/libFIXME: Tony has no clue, please fill this in since most mailing list folks use Macs.
- CCL is thought to work
- SBCL is known to work
For OS X: change the parameters as suggested in the file. Both BLAS and LAPACK are pre installed on Mac OSX.
LispCabinet has it preinstalled, and you can use that version to upgrade
Debian/Ubuntu also allow you access to a (possibly outdated) version. Not sure about upgrade potential.
On Linux and MacOSX, I would recommend using the instructions at the QuickLisp www site (http://www.quicklisp.org).
Unfortunately, as much as we really would like to get rid of this PITA stage, we are pre-alpha, and that means no chance, unless you want to fix your own bugs and copy/paste fixes, etc. Much simpler to figure out a small bit of git.
Hopefully, your distribution (Linux) has it, and instructions for getting it, along with tutorials and documentation, can be found for MacOSX and Microsoft Windows on http://www.git-scm.org/
GitHub also has a Microsoft Windows application that might be useful for fetching and working with GitHub repositories (including this one).
sudo port install gitUsing Debian, Ubuntu, or other apt-get based distributions as an example:
sudo apt-get install gitAt this stage, we need to identify where you will put the D/L’d package. If you are have an existing, highly tuned quicklisp setup, please figure it out and jump to the next stage, but basically you need to pull common-lisp-stat from Blindglobe’s repository, along with a few others.
cd ~/quicklisp/local-projects/
git clone https://github.com/blindglobe/common-lisp-stat.gitRegarding the “few others”, all of them are currently part of QuickLisp, so you only need them if you want to develop with them. Which isn’t entirely a bad idea.
These would include:
| lisp-matrix | CLS | generic interface to BLAS and LAPACK using matrix like API |
| cl-blapack | lisp-matrix | BLAS / LAPACK FFI |
| fnv | lisp-matrix | foriegn-numeric-vectors, C-storage for lisp-matrix |
| ffa | lisp-matrix | foriegn-??-arrays, LISP storage for lisp-matrix |
| lift | CLS | unit-testing |
| listoflist | CLS | list as an array data structure |
| xarray |
Start up your Common Lisp implementation and type in:
(ql:register-local-projects)
(ql:quickload :cls)Retire for a well earned coffee and upon your return you should find the package completely installed. Obviously, potential errors can creep in with spelling the filenames correctly, so be careful.
And now, everything should be working. This is the case for at least one person, so data on failures is very welcome.
Now, load into your IDE or lisp, the files in the examples directory, such as:
| 00-loadingData.lisp |
| 02-DSVloading.lisp |
| 04-dataManipulation.lisp |
| 10-basicEDA.lisp |
| 20-visual-2d-cairo2.lisp |
| 50-TTestExample.lisp |
| 60-regressionExamples.lisp |
| linear-regression.lisp |
| ls-demo.lisp |
| ls-demo-ls1.lisp |
| XX-readMe.lisp |
And more
Load the example lisp files
This is just for directory convenience, not for any real reason.
cd ~/quicklisp/local-projectssbclor
CCL(ql:quickload :cls)
(in-package :cls)Initially we will work in the cls package as all the basic functions we would need are present
For serious work we would create our own workspace and save it in a separate package, but for now we will take this short cut.
(normal-rand 20)
(setf mytest (normal-rand 20))and see if they work (basic CFFI functionality for external C library, LIFT package for unit-testing framework to ensure run time stability).
In Common Lisp, you need to select and setup namespace to store data and functions. There is a scratch user-package, or sandbox, for CLS, cls-user , which you can select via:
(in-package :cls-user)and this has some basic modules from CLS instantiated (dataframes, probability calculus, numerical linear algebra, basic summaries (numerical and visual displays).
However, it can be better is to create a package to work in, which pulls in only desired functionality:
(in-package :cl-user)
(defpackage :my-package-user
(:documentation "demo of how to put serious work should be placed in
a similar package elsewhere for reproducibility. This hints as to
what needs to be done for a user- or analysis-package.")
(:nicknames :my-clswork-user)
(:use :common-lisp ; always needed for user playgrounds!
:lisp-matrix ; we only need the packages that we need...
:common-lisp-statistics
:cl-variates
:lisp-stat-data-examples) ;; this ensures access to a data package
(:shadowing-import-from :lisp-stat
;; This is needed temporarily until we resolve the dependency and call structure.
call-method call-next-method
expt + - * / ** mod rem abs 1+ 1- log exp sqrt sin cos tan
asin acos atan sinh cosh tanh asinh acosh atanh float random
truncate floor ceiling round minusp zerop plusp evenp oddp
< <= = /= >= > > ;; complex
conjugate realpart imagpart phase
min max logand logior logxor lognot ffloor fceiling
ftruncate fround signum cis
<= float imagpart)
(:export summarize-data summarize-results this-data this-report))
(in-package :my-clswork-user) ;; or :my-package-user
(setf my-data
(let ((var1 )) ))We need to pull in the packages with data or functions that we need; just because the data/function is pulled in by another package, in that package’s namespace, does NOT mean it is available in this name space. However, the common-lisp-statistics package will ensure that fundamental objects and functions are always available.
One can build a package, or save an image (CL implementation dependent), or save text files.
mailto:blindglobe@gmail.com if there is anything wrong, or even if something happens to work.
Current beliefs:
- SBCL is target platform. CCL and CMUCL should be similar.
- CLISP is finicky regarding the problems that we have with CFFI conversation. In particular that we can not really do typing that we need to take care of. I think this is my (Tony’s) problem, not someone elses, and specifically, not CLISP’s
- Need to test ECL.
“Languages shape how we …” Need to get and insert this quote that Duncan Temple-Lang found.
The API should distinguish between the realization and the statistical interpretation. Goal is to teach statisticians how to think “systems-computationally”, and programmers, comp-sci types, informaticists and other “data scientists” how to think “statistically”, in order to get a jump on the competition.
The goal of this system is to promote a change in thinking, to move the data analysis approach, currently stuck in a mix of 70s-early 90s approaches, into a new generation/level.
The approach we are taking is one where we provide a general method, and some fundamental building blocks, but don’t force users into approaches in order to allow for experimentation.
DSL’s should be built on top of the core packages, as needed or wanted.
(TonyR:) The DSL I want to build is a verbose statistically precise computing language, but we need quality code underneathe (which others could use for specialized terse DSL’s).
DSL: domain specific language.
See files in file:Doc/ for history, design considerations, and random, sometimes false and misleading, musings.
Initial development, 1989 time frame, partially developed during a visit by Luke Tierney to Bell Labs.
This system.
Since this project is
git clone git://github.com/blindglobe/common-lisp-stat.git
cd common-lisp-statwill pull the whole repository, and create a “master” branch to work on. If you are making edits, which I’d like, you don’t want to use the master branch, but more to use a topic-centric branch, so you might:
git checkout -b myTopicBranchand then work on myTopicBranch, pulling back to the master branch when needed by
git checkout master
git pull . myTopicBranch(or
git rebase myTopicBranch)
BETTER DOCUMENTATION EXAMPLES EXIST ON-LINE (on the git WWW site mentioned above)!! PLEASE READ THEM, THE ABOVE IS SPARSE AND MIGHT BE OUTDATED!
Alternatively, one can work on the github repositories as well. They are a bit differently organized, and require one to get a github account and work from there.
basically, fork the repository on github on the WWW interface, then make a branch (as below), push back the branch to github, and notify the main repository that there is something to be pulled. And we’ll pull it back in.
of course, perhaps you want to contribute to the mob branch. For that, after cloning the repository as above, you would:
git checkout -b mob remotes/origin/mob(work, work, work… through a cycle of
<edit>
git add <files just edited>
git commit -m "what I just did"ad-nauseum. When ready to commit, then just:
git push git+ssh://mob@repo.or.cz/srv/git/CommonLispStat.git mob:mob)
and it’ll be put on the mob branch, as a proposal for merging.
Another approach would be to pull from the topic branch into the mob branch before uploading. Will work on a formal example soon.
(the basic principle is that instead of the edit cycle on mob, do something like:
git checkout mob
git pull . myTopicBranch
git push git+ssh://mob@repo.or.cz/srv/git/CommonLispStat.git mob:mob)
We currently are using and recommend the MIT style license approach.
[fn:1] I´m not including instructions for Emacs or git, as the former is dealt with other places and the latter was required for you to get this. Since disk space is cheap, I´m intentionally forcing git to be part of this system. Sorry if you hate it. Org-mode, org-babel, and org-babel-lisp, and hypo are useful for making this file a literate and interactively executable piece of work.