add experimental package csdp

[dimpase]

CSDP is a fast semidefinite programming solver, one of the very best around, written in C, with a callable
shared library
https://projects.coin-or.org/Csdp/
(unfortunately is has a GPL-incompatible licence, which means it cannot be made standard package)

This ticket adds CSDP: a stand-alone executable, a libtool-made shared library, and headers, to Sage. One interesting software component, using Sage and CSDP, is Flagmatic : http://flagmatic.org/

The modifications to CSDP needed for Sage support, basically, libtoolization and the spkg infrastructure,
is hosted here: https://github.com/dimpase/csdp

A Cython interface will be added on a separate ticket.

The spkg is here: http://boxen.math.washington.edu/home/dima/packages/csdp-6.2.spkg

Apply to $SAGE_LOCAL/bin/

• attachment: trac_14505.patch

CC: @ppurka @vbraun @sagetrac-mehdi-ghasemi

Component: packages: experimental

Keywords: semidefinite programming

Author: Dmitrii Pasechnik

Reviewer: Punarbasu Purkayastha, Nathann Cohen

Merged: sage-5.11.rc0

Issue created by migration from https://trac.sagemath.org/ticket/14505

[nathanncohen]

comment:2

HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much :-/

Nathann

[dimpase]

comment:3

Replying to @nathanncohen:

HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much :-/

Nathann

How the hell one is supposed to libtoolize the thing otherwise?
That's the only modification (and some header layout was changed to conform to autotools standards).
I also wanted to stay with git rather than using svn, so I imported the original svn repo.
(and the latter was not changed for, like, 6 years, you know...)

I hope that my changes will be accepted by upstream, eventually. It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

[dimpase]

comment:4

I added a bit more to SPKG.txt and updated the spkg accordingly (i.e. only SPKG.txt was changed).

[nexttime]

comment:5

Replying to @dimpase:

It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)

[nexttime]

comment:6

So here's a pull request:

diff --git a/spkg-install b/spkg-install
--- a/spkg-install
+++ b/spkg-install
@@ -17,7 +17,7 @@
 echo "Configuring csdp..."
 ./configure --prefix="$SAGE_LOCAL"
 if [ $? -ne 0 ]; then
-   echo "Error configuring csdp."
+   echo >&2 "Error configuring csdp."
    exit 1
 fi
 

B)

[dimpase]

comment:7

Replying to @nexttime:

Replying to @dimpase:

It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)

But this is kind of stupid... What actually makes sense to review, my autotools config files, are going to be buried under lots of automatically generated files.
And, strictly speaking, I might even choose not to include them into the spkg :–)

[dimpase]

comment:8

Replying to @nexttime:

So here's a pull request:

Thanks! Fixed and updated the spkg.

[nexttime]

comment:9

Well, probably consider everything in src/ your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.

[dimpase]

comment:10

Replying to @nexttime:

Well, probably consider everything in src/ your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.

well, I just followed the example of libGAP spkg.

[ppurka]

comment:11

@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.

[dimpase]

comment:12

Replying to @ppurka:

@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.

in this case you need autotools, and Sage doesn't have them all.

[ppurka]

comment:13

Replying to @dimpase:

in this case you need autotools, and Sage doesn't have them all.

I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:

$ tar xf csdp-source.tbz2
$ cp -a csdp-source{,.new}
$ cd csdp-source.new && # create your configure.ac
$ ./autoreconf etc etc
$ cd ..
$ diff -Naur csdp-source csdp-source.new > csdp.patch

And then use the patch file to patch the src/ directory.

[nexttime]

comment:14

Replying to @ppurka:

Replying to @dimpase:

in this case you need autotools, and Sage doesn't have them all.

I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:

$ tar xf csdp-source.tbz2
$ cp -a csdp-source{,.new}
$ cd csdp-source.new && # create your configure.ac
$ ./autoreconf etc etc
$ cd ..
$ diff -Naur csdp-source csdp-source.new > csdp.patch

And then use the patch file to patch the src/ directory.

... making sure that configure etc. are newer than the files they're generated from.

However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside src/ isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.

[dimpase]

comment:15

Replying to @nexttime:

However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside src/ isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.

As I already said, libGAP spkg does not follow the latter "requirement":

$ pwd
/usr/local/src/sage/sage-5.9.rc0/spkg/standard/libgap-4.5.7.p2
$ hg log
changeset:   1:c5d34c333de4
tag:         tip
user:        Jeroen Demeyer <jdemeyer@cage.ugent.be>
date:        Thu Mar 28 19:21:53 2013 +0000
summary:     Added tag libgap-4.5.7.p2 for changeset c034d0063012

changeset:   0:c034d0063012
tag:         libgap-4.5.7.p2
user:        Volker Braun <vbraun@stp.dias.ie>
date:        Thu Feb 28 13:10:53 2013 -1000
summary:     This is not the true repository, read SPKG.txt

$ 

I don't see why you'd like to disallow me doing the same here.

[vbraun]

comment:16

LibGAP will eventually also become an independent "upstream" and release tarballs, with a more conventional spkg. The reason for the current state is just that it took years of fiddling and trying different approaches to make the thing work. It was definitely not clear from the outset how it can be made to work, or whether it is appropriate to shoehorn it into a spkg at all. But over time I realize that tracking libgap as a patch on a subset of the GAP distribution with some scripts that munge source code is the right way.

[nexttime]

comment:17

$SAGE_LOCAL/bin/csdp needs to get added to .hgignore (in the same folder).

[dimpase]

Attachment: trac_14505.patch.gz

sage-scripts patch

[dimpase]

comment:19

Replying to @nexttime:

$SAGE_LOCAL/bin/csdp needs to get added to .hgignore (in the same folder).

done.

[nexttime]

Work Issues: ignore

[nexttime]

Changed work issues from ignore to none

[dimpase]

comment:25

updated spkg with proper hg history
http://sage.math.washington.edu/home/dima/packages/csdp-6.2.spkg

All the changes to SPKG.txt, spkg-install, etc (but not in src/) are properly tracked - well there is one more extra commit on top, moving files to the proper place, but I hope this is OK.

The history extraction is automatic, done by hg convert extension, see
https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

[nathanncohen]

comment:37

@nathanncohen - that's very little information. What OS and what version of the OS are you running? What options are you giving to tar? Is your tar symlinked to busybox (very unlikely), for instance?

Oh sorry. I had sent a couple of emails about that to Dima, that's why I was so vague.

I just don't get what is happening. We checked that my version of the spkg had the right md5sum (*cfcba3f), but when I run

tmp$ tar xjvf csdp-6.2.spkg 

nothing at all happens. It just waits forever. And here's the tar I use

~$ tar --version
tar (GNU tar) 1.26

But well.. O_o

Nathann

[nathanncohen]

comment:38

Okayyyyyyyyyyyyyy... Turns out there was something wrong with some automounting stuff I failed to properly install a week ago. Now it doesn't hang anymore. Mystery cleared : everything that is automatic and/or is remotely related to windows fs should never be installed >_<

Nathann

[ppurka]

comment:39

Is your tar aliased to something? Can you run

\tar xjvf csdp-6.2.spkg

yes, with the \.

[ppurka]

comment:40

Oh ok. It is all fixed for you, then it's good. Yeah, sometimes it is better to avoid mixing windows stuff with linux stuff. Things break down in mysterious ways.

[dimpase]

comment:41

Replying to @nathanncohen:

Well. If you say that this thing has not been updated for years, I guess that modifying its sources to make it run is not that bad ... This being said I haven't been able to find a mention of CSDP 6.2 on internet, only 6.1.1 seems to be available on the original website.

svn repo has 6.2 (beta); here is this commit imported: dimpase/csdp@7708179

Could you also explain somewhere how this spkg can be built from scratch ? I remember having seen somewhere in a spkg a "create spkg" file with an internal variable equal to the url of the .tar.gz file, and running that ended up in creating the SPKG itself.

this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

[nathanncohen]

comment:42

Yo !

svn repo has 6.2 (beta); here is this commit imported: dimpase/csdp@7708179

Ahahah. Cool :-D

this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

Yep, it would be nice ! I just downloaded your git repository and ran this make-spkg.sh script but it ends up like that :

/tmp/csdp/spkg$ ./make-spkg.sh 
make: *** No rule to make target `dist'.  Stop.

Annnnd I have no idea if what I do is what I should to build the spkg :-)

Nathann

[dimpase]

comment:43

Replying to @nathanncohen:

Yo !

svn repo has 6.2 (beta); here is this commit imported: dimpase/csdp@7708179

Ahahah. Cool :-D

this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

Yep, it would be nice ! I just downloaded your git repository and ran this make-spkg.sh script but it ends up like that :

/tmp/csdp/spkg$ ./make-spkg.sh 
make: *** No rule to make target `dist'.  Stop.

Annnnd I have no idea if what I do is what I should to build the spkg :-)

right, you need to populate the tree with makefiles first, by ./configure
at the top level.

Nathann

[nathanncohen]

comment:44

O_o

There is no configure file in at the top level O_o

/tmp/csdp/spkg$ ls
convert_mapfile  install.sh  LICENSE  make-spkg.sh  README.md  spkg-install  SPKG.txt
/tmp/csdp/spkg$ cd ..
/tmp/csdp$ ./configure
bash: ./configure: No such file or directory

Nathann

[dimpase]

comment:45

Replying to @nathanncohen:

O_o

There is no configure file in at the top level O_o

Oops, sorry.

Welcome to the autoHell :-)

I've added the detailed instructions on bilding the spkg to the SPKG.txt and updated the spkg file. Hopefully it's clear now.

[ppurka]

comment:46

There is still something missing. I had to add the line

convert =

in my [extensions] section of ~/.hgrc. Even so, it fails with the following error on running make-spkg.

8 added support to make Sage spkg, and the corresponding files
7 explained NOSHORTS thing in SPKG.txt
6 Update README.md
5 added missing >&2
4 further rectifications of NOSHORTS and SPKG.txt
3 an attempt to use hg convert to fake the history in spkg/
2 tweaking "hg convert" thing
1 yet more tweaks to make_spkg
0 added instructions to build spkg
updating bookmarks
mv: cannot stat ‘csdp-hg/.hg’: No such file or directory

[dimpase]

comment:47

Replying to @ppurka:

There is still something missing. I had to add the line

convert =

in my [extensions] section of ~/.hgrc.

OK, I've updated SPKG.txt mentioning this, as well as git and autotools/libtool requirements for spkg creating.

Even so, it fails with the following error on running make-spkg.

weird. Do have have an old/super-new hg?

Does it create csdp-hg/ in some other place for you?

(I presume you do ./make-spkg.sh)

[ppurka]

comment:48

There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running ./make-spkg.sh

/tmp/csdpnew» sage -sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/mnt/usb/Installations/sage-5.10.rc0
/tmp/csdpnew» hg --version
Mercurial Distributed SCM (version 2.2.2)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2012 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/tmp/csdpnew» which hg
/mnt/usb/Installations/sage-5.10.rc0/local/bin/hg
/tmp/csdpnew»
Exited Sage subshell.
/tmp/csdpnew» hg --version
Mercurial Distributed SCM (version 2.4.2)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2012 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/tmp/csdpnew» which hg
/usr/bin/hg
/tmp/csdpnew» ls
aclocal.m4	config.guess   config.sub	depcomp  install-sh  m4		    matlab     sage	 theta
AUTHORS		config.h       configure	doc	 lib	     Makefile	    missing    solver
autom4te.cache	config.h.in    configure.ac	example  libtool     Makefile.am    NEWS       spkg
ChangeLog	config.log     COPYING		include  LICENSE     Makefile.in    README     stamp-h1
compile		config.status  csdp-6.2.tar.gz	INSTALL  ltmain.sh   Makefile.orig  README.md  test
/tmp/csdpnew» cd spkg
/tmp/csdpnew/spkg» ls
convert_mapfile  csdp-6.2  install.sh  LICENSE	make-spkg.sh  README.md  spkg-install  SPKG.txt
/tmp/csdpnew/spkg» cd -
/tmp/csdpnew
/tmp/csdpnew» find . -name 'csdp-hg'
/tmp/csdpnew»

[dimpase]

comment:49

Replying to @ppurka:

There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running ./make-spkg.sh

/tmp/csdpnew» sage -sh

things in /tmp might be a bit funny...

Also, do you have anything unusual in ~/.hgrc ?
I imagine hg-git might interfere...

[ppurka]

comment:50

No. Nothing special in that file:

[ui]
username = Punarbasu Purkayastha <ppurka snip...>

[extensions]
rebase =
mq =
fetch =
pager =
graphlog =
color =
record =
hgext.extdiff =
convert =

[diff]
git = true

[extdiff]
cmd.merge_hg =

[merge-tools]
merge_hg.args = "$local" "$other" "$base"

[pager]
pager = less
attend = annotate, cat, diff, log, glog, qdiff, export

[alias]
mylog = glog -l15 --template '\033[34;40m{rev} \033[00;40m- \033[01;33;40m{tags} \033[0;37;40m{desc|firstline|fill68} \033[0;32;40m{date|age}\033[0;37;40m <{author|person}> \033[37;40m \033[35;40m{branches}\033[37;40m\n'
l = mylog

[dimpase]

comment:51

Replying to @ppurka:

did you try building outside /tmp ?
I am unable to reproduce your issue.

[ppurka]

comment:52

Replying to @dimpase:

did you try building outside /tmp ?
I am unable to reproduce your issue.

That is indeed very weird. It works outside /tmp. Must be some quirk of hg.

[ppurka]

comment:53

How do I test the installation? I think there should be something to test the installation of this package, without having to go through pycsdp or flagmatic.

I tried the example.c file in example directory, but the compilation seems to work only if I actually compiled csdp and compiled example.c from within the sources directory.

[nathanncohen]

comment:54

Well, if the binary is correctly build and runs, I would say that the spkg does its job :-)

You can set this ticket to positive review from my behalf too once that you will be satisfied with what it does :-)

Nathann

[ppurka]

Changed reviewer from Punarbasu Purkayastha to Punarbasu Purkayastha, Nathann Cohen

[ppurka]

comment:55

The binary does build and run. And I have tested the compiled binay with flagmatic earlier. Since you are ok with it, let's give it a +1. ;-)

[haraldschilly]

comment:56

spkg is on the server.

[jdemeyer]

Merged: sage-5.11.rc0

[kcrisman]

comment:58

Possible followup: #17391.

[williamstein]

comment:59

Any interest in moving this to optional? I have user requests for this on SageMathCloud. E.g.,

Dear SageMath,

I encountered a problem at 

The package csdp (used by lovasz_theta() function) is not installed. The 
current error message is somewhat misleading, as it tells me to run 'sage 
-i csdp'. Doing so fails due to insufficient permissions.

NOTE: The latter thing about "insufficient permissions" is yet another way that we need to distinguish between system-wide and private sage installs.

[dimpase]

comment:60

I am all for it.