Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

D3.3: Support for the SCSCP interface protocol in the distribution of all relevant components ( Sage , GAP , etc.) #62

Closed
minrk opened this issue Sep 8, 2015 · 54 comments

Comments

@minrk
Copy link
Contributor

@minrk minrk commented Sep 8, 2015

#- WP3: Component Architecture

  • Lead Institution: University of St Andrews
  • Due: 2016-08-31 (month 12)
  • Nature: Other
  • Task: T3.2 (#51)
  • Proposal: p.43
  • Final report: due 2017-02-28 [PDF]

SCSCP stands for the Symbolic Computation Software Composability Protocol - the remote procedure call framework by which different software components (primarily mathematical software systems) may offer computational services to a variety of possible clients using the OpenMath encoding both for the data and protocol instructions (see the SCSCP specification for further details).

SCSCP has been developed in the EU FP6 project 026133 SCIEnce - Symbolic Computation Infrastructure for Europe. In the duration of the project (2006-2011) and subsequent years, several native CAS implementations of SCSCP client and server, and also APIs for Java, C and C++ had appeared (see the complete list here). However, there were no Python OpenMath SCSCP implementations (except a prototype quality client supporting only lists of integers) and that hindered further extension of the SCSCP framework.

In this deliverable, we have extended support for SCSCP to other relevant systems involved in the project. This builds foundation to D3.9 "Semantic-aware Sage interface to GAP" (#68) and other activities outlined in our paper "Interoperability in the OpenDreamKit Project: The Math-in-the-Middle Approach" (Intelligent Computer Mathematics. CICM 2016. Lecture Notes in Computer Science, vol 9791. Springer). More specifically, we have achieved the ability to communicate using SCSCP protocol to the following systems/languages:


Remarks:

@minrk minrk added this to the D3.3 milestone Sep 8, 2015
@nthiery nthiery changed the title D3.3: Support for the http://www.symbolic-computing.org/SCSCP interface protocol in all relevant components ( Sage , GAP , etc.) distribution D3.3: Support for the SCSCP interface protocol in the distribution of all relevant components ( Sage , GAP , etc.) Jan 29, 2016
@nthiery nthiery modified the milestones: Month 12: 2016-08-31, D3.3 Mar 22, 2016
@alex-konovalov alex-konovalov self-assigned this Jun 29, 2016
@defeo
Copy link
Contributor

@defeo defeo commented Jun 30, 2016

Some ideas for nice useful demonstrators that could come out of this work:

  • Maple has an SCSCP interface (probably not activated by default on user machines, though). We could use SCSCP to improve the SageMath <-> Maple interface.
  • Same goes for Mathematica.
  • Databases that have an SCSCP interface could be queried live from the Sage command line. This applies to a couple of DBs maintained by @alex-konovalov, and to those that will get SCSCP support through MMT (think LMFDB, FindStat).

Note: distributed computing sounds like an obvious application, however, it is hard to imagine a real world application where it wouldn't be more convenient to run Sage on all nodes and exchange data via Python distributed computing facilities.

@defeo
Copy link
Contributor

@defeo defeo commented Jul 18, 2016

@alex-konovalov, what transport is relevant for SCSCP? Shall we implement the XML Processing Instructions over TCP protocol described in the SCSCP specs, Sec. 5, or shall we wrap it in SOAP? Which protocol do SCSCP clients in the wild use?

@tkw1536, zeep looks like a good library for SOAP. For raw TCP, I don't know what's the best option to support both Py2 and 3.

@tkw1536
Copy link
Contributor

@tkw1536 tkw1536 commented Jul 18, 2016

@defeo To my knowledge all SCSCP clients do it via raw sockets, so I am not sure if we should use something on top of that. Unfortunately I do not know any library for sockets that does both python 2 and 3 either.

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Jul 19, 2016

@defeo:

@alex-konovalov, what transport is relevant for SCSCP? Shall we implement the XML Processing Instructions over TCP protocol described in the SCSCP specs, Sec. 5,

yes, I suggest that you do exactly this so you do not need SOAP at all. This is the way how all existing implementations work by now.

@defeo
Copy link
Contributor

@defeo defeo commented Jul 19, 2016

Ok, thanks to both. I'll think a bit about more the best way to implement in python and come back to you, @tkw1536

@defeo
Copy link
Contributor

@defeo defeo commented Jul 22, 2016

Just talked to J. Gerhard of Maplesoft. Maple has an SCSCP client in the client version. The SCSCP server is only available in MapleNet. Bad luck for the Sage <-> Maple interface :/

@markuspf
Copy link
Member

@markuspf markuspf commented Jul 26, 2016

@defeo @tkw1536 what is the status of the SCSCP interfaces you are assigned to above?

Also, what about singular? Can @sebasguts maybe comment?

@tkw1536
Copy link
Contributor

@tkw1536 tkw1536 commented Jul 26, 2016

@markuspf Inside MMT / MathHub we have a working client implementation but it is not really integrated yet.

@defeo
Copy link
Contributor

@defeo defeo commented Jul 26, 2016

The Sage implementation is on hold as I am visiting UWaterloo ATM. Will resume the work in one week.

@markuspf
Copy link
Member

@markuspf markuspf commented Jul 26, 2016

Please note that this is due on 2016-08-31, so 5 weeks from now.

@defeo
Copy link
Contributor

@defeo defeo commented Jul 26, 2016

Yup, I know. No worries, 5 weeks is enough.

@bpilorget
Copy link
Contributor

@bpilorget bpilorget commented Aug 30, 2016

@defeo Is the report almost over?

@bpilorget
Copy link
Contributor

@bpilorget bpilorget commented Aug 30, 2016

The README gives info on how to write deliverables: https://github.com/OpenDreamKit/OpenDreamKit

@defeo
Copy link
Contributor

@defeo defeo commented Aug 30, 2016

Next week.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Sep 6, 2016

Just in case that could be relevant: I stumbled on mathdom, a Content MathML DOM parser for Python. Incidentally mathdom is developped by @scoder, core dev of Cython and of LXML, the latter being a fast XML Dom parser written in Cython.
For the record, Content MathML is one of the official idioms for OpenMath, and is likely to be endorsed by the OpenMath society as the recommended OpenMath idiom.
cc: @kohlhase

@defeo
Copy link
Contributor

@defeo defeo commented Oct 17, 2016

Hello, work on this on the Sage side is essentially done: https://github.com/OpenMath/py-scscp. Of course, it may be helpful to add documentation and comment the code 😇

How's work on the other components? @tkw1536 said:

Inside MMT / MathHub we have a working client implementation but it is not really integrated yet.

anything new? Singular?

I can start writing the Sage part of the report, but maybe @alex-konovalov wants to setup the scaffold before that?

@kohlhase
Copy link
Member

@kohlhase kohlhase commented Oct 18, 2016

I am thrilled to hear about progress on the SageMath side.
For me the next step would be to test the communication between the (three) systems that is possible now. I guess until the OpenMath Interface of Sage is further, it would be a good first step to send around integers between SageMath, MMT, and GAP as a hello world.
After that we should really start working on getting the Use Case we talked about in Bremen running.

@defeo
Copy link
Contributor

@defeo defeo commented Oct 18, 2016

I agree. Unfortunately I was not able to obtain a copy of gap with a working scscp package (did not try very hard, but the linux arch package is currently broken). Any help appreciated.

@jgmbenoit
Copy link

@jgmbenoit jgmbenoit commented Oct 18, 2016

have you tried to play with the gap-scscp debian package on a (testing)) box ?

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Oct 18, 2016

@defeo You need GAP and also IO and OpenMath packages. Install GAP from the source distribution, and build packages, including the IO - then all should work.

@tkw1536
Copy link
Contributor

@tkw1536 tkw1536 commented Jan 12, 2017

@alex-konovalov Did you already start a report on this? Is there a plan on how to split it up between the different systems? @kohlhase has asked me to write the report from the KWARC side.

@defeo What is the status of the SCSCP implementation in Sage / Python?

@defeo
Copy link
Contributor

@defeo defeo commented Jan 13, 2017

@tkw1536 pip install scscp :)
(or the equivalent command in Sage)

@kohlhase
Copy link
Member

@kohlhase kohlhase commented Jan 15, 2017

Wow, that sounds good.
@tkw1536 I am looking forward to a report on how this works in an example.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 6, 2017

Dear M12 deliverable leaders,

Just a reminder that reports are due for mid-february, to buy us some time for proofreading, feedback, and final submission before February 28th. See our README for details on the process.

In practice, I'll be offline February 12-19, and the week right after will be pretty busy. Therefore, it would be helpful if a first draft could be available sometime this week, so that I can have a head start reviewing it.

Thanks in advance!

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 6, 2017

FYI, new releases of OpenMath and SCSCP packages for GAP, now hosted on GitHub and GitHub pages:

@kohlhase
Copy link
Member

@kohlhase kohlhase commented Feb 7, 2017

very good to know. What has changed?
I see that the acknowledgements in the OpenMath manual do not include ODK (have not checked SCSCP) and that the issues URL in SCSCP manual has no link.

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 17, 2017

@kohlhase

very good to know. What has changed?

Both packages were not updated for a while (SCSCP since 2013, OpenMath since 2016) and apart from cosmetic updates of all URLs for the new hosting, there were various compatibility fixes to work with the new GAP release and GAP development version, cleaning up of obsolete variables, improving testing setup, etc. Also, <OMOBJ> tags now include xmlns and version attributes.

I see that the acknowledgements in the OpenMath manual do not include ODK (have not checked SCSCP)

Fixed for both packages, in manuals and their websites

and that the issues URL in SCSCP manual has no link.

Where? Please send me a link to the exact page.

@tkw1536
Copy link
Contributor

@tkw1536 tkw1536 commented Feb 18, 2017

@alex-konovalov @defeo
For testing, I have just deployed an MMT SCSCP server instance on ash.eecs.jacobs-university.de on port 26133, (212.201.49.2, I haven't tested IPV6). It is minimal in the sense that it can currently only do scscp_transient_1 / addition and no other computations. I did some testing with the MMT client and that worked without problems. Concerning GAP and Python / Sage, I did test this a few weeks ago and it worked back then, but it would be good if you could verify this.

/cc @Jazzpirate @kohlhase

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 24, 2017

@nthiery I've edited issue description (one item from @tkw1536 still needs at least a couple of words).

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 24, 2017

Thanks; that's useful info!

Can you add an abstract of a couple paragraphs with some words of context (how this fits within ODK's aim, a brief description of what SCSCP is, what was achieved)? In case that feels too redundant with the introduction in report.tex, then maybe that introduction should become the abstract by being moved here?

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 24, 2017

@nthiery Updated issue description again - please check #62

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 24, 2017

P.S. @nthiery is there a tool to convert markdown to latex, or I have to do it manually?

@defeo
Copy link
Contributor

@defeo defeo commented Feb 24, 2017

@alex-konovalov, if you're trying to include the issue description, just use the command \githubissuedescription. Otherwise, I recommend pandoc.

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 24, 2017

@defeo \githubissuedescription is there, but it does not convert updated github-issue-description.md to github-issue-description.tex

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 24, 2017

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 24, 2017

@defeo indeed pandoc -o github-issue-description.tex github-issue-description.md works. Need manual tweaking to enter new lines when the final version will be out.

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 27, 2017

@nthiery: we will be waiting for an example from @tkw1536 on Tuesday morning.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

@tkw1536: Tuesday morning is over :-)

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 28, 2017

Checklist:

  • finish conclusions
  • add authors
  • update GitHub issue in the pdf
  • hopefully one more example from @tkw1536
  • @nthiery do you know how to put a space before DELIVERABLE in the 1st line of the title page ?
@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

  • @nthiery do you know how to put a space before DELIVERABLE in the 1st line of the title page ?
    Fixed in fdc63d4 and pushed.
@tkw1536
Copy link
Contributor

@tkw1536 tkw1536 commented Feb 28, 2017

Sorry for not doing anythign this mornign, I was working on D4.3. I am working on the example right now.

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 28, 2017

@tkw1536 thanks. @nthiery as far as @stevelinton and me are concerned, after having a look at @tkw1536 new example this will be ready to go. The issue description above has been updated last time this morning, if you wish, you can convert it to latex in the meantime.

@tkw1536
Copy link
Contributor

@tkw1536 tkw1536 commented Feb 28, 2017

@alex-konovalov I have just pushed the second example. I also re-did the first example and it now has proper syntax highlighting.

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 28, 2017

@tkw1536 thanks, that looks good. @nthiery this is now ready to go, only needs your magic to fetch the latest issue description from GitHub.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

@alex-konovalov: could you fix the "Distrubuted" typo in READMEs/Parallel-GAP-SCSCP.pdf?
Then I'll submit.

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Feb 28, 2017

@nthiery Fixed! Also updated issue description in both places to avoid overfull line with the long URL. Please submit!

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

Submitted!
Thanks @defeo for your hard work on implementing SCSCP for Python, @alex-konovalov for leading this very thorough report and spearheading SCSCP, and everyone who helped.
Now let's see how this supports the wider adoption of SCSCP. Keep it up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.