Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

D3.2: Understand and document SageMathCloud backend code #61

Closed
minrk opened this issue Sep 8, 2015 · 56 comments
Closed

D3.2: Understand and document SageMathCloud backend code #61

minrk opened this issue Sep 8, 2015 · 56 comments

Comments

@minrk
Copy link
Contributor

@minrk minrk commented Sep 8, 2015

  • WP3: Component Architecture
  • Lead Institution: Université Paris-Sud
  • Due: 2016-02-28 (month 18; originally month 6)
  • Nature: Report
  • Task: T3.6 (#55): Document and modularise SageMathCloud’s codebase
  • Proposal: p. 43
  • Final report

SageMathCloud (SMC) is both an open source software project (https://github.com/sagemathinc/smc) and an online instance of that software (hosted at https://cloud.sagemath.com/) that provides an interactive, collaborative environment for teaching and research in science, technology, engineering, and mathematics. See e.g. D2.3 (#43) for a description of this emergent technology.

SMC predates OpenDreamKit (ODK) and acts as prototypical example of VRE that can be built from the ecosystem OpenDreamKit aims at fostering. In particular, SMC is one of the main channels through which some of the most important technologies of ODK, such as Jupyter and SageMath, are distributed on-line. It makes it a good mean to distribute some of the newly developed ODK features, like the new Jupyter kernels of D4.4 (#93). It is very probable that many users will benefit from some of the ODK new developments through SMC. Reciprocally, the inner technologies of SMC are of special interest to ODK developers: they show advanced uses of cutting-edge web technologies and explore new leads that could inspire the work we do in ODK.

For all these reasons, is has been planned since the beginning for ODK to collaborate actively with SMC. For example, in D2.4 (#44), we have developed a short course for educators who wish to adopt SMC and related ODK technology in order to enhance their teaching.

In this deliverable, we start exploring the main layers of SMC's backend code and give a general overview of its functioning. The material we have produced can directly help the platform attract more developers. One of the expected follow-up is an easy install for a local version of SMC especially designed for development which could be part of upcoming D3.5 (#63). The long term goal however is to understand the extent of a full install of a SMC instance on a server or cluster: How hard is it? What is the total cost of ownership? Is it a viable solution for institutions of various scales to deploy and run a local SMC instance?

@minrk minrk added this to the D3.2 milestone Sep 8, 2015
@nthiery nthiery mentioned this issue Sep 11, 2015
3 of 3 tasks complete
@nthiery
Copy link
Contributor

@nthiery nthiery commented Mar 22, 2016

Work on this started at Sage Days 70.
Jeroen, Viviane, David, please report here!

Due to delays in the recruitement in Orsay, and given that this task has no direct impact on other tasks, it was decided with the Project Officer to postpone the milestone, up to Month 18. Still we should do further progress before June, and report during the informal review in June. I am thus updating the milestone accordingly.

@nthiery nthiery modified the milestones: First informal review, D3.2 Mar 22, 2016
@kohlhase
Copy link
Member

@kohlhase kohlhase commented Jun 14, 2016

so I guess that you should change milestone as well.

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Jun 24, 2016

Some progress report:

We have started working on this with @jdemeyer going through the notes of https://wiki.sagemath.org/days70 with the goals:

  • Documenting installation of development version of SMC in SMC or on personal laptops, following William Stein's talk
  • Started to look at SMC code base and intending to document its basic structure
@bpilorget
Copy link
Contributor

@bpilorget bpilorget commented Nov 21, 2016

@defeo (WP leader) and @VivianePons (lead beneficiary)
This deliverable is due for February 2017

@nthiery
Copy link
Contributor

@nthiery nthiery commented Jan 6, 2017

Dear SageMathCloud fans,

Let's face it: there has been exciting long term developments elsewhere (e.g. cygwin port, packaging, ...) and we (ODK partners) have not been devoting as much manpower as planned (10PM total) on the SageMathCloud deliverables (#61, #63, #64). After a head start at Sage Days things been lagging behind. #61 has already renegotiated to be delivered on Month 18 which is in less than two months from now.

It's still time to do something about it and we should! Technically speaking because we have signed a contract and have to deliver as planned there. More importantly because SMC is an important piece in our ecosystem and we want it to flourish. And also because we promised William to give a hand.

The description of the deliverables is rather open ended. That's by design: as expected much has happened since we wrote the proposal two years ago. Now we still have the flexibility to choose what's the most useful course of action. For example, #63 has basically been already implemented by the SMC devs (thanks guys!); so we will get to choose how far further we will want to push it.

But let's focus on #61 for now. We have started to brainstorm yesterday with Luca, Viviane and Erik about what could be a useful and achievable goal for this deliverable. At this stage we are thinking about the following:

  • Expand the existing tutorial on installing SMC on a personal machine and/or SMC itself; plan: follow the tutorial on a bunch of different machines and fill in holes as needed.

  • Refine the existing SMC architecture description

  • Report on the experience gained during the installation of new instances of SMC (on personal machines, in Zürich, ...), trying to assess from those what use cases we can hope SMC to fill in by the end of ODK with respect to our overall goal of "easy to setup Virtual Research Environments".

A tricky part is that some of the above will require discussions with the SMC devs (William, Harald, ...) and we should not use up more of their time than we contribute back!

Plan: we are considering running a 2-3 days focused meeting in the week 6-10 of February. Depending on availability, this would be either online or a physical workshop in Orsay.

Please:

  • Comment on what would actually be most useful for SMC, for #61 and for the longer run
  • (especially for ODK participants) specify whether you would be interested in devoting some man power on #61 or its followups
  • If you are interested in joining the workshop, online or on site, fill in the date poll

cc: @haraldschilly, @williamstein, @mikecroucher, @pdehaye, @VivianePons, @embray, @defeo, @jdemeyer, @dimpase, @minrk

@williamstein
Copy link
Contributor

@williamstein williamstein commented Jan 6, 2017

@nthiery
Copy link
Contributor

@nthiery nthiery commented Jan 6, 2017

@embray
Copy link
Collaborator

@embray embray commented Jan 9, 2017

Since I'm planning on helping with this task I can definitely help bang away at the docker issues.

ISTM that for personal installs Docker is the fastest, easiest way to get up and running. But it would also be good to expand on the documentation for getting it working without Docker too, as well as expanding on the documentation of the overall system architecture, how Kubernetes is used, etc. (though the latter sounds like it's in flux?)

It's too bad about RethinkDB--I thought it had a really elegant design. Is Postgres proving to be reasonable to deal with? I've used it before, but only superficially. I'm not crazy about its command-line interfaces.

@williamstein
Copy link
Contributor

@williamstein williamstein commented Jan 9, 2017

Since I'm planning on helping with this task I can definitely help bang away at the docker issues.

Very good -- thanks. If you need access to any build infrastructure, let me know. I have a very powerful dedicated machine at the university that can be used for docker build, and I can make an account for you.

how Kubernetes is used, etc. (though the latter sounds like it's in flux?)

Very much so -- it will definitely significantly change. It may not remain entirely open source, since it's exactly the sort of thing that is not needed for personal or small installs, but is needed for running the main site or for major enterprise level private installs.

It's too bad about RethinkDB--I thought it had a really elegant design.

It is too bad. It does have an elegant design.

Is Postgres proving to be reasonable to deal with?

Yes, it's very good and mature, though I had to design and implement changefeeds from scratch, built on their LISTEN/NOTIFY functionality.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Jan 13, 2017

@haraldschilly, @mikecroucher, @pdehaye, @VivianePons, @defeo, @dimpase, @minrk, @fcayre

In case you'd be interested in participating to the tentative (online?) sprint on SMC in February, please fill in urgently the date poll. So far, only Erik and Jeroen signed in (thanks!).

@nthiery nthiery changed the title D3.2: Understand and document SageMathCloud backend code. D3.2: Understand and document SageMathCloud backend code Jan 19, 2017
@embray
Copy link
Collaborator

@embray embray commented Jan 19, 2017

For what it's worth, I've spent the last couple days diving into understanding how SMC works, and am working on writing an introductory document to its architecture (as I understand it) and more importantly all the different technologies it depends on. It's quite daunting, especially for anyone who hasn't worked with many modern web development technologies, or the Node.js ecosystem in particular. So I think my guide will make it much easier for anyone else who wants to look to understand what it is they're looking at. So if you haven't already taken a shot at it, hang tight for my writeup as it will likely help.

I've also tried not to look too much at any existing documentation because:
a) A lot of it is out of date (and so will my docs eventually, but a lot of what I'm writing is very general as well) and
b) So as to not be unduly influenced or lulled into a sense of thinking I understand things without actually figuring them out myself.

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Jan 19, 2017

@dimpase
Copy link
Contributor

@dimpase dimpase commented Jan 19, 2017

here is a good link on modern web technologies.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Jan 19, 2017

Hi Erik,
Thanks for your work on this! I disscussed with @fcayre: he is interested in joining the effort and should be able to participate to a local sprint. He will bring plenty of expertise on web technologies.

@williamstein
Copy link
Contributor

@williamstein williamstein commented Jan 19, 2017

@embray
Copy link
Collaborator

@embray embray commented Jan 19, 2017

@dimpase I feel much the same way--although many of the technologies used in SMC are not entirely new to me (I've played around with ReactJS before, for example) the last time I built a large web application myself, "ajax" was still a buzzword, and for many people even that has passed them by. William is right too though--there's no way he could have almost single-handedly built SMC without all this stuff. Today you can build the basics of a web-based chat server (client side and server side) in like 10 minutes. Used to be that was a real challenge.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Jan 31, 2017

Brief update: @embray has been working hard digging in, and has a preliminary architecture writeup. On Friday, he met with @fcayre and @defeo to discuss this further, and they will probably meet again soon.
Altogether, this seems on a good track.
@jdemeyer: given the progress, it's probably not worth the trouble to make you come over as we had previouly discussed.

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Feb 25, 2017

Ok, I proof-read the whole document, edited and updated the github description and edited a bit the introduction. I also added Erik's "overview.rst" documentation as an appendix. For me, it's all good, I let @nthiery @defeo and @embray check if they're happy with it.

Thank you again Erik for such an impressive work. Your overview document is really great, it gave me a much clearer understanding on how this whole thing works!

@williamstein
Copy link
Contributor

@williamstein williamstein commented Feb 26, 2017

@VivianePons -- where? Can you make a PR to SMC?

@dimpase
Copy link
Contributor

@dimpase dimpase commented Feb 26, 2017

@williamstein
Copy link
Contributor

@williamstein williamstein commented Feb 26, 2017

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Feb 26, 2017

@embray
Copy link
Collaborator

@embray embray commented Feb 27, 2017

Like @VivianePons said, I think the copy in this repo can just be considered like a tagged snapshot. I think the report itself should still also link to the copy in SMC's repo. I think I mentioned in the report that it was added to the upstream docs, but I'll double-check that.

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Feb 27, 2017

@embray
Copy link
Collaborator

@embray embray commented Feb 27, 2017

@VivianePons I noticed you added an \includepdf{overview/overview.pdf}. Would you like me to create a PDF version of that document?

(In fact, I have already done it, but I wanted to make sure you weren't working on it first.)

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Feb 27, 2017

@embray
Copy link
Collaborator

@embray embray commented Feb 27, 2017

Oh, turns out I accidentally added the overview.pdf in an unrelated commit. How do we feel about rebasing?

@dimpase
Copy link
Contributor

@dimpase dimpase commented Feb 27, 2017

all your base are belong to us!

@defeo
Copy link
Contributor

@defeo defeo commented Feb 27, 2017

How do we feel about rebasing?

Bad. Your commit is already buried pretty deep, it would be very painful to make sure that anyone with push rights checks out the new head, without eventually losing local modifications, and discards the old history.

I would only rebase for security critical reasons (e.g., someone commits a password in the history).

@defeo
Copy link
Contributor

@defeo defeo commented Feb 27, 2017

Just finished proofreading. I made a pull request to SMC with some typo fixes.

Thanks @embray for this thorough overview.

@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

Bad. Your commit is already buried pretty deep, it would be very painful to make sure that anyone with push rights checks out the new head, without eventually losing local modifications, and discards the old history.

That was my feeling as well. Maybe in the future it will be better to work in a branch until the final is complete.

@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

Thanks @defeo and @VivianePons for helping finish this up. I think this is ready to go, unless @nthiery has any last comments?

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Feb 28, 2017

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

Hi @embray, @VivianePons,
I have been through the report; it's really nice.
I made some tiny edits and, for more consistency, switched to a systematic use of SMC / ODK (rather than SageMathCloud / OpenDreamKit. I also reworked the github issue description for consistency with D2.4. You may want to proofread.
Just one point: the font in the appendix is kind of small. In case it would be quick, could overview.pdf be regenerated using a larger font?
Other than that, it's good to go!

@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

Yes, I could increase the font size.

@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

Agreed on the consistency with D2.4 (which I didn't even really know about until this morning when I read the final report, and realized it had a tiny bit of overlap with this one).

@VivianePons
Copy link
Contributor

@VivianePons VivianePons commented Feb 28, 2017

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

Ok, thanks! Ping me when overview.pdf with larger fonts is committed, and I'll submit!

@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

@nthiery I'm not sure I agree with the mass replacement of SMC -> \SMC. The latter expands to the full name "SageMathCloud", and so I deliberately didn't use it in most places, in favor of the acronym, for the sake of flow. Is it clearer without using the acronym? I'm not so sure...

@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

Oh nevermind, I see you redefined the \SMC command. That makes sense then.

@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

@nthiery Okay, I've made the font 25% larger. Let me know how that looks.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

Looks great, thanks! About to submit.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Feb 28, 2017

Submitted!
@embray: that was an impressive sprint; you really saved that deliverable :-)
@VivianePons, @defeo, @williamstein, @fcayre, ... thanks for the help!

@nthiery nthiery closed this Feb 28, 2017
@embray
Copy link
Collaborator

@embray embray commented Feb 28, 2017

Thanks for submitting! Relieved to have this "done".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.