# D3.5: Integration between SageMathCloud and Sage 's TRAC server #64

Closed
opened this Issue Sep 8, 2015 · 44 comments

Projects
None yet
10 participants
Contributor

### minrk commented Sep 8, 2015 • edited by nthiery

 WP3: Component Architecture Lead Institution: Université Paris-Sud Due: 2017-08-31 (month 24) Nature: Other Task: T3.7 (#56) Final report Task T3.7 (#56) aims to improve the development workflow in mathematical software, to ease contributions by all actors, from experts developers on those projects or coming from other projects, to students, novice users, or non-programmers. For this deliverable we focused on engaging newcomers to become contributors by reducing the barrier of entry to Sage development. In particular, trivial contributions ought to be trivial: for example, a user or technical writer that notices a typo or a potential phrasing improvement in the documentation should be encouraged and empowered to fix it on the spot and submit it for review and integration into Sage. The Sage community has a history of blurring the line between developer and user, with even new users often encouraged to make code contributions. However, the core group of frequent development contributors remains relatively small, and has a well-established workflow and culture surrounding its existing development tools--in particular the source code hosting and issue tracking service Trac. This works well for experienced Sage developers, but also creates certain barriers to entry for new contributors, especially those who are new to open source development. To tackle those barriers, we aimed for a better integration between Sage's Trac-based workflow with platforms that are more modern and familiar to open source novices, such as GitHub and GitLab, without immediately breaking the Sage project's existing workflow [1]. Beyond tackling the immediate goal of this deliverable, we believe that this approach also opens the doors to long-term improvements in Sage's development workflow, as the advantages of using these new tools become more apparent. Part of this work also serves as a prerequisite to improvements in Sage's continuous integration practices, which are being explored in D3.8 (#67). An aspect that Sage (and indeed any open source mathematics software) prides itself on over closed-source alternatives is the ability to peer inside the software to see how it works. It is not a black box and one does not have to take its results for granted--it is possible to look at the source code in order to understand how a particular algorithm works, or even modify and improve it. To this end we sought ways to more tightly couple Sage's API documentation to its source code and, through access to the source code, encourage contribution of improvements. To these ends, this deliverable includes the following improvements to Sage's development platform and documentation: Enable users with existing GitHub credentials to log into Sage's Trac site. Create a presence for Sage on the GitLab collaborative source code sharing service, including the ability to accept patches to Sage's documentation and source code in the form of merge requests (a.k.a. pull requests). Link GitLab merge requests to tickets in Sage's Trac issue tracker in a manner that fully integrates with Sage's existing development workflow. Create a direct path from Sage's online API documentation to the source code, and from the source code directly an interface to edit the source code and submit a patch (as a merge request) all without the user having to leave their web browser. [1]: Upon writing the proposal we envisioned using SageMathCloud (now CoCalc) as a major development platform for Sage, hence the title of this deliverable. However, while CoCalc remains quite viable for this purpose, it became evident that there was a need for solutions not explicitly tied to CoCalc. See the report for a discussion.

Open

Closed

Open

Collaborator

Contributor

### nthiery commented Jun 29, 2017

 Hi Erik! Thanks for the thoughts. Just as an additional data point: this deliverable indeed takes its origins from discussions about how to simplify contributions. And in particular about how to make it trivial to do trivial contributions, in order to enpower end users and crowdsource the polishing of the zillion tiny rough edges of Sage. Typical scenario: Alice is working in the VRE, reads some (Sage/...) documentation, spots a typo, has a suggestion to fix it. We would want a very simple workflow to submit and review the fix. Of the kind "Edit on github".
Collaborator

### embray commented Jun 30, 2017

 I mean, for Sage specifically, I don't really see that as a "VRE" issue at all, or at least not as much. All we really want is an "Edit on GitHub" link for Sage's HTML docs. Hypothetically something similar in the interactive Python docs as well, though I'm not exactly sure what that would look like.
Collaborator

### embray commented Jun 30, 2017

 For anything other than documentation this becomes, I think, non-trivial. Though I suppose for functions too it could be possible to have a way to go directly to their source code.
Contributor

### nthiery commented Jul 1, 2017

 On Fri, Jun 30, 2017 at 07:57:56AM +0000, Erik Bray wrote: I mean, for Sage specifically, I don't really see that as a "VRE" issue at all, or at least not as much. All we really want is an "Edit on GitHub" link for Sage's HTML docs. Hypothetically something similar in the interactive Python docs as well, though I'm not exactly sure what that would look like. I agree that it's not tightly coupled to the VRE. Small things that a VRE integration could bring: - "Edit on GitHub" (or trac/...) requires to login on github, and therefore have an account there. Since the user is already authenticated on the VRE, that could be used as credentials in the workflow. - The edition could take place within the VRE, letting the user test himself and immediately benefit from his own changes Within jupyter(lab), one could imagine something like: %%edit sage.foo.bar ... %%submit sage.foo.bar Just throwing ideas in the air ...
Collaborator

### embray commented Jul 3, 2017

 That seems like a possible lot of work with many potential pitfalls for not much benefit. How many people are really going to edit source code or even documentation on jupyter lab, as opposed to the editors they already have working locally? I could see this as potentially useful on a small scales for users wanting to submit issues or small changes to a notebook, but not serious editing of source code (especially for anything across multiple files).
Contributor

### bpilorget commented Aug 16, 2017

 @defeo This deliverable is officially due for 31/08/2017. Can you please report on the progress you have achieved? Thanks in advance
Collaborator

### embray commented Aug 21, 2017

 Last week @nthiery and I discussed a possible different direction for this deliverable, involving developing better integration between GitHub and Sage's Trac--so doing would potentially enable easier editing of Sage's documentation, among other things. This new direction wouldn't necessarily relate to SageMathCloud, but in retrospect integration with SMC is not necessarily the best way to achieve the original goals of this task.
Contributor

### defeo commented Feb 5, 2018

 @embray, what's the status of this deliverable? Do you need help to finalize it?
Contributor

### slel commented Feb 5, 2018

 Regarding better integration between GitHub and Sage's Trac, there once was a bot called "sageb0t" who converted GitHub issues to Sage Trac ticket. Hopefully @robertwb can comment, since I think he was the author and/or maintainer of sageb0t. See a 2016-07 discussion on the sage-devel mailing list:
Contributor

### slel commented Apr 3, 2018

 Regarding using CoCalc for Sage development, especially at Sage days, David Roe (@roed314) cooked up a fairly automatized way to set up a communal Sage Days environment in a CoCalc project, which has been used and refined at various Sage Days, including Sage Days 87, 88, 90, 91. Find it at:
Contributor

Collaborator

Contributor

### defeo commented Jun 28, 2018

 I believe this deliverable remains plagued by ill definition. It is. While some of us (e.g. I and @saraedum) have been (slowly) making progress on trying to make it easier to make new contributions to Sage, that work is very specific to Sage and making its development process more accessible. Worse case, the deliverable will be about your experience, and how it can benefit other projects. The description of task T3.7 which encapsulates this deliverable also mentions... I say, let's not try to stick too close to something that was written 3 years ago, based on a half-formed idea. The spirit of the task should guide our work, let's do something useful, but let's not waste resources by trying to interpret the wording of the proposal.
Contributor

### defeo commented Jun 28, 2018

 @haraldschilly What you're talking about seems to fall more under the rubric of a "help desk" (Zendesk; that sort of thing) albeit with some sort of notebook integration. But how would that work? Who would provide first level support for such a thing? Would it depend on the project(s) a user is using? If a user is making a notebook that uses, say, TensorFlow and Sage, and they need help, whose help desk does it go to? I may already have talked about this with some of you. I have been thinking for a while of having a field in Jupyter's kernel.json giving a url where one can post bug reports in some standardised way. Then a Jupyter add-on (although this is not really useful if it doesn't get installed by default) could help forming a bug report and sending it to that url. The add-on could: Automatically extract the notebook contents and format them in the report; Show a notification inviting the user to submit a bug report whenever the kernel dies; Add a menu somewhere to write and send bug reports; Use some anti-spam measures? There could even be some other deeper hooks that trigger a notification. For example, any exception trace could be accompanied by a small message "Think this is a bug? Report it". Jupyter makes it easy to do this in a uniform way for many systems. Writing a prototype could be done rather quickly. However not much of this makes sense outside of Jupyter, unless we want to write a standard describing how the HTTP request should be formed. But this seems too simple to deserve writing a standard.
Contributor

### defeo commented Jun 28, 2018

 If a user is making a notebook that uses, say, TensorFlow and Sage, and they need help, whose help desk does it go to? Of course, that's a tricky question. My proposal says that the kernel is responsible for receiving bug reports. This makes some sense for Sage, but not at all for Python. One can imagine that the kernel may have an internal mechanism to dynamically decide what the reporting URL should be. For example ipython may allow packages to set the URL at import time. It could form a list of packages that have registered a reporting URL, and give the user the choice to which component to send the report.
Collaborator

### embray commented Jun 28, 2018

 I may already have talked about this with some of you. I have been thinking for a while of having a field in Jupyter's kernel.json giving a url where one can post bug reports in some standardised way. Then a Jupyter add-on (although this is not really useful if it doesn't get installed by default) could help forming a bug report and sending it to that url. While there's some potential value in tying something like this to the kernel itself (as in the case of the Sage kernel), I think in general that's not going to be useful for a number of reasons which I've mentioned here before. Namely: Who do the bug reports go to? Surely, we do not want the standard Python kernel to send bugs to bugs.python.org, as the majority of even suspected bugs that someone might encounter in a notebook is not going to be an upstream Python bug. Within a given notebook a whole slew of technologies might be used, and a given bug report might be applicable to any one of them, or none of them. That's why I proposed maybe having some kind of "homepage" metadata in the metadata of individual notebooks. This would be a homepage for a notebook itself, indicating its upstream origin. If this is a GitHub/GitLab/Bitbucket/etc. repository that could also maybe include a quick link for submitting issues against the notebook's repository (I would be hesitant to go overboard with this though--I don't think we want to end up re-implementing Mylyn in Jupyter (or maybe we do, but that's a big project).
Collaborator

### embray commented Jun 28, 2018

 One can imagine that the kernel may have an internal mechanism to dynamically decide what the reporting URL should be. For example ipython may allow packages to set the URL at import time. It could form a list of packages that have registered a reporting URL, and give the user the choice to which component to send the report. While I like the idea of, say, individual packages providing metadata like this (some do in fact; e.g. if you look up the distribution metadata for Python packages, albeit not in a very consistent manner), this is likely to be overwhelming to users. Most researchers who encounter a problem, unless they are already experts, have no idea what lower-level library (if any) their problem pertains to. Maybe the issue is entirely user error (which is what I think @haraldschilly was getting at). It would almost be more useful to give them a form to create a well-formed StackOverflow question, but that's not always the most appropriate forum either (in fact, although SO is working on cleaning up their culture, I wouldn't want to toss unwary VRE users to the sharks...) But certainly when it comes to integrating issue reporting in notebooks or other VRE-like environments, I think we need to think at a higher level than is our impulse as software development "experts".
Contributor

### nchauvat commented Jun 28, 2018

 On Thu, Jun 28, 2018 at 06:15:01AM -0700, Luca De Feo wrote: One can imagine that the kernel may have an internal mechanism to dynamically decide what the reporting URL should be. For example ipython may allow packages to set the URL at import time. It could form a list of packages that have registered a reporting URL, and give the user the choice to which component to send the report. We use http://getsentry.com/ when we want the application to report its bugs without having to ask the user to do something. Maybe it could be a source of inspiration in this case? … -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances
Collaborator

### embray commented Jun 29, 2018

 @nchauvat Any chance you could expand on that? Who is "we" and how are you using it?
Collaborator

### embray commented Jun 29, 2018

 Interesting. I could definitely see the use for something like that in Sage, although it raises some questions too. First of all, it would have to be opt-in which immediately reduces the usefulness, but maybe not by too much. Something like sentry is pretty explicitly for applications things though, so the question becomes what kinds of errors to report. I suppose in the case of sage we would at least want to report start-up errors, and also crashes of the kernel itself. Even those would not all be of interest (some of them may be perfectly "expected"). I assume there's some way to filter out/automatically close reports of certain errors...?
Contributor

### nchauvat commented Jun 29, 2018

 On Fri, Jun 29, 2018 at 05:08:36AM -0700, E. M. Bray wrote: Something like sentry is pretty explicitly for applications things though, so the question becomes what kinds of errors to report. I suppose in the case of sage we would at least want to report start-up errors, and also crashes of the kernel itself. Even those would not all be of interest (some of them may be perfectly "expected"). If you take a look at https://docs.sentry.io/clients/ and https://docs.sentry.io/integrations/ you will see that sentry has a wide range of communication skills. One could imagine to plug it into jupyter/binder both server-side and js/client-site, to plug it into various kernels and to plug it into notebooks with a special %directive. On the other end, it integrates with several free-software forges. Trac is not one of them, but I suppose the plug-in could be written if needed. I assume there's some way to filter out/automatically close reports of certain errors...? Yes. … -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances
Collaborator

### embray commented Jul 9, 2018

 I'm struggling with how to begin a report for this. The deliverable has morphed away from "bug reporting and contributing features right in the VRE", for which we can find no clear vision, and more towards simply making contributing to Sage more accessible, as well as making Sage's source code more accessible through the documentation. It doesn't even have anything to do with SageMathCloud. I know we are allowed some flexibility on this--that we can sort of "replace" a deliverable with another deliverable of similar work scope if the requirements change over time. But I don't know if that was approved in this case, or at least if the rationale for the change in scope needs to be discussed in the report. Please advise. In the meantime I am doing what I can to write up when I have done and how it fits into things.
Collaborator

### embray commented Jul 9, 2018 • edited

 One thing for this deliverable that would be very useful to have, though I have not made any moves towards implementing yet, is a simplified version of the magics @nthiery described here: #64 (comment) One thing I have implemented for Sage is that the documentation will include an "edit on gitlab" link, which takes the user to the gitlab page for the documentation of some object, where the user can then make edits online and submit their edits as a merge request. All this really requires is the ability to produce a URL to view the given object in an online source tracker. This URL could just as easily be returned by a magic like %editsource . There could then be an interface whereby the current kernel can provide the source link for an object (or return an appropriate message if it can't). In Python, obviously, the source link depends on the package/module that the object belongs to. There could be a magic method or attribute on Python objects like __sourcelink__. This need not be defined on every individual object to work. There could be a resolution order for __sourcelink__ like object -> class (if the object is an attribute of a class) -> module -> sub-package -> package. At the top-level package you might have something like: def __sourcelink__(obj): # look up the filename of the module in which obj is defined, and the line numbers # produce a URL like https://github.com///blob/master/src/#L-  this is already essentially how the [edit on gitlab] link works for the Sage docs. It looks up where some object's source code is defined, and then plugs that into a template for a URL. In the case of __sourcelink__, allowing this to be overridden on a per-module or even per-object basis allows supporting corner cases such as extension modules, where there isn't always an automatic mapping from an object to a .py module, for example. How this would be implemented for other kernels is left as an exercise. I believe something like this could be very useful, and the scheme I describe here is something of a more concrete proposal that I believe captures some of what we've been casting about for in this discussion. However, it would take some time to implement a prototype. Perhaps for starters I could get this working just for Sage which is easier. But I don't know if it would be possible to reach some agreement on a general solution in the time before this deliverable is due. I think it's still worth doing anyways, unless I'm overlooking an obvious problem (in the concept that is--the details are obviously still half-baked).
Contributor

### nthiery commented Jul 9, 2018

 Thanks for bringing this up. Yes, for now, focus on the report as if the title was "Ease contribution to SageMath" or something similar that matches the actual content. Then we will add a section on the rationale for the change of scope, either in the report itself or in the general report (or presumably both, with a more concise version in the general). If you have some preliminary language elements for that rationale, feel free to throw them in. Thanks!
Collaborator

### embray commented Jul 9, 2018

 If you have some preliminary language elements for that rationale, I do have some text to that effect and had planed to include it.
Collaborator

### embray commented Jul 11, 2018

 Almost finished with a draft report of what we have done so far: https://github.com/OpenDreamKit/OpenDreamKit/blob/master/WP3/D3.5/status-report.tex A few more sections to fill in. Also I'm not sure what to write as a "conclusion", especially since none of this work has gone "live" yet. There are no results to report really, except for what work was done and why.

Closed

Contributor

### nthiery commented Aug 22, 2018

 Hi @embray, I just finished reviewing the report. Generally speaking, it's looking good, giving a good account of the motivations, what was achieved, and its relevance. I have done some minor changes here and there, and left some TODO's. The main remaining item is indeed the conclusion; I put some ideas of future steps, but that's not quite a conclusion. Let's discuss this tomorrow.
Collaborator

### embray commented Aug 23, 2018

 @nthiery You wrote \TODO{Use systematically the macros \Sage, \GitHub, \GitLab, \Trac} Originally I used those always, every time I mentioned any of those technologies. But what is the recommendation for that? I found that when reading the rendered page, if I used those always, it was very distracting. My approach was to use those macros only when mentioning a specific technology by itself as the subject or object of a sentence, but not in contexts like "Sage community" or "GitHub workflow".
Collaborator

### embray commented Aug 23, 2018 • edited

 You also changed The Sage development community already has well-established workflow and culture to The Sage community already has a well-established workflow and culture but in my original wording "development" was deliberate. But what do we mean by "Sage community"? I do not think it should necessarily include people who do regular, or even infrequent development on Sage. I don't want to give the impression that the "Sage community" is just the developers, because the "workflow and culture" e.g. of sage-devel and Trac is not representative, I think, of the whole Sage community, and I want to make Sage contributions more accessible beyond that. Perhaps I should clarify that as something like "the existing community of Sage developers..." and later clarify "which we would like to grow, by introducing more familiar tools", etc.
Contributor

### nthiery commented Aug 23, 2018

 but in my original wording "development" was deliberate. But what do we mean by "Sage community"? I do not think it should necessarily include people who do regular, or even infrequent development on Sage. I don't want to give the impression that the "Sage community" is just the developers, because the "workflow and culture" e.g. of sage-devel and Trac is not representative, I think, of the whole Sage community, and I want to make Sage contributions more accessible beyond that. Makes sense indeed. I am too much tainted by Sage's origins where blurring the line between the user and dev community was a vital priority to reach the critical mass:-)
Contributor

### nthiery commented Aug 23, 2018

 My approach was to use those macros only when mentioning a specific technology by itself as the subject or object of a sentence, but not in contexts like "Sage community" or "GitHub workflow". I like that rule of thumb! Let's just make sure we use it consistently.
Collaborator

### embray commented Aug 23, 2018 • edited

 @nthiery \TODO{could we choose another screenshot, maybe of some function with mathematical relevance, some mathjax formulae and examples? Something we would be proud to highlight as good documentation?} I agree. Any recommendations? What is one of the more exciting/important API doc pages you can think of? Particularly one that is interesting in the first paragraph since most of the rest would be cropped.
Collaborator

### embray commented Aug 23, 2018

 @nthiery \TODO{Future steps: simplify non only trivial contributions but also easy one (e.g. involving a code change); possibly from a VRE (e.g. a jupyterhub): make sure that the %edit magic works in-place build of the library live reload??? anything that could help create a PR/ticket from local modifications? I'm not sure how I feel about all this. This seems to be conflating development of Sage itself with development of VREs. I'm having trouble picturing a workflow whereby I would make modifications to Sage sources from within a notebook, much less then rebuild and run those changes (which would require at a minimum restarting the kernel). I just don't see the advantage, and notebooks do not necessarily make good IDEs. But maybe I'm being myopic. I just don't think I could explain a workflow like this when it isn't clear to me how that would be useful. inclusion of git-trac as sagemath package This seems like a technical detail to me, and also distracts from the overall point of enabling GitLab contributions.
Contributor

### nthiery commented Aug 23, 2018

 I'm not sure how I feel about all this. This seems to be conflating development of Sage itself with development of VREs. I'm having trouble picturing a workflow whereby I would make modifications to Sage sources from within a notebook, much less then rebuild and run those changes (which would require at a minimum restarting the kernel). I just don't see the advantage, and notebooks do not necessarily make good IDEs. But maybe I'm being myopic. I just don't think I could explain a workflow like this when it isn't clear to me how that would be useful. The main point is not so much about developing from the notebook: most of the items above are about simplifying the workflow in general. Doing development in a JupyterHub/cocalc/... service (aka "in a VRE") mostly means that you do not need to install and compile Sage locally; it would still involve opening a shell, etc. * inclusion of git-trac as sagemath package This seems like a technical detail to me, and also distracts from the overall point of enabling GitLab contributions. Right; I was just throwing in some random ideas about what could ease development. I agree that the conclusion can/should stay at a higher level.
Collaborator

### embray commented Aug 27, 2018

 @nthiery I believe I've addressed all of your TODOs in the latest draft. I also added a few other bits. I still kept the conclusion a bit vague about specifics, but did tie it again to D3.8, and how it makes it relatively easy to spin up a Sage development container. I'm happy to add more, but I was concerned that listing too many ideas would require too many words spent contextualizing those ideas.
Contributor

### nthiery commented Aug 28, 2018 • edited

 Thanks @embray. Now proofreading! Missing files: screenshots/docs-console screenshots/docs-web
Contributor

### nthiery commented Aug 28, 2018

 Done with the review. Great report Erik; thanks a lot! I did a couple minor changes in 0c74ab2; mostly syntactical. You may want to double check on them. Then, once I have the aforementioned files, I'll recompile and submit. Very good choice of documentation screenshot in Figure 1 btw.
Collaborator

### embray commented Aug 28, 2018

 Added the missing files. FWIW the subject used in the other screenshots is laplace() (transform)
Contributor

### nthiery commented Aug 28, 2018

 I am apparently still missing screenshots/docs-console. Good choice too for laplace.
Collaborator

### embray commented Aug 28, 2018

 Strange. I have right in my bash history \$ git add WP3/D3.5/screenshots/docs-console.png But it didn't go through for some reason.
Contributor

### nthiery commented Aug 28, 2018

 Go figure ... Got it now as well as the final polishing. The report is submitted now. Thanks for all the hard work: one more very useful contribution to Sage and a nice report!

Contributor

### jdemeyer commented Aug 30, 2018

 Good choice too for laplace. In a deliverable which is about "integration", certainly!

Closed

Closed

Open