New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: Open OnDemand: A web-based client portal for HPC centers #622

Closed
whedon opened this Issue Mar 14, 2018 · 51 comments

Comments

Projects
None yet
6 participants
@whedon
Collaborator

whedon commented Mar 14, 2018

Submitting author: @ericfranz (Eric Franz)
Repository: https://github.com/OSC/Open-OnDemand
Version: v1.2
Editor: @danielskatz
Reviewer: @marpierc, @smgallo
Archive: 10.6084/m9.figshare.6265514.v1

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a"><img src="http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a/status.svg)](http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer1 instructions & questions

@marpierc, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @danielskatz know.

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: Does the release version given match the GitHub release (v1.2)?
  • Authorship: Has the submitting author (@ericfranz) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Reviewer2 instructions & questions

@smgallo, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @danielskatz know.

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: Does the release version given match the GitHub release (v1.2)?
  • Authorship: Has the submitting author (@ericfranz) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Collaborator

whedon commented Mar 14, 2018

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @marpierc it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Collaborator

whedon commented Mar 14, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Collaborator

whedon commented Mar 14, 2018

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Mar 14, 2018

@arfon - can you invite @smgallo so that he can be a reviewer?
@marpierc - you automatically be invited as described in the review instructions.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Mar 14, 2018

My one initial comment is that the paper doesn't have any references - the reviewers might think about any things (papers, systems) that should be referenced...

@arfon

This comment has been minimized.

Member

arfon commented Mar 14, 2018

@arfon - can you invite @smgallo so that he can be a reviewer?

Done.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Mar 14, 2018

discussing the references in OSC/Open-OnDemand#30

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Mar 26, 2018

For the record, the reviews from @smgallo and @marpierc are expected by the end of March and April 5, respectively.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Mar 27, 2018

@whedon commands

@whedon

This comment has been minimized.

Collaborator

whedon commented Mar 27, 2018

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the sole reviewer of this submission
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Open the review issue
@whedon start review

🚧 🚧 🚧 Experimental Whedon features 🚧 🚧 🚧

# Compile the paper
@whedon generate pdf

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Mar 27, 2018

@whedon generate pdf

@whedon

This comment has been minimized.

Collaborator

whedon commented Mar 27, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Collaborator

whedon commented Mar 27, 2018

@smgallo

This comment has been minimized.

Collaborator

smgallo commented Mar 27, 2018

@danielskatz

OOD is an excellent package and fills a need for HPC users by simplifying and improving access to complicated HPC resources. HPC centers are able to provide their users with simplified access to resources while making it easier to submit and manage both jobs and data. Single sign-on is supported making it easier to integrate with center or campus-wide authentication systems. Because OOD runs in a web-browser, users do not need to install additional software on their computers to access HPC resources.

Our center recently installed OOD to serve novice and light HPC users and has found that even experienced users have said that using OOD is "so much easier" than logging in via the command line to manage their compute jobs and data. In addition, faculty who use our HPC resources to teach courses are entheuiastic about OOD. Over the years, we had implemented a subset of the functionality that OOD provides to support the needs of our users, but having all of this functionality in a single integrated framework is great. There is an active mailing list which is available for general questions. Multiple authentication methods are supported via Apache modules including shibboleth, LDAP, and CiLogin. We have implemented CiLogin backed by FreeIPA for access to our resources.

OOD supports a number of useful applications including a job status viewer, file manager and editor, shell tool, and an interactive desktop. In addition, there are interactive desktop applications that are supported such as MATLAb, Jupyter, and R Studio, all of which are desired by end users. Development of custom applications is documented (with examples). The interactive desktop application has proven especially useful and allowed us to deprecate a licensed application that was previously provided to our users. A very useful feature of OOD are the Per User NGIX Processes (PUN). This allows users to run their applications via the web server in a process that they own (rather than the web server owning the process) making system and data security far simpler.

General Comments

  1. An RPM install is not available but it would be great to have this (along with an upgrade path from a source install). Is an RPM (or other package manager) version planned?
  2. Individual apps are contained in their own git repos. While this makes extra work when installing the apps, it is nicely compartmentalized.

Versioning

The versioning is a bit confusing and could use clarification in the documentation. OOD is made up of several components including infrastructure and applications. The infrastructure component versions do not appear to be to match the 1.2 version of OOD as a whole and the only application that I could find with a recent 1.23 version number was the dashboard. Other applications seem to have their own version tags but are now far ahead of version 1.2. For example, the myjobs application is now at version 2.8.2 with 1.2 released almost 2 years ago.

Thanks to CCR staff members @aebruno, @dsajdak, and Cynthia Cornelius at CCR for their input. CCR has installed Open on Demand and made it available to our users.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Mar 27, 2018

Thanks @smgallo - it looks like there are some issues you raised for @ericfranz to respond to or resolve. Additionally, I see you did not check the automated tests item, or discuss testing in your comment above. Can you explain?

@smgallo

This comment has been minimized.

Collaborator

smgallo commented Mar 27, 2018

@danielskatz Apologies, I forgot to add that in.

@ericfranz While the documentation for the various components does explain how to use or access the component, there does not appear to be an automated test suite or manual process specified for each component to verify that the component is installed correctly (note that some components do instruct the installer to manually verify). This is a complex software and infrastructure package and it would be useful for each component to have a simple verification script that could perhaps test for the existence of a file or test the data returned from a REST endpoint. I can certainly appreciate the effort involved in creating and maintaining tests of this nature so it would be good to see plans for incremental steps in this direction.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Apr 2, 2018

@marpierc - just a reminder that your review is expected by April 5.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Apr 2, 2018

@ericfranz - Do you have a response to @smgallo's comments? You can certainly address them before @marpierc's review

@ericfranz

This comment has been minimized.

ericfranz commented Apr 2, 2018

Thank you for the comments! I opened three issues in https://github.com/OSC/Open-OnDemand for further discussion.

  1. We have an rpm we are using at OSC now and will be the default installation for OnDemand 1.3 released in April. I opened OSC/Open-OnDemand#31 to address the problem of not advertising our development roadmap.
  2. For each OnDemand release we document all the corresponding versions in the release notes (see 1.2 example). I opened OSC/Open-OnDemand#32 to track the need to put this list in a more prominent location. We are also looking to combine some repos into one in the future to help with this.
  3. We have some unit test coverage for components which target developers, but have no system tests for installation validation which would target admins. I opened an issue OSC/Open-OnDemand#33 to discuss that further.
@smgallo

This comment has been minimized.

Collaborator

smgallo commented Apr 2, 2018

Thanks @ericfranz, I'll follow up on those issues to provide further input as needed. @danielskatz that addresses any comments that I had.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Apr 4, 2018

@marpierc - just a reminder that your review is expected by April 5.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Apr 4, 2018

@marpierc - I think I saw a comment from you about the paper and what it should contain, though I no longer see it, so perhaps it was deleted? But in any case, see http://joss.theoj.org/about#author_guidelines (what should my paper contain)

@marpierc

This comment has been minimized.

Collaborator

marpierc commented Apr 4, 2018

@marpierc

This comment has been minimized.

Collaborator

marpierc commented Apr 5, 2018

I have made comments here: https://github.com/OSC/Open-OnDemand/issues.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Apr 5, 2018

Thanks @marpierc - please update this issue as those issues get resolved.

@marpierc

This comment has been minimized.

Collaborator

marpierc commented Apr 11, 2018

I second @smgallo comments. The major issues with the current release are the complicated installation process and the need for more automated testing. The OOD team should also consider how to handle third party contributions (bug fixes, add-ons, etc).

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Apr 11, 2018

Thanks @marpierc

Just to make the status of this review clear, I think the ball is now with @ericfranz to address these issues and respond.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented Apr 22, 2018

👋 @ericfranz - please let us know when you have addressed these issues.

1 similar comment
@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented May 1, 2018

👋 @ericfranz - please let us know when you have addressed these issues.

@ericfranz

This comment has been minimized.

ericfranz commented May 1, 2018

I believe all issues have been addressed:

  1. The 1.3 release next week will include deployment via rpm to address the complicated installation process
  2. We added a set of auto-generated Rake tasks to test and verify job submission from the web node for each configured cluster (OSC/Open-OnDemand#33 and OSC/ood-dashboard#360) which will be included in the 1.3 release. This should address concerns for tests to help admins verify the installation.
  3. We added a CONTRIBUTING.md file to the root of the OnDemand repo to address third party contributions.

Assuming everything goes well this week, OnDemand 1.3 will be released next Monday. Is there a need to wait for this release or can we proceed with the JOSS publication?

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented May 1, 2018

If the current software meets what reviewers think is needed, we can proceed.

At this point, we need @marpierc and @smgallo to examine this again and see if they can check off all the items in their checklists.

@smgallo

This comment has been minimized.

Collaborator

smgallo commented May 1, 2018

@danielskatz The recent/pending updates to OOD are sufficient for me.

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented May 1, 2018

The recent/pending updates to OOD are sufficient for me.

@smgallo - can you check off the remaining item on your list in this case?

@smgallo

This comment has been minimized.

Collaborator

smgallo commented May 1, 2018

@danielskatz Done!

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented May 7, 2018

@marpierc - can you verify that the remaining point has been addressed and check off the remaining item on your list?

@marpierc

This comment has been minimized.

Collaborator

marpierc commented May 7, 2018

Done

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented May 7, 2018

@marpierc and @smgallo - thanks very much for your reviews

@danielskatz

This comment has been minimized.

Collaborator

danielskatz commented May 7, 2018

👋 Arfon - over to you to continue the acceptance process

@arfon

This comment has been minimized.

Member

arfon commented May 7, 2018

@ericfranz - At this point could you make an archive of the reviewed software in Zenodo/figshare/other service and update this thread with the DOI of the archive? I can then move forward with accepting the submission.

@ericfranz

This comment has been minimized.

ericfranz commented May 10, 2018

@arfon would it be sufficient to upload a zip or a tarball of the related rpms at https://yum.osc.edu/ondemand/1.3/ to figshare? (this is the latest release of OnDemand just released this week, which include suggested modifications). If not, what is your recommendation for how to package in a single tarball software that is spread across several repos and whose installation results in files being placed in the appropriate locations following the Filesystem Hierarchy Standard?

Other approaches I am considering include:

  1. unpack the 1.3 rpms, adding a README at the root for how to proceed with the installation, and tar the parent directory
  2. get tarballs of each git repo, placing them in the appropriate location in the filesystem that an rpm installation would place them, adding a README at the root of how to proceed with installation
@arfon

This comment has been minimized.

Member

arfon commented May 11, 2018

@arfon would it be sufficient to upload a zip or a tarball of the related rpms at https://yum.osc.edu/ondemand/1.3/ to figshare?

The archive needs to include the source code which I don't think would be true in this case right?

If so, option #2 above is probably the best option...

@ericfranz

This comment has been minimized.

ericfranz commented May 14, 2018

@arfon see https://doi.org/10.6084/m9.figshare.6265514.v1

This tarball is also attached to the packaging repo https://github.com/OSC/ondemand/releases/tag/v1.3.6 and is all of the source for the core infrastructure and apps. The README.md in the tarball is markdown formatted and provides instructions for building from source, where to get help when stuck, and recommends using the RPM based installation.

@arfon arfon added the accepted label May 15, 2018

@arfon

This comment has been minimized.

Member

arfon commented May 15, 2018

@whedon set 10.6084/m9.figshare.6265514.v1 as archive

@whedon

This comment has been minimized.

Collaborator

whedon commented May 15, 2018

OK. 10.6084/m9.figshare.6265514.v1 is the archive.

@arfon

This comment has been minimized.

Member

arfon commented May 15, 2018

@marpierc, @smgallo many thanks for your reviews here and to @danielskatz for editing this submission

@ericfranz - your paper is now accepted into JOSS and your DOI is https://doi.org/10.21105/joss.00622 ⚡️ 🚀💥

@arfon arfon closed this May 15, 2018

@whedon

This comment has been minimized.

Collaborator

whedon commented May 15, 2018

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippet:

[![DOI](http://joss.theoj.org/papers/10.21105/joss.00622/status.svg)](https://doi.org/10.21105/joss.00622)

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

@ericfranz

This comment has been minimized.

ericfranz commented Jun 29, 2018

NSF would like us to add an acknowledgement to the paper. After updating the paper.md file to contain an Acknowledgements section similar to what I see in other JOSS publications, what is the next step to get the PDF to be regenerated? OSC/Open-OnDemand#44

@arfon

This comment has been minimized.

Member

arfon commented Jun 29, 2018

Please add the language to your paper.md file in the master branch of your repository and we can regenerate it.

@ericfranz

This comment has been minimized.

ericfranz commented Jul 2, 2018

@arfon ok the paper.md has been updated

@arfon

This comment has been minimized.

Member

arfon commented Jul 3, 2018

@arfon ok the paper.md has been updated

I've updated the PDF too. This may take a few hours to show up on the live site due to the caching we do.

@ericfranz

This comment has been minimized.

ericfranz commented Jul 4, 2018

@arfon Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment