Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: Quantitative Big Imaging #25

Open
whedon opened this Issue Aug 8, 2018 · 28 comments

Comments

Projects
None yet
6 participants
@whedon
Copy link
Collaborator

whedon commented Aug 8, 2018

Submitting author: @kmader (Kevin Mader)
Repository: https://github.com/kmader/Quantitative-Big-Imaging-2018
Version: v2018
Editor: @katyhuff
Reviewer: @arokem, @ThomasA
Archive: Pending

Status

status

Status badge code:

HTML: <a href="http://jose.theoj.org/papers/473509727717e744423e8f3358af4832"><img src="http://jose.theoj.org/papers/473509727717e744423e8f3358af4832/status.svg"></a>
Markdown: [![status](http://jose.theoj.org/papers/473509727717e744423e8f3358af4832/status.svg)](http://jose.theoj.org/papers/473509727717e744423e8f3358af4832)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@arokem & @ThomasA, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://jose.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @katyhuff know.

Review checklist for @arokem

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source for this learning module available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
  • Version: Does the release version given match the repository release (v2018)?
  • Authorship: Has the submitting author (@kmader) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

  • A statement of need: Do the authors clearly state the need for this module and who the target audience is?
  • Installation instructions: Is there a clearly stated list of dependencies?
  • Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

  • Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
  • Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
  • Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
  • Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
  • Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Does the paper clearly state the need for this module and who the target audience is?
  • Description: Does the paper describe the learning materials and sequence?
  • Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
  • Could someone else teach with this module, given the right expertise?
  • Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @ThomasA

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source for this learning module available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
  • Version: Does the release version given match the repository release (v2018)?
  • Authorship: Has the submitting author (@kmader) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

  • A statement of need: Do the authors clearly state the need for this module and who the target audience is?
  • Installation instructions: Is there a clearly stated list of dependencies?
  • Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

  • Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
  • Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
  • Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
  • Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
  • Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Does the paper clearly state the need for this module and who the target audience is?
  • Description: Does the paper describe the learning materials and sequence?
  • Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
  • Could someone else teach with this module, given the right expertise?
  • Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 8, 2018

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @arokem, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 8, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 8, 2018

PDF failed to compile for issue #25 with the following error:

/app/vendor/ruby-2.3.4/lib/ruby/2.3.0/find.rb:43:in block in find': No such file or directory (Errno::ENOENT) from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/find.rb:43:in collect!'
from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/find.rb:43:in find' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/whedon-e0f72c5e8125/lib/whedon/processor.rb:57:in find_paper_paths'
from /app/vendor/bundle/ruby/2.3.0/bundler/gems/whedon-e0f72c5e8125/bin/whedon:32:in prepare' from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor/command.rb:27:in run'
from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in invoke_command' from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor.rb:387:in dispatch'
from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor/base.rb:466:in start' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/whedon-e0f72c5e8125/bin/whedon:99:in <top (required)>'
from /app/vendor/bundle/ruby/2.3.0/bin/whedon:22:in load' from /app/vendor/bundle/ruby/2.3.0/bin/whedon:22:in

'

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Aug 8, 2018

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 8, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 8, 2018

PDF failed to compile for issue #25 with the following error:

/app/vendor/ruby-2.3.4/lib/ruby/2.3.0/find.rb:43:in block in find': No such file or directory (Errno::ENOENT) from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/find.rb:43:in collect!'
from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/find.rb:43:in find' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/whedon-e0f72c5e8125/lib/whedon/processor.rb:57:in find_paper_paths'
from /app/vendor/bundle/ruby/2.3.0/bundler/gems/whedon-e0f72c5e8125/bin/whedon:32:in prepare' from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor/command.rb:27:in run'
from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in invoke_command' from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor.rb:387:in dispatch'
from /app/vendor/bundle/ruby/2.3.0/gems/thor-0.20.0/lib/thor/base.rb:466:in start' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/whedon-e0f72c5e8125/bin/whedon:99:in <top (required)>'
from /app/vendor/bundle/ruby/2.3.0/bin/whedon:22:in load' from /app/vendor/bundle/ruby/2.3.0/bin/whedon:22:in

'

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Aug 8, 2018

Hmm… did you change anything from the Pre-Review stage, @kmader ?

@kmader

This comment has been minimized.

Copy link

kmader commented Aug 8, 2018

@labarba Not that I know of, I don't immediately see what the error message is, is something I can change or try?

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Aug 8, 2018

I think the message means that the paper file is not being found. @arfon ?

@arfon

This comment has been minimized.

Copy link
Member

arfon commented Aug 8, 2018

@arfon

This comment has been minimized.

Copy link
Member

arfon commented Aug 8, 2018

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 8, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Aug 8, 2018

@arfon

This comment has been minimized.

Copy link
Member

arfon commented Aug 8, 2018

I've updated the URL of the repository address to https://github.com/kmader/Quantitative-Big-Imaging-2018 which has fixed this.

@arokem

This comment has been minimized.

Copy link
Collaborator

arokem commented Sep 23, 2018

This submission includes a set of materials that were created for the teaching of an image processing course at the ETH in Spring 2016. The course includes lectures about a large range of topics in image processing, introducing modern methods in computational science along the way. From what I could gather from watching the lecture videos and going through the materials, it looks like the course was fantastic. I wish that these topics would have been taught to me early on in this way.

As I said, the materials provided in the repository, including lecture videos, notebooks, and slide-decks are excellent, and could potentially be made very useful as a set of reusable materials that others could pick and choose from, when designing courses, or even as materials for self-teaching. However, as the submission is currently organized, it carries with it too much of the legacy of its original provenance as it was written to accompany a particular course in a particular time and place. I think that for this to be a valuable contribution in this context, and to facilitate reusability, some of this specific context might need to be set aside, in favor of a more general set of materials, targetted at reuse.

For example, I think that many of the slide-decks are not currently useful to anyone who wasn't in the class, without additional notes that would tell us what to do with these slides. For example, the slides for the second lecture are useful as an accompaniment to the lecture that Dr. Kaestner gave, but are not necessarily useful to anyone without detailed notes from that lecture. A similar issue arises with Dr. Prummer's slides (from May 17th), but even worse, because no video is provided for this lecture. Similar can be said for many other lecture materials, that do not stand on their own, and I believe cannot currently be effectively reused, without substantial additional effort.

An overall guide in the readme that tells potential users of these materials (learners and instructors) how they should use the materials would be useful. Detailed narrative notes to accompany the lectures or at least more notes in the slides and notebooks would help.

The profusion of materials and lack of guidance on how to proceed through the course (either as a learner or as an instructor) is further exacerbated by the presence of deprecated materials from previous years. I would recommend to remove these materials, or at least relegate them to somewhere less prominent.

Because materials are spread between Youtube, Github, Kaggle and other places I may have missed, I worry also that some of these materials may not be available in the long run. This dispersion of the materials would also make it impossible to archive the entire set of materials together in something like a Zenodo DOI.

I would really love to see these materials better organized because I think that this set of materials could potentially serve as a very valuable contribution to JOSE, but I think that a major revision of the materials would be required to achieve that goal.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Nov 12, 2018

Hi @kmader👋
Wondering here what your sense is on re-organizing the materials for better re-usability, according to @arokem's review. Thanks!

@kmader

This comment has been minimized.

Copy link

kmader commented Nov 14, 2018

@labarba @arokem thanks for the feedback, I definitely agree with all of it and appreciate the effort you have put in. The changes look fairly substantial and I am currently swamped with other projects. I was hoping to get to some of the issues over the coming holiday breaks.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jan 1, 2019

👋 @kmader Happy New Year!
How are the revisions coming along? Give us a quick update, when you can. Thanks!

@kmader

This comment has been minimized.

Copy link

kmader commented Jan 1, 2019

Hi @labarba thanks, you as well! I've made a number of the big changes already and have them as pull requests for @arokem to review when they have time.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Jan 20, 2019

@arokem — I think we're waiting on you to review the latest changes. Can you give us an update? Thanks!

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Feb 2, 2019

Status update, @arokem ?

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Feb 9, 2019

Followed up by email with @arokem now.

👋 @ThomasA — I don't see any checked items on your list. Have you been able to work on this review for JOSE? Give us an update when you can. Thanks!

@arokem

This comment has been minimized.

Copy link
Collaborator

arokem commented Feb 9, 2019

Sorry for the slowness. I will comment on that PR.

Any other thoughts about the comments I made above? It looks like much of my feedback in these comments has not been incorporated yet.

@kmader

This comment has been minimized.

Copy link

kmader commented Feb 10, 2019

@arokem I focused on the concrete issues first because they were easier to address. As for the comments

For example, I think that many of the slide-decks are not currently useful to anyone who wasn't in the class, without additional notes that would tell us what to do with these slides. For example, the slides for the second lecture are useful as an accompaniment to the lecture that Dr. Kaestner gave, but are not necessarily useful to anyone without detailed notes from that lecture. A similar issue arises with Dr. Prummer's slides (from May 17th), but even worse, because no video is provided for this lecture. Similar can be said for many other lecture materials, that do not stand on their own, and I believe cannot currently be effectively reused, without substantial additional effort.

Unfortunately I was only able to ensure recordings on the lecture I held myself and thus the others were not done.

An overall guide in the readme that tells potential users of these materials (learners and instructors) how they should use the materials would be useful. Detailed narrative notes to accompany the lectures or at least more notes in the slides and notebooks would help.

Detailed narrative notes would be nice but the time required to remake all of the slides notes and narratives would be on the order of weeks to months (which I don't imagine having in the foreseeable future)

The profusion of materials and lack of guidance on how to proceed through the course (either as a learner or as an instructor) is further exacerbated by the presence of deprecated materials from previous years. I would recommend to remove these materials, or at least relegate them to somewhere less prominent.

The course is meant to be followed chronologically as most lectures build upon ideas in the previous, but many students manage it out of order. I kept the profusion of materials at the specific request of students who found it challenging to go between 4 years of websites. In addition, given the diversity of students in the class (ranging from computer science grad students to Art History undergrads) means it was always necessary to offer multiple parallel paths of exercises in Python (for more advanced students) and KNIME/ImageJ (for less technical ones). I tried to keep these clearly labeled but it is quite a lot to keep straight.

Because materials are spread between Youtube, Github, Kaggle and other places I may have missed, I worry also that some of these materials may not be available in the long run. This dispersion of the materials would also make it impossible to archive the entire set of materials together in something like a Zenodo DOI.

Unfortunately it would be exceptionally difficult to have a real, useful course on a single platform. All have their strengths and weaknesses and this certainly makes long run sustainability more challenging, it was done for the benefit of the students in the course today.

  • Quantitative 'Big' Imaging requires hosting of datasets and computing resources which exceed what github and mybinder currently offer (thus kaggle datasets and kaggle kernels)
  • Reproducibility requires that students are able to setup their own environment without being forced to put their data on a commercial site thus mybinder and repo2docker for the structure so they can recreate the exact setup everywhere and travisci ensures that the setup is valid and every notebook completes without error
  • Having videos as piles of MP4 files is quite difficult to store and navigate (42 hours worth) using YouTube provides quick access, playing at 2x speed and subtitles with translations for other languages
@labarba

This comment has been minimized.

Copy link
Member

labarba commented Feb 11, 2019

Here's the thing, @kmader. You have collected years of materials from your course in one GitHub repository, but this is not what JOSE was designed to publish. One of the key ideas for JOSE publications is that the material be reusable by other instructors or self-learners. Also, we write in the journal's scope:

What do you mean by "open-source educational materials"?
Examples include Jupyter notebooks or plaintext/markup language documents like LaTeX, R Markdown, and ReST for course/lesson content and associated notes, with embedded or associated code snippets/programs.
We do not mean openly available slides, lecture notes, or YouTube videos, though these may be acceptable as supplementary materials.

As a general rule, it's nearly impossible for instructors to re-use slide decks prepared by others. We also explicitly exclude multi-media content (e.g., videos), because of the focus on open source content, that can be modified, version controlled, forked and built on.

From your collection of materials, if you were to organize only the Jupyter notebooks into a coherent, self-contained learning module, that's a JOSE publication. Imagine if you were giving an intense, three-day tutorial at a conference, and you polished just the notebooks for that purpose, expecting to do fast-paced live coding, plus leave the audience with fully narrated materials to review at their own pace after the event. Those materials would be a JOSE module.

@kmader

This comment has been minimized.

Copy link

kmader commented Feb 11, 2019

Thanks for that info @labarba , I guess in that case it won't make sense to make this repo as it is into a JOSE publication (students still use the repo and it would not be fair to them to remove possibly useful materials to make it conform with JOSE's requirements).
I think I had originally misunderstood the exact purpose of JOSEs work and had incorrectly assumed it was meant to be open-source materials for students (all figures, examples and diagrams driven by tested, continuously integrated code rather than static slides). I had never intended for my course to be particularly useful for lecturers preparing other courses and as such it is probably a bad fit.
When I have time, I'll try to put together some of the material into a new repository just for JOSE, but I guess I can make that as a new separate submission and we can safely reject and close this one.

Thanks in any event for the helpful feedback I think it has made the material more accessible for students.

@labarba

This comment has been minimized.

Copy link
Member

labarba commented Feb 11, 2019

Here's an idea: you could create a new repository with the Jupyter notebooks, polish them a little bit to make them self-contained with references out to the vids, etc. as supplementary, and then include that repo in your main course one as a GitHub module. The JOSE paper would be associated with the sub-module repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.