Conclusions #370

agitter · 2017-05-02T04:42:56Z

@cgreene see if you agree with the position I took, and feel welcome to tear down this entire section if you have a different stance.

This will be one focal point of the review, so others can provide feedback as well.

enricoferrero · 2017-05-02T08:09:50Z

sections/07_conclusions.md

+its dominance over competing machine learning approaches in many of the areas
+reviewed here and quantitative improvements in predictive performance, deep
+learning has not yet qualitatively "solved" those problems that were previously
+"unsolved".


Is

that were previously "unsolved"

needed here? It sounds like a bit of a repetition to me.

enricoferrero · 2017-05-02T08:12:50Z

sections/07_conclusions.md

+finally approaching or exceeding human performance in the past year
+[@arxiv:1610.05256 @arxiv:1703.02136] `TODO: working on a second source for this
+error trajectory from a talk by Eric Horvitz`. The phenomenal  improvements on
+benchmark datasets are undeniable, but the successes of the early 2010s did not


Maybe it's just me but I don't think it's overly clear what the

successes of the early 2010s

are.

Good point, this is unclear. I'm thinking of the left image from https://twitter.com/amram/status/845748240033050624 However, I can't directly use it because I don't know the source, and Eric is likely too busy being the new head of MSR to reply. http://www.businessinsider.com/ibm-edges-closer-to-human-speech-recognition-2017-3 shows comparable numbers but with less granularity.

I changed this to "...are undeniable, but halving the error rates on these benchmarks did not fundamentally transform...". Still not perfect, but that's my intended message.

No one is pushing back on my conversational speech example, so I removed: TODO: this is debatable, maybe need a different example or to clarify what is meant by "conversational" speech

enricoferrero · 2017-05-02T08:16:17Z

sections/07_conclusions.md

+fundamentally transform the domain. `TODO: this is debatable, maybe need a
+different example or to clarify what is meant by "conversational" speech`
+Widespread adoption of these technologies will requires not only improvements
+over baseline methods but truly "solving" the problem, in this case exceeding


Since this is the second time you use 'solve' in this section I think the double quotes are unnecessary.

enricoferrero · 2017-05-02T08:20:53Z

sections/07_conclusions.md

+[@doi:10.1001/jama.2016.17216], diabetic macular edema
+[@doi:10.1001/jama.2016.17216], and skin lesion [@doi:10.1038/nature21056]
+classifiers are highly accurate and comparable to dermatologist performance in
+the latter case. `TODO: more imaging examples or other examples that might be at


comparable to dermatologist performance in the latter case

sounds too specific for this section, maybe consider something more generic like

comparable to human performance

Agreed - reads easier.

Changed to "comparable to clinician performance" to differentiate between an expert and average human, but I can reword this further if you would like

enricoferrero · 2017-05-02T08:21:46Z

sections/07_conclusions.md

+[@doi:10.1001/jama.2016.17216], diabetic macular edema
+[@doi:10.1001/jama.2016.17216], and skin lesion [@doi:10.1038/nature21056]
+classifiers are highly accurate and comparable to dermatologist performance in
+the latter case. `TODO: more imaging examples or other examples that might be at


Agree, maybe you can add #366 here?

Added it. I like the idea about using experts to review a smaller set of images where two CNNs disagree, but that may be better for the Categorize sub-section instead of expanding the conclusions.

enricoferrero · 2017-05-02T08:26:58Z

sections/07_conclusions.md

+[@doi:10.1001/jama.2016.17216], and skin lesion [@doi:10.1038/nature21056]
+classifiers are highly accurate and comparable to dermatologist performance in
+the latter case. `TODO: more imaging examples or other examples that might be at
+or close to "transformative"?`  In other domains, perfect accuracy will not be


Do we want to argue that in some domains perfect accuracy might not even be possible? Not sure what arguments we could bring here but intuitively some complex problems (e.g: gene regulation in disease) might not be within DL reach, no matter how good the algorithm is.

Just a thought, I appreciate it's a rather subjective stance and not everyone might agree.

I agree with @enricoferrero that in some areas perfect performance may be impossible for any algorithm due to the stochastic/critical nature of biological systems. Maybe just one closing sentence to this paragraph since it's the perfect accuracy may not... paragraph. Also, I think you may want to start a new paragraph at "In other domains...".

@agitter : if you want to create an issue for this, I or @enricoferrero can likely address afterwards. Up to you on what you prefer.

I completely agree with this sentiment. I reworked this slightly; see if it needs more editing and a new issue. My intention is to keep the two examples where deep learning is closer to practical impact (medical imaging and chemical screening) together. Then I split a new paragraph for the more negative examples.

enricoferrero · 2017-05-02T08:31:03Z

sections/07_conclusions.md

+approaches that would be infeasible with other machine learning techniques.
+Unsupervised methods are currently less-developed than their supervised
+counterparts, making them an attractive target for future research in this
+domain.  `TODO: still working on a strong closing line`


I feel this sentence on unsupervised learning could be expanded a little bit. Maybe we could argue that since a lot of biomedical data is unlabelled and labelling has to be done manually, accurate deep unsupervised methods could also be transformative in high impact fields such as patient stratification for precision medicine approaches.

Updated per @cgreene's suggestion below

enricoferrero

@agitter Great section, thanks. I've left a few minor comments here and there.

agitter · 2017-05-02T12:04:09Z

@enricoferrero thanks for these comments. These are excellent suggestions, and I'll make point-by-point responses and text updates, hopefully tonight.

cgreene

I had a few thoughts. I also had a couple points where I need more clarity to provide a helpful review. I love where this is going though! I suggested a potential final sentence.

cgreene · 2017-05-02T12:11:35Z

sections/07_conclusions.md

+[@doi:10.1001/jama.2016.17216], and skin lesion [@doi:10.1038/nature21056]
+classifiers are highly accurate and comparable to dermatologist performance in
+the latter case. `TODO: more imaging examples or other examples that might be at
+or close to "transformative"?`  In other domains, perfect accuracy will not be


I agree with @enricoferrero that in some areas perfect performance may be impossible for any algorithm due to the stochastic/critical nature of biological systems. Maybe just one closing sentence to this paragraph since it's the perfect accuracy may not... paragraph. Also, I think you may want to start a new paragraph at "In other domains...".

@agitter : if you want to create an issue for this, I or @enricoferrero can likely address afterwards. Up to you on what you prefer.

cgreene · 2017-05-02T12:14:21Z

sections/07_conclusions.md

+For example, in chemical screening for drug discovery, a deep learning system
+that successfully identifies dozens or hundreds of target-specific, active
+small molecules from a massive search space would have immense practical value
+even if its overall precision is modest. Conversely, the most challenging tasks


I would need some clarification here to decide if this should be elaborated on. Why are these potentially the most challenging? I guess I am missing the rationale for this statement.

Perhaps I mean to say that errors may be magnified in some cases if we rely on predictions for some secondary task. For instance, suppose I rely on predictions of TF binding from a neural net to predict TF binding and then use those genome-wide predictions to model gene expression. I'll need to be more accurate than if I were going to use those predictions to follow up on a few interesting or high-confidence binding sites.

cgreene · 2017-05-02T12:14:39Z

sections/07_conclusions.md

+challenges beyond improving training and predictive accuracy, such as preserving
+patient privacy and interpreting models.  Ongoing research has begun to address
+these problems and shown they are not insurmountable.  Deep learning offers the
+flexibility to model data in its most natural form, spurring creative modeling


What do you mean by "most natural form" here?

I'm thinking of things like graph convolutional networks that allow one to work directly on a molecular graph for chemical modeling. Previously, pre-computed lossy feature representations are more common when using other machine learning approaches. I could probably come up with examples in other domains as well.

cgreene · 2017-05-02T12:18:00Z

sections/07_conclusions.md

+these problems and shown they are not insurmountable.  Deep learning offers the
+flexibility to model data in its most natural form, spurring creative modeling
+approaches that would be infeasible with other machine learning techniques.
+Unsupervised methods are currently less-developed than their supervised


Unsupervised methods are currently less-developed than their supervised counterparts, but they may have the most potential. When deep learning algorithms can summarize very large collections of input data into interpretable models that spur scientists to ask questions that we didn't know to ask, it will be clear that deep learning has transformed biology and medicine.

agitter · 2017-05-02T14:13:33Z

@cgreene thanks, I'll be able to clarify these parts in my next pass.

agapow

Looks good - my comments should be taken largely as suggestions at to how to polish it.

agapow · 2017-05-02T14:58:50Z

sections/07_conclusions.md

@@ -1,15 +1,60 @@
 ## Conclusions

-Final thoughts and future outlook here. The Discussion will give an overview
-and the Conclusion will provide a short, punchy take home message.
+Deep learning-based methods now represent the state of the art in a diverse


I'm a little uncomfortable about "state of the art" in that it's making an assertion that deep learning is the state of the art. Perhaps instead say "deep learning now matches or suprasses previous state of the art"

Yes, that's better

agapow · 2017-05-02T15:00:02Z

sections/07_conclusions.md

+its transformative potential or induced a strategic inflection point.  Despite
+its dominance over competing machine learning approaches in many of the areas
+reviewed here and quantitative improvements in predictive performance, deep
+learning has not yet qualitatively "solved" those problems that were previously


instead of "qualitatively" maybe "definitively" / "inarguably"

agapow · 2017-05-02T15:01:18Z

sections/07_conclusions.md

+human-level performance, as well as convincing users to embrace the technology
+[@tag:Speech_recognition].  We see parallels to the healthcare domain, where
+achieving the full potential of deep learning will require outstanding
+predictive performance as well as adoption by biologists and clinicians.


maybe "acceptance and adoption"

agapow · 2017-05-02T15:01:42Z

sections/07_conclusions.md

+[@doi:10.1001/jama.2016.17216], diabetic macular edema
+[@doi:10.1001/jama.2016.17216], and skin lesion [@doi:10.1038/nature21056]
+classifiers are highly accurate and comparable to dermatologist performance in
+the latter case. `TODO: more imaging examples or other examples that might be at


Agreed - reads easier.

agapow · 2017-05-02T15:03:29Z

sections/07_conclusions.md

+classifiers are highly accurate and comparable to dermatologist performance in
+the latter case. `TODO: more imaging examples or other examples that might be at
+or close to "transformative"?`  In other domains, perfect accuracy will not be
+required because deep learning will be used primarily to prioritize experiments.


I like this point a lot: imperfect solutions can be damn useful. Perhaps it needs to be broken out to a separate paragraph to emphasis? I'm thinking there's a lot of value in "assisted discovery" and highlighting areas for further investigation. Conversely, this could blow out into far too long a discussion.

Glad you like it. I made a separate paragraph and added an example from #366. We could probably add at least one more to make the point stronger if you have ideas.

I should add, I'm somewhat concerned the false negatives in the deep learning-assisted strategy proposed in #366.

agapow · 2017-05-02T15:06:04Z

sections/07_conclusions.md

+decision-making, especially in the clinic. `TODO: elaborate more on this idea
+or split in a new paragraph?`
+
+Even if deep learning in biology and healthcare is not yet transformative today,


... and it's early days yet, full potential not explored, deep learning still evolving, yadda yadda yadda

Added a line

agitter · 2017-05-03T15:13:53Z

Thanks again @enricoferrero @cgreene @agapow. I updated the text or commented in response to all feedback above. There are still some open questions we can resolve before merging, mostly in response to @cgreene.

agapow

Looks good - one small language suggestion

agapow · 2017-05-03T19:39:15Z

sections/07_conclusions.md

-"unsolved".
+Deep learning-based methods now matches or surpasses the previous state of the
+art in a diverse array of tasks in patient and disease categorization,
+fundamental biological study, genomics, and treatment development.  We return to


"Returning to our central question": less passive

Yes, that's better. I changed it.

enricoferrero

LGTM

cgreene

One minor change then LGTM 👍 . Very nice!

cgreene · 2017-05-04T11:18:43Z

sections/07_conclusions.md

@@ -1,15 +1,70 @@
 ## Conclusions

-Final thoughts and future outlook here. The Discussion will give an overview
-and the Conclusion will provide a short, punchy take home message.
+Deep learning-based methods now matches or surpasses the previous state of the


subject/verb agreement

agitter · 2017-05-04T14:16:04Z

Made the last change from @cgreene and addressed a couple TODOs. Merging now.

I added this example to clarify one of my remarks: "As an example, errors in a predicted protein contact map could be amplified if that contact map is used directly for 3D structure prediction." @j3xugit, is this statement correct in your opinion?

This build is based on 2dfa08a. This commit was created by the following Travis CI build and job: https://travis-ci.org/greenelab/deep-review/builds/228757316 https://travis-ci.org/greenelab/deep-review/jobs/228757317 [ci skip] The full commit message that triggered this build is copied below: Conclusions (#370) * Initial draft of conclusions * Respond to feedback * Rephrasing * Address TODOs and grammar * Minor rewording

j3xugit · 2017-05-04T17:31:13Z

In fact, we can directly use predicted contact maps to predict 3D structures and for many proteins we indeed produce very good 3D modeling. The prediction error of an individual contact may be big, but when multiple predicted contacts are used together for 3D structure modeling, the impact of an individual contact is usually reduced instead of amplified. By the way, I believe that the change of ab initio folding is transformative in the past few years due to 1) significantly improved co-evolution analysis; and 2) significantly improved contact prediction by deep learning.

…

On Thu, May 4, 2017 at 9:16 AM, Anthony Gitter ***@***.***> wrote: Made the last change from @cgreene <https://github.com/cgreene> and addressed a couple TODOs. Merging now. I added this example to clarify one of my remarks: "As an example, errors in a predicted protein contact map could be amplified if that contact map is used directly for 3D structure prediction." @j3xugit <https://github.com/j3xugit>, is this statement correct in your opinion? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#370 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKR63opN4qoO1559KyTxbWAMZQ2wZR0oks5r2d2mgaJpZM4NNtGm> .

--

_________________________________________ Professor Toyota Technological Institute at Chicago 6045 S. Kenwood Ave. Chicago, IL 60637 fax: 773 834 2557, Google Voice: 773 359 3721 http://ttic.uchicago.edu/~jinbo/

agitter · 2017-05-05T14:17:26Z

@j3xugit thanks for the correction. I opened #376 to address this and will make a new pull request.

Initial draft of conclusions

406af08

agitter requested a review from cgreene May 2, 2017 04:42

enricoferrero reviewed May 2, 2017

View reviewed changes

cgreene reviewed May 2, 2017

View reviewed changes

agitter mentioned this pull request May 2, 2017

Data/code sharing and data limitations #367

Merged

agapow approved these changes May 2, 2017

View reviewed changes

Respond to feedback

3955331

agapow approved these changes May 3, 2017

View reviewed changes

Rephrasing

85a567b

enricoferrero approved these changes May 4, 2017

View reviewed changes

cgreene approved these changes May 4, 2017

View reviewed changes

agitter added 2 commits May 4, 2017 09:13

Address TODOs and grammar

e979cf3

Minor rewording

934fa5a

agitter merged commit 2dfa08a into greenelab:master May 4, 2017

agitter deleted the conclusions branch May 4, 2017 14:24

agitter mentioned this pull request May 5, 2017

Protein structure in conclusion #376

Closed

Conclusions #370

Conclusions #370

Conversation

agitter commented May 2, 2017

enricoferrero May 2, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enricoferrero May 2, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agitter May 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enricoferrero left a comment

Choose a reason for hiding this comment

agitter commented May 2, 2017

cgreene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgreene May 2, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agitter commented May 2, 2017

agapow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agitter commented May 3, 2017

agapow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enricoferrero left a comment

Choose a reason for hiding this comment

cgreene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agitter commented May 4, 2017

j3xugit commented May 4, 2017 via email

agitter commented May 5, 2017

enricoferrero May 2, 2017 •

edited

enricoferrero May 2, 2017 •

edited

agitter May 3, 2017 •

edited

cgreene May 2, 2017 •

edited