Missing cell types in the results

[jracle85]

Hello,
First, congrats for your paper and thank you for including some additional discussion in it, as I had suggested. It's great to see that EPIC was top-performing in this independent benchmark :-)

It's also cool that you wrote this package so that people can easily use the one or other method, even though this might hide some options that the original packages are proposing to their users, like the use of their own cell reference profiles.

After testing your package, I've seen an issue: some cell types predicted by a given method are missing from the results. In particular, for EPIC, when doing the deconvolution for non-tumor tissue, we also predict the neutrophils but they aren't returned. And both for the tumor and non-tumor cases, EPIC also adds the predictions of the "otherCells". This is a novel and very important feature that has been introduced by EPIC, representing mostly the cancer cells or other cell types without reference profiles that are present in the bulk sample. We feel that these cell types should thus be added in your cell_type_mapping file to include all cell types predicted by EPIC.

Similarly, for xCell, I understand that you didn't want to output all cell subtypes due to the potential spillover effects affecting this method. But this also hides the fact that xCell would also return predictions for other cell types. So, it might be good to implement an option allowing the user to select doing the deconvolution for his cell types of interest (the "expected_cell_type" option seem to only work to further restrain the set of cell types, but not to add other cell types absent from the cell_type_mapping table).

Thanks! Cheers,

Julien

[mlist]

Dear Julien, Thanks! Those were really good comments :-) We really appreciate that you had a close look at the R package. Gregor will check how we can add those missing cell types and get back to you soon. Any chance we'll see you in Basel for the ISMB next week? Best, Markus Julien Racle <notifications@github.com> schrieb am Do., 18. Juli 2019, 11:06:

…

Hello, First, congrats for your paper and thank you for including some additional discussion in it, as I had suggested. It's great to see that EPIC was top-performing in this independent benchmark :-) It's also cool that you wrote this package so that people can easily use the one or other method, even though this might hide some options that the original packages are proposing to their users, like the use of their own cell reference profiles. After testing your package, I've seen an issue: some cell types predicted by a given method are missing from the results. In particular, for EPIC, when doing the deconvolution for non-tumor tissue, we also predict the neutrophils but they aren't returned. And both for the tumor and non-tumor cases, EPIC also adds the predictions of the "otherCells". This is a novel and very important feature that has been introduced by EPIC, representing mostly the cancer cells or other cell types without reference profiles that are present in the bulk sample. We feel that these cell types should thus be added in your cell_type_mapping file to include all cell types predicted by EPIC. Similarly, for xCell, I understand that you didn't want to output all cell subtypes due to the potential spillover effects affecting this method. But this also hides the fact that xCell would also return predictions for other cell types. So, it might be good to implement an option allowing the user to select doing the deconvolution for his cell types of interest (the "expected_cell_type" option seem to only work to further restrain the set of cell types, but not to add other cell types absent from the cell_type_mapping table). Thanks! Cheers, Julien — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/grst/immune_deconvolution_benchmark/issues/26?email_source=notifications&email_token=AAKWBXMFFFAXFEDSLPF24W3QAAXCVA5CNFSM4IEYRDH2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G76LJ2Q>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKWBXPNJPG52ZZNWTI2UKLQAAXCVANCNFSM4IEYRDHQ> .

[jracle85]

Dear Markus,

Thank you for the message. Unfortunately I won't be able to join ISMB next week. But I'll be at BC2 in September, also in Basel. Maybe you'll be there as well?

And sorry, but I see that I posted this message in the wrong repository... I had both repo opened but this issue should of course relate to the package, not to the benchmark where the cell types are fixed...

I'll thus put a link to this also in the package in case someone else is interested by this question (but if you prefer we can also delete it from here and recopy in the other).

Thank. Cheers,

Julien

[grst]

Moved the issue ;)
Will reply in detail later...

[jracle85]

Great, thanks for moving it; I'm still quite new to GitHub and don't master all of it ;-)

[grst]

Hi Julien,

thanks for your feedback, this is really helpful!

It's also cool that you wrote this package so that people can easily use the one or other method, even though this might hide some options that the original packages are proposing to their users, like the use of their own cell reference profiles.

I believe it would be ideal to access all the methods' options through a consistent interface, however this is pretty challenging due to the conceptual differences of the methods. One example would be to specify custom signatures. This is something that has been proposed to me earlier via email. I now created #15 to discuss how to implement this.

some cell types predicted by a given method are missing from the results. In particular, for EPIC, when doing the deconvolution for non-tumor tissue, we also predict the neutrophils but they aren't returned. And both for the tumor and non-tumor cases, EPIC also adds the predictions of the "otherCells".

Well the reason why they are not included is that first I created this package as helper functions for the benchmark pipeline. Since the scRNA-seq dataset does not include Neutrophils (and all the different xCell types), there was no point in adding them.

However I agree that they should be included now!

As you may have figured out it's all in this excel sheet, and it should be easy to add the additional cell types:
https://github.com/grst/immunedeconv/blob/master/inst/extdata/cell_type_mapping.xlsx

I am hoping to do next week. Maybe you could review the final mapping? What do you think?

Cheers,
Gregor

[jracle85]

Hi Gregor,

Thanks for the reply.

I am hoping to do next week. Maybe you could review the final mapping? What do you think?

Yes sure, I can check the updated mapping once it's done (but I won't have much time next week anyway). I can check for EPIC and have a look at the other methods as well but I don't know all the details from all the methods.

I believe it would be ideal to access all the methods' options through a consistent interface, however this is pretty challenging due to the conceptual differences of the methods.

Another simple option would be that you add help pages for all the methods, quoting back the full help page of the given function to indicate the various options available to it. You already did kind of that (a simplified help page though) for quanTIseq and Timer where there are separate help documents. The options from each method are kind of already available through your interface through the .... There are nevertheless still two issues that need to be considered:

• Some of the options of a method are defined based on your other parameters (e.g. for EPIC the referenceoption is depending on your tumor option), so if both options are given, it should be the option specific to the original method that should be used.

• If a method gets updated with new parameters, you'd then have to also update manually the corresponding help document. Possibly bigger changes in the functions might also need other adaptations to your parameter mapping between the methods.

  • Maybe instead of rewriting the full help page, you could also just give a link to the help document of the function found directly in it's package of origin (when such a help pages exists). Maybe in such a case, you should indicate somewhere the mapping that you did between the parameters such that one knows which parameters of the function of origin are already set by your method.

Cheers,

Julien

[grst]

Hi Julien,

I'm sorry, I didn't manage to update the package during the conference and I'm off to a bikepacking trip for the next four weeks...

But I'll take care of this as soon as I'm back!

[grst]

Hi @jracle85,

I added the missing cell types and am working on improving the documentation in #19.

• I would rather not include EPIC's "otherCells" as an own cell type, because technically it isn't and doesn't really fit in the cell-type tree. The fraction is already available as 1-sum(other_cells) and that way it's consistent with quanTIseq. Instead I would dedicate a new section in the documentation to "absolute scores" explaining that. Is that ok with you?

• It would be great if you could have a quick look at the new cell type mapping. I now added the Neutrophils and also the remaining immune cell types of xCell. Here is the mapping as a google doc, where you could add comments directly.

And here is the new cell-type tree:

[grst]

As a good deal of the re-mapping deals with xCell signatures, maybe @dviraran could have a quick look at the mapping as well?

• The mapping is in this google doc
• Sheet1 defines the cell type tree above
• Sheet2 defines how the signatures of the different methods are mapped to nodes in the cell-type tree.

Your help would be greatly appreciated!

Cheers,
Gregor

[jracle85]

Hi @grst,

Thanks for the update.

It would be great if you could have a quick look at the new cell type mapping. I now added the Neutrophils and also the remaining immune cell types of xCell. Here is the mapping as a google doc, where you could add comments directly.

The mapping for EPIC cell types seems fine. I didn't check for xCell.

I would rather not include EPIC's "otherCells" as an own cell type, because technically it isn't and doesn't really fit in the cell-type tree. The fraction is already available as 1-sum(other_cells) and that way it's consistent with quanTIseq. Instead I would dedicate a new section in the documentation to "absolute scores" explaining that. Is that ok with you?

I agree that you could find it back by 1 - sum(other_cells). The only drawback of not including a value directly for these in the results, is that I fear that people with less experience working with EPIC or that didn't read the documentation in detail, would miss the point that we are also returning this value, which is a meaningful value (often corresponding directly to the fraction of cancer cells). It is not straightforward that we can obtain the remaining cells by 1 - sum(other_cells): some of the other methods return values for various cell types but without this equation to be meaningful. So, if the value appears in the results directly, then the user will see there is this cell type and if he doesn't understand what it means he can simply check the documentation. But if it is not visible, he will not necessarily think that he could do such a transformation to get the cell fractions from an additional cell type.

If you nevertheless still rather don't include this cell type in the results, please make sure that the documentation clearly states that we can obtain also the fraction of the remaining cells with above's equation.

Thanks. Best,

Julien

[grst]

Hi Julien,

I re-thought this and think you are right. Other cells are now included for both EPIC and quanTIseq in the main result. Wasn't too complicated anyway.

#19 will be merged today, and then it's available.

Cheers and thanks for your input,
Gregor

[grst]

Close #14 per #19.