Adding files from scib

[mumichae]

Scripts added:

• scripts/preprocessing.py
• scripts/run_normalize.sh
• scripts/sim_norm.py
• R/visualization/*
• notebooks/*

Corresponding PRs:

• scib: Moving files to scib-pipeline and scib-reproducibility scib#217
• scib-pipeline: Pruning files from old repo scib-pipeline#1

[LuckyMD]

ideally add the R/visualization stuff already now. But fine to do it in stages as well.

[mumichae]

@martaint Is it safe to move all the R/visualization code here or are you still working on it?

[martaint]

I think it's safe! I'm not working on it :)

…

On Sat, 6 Mar 2021, 06:05 Michaela Mueller ***@***.***> wrote: @martaint <https://github.com/martaint> Is it safe to move all the R/visualization code here or are you still working on it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHR32OQTFPRXYMLCF3BZLB3TCGZX7ANCNFSM4YVK5JYQ> .

[lazappi]

Not sure where this blank file came from but can probably delete it.

[lazappi]

Wow, that's a lot of notebooks! I assume somebody has already checked these are all still relevant?

• Do we need the .ipynb_checkpoints directories?

Should we move the data files in the visualization/ directory to the top data/ directory? At least those that are for the paper not just examples for the plotting functions (@martaint might be able to help with that).

[mumichae]

Wow, that's a lot of notebooks! I assume somebody has already checked these are all still relevant?

We removed notebooks some time back (PR), so I think things should be OK. @LuckyMD @danielStrobl Could you double-check if there are any notebooks we don't need anymore?

* Do we need the `.ipynb_checkpoints` directories?

Good point, didn't realise that I was adding those. They should be removed and ignored now.

Should we move the data files in the visualization/ directory to the top data/ directory? At least those that are for the paper not just examples for the plotting functions (@martaint might be able to help with that).

I agree, that would be better. If we move the plot data, we just need to update the R scripts accordingly.

[LuckyMD]

Things I think might be removable:

1. The empty file @lazappi highlighted
2. Mouse atlas notebooks (@danielStrobl and @mumichae)
3. notebooks/Visualization_ATAC.ipynb from @kridsadakorn
4. notebooks/silhouette_graph.ipynb, notebooks/trajectory_plots and notebooks/visualization.ipynb from @danielStrobl
5. visualization/atlas_heatmap.ipynb from @mumichae

Would love to hear if you think these can be removed or if this is important for reproducibility. If important, then maybe we need a further folder.

[kridsadakorn]

notebooks/Vitualisation_ATAC.ipynb can be deleted because the plots were replaced by Daniel's version.

Naively, I tried to find a way to comment right under this file, but I couldn't figure out how to do.

PS. there is even a typo in the file name, poor me ...

[danielStrobl]

The mouse atlas notebooks can be removed I think. I would keep the trajectory_plots and visualisation notebooks as the figures for the supplementary were generated with those. The silhouette graph notebook can be removed.

[danielStrobl]

One thing I noticed is that the notebooks/visualisation.ipynb in master is not the most recent one, it's actually in the recompute_cluster_metrics branch

[LuckyMD]

Thanks for comments @danielStrobl. Could you say how you would name the folder (or in which existing folder) the notebooks to keep should be put? Just visualization?

[danielStrobl]

Yes, sounds good

[lazappi]

One thing I noticed is that the notebooks/visualisation.ipynb in master is not the most recent one, it's actually in the recompute_cluster_metrics branch

@danielStrobl Can you open a separate PR for the latest version (or add it here)? Or open an issue I guess, just don't want to forget about this.

P.S. I think you have a pending invite to join this repo, I can resend if you need.

[danielStrobl]

Ok, I'll open a PR for that. That would be great if you could resend the invite.

[mumichae]

I agree that the heatmap and mouse atlas notebooks can be removed.

[mumichae]

I removed the heatmap plotting script (duplicate) and moved the preprocessing scripts to the Masterinternship_2019 repository.

[LuckyMD]

also, new analysis notebook updates from scIB.

[lazappi]

Summarising things that have come up in comments:

Done

• Resend invite to @danielStrobl
• Remove .ipynb_checkpoints directories
• Remove mouse atlas notebooks
• Remove empty visualization/img/init file
• Remove `notebooks/Vitualisation_ATAC.ipynb
• Remove visualization/atlas_heatmap.ipynb
• Remove notebooks/silhouette_graph.ipynb
• Move visualisation notebooks to notebooks/visualization/

Todo

• Move the visualisation data files to the data/ directory (this probably means other changes to scripts so maybe can be a later PR)`
• Update notebooks/visualisation.ipynb to most recent version
• Other notebook updates from scIB?
• Add content to empty dataset READMEs

Anything that's incorrect or I missed?

Updated 2021-04-22

[lazappi]

Also just noticed that each of the dataset notebook folders has an empty README. We should probably either add some info about the datasets or just delete them. @LuckyMD what do you think?

[LuckyMD]

Could you copy over some text from the dataset description in the paper to the README?

[lazappi]

Could you copy over some text from the dataset description in the paper to the README?

That's definitely possible. Alternatively could describe what is in the notebooks (or both).

[LuckyMD]

Or maybe better: everyone could write a small blurb (3-4 sentences of what is in the dataset). I think that could even be from the SI sections.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[lazappi]

Did some of the minor things, see updated checklist #4 (comment). Main things left to do are double checking notebooks are up to date (@danielStrobl can you help with that?) and deciding what to do with the READMEs.

[martaint]

@lazappi I added a file in scib (theislab/scib#234) which should also be added here in the respective notebook folder. Could you take care of that? Don't want to mess with this PR :) thanks a lot!