Organize the Examples Folder

[awaelchli]

Outline & Motivation

Our examples folder is messy, it currently looks like this: https://github.com/Lightning-AI/lightning/tree/master/examples
It does not matter if you are an expert or a beginner, if you arrive here in this repository, the examples folder sticks out as an obvious place to start exploring. But it's a mess. Where to start? What is important? What is new, what is old? The Readme is outdated.

Pitch

Organize the top level folders into:

examples/
    app
    fabric
    pytorch

Organize the readme, either at the top level or in each subfolder, with sections for

Beginner
Intermediate
Expert

(or a similar structure organized by concepts).

Delete examples that are too outdated or irrelevant. Make everything as concise as possible.

Additional context

No response

cc @Borda @justusschock @awaelchli

[Borda]

that sounds good to me, just would split the app into app and component
or we can still utilize the prefix to keep it rather flat?

examples/
    app_...
    component_...
    fabric_...
    pytorch_...

cc: @lantiga @Lightning-AI/core-lightning

[lantiga]

I think we should get rid of the distinction between app and component, every component should be runnable in isolation (as a simple app), so I'd just keep app

[Borda]

I think we should get rid of the distinction between app and component, every component should be runnable in isolation (as a simple app), so I'd just keep app

ok, but go with folders or flat and prefix? 🦦

[lantiga]

Just like @awaelchli suggested

examples/
    app
    fabric
    pytorch

inside app, just have the current app examples (no need to further split between app and component, that was my suggestion)

[andrewaf1]

Would be happy to take this one.

[awaelchli]

@andrewaf1 the stage is yours :)

[Borda]

@andrewaf1, how is it going, can I help somehow? :)

[andrewaf1]

So, I've been going through everything, making sure everything makes sense, is relevant, runs according to the readme instructions, etc.

I wanted to clarify what is going to go into the "pytorch" folder. Is that supposed to be the examples with a "pl_" prefix (in which case the folder should be called something else, right?) Or is that intended to contain the plain pytorch examples that are bundled with the fabric examples?

[andrewaf1]

And another thing that has me a little confused. One of the examples throws the error:

Traceback (most recent call last):
  File "/home/andrew/lightning/examples/pl_basics/autoencoder.py", line 26, in <module>
    from pytorch_lightning import callbacks, cli_lightning_logo, LightningDataModule, LightningModule, Trainer
ImportError: cannot import name 'callbacks' from 'pytorch_lightning' (unknown location)

I can do from lightning.pytorch import callback successfully. I feel like I'm not up to speed on something here.

[awaelchli]

@andrewaf1 Good questions. Yes you are right, with the pytorch folder we mean anything that was previously prefixed with pl_. The PyTorch Lightning examples. The Lightning package, as it currently is on master, will have these three modules:

lightning.app: The app framework
lightning.pytorch: What was previously pytorch_lightning
lightning.fabric: The new Fabric code base

This is what motivates the new example structure I proposed. The examples/fabric folder already follows that pattern, so this, and the pure torch files inside of it, can be left as they are.

I can do from lightning.pytorch import callback successfully. I feel like I'm not up to speed on something here.

This is simply because the imports in the examples haven't been updated yet. from pytorch_lightning import should become from lightning.pytorch import x. This will get done in #16615.

[Borda]

@andrewaf1, how is it going here? :)

[aniketmaurya]

hey @Borda, please assign this issue to me. Would love to work on this!