Example notebooks

[st--]

I've played around a bit with the example notebooks. This isn't intended for merging as-is (if anything, it probably makes sense to split it into one PR per notebook), just to see where I've gone so far - happy to hear your thoughts! Heavily builds on top of @theogf's examples branch.

Following #303 and #314, examples are now split into separate PRs:

• Gaussian process priors -> Example notebook: Gaussian process priors #316
• training kernel parameters -> Example notebook: training kernel parameters #317
• kernel ridge regression -> Example notebook: kernel ridge regression #320
• support vector machine -> Example notebook: support vector machine #321
• deep kernel learning -> Example notebook: deep kernel learning #322 (maybe move this into AbstractGPs.jl?)

[devmotion]

Regarding the general structure, I think we should copy the configuration in AbstractGPs. I assume it would also make sense to move examples that are based or require AbstractGPs to the AbstractGPs repo.

[devmotion]

Generally, I think it would be nice to use a separate environment and hence a separate Project.toml + Manifest.toml for probably each of the examples such that they are easily reproducible and the dependencies are separated clearly. One would have to change this in AbstractGPs as well.

[devmotion]

IMO the example should be moved to AbstractGPs.

[st--]

Happy to move this one, I think it's a bit less about kernels themselves. @theogf as this was yours, would like to hear what you think before removing it here ?

[theogf]

I replied more generally on the main comments.

[devmotion]

Suggested change

# -*- coding: utf-8 -*-

[st--]

It's from jupytext. I'd like to see if it's possible to get Literate.jl and jupytext to play along 100% instead of just 98%... fredrikekre/Literate.jl#131

[devmotion]

Even if it is supported, I would prefer the standard Literate syntax. Otherwise one either ends up with a mix of different styles or has to pay attention to use the non-standard jupytext formatting (at least non-standard in this environment). To me it seems this would add more confusion but not provide any benefits. If you want to play around and tune the code, I can recommend VSCode or the outdated Juno environment which support executing single blocks and lines of literate files automatically in the REPL.

[st--]

but not provide any benefits

We can discuss trade-offs, but I don't think this claim is fair; I've explained my motivation on the Literate.jl issue I linked. Having the example notebooks in a format that can directly be edited interactively in jupyter notebooks makes it easier to contribute notebooks for people who are comfortable with that workflow (like me, but it's a widely popular workflow), without requiring people to use VSCode (which is a lot harder to get into than firing up a jupyter notebook).

[devmotion]

My impression is that in the Julia community the common workflow for writing Literate or Weave documents is to work with Juno or VSCode, to copy and run code in a REPL on the side, or rebuild the output with Literate and Weave intermittently. I think it's fine to support additional workflows but I think

• it should be officially support by Literate and Weave
• it should not impact the other apparently more common scenarios.

From the issue I get the impression that there won't be an official support for jupytext in the near future. And even if it is added, the second point seems still problematic. It seems jupytext requires a different syntax that deviates from the official syntax for Literate and Weave - if someone edits the document with jupytext the syntax of the document would have to change and so we might end up with a mix of the regular and the jupytext syntax. This seems confusing.

[devmotion]

I have never seen the notation \cdot', IMO k(\cdot, \cdot) already indicates that k is a function of two distinct arguments.

[theogf]

Suggested change

	# A Gaussian process (GP) is defined by its mean function $m(\cdot)$ and its covariance function or kernel $k(\cdot, \cdot')$:
	# A Gaussian process (GP) is defined by its mean function $m(x)$ and its covariance function or kernel $k(x, x')$:

[theogf]

About the GP example(s) here are my thoughts.
It would be nice to have the following:

• One being Gaussian Process regression but where we do not use AbstractGPs at all (although we could mention it of course), we just reformulate the theory, maybe we could follow exactly the Rasmussen and Williams algorithm. We could additionally merge this example with the kernel parameters optimization
• One with the automated statistician where the kernel selection is made automatically. I think it's really fine to depend on AbstractGPs when the focus is on the kernels really.
• Same thing with the deep kernel learning, the GP layer is not the interesting one, I don't think depending on AbstractGPs make it necessary to move it to other docs

[theogf]

Should this be removed?

[theogf]

Suggested change

	# A Gaussian process (GP) is defined by its mean function $m(\cdot)$ and its covariance function or kernel $k(\cdot, \cdot')$:
	# A Gaussian process (GP) is defined by its mean function $m(x)$ and its covariance function or kernel $k(x, x')$:

[devmotion]

Regarding the AbstractGPs discussion, I still think that examples that use AbstractGPs should be added to the other repo or not use AbstractGPs if it is not the main focus. Mainly, because examples that use AbstractGPs would be interesting for users of AbstractGPs but would not show up when they go to the documentation of the package. While probably kernels are most commonly used with GPs, it seems more natural and easier to find for users if the examples in KernelFunctions are focused on, well, kernel functions and the ones in AbstractGPs on GPs.

[theogf]

Regarding the AbstractGPs discussion, I still think that examples that use AbstractGPs should be added to the other repo or not use AbstractGPs if it is not the main focus. Mainly, because examples that use AbstractGPs would be interesting for users of AbstractGPs but would not show up when they go to the documentation of the package. While probably kernels are most commonly used with GPs, it seems more natural and easier to find for users if the examples in KernelFunctions are focused on, well, kernel functions and the ones in AbstractGPs on GPs.

I get your point but it seems also absurd to restrict ourselves to only non-GP examples.
For example for the automated kernel selection, the fact that a GP is used is a minor point.

If this is a problem this is where the general website we set up could be useful. We could have these more complex examples there and mention them in the docs of each package.

[st--]

Hi, it's been a while but I'd still like to get more & better examples into the KernelFunctions docs :) what do you think would be the best way forward ?

[theogf]

Hi, it's been a while but I'd still like to get more & better examples into the KernelFunctions docs :) what do you think would be the best way forward ?

Nice to see you back! The main problem is the discussion on which examples should belong here, or AbstractGPs. TBH I think it's a bit pointless to discuss this before the tutorials are even finished 😆 . We should first finish them and then decide where they should go

I guess you can start by adding the review suggestions and try to get the Documentation action to pass so we can have previews :)

[st--]

Closing this as it's now split up into individual branches and PRs for each example (#316, #317, #320, #321, #321).