pyspark kernel created using sparkmagic is not showing in the kernel list of jupyter extension in vs code

[GaryLiuTelus]

Environment data

• VS Code version: 1.62.2
• Jupyter Extension version (available under the Extensions sidebar): v 2021.10.110
• Python Extension version (available under the Extensions sidebar): v. 2021.10.11422169775
• OS (Windows | Mac | Linux distro) and version: Windows 10 Enterprise 20H2 build 19042.1288
• Python and/or Anaconda version: python 3.8.8, Anaconda 3 (64 bits)
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): NA
• Jupyter server running: Local | Remote | local

Expected behaviour

All kernel visible/working in Conda Jupyter Notebook should be the same in VS code jupyter extension

Actual behaviour

pyspark kernel installed using sparkmagic did not show in vs code jupyter extension kernel list, even it worked well with Conda Jupyter Notebook and it showed with command of jupyter kernelspec list.

Steps to reproduce:

1. Install sparkmagic extension (instruction: https://github.com/jupyter-incubator/sparkmagic)
2. test in Jupyter Notebook, and it should be available in the kernel list
3. Open vs code, choose kernel, it is not showing in the list.

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. XXX

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

XXX

[rchiodo]

Thanks for the bug. This isn't working because the 'python' specified by the kernelspec can't be found so we don't know how to launch it.

This code here is the culprit:

vscode-jupyter/src/client/datascience/kernel-launcher/localPythonAndRelatedNonPythonKernelSpecFinder.ts

Line 168 in 3f523db

const matchingInterpreter = await this.findMatchingInterpreter(item.kernelSpec, interpreters);

Kernel spec looks like this:

{
  "argv": [
    "python", <-- This is ambiguous so we cannot tell how to start this kernel
    "-m",
    "sparkmagic.kernels.pysparkkernel.pysparkkernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "PySpark",
  "language": "python"
}

We might be able to change this to look for an interpreter that has that module installed (and then we can be sure that launching it will work).

Or you can change the kernelspec created such that it points to the correct virtual environment that has spark magic installed.

For me it works if I change it to this:

{
  "argv": [
    "D:\\Source\\Testing_SparkMagic\\.venv\\Scripts\\python.exe", <-- Full path to python from venv that has sparkmagic installed
    "-m",
    "sparkmagic.kernels.pysparkkernel.pysparkkernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "PySpark",
  "language": "python"
}

You might also log a request on sparkmagic to have their kernelspec have the full path to python in it.

[GaryLiuTelus]

It worked for me.
Thank you very much for your help!

[greazer]

One idea. List all the kernels found, but if it's ambiguous as to which interpreter to use, then ask the user to choose which one to use.

[DonJayamanne]

I'm closing this issue as we've resolved the issues around displaying custom kernels (in the latest dev branch, soon to be released).
If the issue persists, please feel free to re-open this issue or create a new one and we'll be happy to look into them.

[fkamp]

I'm closing this issue as we've resolved the issues around displaying custom kernels (in the latest dev branch, soon to be released). If the issue persists, please feel free to re-open this issue or create a new one and we'll be happy to look into them.

@DonJayamanne, I still faced this issue in this version:
VSCode 1.71.2
Jupyter Extension v2022.8.1002431955
Python Extension v2022.14.0

This was missing until I updated "python" to the full path

{
  "argv": [
    "python",
    "-m",
    "aws_glue_interactive_sessions_kernel.glue_pyspark.GlueKernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Glue PySpark",
  "language": "python"
}

Interestingly, this works fine without it:

{
  "argv": [
    "python",
    "-m",
    "aws_glue_interactive_sessions_kernel.glue_spark.GlueKernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Glue Spark",
  "language": "scala"
}

[jpmaterial]

If anyone is wondering where to find the kernelspec: jupyter kernelspec list gives a list of all directories specifying kernels, which will contain kernel.json files.