`jupyter`: warn when kernel apparently doesn't support inline expressions (was: Stata inline code error)

[hugetim]

Bug description

My attempt to use Stata inline code results in an apparent Python error.

Steps to reproduce

Sample doc:

---
title: "Stata inline code"
format: html
jupyter: nbstata
---

```{stata}
disp 3
```

Testing inline `{stata} disp 3`.

I attempt to create a preview from within VSCode and get the following terminal output:

Starting nbstata kernel...Done

Executing 'stata-inline.quarto_ipynb'
  Cell 1/1: ''...Done
ERROR: 

'NoneType' object has no attribute 'get'

Expected behavior

No response

Actual behavior

No response

Your environment

• IDE VSCode 1.94.0 (with Quarto extension v1.115.0)
• OS: Windows 11

Quarto check output

Quarto 1.5.57
[>] Checking versions of quarto binary dependencies...
      Pandoc version 3.2.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
      Typst version 0.11.0: OK
[>] Checking versions of quarto dependencies......OK
[>] Checking Quarto installation......OK
      Version: 1.5.57
      Path: C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\bin
      CodePage: 1252

[>] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[>] Checking LaTeX....................OK
      Tex:  (not detected)

[>] Checking basic markdown render....OK

[>] Checking Python 3 installation....OK
      Version: 3.10.12 (Conda)
      Path: C:/Users/tjhuegerich/AppData/Local/anaconda3/envs/nbstata_demo/python.exe
      Jupyter: 5.3.1
      Kernels: nbstata, python3

[>] Checking Jupyter engine render....OK

[>] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

[cscheid]

Because this is a proprietary kernel, we can't really help much with its debugging. But you could use the advice here.

[hugetim]

Thank you. I hadn't seen that section of the docs. Unfortunately, the debug mode didn't add anything helpful---the closest thing is an additional line that says "- Error response received".

What might be helpful for me is the full traceback for the apparent Python error 'NoneType' object has no attribute 'get'

(I actually maintain the open source nbstata kernel. It calls the proprietary 'pystata' under the hood, but you can see how it does that here: https://github.com/hugetim/nbstata/blob/master/nbstata/kernel.py. edit: Oh, but now I see the problem is that you can't reproduce the issue. I'll see if I can reproduce with an "echo" kernel.)

Here's the temporary notebook file Quarto creates:

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "title: \"Stata inline code\"\n",
        "format: html\n",
        "jupyter: nbstata\n",
        "execute:\n",
        "  debug: true\n",
        "---"
      ],
      "id": "8c10f86e"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "disp 3"
      ],
      "id": "455eb13a",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Testing inline `{stata} disp 3`.\n"
      ],
      "id": "ba4bbe67"
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "nbstata",
      "language": "stata",
      "display_name": "Stata (nbstata)",
      "path": "C:\\Users\\tjhuegerich\\AppData\\Roaming\\jupyter\\kernels\\nbstata"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}

And here's Quarto's jupyter log output:

2024-10-07 15:57:34,221 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:273 - starting notebook server subprocess
2024-10-07 15:57:36,578 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:278 - running notebook server subprocess
2024-10-07 15:57:36,579 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:106 - creating notebook server (tcp: C:\Users\tjhuegerich\AppData\Local\quarto\jt\468ba56d00cf61952166)
2024-10-07 15:57:36,582 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:132 - notebook server bound to port 57715
2024-10-07 15:57:37,118 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:34 - handling server request
2024-10-07 15:57:37,130 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:65 - executing notebook
2024-10-07 15:57:37,137 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:164 - inside notebook_execute
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:186 - notebook was read
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:434 - {
  "name": "nbstata",
  "language": "stata",
  "display_name": "Stata (nbstata)",
  "path": "C:\\Users\\tjhuegerich\\AppData\\Roaming\\jupyter\\kernels\\nbstata"
}
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:445 - No stata directory found in C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\lang\stata
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:446 - Will look for explicit quarto setup cell information in kernelspec dir
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:453 - No quarto_setup_cell file found in C:\Users\tjhuegerich\AppData\Roaming\jupyter\kernels\nbstata
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:454 - C:\Users\tjhuegerich\AppData\Roaming\jupyter\kernels\nbstata\quarto_setup_cell
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:134 - not using cache
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:208 - Will attempt to create notebook
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:210 - type of notebook: <class 'nbformat.notebooknode.NotebookNode'>
2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:381 - Creating NotebookClient
2024-10-07 15:57:43,348 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:222 - NotebookClient created
2024-10-07 15:57:43,348 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 0
2024-10-07 15:57:43,352 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:507 - cell_execute with eval=True
2024-10-07 15:57:43,352 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:293 - Executed cell 0
2024-10-07 15:57:43,355 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:309 - Handling quarto metadata
2024-10-07 15:57:43,356 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:310 - {
  "id": "d99f783b",
  "cell_type": "code",
  "metadata": {},
  "execution_count": 1,
  "source": "",
  "outputs": []
}
2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 1
2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:293 - Executed cell 1
2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 2
2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:507 - cell_execute with eval=True
2024-10-07 15:57:45,431 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:293 - Executed cell 2
2024-10-07 15:57:45,431 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:328 - Done
2024-10-07 15:57:45,431 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 3

[hugetim]

After setting QUARTO_PRINT_STACK, I can see this traceback now:

'NoneType' object has no attribute 'get'
ERROR: Error
    at renderFiles (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:78081:29)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async render (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:82929:21)
    at async Command.actionHandler (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:83077:32)
    at async Command.execute (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:8017:13)
    at async Command.parseCommand (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:7907:20)
    at async quarto (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118224:9)
    at async file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118244:9
    at async mainRunner (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118128:9)
    at async file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118235:5

[cscheid]

(I actually maintain the open source nbstata kernel. It calls the proprietary 'pystata' under the hood, but you can see how it does that here: https://github.com/hugetim/nbstata/blob/master/nbstata/kernel.py. edit: Oh, but now I see the problem is that you can't reproduce the issue. I'll see if I can reproduce with an "echo" kernel.)

Hey, nice to meet you 👋 ! I'm sorry Quarto isn't working well in this scenario.

The code we use to execute inline expressions in a kernel is here:

quarto-cli/src/resources/jupyter/notebook.py

Lines 544 to 602 in 0452446

	def cell_execute_inline(client, cell):
	# helper to raise an error from a result
	def raise_error(result):
	ename = result.get('ename')
	evalue = result.get('evalue')
	raise Exception(f'{ename}: {evalue}')
	# helper to clear existing user_expressions if they exist
	def clear_user_expressions():
	if "metadata" in cell:
	metadata = cell.get("metadata")
	if "user_expressions" in metadata:
	del metadata["user_expressions"]
	# find expressions in source
	language = client.nb.metadata.kernelspec.language
	source = ''.join(cell.source)
	expressions = re.findall(
	fr'(?:^|[^`])`{{{language}}}[ \t]([^`]+)`',
	source,
	re.MULTILINE
	)
	if len(expressions):
	# send and wait for 'execute' kernel message w/ user_expressions
	kc = client.kc
	user_expressions = dict()
	for idx, expr in enumerate(expressions):
	user_expressions[str(idx).strip()] = expr
	msg_id = kc.execute('', user_expressions = user_expressions)
	reply = client.wait_for_reply(msg_id)
	# process reply
	content = reply.get('content')
	if content.get('status') == 'ok':
	# build results (check for error on each one)
	results = []
	for key in user_expressions:
	result = content.get('user_expressions').get(key)
	if result.get('status') == 'ok':
	results.append({
	'expression' : user_expressions.get(key),
	'result': result
	})
	elif result.get('status') == 'error':
	raise_error(result)
	# set results into metadata
	if not "metadata" in cell:
	cell["metadata"] = {}
	cell["metadata"]["user_expressions"] = results
	elif content.get('status') == 'error':
	raise_error(content)
	else:
	clear_user_expressions()
	# return cell
	return cell

Concretely,

• client is an instance of NotebookClient from the nbclient Python library.
• we call execute with an empty statement and a non-empty user_expressions parameter

From there, you should be able to create a minimal example using only nbclient in your own kernel to test things more directly. Let me know if you need further info.

[cscheid]

@hugetim If you want to debug this further, I'd also highly recommend you install a development build of Quarto, which will give you better stack traces because we don't run a javascript bundle in the dev builds.

[cscheid]

If it helps further, it seems that user_expressions is implemented in the ipykernel as such:

https://github.com/ipython/ipykernel/blob/8d60e674db5c10738b65ced587aa76242f693dfd/ipykernel/ipkernel.py#L472-L476

[hugetim]

If it helps further, it seems that user_expressions is implemented in the ipykernel as such:

https://github.com/ipython/ipykernel/blob/8d60e674db5c10738b65ced587aa76242f693dfd/ipykernel/ipkernel.py#L472-L476

That is extremely helpful. I see I need to add support for user_expressions to the kernel, and that should be straightforward.

My guess is that nbstata's lack of support for user_expressions results in the error occurring here, trying to call 'get' on result, which will be None:

quarto-cli/src/resources/jupyter/notebook.py

Lines 581 to 583 in 174d6ff

	for key in user_expressions:
	result = content.get('user_expressions').get(key)
	if result.get('status') == 'ok':

It may be helpful for others to check whether 'result' is None, raising an informative error if not.

Might I also suggest adding something about inline expressions' use of user_expressions to the docs here and/or adding basic "echo" user_expressions handling to the model Quarto kernel here?

[cscheid]

Might I also suggest adding something about inline expressions' use of user_expressions to the docs here and/or adding basic "echo" user_expressions handling to the model Quarto kernel here?

We definitely need that. Can you create a documentation issue here and a bug report on quarto_echo_kernel about this? Thanks.