Magics support %run and %import_ipynb

[flowy0]

Hi,

Can i check if there is support for %run and %import_ipynb in the Pyspark kernel?

[dosubot[bot]]

🤖

Hello @flowy0! 👋

I'm Dosu, a friendly bot here to lend a hand while the human maintainers are away. I'm like your digital sidekick, helping you squash bugs, answer questions, and become a contributor.

So, while we wait for a human to join us, feel free to toss any questions or issues my way! 🚀

Based on the current version of sparkmagic (0.21.0), it does not support the %run magic command in the PySpark kernel. The RemoteSparkMagics class, which handles the magic commands, only supports the following subcommands: info, config, add, delete, cleanup, logs, and run. The run subcommand is used to execute Spark code against a session, not to run other notebooks or scripts like the %run command in IPython. You can refer to the source code for more details.

As for the %import_ipynb magic command, there is no support for it in the PySpark kernel for the current version of sparkmagic (0.21.0). The SparkMagicBase class provides the %spark magic command and methods for executing Spark commands and SQL queries, sending data to Spark, and displaying session information. There is no mention or implementation of the %import_ipynb magic command. You can check the source code for more information.

Please let me know if you need further clarification or have any other questions.

Sources

• sparkmagic/sparkmagic/magics/remotesparkmagics.py
• sparkmagic/sparkmagic/magics/sparkmagicsbase.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[devstein]

Hey @flowy0 Dosu is right, but its explanation isn't slightly off.

Sparkmagic is a wrapper around the Apache Livy REST API. All executed notebook cells are sent to the the Spark cluster via the Apache Livy's REST API, which means all cells are executed in the context of the Spark cluster.

So in theory, %run and %import_ipynb are supported, but they run in the context of the Spark Executor machine. This means you cannot reference any local files that aren't also available on the same path on the Spark Executor.

I don't know what you're Spark cluster setup is, but this is often more complex than it's worth, so the short answer is no 😄

Hope this helps!