Merge pull request #96 from ISISComputingGroup/scripting_python_edit

Copyedit scripting & python sections

Author: rerpha

doc/Scripting.md

@@ -1,7 +1,7 @@
 # Scripting (Python)
 
 ```{note}
-These pages document user-facing scripting workflows. Looking for developer information about python? See [Python](system_components/Python).
+These pages document python libraries or workflows which users may use directly from their scripts. Looking for information about how Python is used in the IBEX backend? See [Python](system_components/Python).
 ```
 
 ```{toctree}

doc/conf.py

@@ -100,6 +100,7 @@
     "client/coding/Migrating-or-adding-a-button-to-the-E4-perspective-switcher": "Perspectives.html",  # noqa E501
     "client/coding/Adding-a-Button-to-the-Perspective-Switcher": "Perspectives.html",
     "client/testing/Using-Mockito-for-Testing-in-the-GUI": "Mockito.html",
+    "scripting/uv": "../system_components/python/Python-venvs.html",
 }
 
 intersphinx_mapping = {
@@ -108,4 +109,7 @@
     # If it does, then fall back to local inventory. This local inventory should be
     # updated periodically.
     "EPICS": ("https://docs.epics-controls.org/en/latest/", (None, "epics-controls.inv")),
+    "genie_python": ("https://isiscomputinggroup.github.io/genie/", None),
+    "ibex_bluesky_core": ("https://isiscomputinggroup.github.io/ibex_bluesky_core/", None),
+    "matplotlib": ("https://matplotlib.org/stable/", None),
 }

doc/scripting/Bluesky-scanning.md

@@ -1,17 +1,17 @@
 # Bluesky
 
 For information about scanning using bluesky, including information about:
-- The bluesky run engine, `RE`
-- Bluesky ophyd-async devices (for example `BlockR`, `BlockRw`, `BlockRwRbv`, `ReflParameter`, or `SimpleDae`)
+- The bluesky run engine, {external+ibex_bluesky_core:py:obj}`RE <ibex_bluesky_core.run_engine.get_run_engine>`
+- Bluesky ophyd-async devices (for example {external+ibex_bluesky_core:py:obj}`BlockR <ibex_bluesky_core.devices.block.BlockR>`, {external+ibex_bluesky_core:py:obj}`BlockRw <ibex_bluesky_core.devices.block.BlockRw>`, {external+ibex_bluesky_core:py:obj}`BlockRwRbv <ibex_bluesky_core.devices.block.BlockRwRbv>`, {external+ibex_bluesky_core:py:obj}`ReflParameter <ibex_bluesky_core.devices.reflectometry.ReflParameter>`, or {external+ibex_bluesky_core:py:obj}`SimpleDae <ibex_bluesky_core.devices.simpledae.SimpleDae>`)
   * If you are looking at bluesky code which uses `async def` and `await`, and imports `ophyd_async`, you're probably looking at a device.
 - Bluesky plans, (for example `scan`, `adaptive_scan`)
   * If you are looking at bluesky code with lots of `yield from` statements, you're probably looking at a plan.
-- Bluesky callbacks (for example `LivePlot`, `LiveFit`, `ISISCallbacks`, file-writing)
+- Bluesky callbacks (for example {external+ibex_bluesky_core:doc}`LivePlot <callbacks/plotting>`, {external+ibex_bluesky_core:doc}`LiveFit <callbacks/fitting/fitting>`, {external+ibex_bluesky_core:doc}`ISISCallbacks <callbacks/isiscallbacks>`, {external+ibex_bluesky_core:doc}`file-writing <callbacks/file_writing>`)
 
-**See [ibex_bluesky_core docs](https://isiscomputinggroup.github.io/ibex_bluesky_core).**
+**See {external+ibex_bluesky_core:doc}`index`.**
 
 ---
 
-For troubleshooting, see [ibex_bluesky_core troubleshooting guide](https://isiscomputinggroup.github.io/ibex_bluesky_core/dev/troubleshooting.html#). 
+For troubleshooting, see {external+ibex_bluesky_core:doc}`ibex_bluesky_core troubleshooting guide <dev/troubleshooting>`.
 
-Please add troubleshooting information to that guide (via a pull-request on `ibex_bluesky_core`) rather than this page.
+Please add troubleshooting information to that guide via a pull-request onto the [ibex_bluesky_core repository](https://github.com/ISISComputingGroup/ibex_bluesky_core/), rather than this page.

doc/scripting/Genie-Python.md

@@ -14,4 +14,4 @@ genie_python/*
 ### User documentation
 
 - {external+ibex_user_manual:doc}`Scripting in the user manual <Scripting>`
-- [genie_python reference](http://shadow.nd.rl.ac.uk/genie_python/sphinx/genie_python.html)
+- {external+genie_python:doc}`genie_python reference manual <genie_python>`

doc/scripting/Matplotlib.md

@@ -1,37 +1,38 @@
 # Matplotlib
 
 ```{seealso}
-{external+ibex_user_manual:doc}`scripting/Matplotlib` on the user manual.
+- {external+ibex_user_manual:doc}`scripting/Matplotlib` on the user manual
+- {external+matplotlib:doc}`index` (reference documentation)
 ```
 
-We allow plotting in IBEX via `matplotlib`. These plots will appear in the IBEX client.
+We allow plotting in IBEX via {external+matplotlib:doc}`matplotlib <index>`. These plots will appear in the IBEX client.
 
-As of [ticket 6719](https://github.com/ISISComputingGroup/IBEX/issues/6719), we are using a custom matplotlib backend. This backend comprises two main components:
-- The websocket backend in `genie_python\matplotlib_backend\ibex_websocket_backend.py`, which is based on matplotlib's `WebAgg` backend. This serves plots over a websocket connection.
-- The front-end code in `/uk.ac.stfc.isis.ibex.ui.graphing` in the GUI. This reads data from the websockets published by the backend. It is effectively a Java reimplementation of `WebAgg`'s front-end code, which is in `python\Lib\site-packages\matplotlib\backends\web_backend`.
+As of [ticket 6719](https://github.com/ISISComputingGroup/IBEX/issues/6719), we are using a custom {external+matplotlib:doc}`matplotlib backend <users/explain/figure/backends>`. This backend comprises two main components:
+- The websocket backend in `genie_python\matplotlib_backend\ibex_websocket_backend.py`, which is based on matplotlib's {external+matplotlib:py:obj}`WebAgg <matplotlib.backends.backend_webagg>` backend. This serves plots over a websocket connection.
+- The front-end code in the `uk.ac.stfc.isis.ibex.ui.graphing` plugin in the GUI. This reads data from the websockets published by the backend. It is effectively a Java reimplementation of {external+matplotlib:py:obj}`WebAgg <matplotlib.backends.backend_webagg>`'s front-end code.
 
 ## Python (backend) component
 
-Our custom backend is defined in `genie_python\matplotlib_backend\ibex_websocket_backend.py`. It uses/builds on `WebAggCore` under the hood. 
+Our custom backend is defined in `genie_python\matplotlib_backend\ibex_websocket_backend.py`. It uses/builds on {external+matplotlib:py:obj}`WebAggCore <matplotlib.backends.backend_webagg_core>` under the hood. 
 
 Fundamentally, it is a relatively simple backend which publishes a websocket. This websocket can send or receive:
 - Binary messages, send only. These are PNG-encoded frames which should be published by the frontend.
 - String messages, send & receive. These are encoded using JSON. They can be used, for example:
   * To notify the front-end that a new frame is available
   * To allow the front-end to request a new frame
   * To allow the front-end to resize a plot
-  * To allow the front-end to pan/zoom the plot, via mouse handlers. Note the front end simply sends mouse events to the backend, `WebAgg` then interprets these and redraws an appropriate plot.
+  * To allow the front-end to pan/zoom the plot, via mouse handlers. Note the front end simply sends mouse events to the backend, {external+matplotlib:py:obj}`WebAgg <matplotlib.backends.backend_webagg>` then interprets these and redraws an appropriate plot.
 
 We have modified the backend to:
 - Be non-blocking, so that scientists can update their plot and still continue running scripts in the background
-- To notify the IBEX client on calling `plt.show()`, so that IBEX can show the plot windows.
+- To notify the IBEX client on calling {external+matplotlib:py:obj}`matplotlib.pyplot.show`, so that IBEX can show the plot windows.
 
 
 ## GUI (java) component
 
-`WebAgg` is published by matplotlib as a browser-based backend. IBEX previously rendered plots in an embedded web browser view within an OPI, but this had significant reliability problems.
+{external+matplotlib:py:obj}`WebAgg <matplotlib.backends.backend_webagg>` is published by matplotlib as a browser-based backend. IBEX previously rendered plots in an embedded web browser view within an OPI, but this had significant reliability problems.
 
-We have since moved to reimplementing the javascript based frontend code in java, as it is very simple. When matplotlib is updated, we should verify that plots still work and update the protocol in the GUI if the underlying protocol in `WebAgg` has changed or been extended. However this should be relatively rare as WebAgg is designed to be embedded in other applications (per [mpl docs](https://matplotlib.org/stable/gallery/user_interfaces/embedding_webagg_sgskip.html)).
+We have since moved to reimplementing the javascript based frontend code in java, as it is very simple. When matplotlib is updated, we should verify that plots still work and update the protocol in the GUI if the underlying protocol in {external+matplotlib:py:obj}`WebAgg <matplotlib.backends.backend_webagg>` has changed or been extended. However this should be relatively rare as WebAgg is designed to be embedded in other applications (per {external+matplotlib:doc}`gallery/user_interfaces/embedding_webagg_sgskip`).
 
 The front-end receives binary messages from the backend, and displays the binary message content as a PNG in the user interface. The user interface also occasionally forces plots to be redrawn, so that any missed updates will not cause a plot to completely freeze/fail to update.
 

doc/scripting/genie_python/genie_python-Troubleshooting.md

@@ -4,7 +4,7 @@
 
 ### Where are the logs?
 
-Genie_python writes its logs to `C:\Instrument\var\logs\genie_python`.
+`genie_python` writes its logs to `C:\Instrument\var\logs\genie_python`.
 
 ### What does `ERROR: CAException` mean when it is in the log?
 
@@ -15,35 +15,54 @@ The following is written to the log when a virtual circuit disconnects from the
     2018-05-21T15:21:31	(CA)	(15712)	ERROR: CAException: type=6 state=192 op=0 file=..\getCopy.cpp lineNo=92
 ```
 
-The time stamp on these are for the first `get_pv` call or equivalent after a disconnect.
+The time stamp on these are for the first {external+genie_python:py:obj}`get_pv <genie.get_pv>` call or equivalent after a disconnect.
 
 ## Command problems
 
-### Can not set or get a block reports disconnected
+### Can not set or get a block; reports disconnected
 
 There can be multiple problems, check:
 
 1. Block exists
 1. Block is spelt correctly, use `b.` and autocomplete
-1. Try getting the underlying PV `g.get_pv("IN:<instrument>:CS:SB:<Block name?")`
-1. Restart the GUI genie_python console
+1. Try getting the underlying PV `g.get_pv("IN:<instrument>:CS:SB:<Block name>")`
+1. Restart the GUI Python console
 
-## Import problems
+## Problems finding a python interpreter
 
-### Can't find Python 3
+### Can't find the Uktena Python distribution
 
 If you get an error message similar to
 
 ```
 *** Cannot find GENIE-PYTHON 3 - some things are not likely to work ***
 ```
-on running `config_env.bat`, you need to ensure you have Python 3 available on your system.
+on running `config_env.bat`, you need to ensure you have the {doc}`/system_components/Python` installed on your system.
+You may need to follow the {doc}`/system_components/python/Building-and-installing-uktena` instructions.
 
-## Other Issues
+```{note}
+As of October 2025, many processes in the IBEX backend depend on the Uktena python distribution. However, we are
+gradually migrating these to use {doc}`python virtual environments </system_components/python/Python-venvs>`; the eventual goal is that Uktena will not be required
+for the IBEX server backend.
+```
+
+### No appropriate `venv` has been created for a backend process
 
-### Can not set change users
+We are gradually migrating our server-side processes to use {doc}`uv python environments </system_components/python/Python-venvs>`.
 
-Users seems not to get set properly using g.change_users, see ticket [5812](https://github.com/ISISComputingGroup/IBEX/issues/5812). Look into this it is more than a one off.
+These virtual environments are created on instruments [by the IBEX deployment script](https://github.com/ISISComputingGroup/ibex_utils/blob/b5998462ddd4d5aa4123e30104166043151cefea/installation_and_upgrade/ibex_install_utils/tasks/system_tasks.py#L146). If that step has previously been missed or failed during a deployment, it will need to be re-run.
+
+If you need to do this manually for one specific module, you may run:
+```
+rmdir /s /q .venv
+uv venv .venv
+uv pip sync requirements-frozen.txt
+```
+at the top-level of the relevant module. Modules using this approach will have a `requirements-frozen.txt`.
+
+On a developer machine, the virtual environments are created by the [developer update script](https://github.com/ISISComputingGroup/ibex_utils/blob/master/installation_and_upgrade/developer_update.bat); if you are missing virtual environments on your developer machine, you will need to re-run that script.
+
+## Other Issues
 
 ### Repeated error messages in console while waiting
 
@@ -55,9 +74,13 @@ If you get repeated errors of the form:
 ```
 You may need to restart the genie_python session. The root cause of this issue is currently unknown. See ticket [5893](https://github.com/ISISComputingGroup/IBEX/issues/5893) for details, including a script which can scan all instruments for occurrences of this issue. If this issue is seen again, please create a new ticket to investigate further and also link it here.
 
-### can read local PVs from instrument but not e.g. `CS:INSTLIST` or accelerator ones like beam current
+### Can read local PVs but not central PVs
 
-In one case this was due to the firewall rule for `A:\python3\python.exe` had been disabled - a process firewall exception is needed to allow it to receive the UDP name query reply 
+Central PVs could include:
+- `CS:INSTLIST` (the instrument list)
+- Accelerator PVs, for example beam current
+
+In one case, this was due to the firewall rule for `A:\python3\python.exe` which had been disabled - a process firewall exception is needed to allow it to receive the UDP name query reply.
 
 ### Can't import channel-access modules
 
@@ -89,7 +112,7 @@ When trying to import channel access libraries (`CaChannel`, `pcaspy`, `aioca`,
 
 It's also technically possible to replace `import pip` with:
 
-```
+```python
 import _distutils_hack
 _distutils_hack.remove_shim()
 import pip
@@ -101,4 +124,4 @@ But this is not recommended for obvious reasons.
 
 Pyright keeps a cache directory in `c:\Users\<user>\.cache\pyright-python`, this can get corrupted, if it does get corrupted pyright will entirely fail to execute. This cache directory can be deleted (at the cost of the next script-check operation being much slower).
 
-Error from `g.load_script` will be a `json.decoder.JSONDecodeError` as pyright does not return JSON in this case (but rather, returns some non-JSON error message).
+Error from {external+genie_python:py:obj}`g.load_script <genie.load_script>` will be a `json.decoder.JSONDecodeError` as pyright does not return JSON in this case (but rather, returns some non-JSON error message).

doc/scripting/genie_python/genie_python-linter.md

@@ -1,19 +1,28 @@
 # `genie_python` script checking (linting)
 
-On `g.load_script` genie_python now runs pylint on the scripts. This help the user see errors before the script is run. However sometimes this causes its own issues. There is a {external+ibex_user_manual:doc}`page in the user manual <scripting/Error-Checking-Troubleshooting>` describing what the user should do.
+On {external+genie_python:py:obj}`g.load_script <genie.load_script>` genie_python now runs pylint on the scripts. This help the user see errors before the script is run. However sometimes this causes its own issues. There is a {external+ibex_user_manual:doc}`page in the user manual <scripting/Error-Checking-Troubleshooting>` describing what the user should do.
 
-Here are some more tips that we don't necessarily want to encourage. 
+If a warning cannot be fixed, and you are sure the code is otherwise correct, warnings can be ignored from a line by adding a comment:
 
-Warnings can be removed from a line by adding a comment
+```python
+some_code()  # pylint disable=<warning name>
+```
 
-`<line> # pylint disable=<warning name>`
+For example:
+```python
+from IMAT_library import *  # pylint: disable=wildcard-import, unused-wildcard-import
+```
 
-e.g. `from IMAT_library import *  # pylint: disable=wildcard-import, unused-wildcard-import`
+If the error is a pyright (typing) error, as opposed to a pylint error, then the comment syntax is:
 
+```python
+some_code()  # pyright: ignore
+```
 
 ### Linting dynamically defined variables
+
 Python allows programmers to set attributes dynamically using expressions like `locals()['foo'] = 1`, which creates a local variable called `foo` with value `1`. Pylint doesn't support dynamic assignment, so if `foo` was subsequently referenced in the code it would be counted as an undefined variable.
 
-In cases where we need to lint scripts containing dynamic assignments, we can write a Pylint [transform plugin](http://pylint.pycqa.org/en/latest/how_tos/transform_plugins.html) to let Pylint know that the dynamically assigned variables are OK and should not be counted as undefined. This has been done [here](https://github.com/ISISComputingGroup/genie_python/blob/0a5f5093486e85e550b8168810e3d5cd762e34ff/Lib/site-packages/genie_python/scanning_instrument_pylint_plugin.py) to support dynamic assignments in the [SCANS library](https://github.com/ISISComputingGroup/IBEX/issues/5214), where some scripts were dynamically adding all methods of classes derived from `ScanningInstrument` to the local module.
+In cases where we need to lint scripts containing dynamic assignments, we can write a Pylint [transform plugin](http://pylint.pycqa.org/en/latest/how_tos/transform_plugins.html) to let Pylint know that the dynamically assigned variables are OK and should not be counted as undefined. This has been done [here](https://github.com/ISISComputingGroup/genie/blob/19cdfeedaf5ec9107f0328a69204b621858c859a/src/genie_python/scanning_instrument_pylint_plugin.py#L14) to support dynamic assignments in the [SCANS library](https://github.com/ISISComputingGroup/IBEX/issues/5214), where some scripts were dynamically adding all methods of classes derived from `ScanningInstrument` to the local module.
 
 

doc/scripting/instrument-scripts.md

@@ -1,6 +1,14 @@
 # Shared Instrument Scripts
 
-The general instrument scripts are in this [repository](https://github.com/ISISNeutronMuon/InstrumentScripts) and include scripts like `do_sans` and `do_trans`, as well as the (old) scans library.
+The general instrument scripts are in this [repository](https://github.com/ISISNeutronMuon/InstrumentScripts). These scripts include:
+- The ['old' scans library](https://pygenie-scans.readthedocs.io/en/latest/index.html); as of October 2025, this is used by SANS instruments and reflectometers, but is being gradually replaced by {doc}`Bluesky-based scans <Bluesky-scanning>`
+- A shared SANS scripting framework. Contains definitions of `do_sans` and `do_trans`
+- Shared utilities for Muon beamlines, for example including:
+  * Background plot configuration (using {doc}`/specific_iocs/other/Background-Script-IOC` and {doc}`Matplotlib`)
+  * {doc}`Zero-field system </specific_iocs/magnets/Zero-field-controller>` calibration routines
+  * {doc}`DAE pre & post commands </specific_iocs/dae/DAE-Pre-and-Post-commands>`
+
+## Deployment
 
 During IBEX deployment, an optional step has been added to the deploy script (see [ticket](https://github.com/ISISComputingGroup/IBEX/issues/7914)
 for details) to pull the latest master branch of the scripts repository and attempt an automatic merge with the local branch.

doc/system_components/Python.md

@@ -1,4 +1,4 @@
-# Python (Uktena)
+# Python (including Uktena, `uv`)
 
 ```{toctree}
 :glob:

doc/scripting/uv.md renamed to doc/system_components/python/Python-venvs.md

@@ -1,5 +1,5 @@
 {#uv}
-# `uv` and python environments
+# Virtual environments (`uv`)
 
 [`uv`](https://github.com/astral-sh/uv) is a Python package and project manager. See [`uv`'s docs](https://docs.astral.sh/uv/) for more information.