Python virtual environment

[gswifort]

Hi,
I have a question: at what point does PyRx decide on the choice of a specific python interpreter? During installation or when loading into AutoCAD? I would like to run Autocad from a python virtual environment with a separate interpreter (e.g. .\venv\Scripts\python.exe), but PyRx in Autocad is always set to the global interpreter.

[CEXT-Dan]

Hi,

Currently, nothing is implemented to handle virtual environments. Each instance of AutoCAD fires up a new interpreter by calling Py_Initialize(), The module loads python312.dll into the AutoCAD process, python.exe is never called

I’m not entirely sure how to handle virtual environments in an embedded context, but I will look into this

[gswifort]

Thanks for the answer.
Yet on another topic. I haven't found an example of how to open a new empty document, add elements to it and save it. On various attempts I received:

RuntimeError:
Exception! (eWrongDatabase) in function appendAcDbEntity ,Line 2322, File PyDbSymbolTableRecord.cpp

[CEXT-Dan]

Hi do you mean a new .DWG, as in a side database?
Have a look at this sample

import traceback
import PyRx as Rx
import PyGe as Ge
import PyGi as Gi
import PyDb as Db
import PyAp as Ap
import PyEd as Ed

#scope so everything is closed
def processDb(db : Db.Database):
    line = Db.Line(Ge.Point3d(0,0,0),Ge.Point3d(100,100,0))
    line.setDatabaseDefaults(db)
    model = Db.BlockTableRecord(db.modelSpaceId(), Db.OpenMode.kForWrite)
    model.appendAcDbEntity(line)
    
def PyRxCmd_doit() -> None:
    try:
        sideDb = Db.Database()
        processDb(sideDb)
        sideDb.saveAs("e:\\newdwg.dwg")
    except Exception as err:
        traceback.print_exception(err)

[gswifort]

Yes, that was it, thank you!

Coming back to the virtual environment, python312.dll can always be the same, so it all comes down to loading <venv>\Lib\site-packages into sys.path instead of %localappdata%\programs\python\python312\Lib\site-packages or at least site-packages from a virtual environment before global ones

[gswifort]

I think I found the solution, just set PYTHONPATH to <venv>\Lib\site-packages

[CEXT-Dan]

Great!

Note: I’m not happy with the current installer, I don’t want to have to write anything to the registry.
So, I plan on writing a loader module that sets AutoCAD’s local ENV to the DLL paths, then loads the python wrapper module.

This loader module will expose PyConfig, https://docs.python.org/3/c-api/init_config.html#init-path-config , I’m not sure the format yet, maybe an .INI or XML configuration file that sits in the same folder.

This is some time off though, as I’m still writing wrappers

[gswifort]

Yes, this should also solve the virtual environment issue (take into account the VIRTUAL_ENV environment variable). Also keep in mind that AutoCAD can be launched using the COM interface (win32com.client.Dispatch), and so far I have encountered a problem where AutoCAD does not see the modules located in the PyRxStubs folder (I also solved this by adding to PYTHONPATH, but this is a bit of a monkey-patching solution)

[schoeller]

@gswifort
I have set up venv with launch.json as below:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote Attach",
            "type": "python",
            "request": "attach",
            "port": 5678,
            "host": "127.0.0.1",
            "justMyCode": false,
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "."
                }
            ],
            "env": {
                "PYTHONPATH": "$PYTHONPATH:C:/ProgramData/Autodesk/ApplicationPlugins/PyRx.bundle/Contents/PyRxStubs" 
            },
        }
    ]
}

VS Code states that env property is not allowed. I have tried adding PYTHONPATH="C:/ProgramData/Autodesk/ApplicationPlugins/PyRx.bundle/Contents/PyRxStubs" to the .env file. I still receive below error

Any hints?

Best regards

Sebastian

[gswifort]

I have the following debugging settings:

launch.json

{
    "name": "Autocad attach",
    "type": "debugpy",
    "request": "attach",
    "connect": {
        "host": "127.0.0.1",
        "port": 5678
    },
    "justMyCode": false
}

settings.json

{
    "python.envFile": "${workspaceFolder}/.env"
}

.env

PYTHONPATH = c:\ProgramData\Autodesk\ApplicationPlugins\PyRx.bundle\Contents\PyRxStubs

However, the error you show will not be resolved by setting the PYTHONPATH to the "stubs" folder. The "stubs" folder contains only *.pyi interfaces and Pyxx modules (containing the source code) are only available in the AutoCAD environment.

[CEXT-Dan]

Not quite sure how to deal with that vscode warning. The problem is PyLance can’t see python embedded inside C++
Those modules PyRx, PyGe etc.. are inside the .ARX file
I’ve semi-resolved the issue by hiding it, see issue #3

[gswifort]

If the problem is only the display of errors, you can simply ignore them, as follows:

import PyRx as Rx  # type:ignore
import PyGe as Ge  # type:ignore
import PyGi as Gi  # type:ignore
import PyDb as Db  # type:ignore
import PyAp as Ap  # type:ignore
import PyEd as Ed  # type:ignore

If you use isort then also:

import PyRx as Rx  # isort:skip # type:ignore
import PyGe as Ge  # isort:skip # type:ignore
import PyGi as Gi  # isort:skip # type:ignore
import PyDb as Db  # isort:skip # type:ignore
import PyAp as Ap  # isort:skip # type:ignore
import PyEd as Ed  # isort:skip # type:ignore

[schoeller]

Thanks for the work. My current environment foresees the following working steps

1#Start BCAD and VS Code in parallel
2#Load script in BCAD manually via _PYLOAD
3#Start pydebug and call the remote debugger in VS Code
4#Alter and save script and restart from 2# via _PYLOAD (hopefully not killing BCAD)

Thus all modules must be installed accesible to the interpreter in BCAD (which is system-wide Python). VDEV is only available to VS Code. Is there any chance that I may have missed something and may directly launch and debug from within VS Code or reorganize my Python installation?

Best

Sebastian

[gswifort]

For BricsCad to see the virtual environment (installed packages), the directory <virtualenv>\Lib\site-packages must be added to sys.path
To do this you have two options:

1. Set PYTHONPATH:

# <virtualenv>\Scripts\activate.bat
...
set PYTHONPATH=%VIRTUAL_ENV%\Lib\site-packages;%PYTHONPATH%

and run BricsCad from the console with the virtual environment activated (for Autocad it is acad, for BricsCad probably bcad).
For me, when I run Autocad from a virtual environment, the embedded Python does not see the PyRxStubs directory, so it also needs to be added to PYTHONPATH (You probably won't need this if you have the global PYTHONPATH variable set correctly - it is set during installation):

# <virtualenv>\Scripts\activate.bat
...
set PYTHONPATH=%VIRTUAL_ENV%\Lib\site-packages;C:\ProgramData\Bricscad\ApplicationPlugins\PyRx.bundle\Contents\PyRxStubs;%PYTHONPATH% 
# check if the path is correct, I only replaced Autocad → Bricscad.

2. Add a path from the BCAD console:

>>> PYCMDPROMPT
>>> import sys
>>> sys.path.insert(0, ".\<virtualenv>\Lib\site-packages")

[CEXT-Dan]

Some ideas:

-create loader modules for each host application.
-The loader module will replace registry entries by adding env paths at runtime.
-The loader module will be able to read a human editable configuration file so users can customize it.

-the installer will install main bulk of the package, i.e. wrappers, stubs, samples
Users*username*\AppData\Local\Programs\PyRx

-Only the AutoCAD loader modules and PackageContents.xml will be installed in ApplicationPlugins
-The loader modules for the clones will be installed in the PyRx folder where they can be added to a startup suite

• or maybe this can be a separate thing

[gswifort]

What do you mean by loader module?

[CEXT-Dan]

Another .ARX module that loads into Autocad, I'll need to hook into Autocads DLL paths and add the necessary paths

[CEXT-Dan]

Hi,
Can you test if v1.3.002 is enough to solve this task?

I’ve added these functions:
PyPreConfig_InitPythonConfig
PyConfig_InitPythonConfig

But I don’t quite understand how they are used, but the goal is to be able to add configurations to the INI, that I can pass along.
But I don’t know which ones you might need

[gswifort]

Hi, Wednesday at the earliest.

[gswifort]

Hi, I don't know how to use what you did.
From what I see, I can use PyRx.ini, but only globally, this file is not searched in the current directory (virtual environment).
For the virtual environment to work properly, it is necessary to adopt a different installation path scheme, as python.exe does in the virtual environment (python3x.dll does not set paths in the same way as python.exe, it has to be done manually).
issue22213 and PEP 587 should be helpful.

[CEXT-Dan]

Yeah, It clear I don’t understand how all this works.

“From what I see, I can use PyRx.ini, but only globally, this file is not searched in the current directory (virtual environment).”

It’s not meant to be, the design is to tell AutoCAD the path to the DLLs. And to pass information to
Py_PreInitialize and Py_InitializeFromConfig, I just don’t have a clue what parameters are needed

How about we start with:
PYTHONISOLATED = 1
PYTHONEXECUTABLE = C:\path\to\venv\Scripts\python.exe

I’ll pass these to Py_InitializeFromConfig

[gswifort]

Yes, it is possible that this will be enough.
However, you should probably pay attention to one more thing - which python3x.dll file to use. When creating a virtual environment, a pyvenv.cfg file is created in the %VIRTUAL_ENV% directory, containing, among others:

home = C:\Users\gswi\AppData\Local\Programs\Python\Python312

use the file located in this directory (we assume that the user may have different versions of python installed).

[schoeller]

@gswifort is this design platform-independant?

[gswifort]

I don't have access to macOS to test, but it seems so. PEP 587

[gswifort]

Analyzing the python venv module also shows that it is platform-independent

[gswifort]

If you run AutoCAD from the console (type acad in the console), it will see it.

[CEXT-Dan]

Acad didn’t launch for me, but it sounds reasonable that would be the case.
I will try to resolve VIRTUAL_ENV first, then fall back to the .INI for now

[CEXT-Dan]

it sees it

Customization file loaded successfully. Customization Group: FEATUREDAPPS
Regenerating model.
true-C:\path\to\myenv
AutoCAD menu utilities loaded.Cancel

[gswifort]

Going back to the ini file:

I copied these 4 files to D:\GSWI\Documents\pyrx\

in the ini file I have:

[PYRXSETTINGS]
; ignored if this exact path is already found in env
PYTHONINSTALLEDPATH =c:\users\gswi\appdata\local\programs\python\python312

; this is the path to wxPython
WXPYTHONPATH =c:\users\gswi\appdata\local\programs\python\python312\Lib\site-packages\wx

; this is the path to the auto complete stubs and activex modules
PYRXSTUBPATH =C:\Users\gswi\AppData\Local\Programs\PyRx\Stubs

;uses PyConfig_InitIsolatedConfig
PYTHONISOLATED=0

;applied config.executable ignored if isolaed is 0
PYTHONEXECUTABLE = D:\GSWI\Documents\pyrx\venv\Scripts\python.exe                                                               

in autocad I load D:\GSWI\Documents\pyrx\RxLoader24.1.arx
then I load module:

import PyRx as Rx # isort:skip
import PyGe as Ge # isort:skip
import PyGi as Gi # isort:skip
import PyDb as Db # isort:skip
import PyAp as Ap # isort:skip
import PyEd as Ed # isort:skip


def PyRxCmd_doit():
     import sys

     print(sys.executable)

and after calling DOIT I get C:\Program Files\Autodesk\AutoCAD 2022\acad.exe instead of D:\GSWI\Documents\pyrx\venv\Scripts\python.exe

[gswifort]

And if I understand this document correctly, setting sys.executable correctly should take care of most of the configuration for the virtual environment.

[CEXT-Dan]

PYTHONISOLATED should be 1

[gswifort]

Sorry, I didn't add that I tested for PYTHONISOLATED 0 and 1, it always sets sys.executable to acad.exe.

[CEXT-Dan]

[PYRXSETTINGS]
PYTHONINSTALLEDPATH =c:\users\dan\appdata\local\programs\python\python312
WXPYTHONPATH =C:\path\to\myenv\Lib\site-packages\wx
PYRXSTUBPATH =M:\Dev\Projects\PyRxGit\PyRxStubs
PYTHONISOLATED=1
PYTHONEXECUTABLE=C:\path\to\myenv\Scripts\python.exe

def PyRxCmd_doit():
    try:
        print(sys.executable)
    except Exception as err:
        traceback.print_exception(err)

[CEXT-Dan]

I probably have the search order messed up

[CEXT-Dan]

Try renaming the .INI in the install folder, you may get a warning messages

[gswifort]

Still nothing.

[CEXT-Dan]

Ok, will have to do some sort of logging to sort this out
you are running v1.3.009 yes?

I have confidence that this will work, basically,
If PYTHONISOLATED=1 and PYTHONEXECUTABLE is a valid path

static bool initIsolated()
{
    PyConfig config;
    PyConfig_InitIsolatedConfig(&config);

    auto [es, venv_executable] = PyRxINI::pythonvenv_path();
    if (es == false)
        return false;

    auto status = PyConfig_SetString(&config, &config.executable, venv_executable.c_str());
    if (PyStatus_Exception(status))
    {
        PyConfig_Clear(&config);
        acutPrintf(_T("\nPyConfig_SetString failed %ls: "), __FUNCTIONW__);
        return false;
    }

    status = Py_InitializeFromConfig(&config);
    PyConfig_Clear(&config);
    if (PyStatus_Exception(status))
    {
        acutPrintf(_T("\nInitializeFromConfig failed %ls: "), __FUNCTIONW__);
        return false;
    }
    return true;
}

[gswifort]

Yes, 1.3.009

[CEXT-Dan]

I’ve uploaded a new version (v1.3.10) to test at your convenience
It has a bit of logging, hopefully will shed some light on where the issue is.

You will have to turn it on from lisp and restart CAD
(setenv "PYRX_LOG" "1") on
(setenv "PYRX_LOG" "0") off

I also noticed wx path may be incorrect. I assume every venv should have it’s own copy

c:\users\gswi\appdata\local\programs\python\python312\Lib\site-packages\wx
should be:
D:\GSWI\Documents\pyrx\venv\Lib\site-packages\wx

Reading VIRTUAL_ENV should work now, but assumes a default folder structure.
With this option, .INI is skipped

[CEXT-Dan]

@gswifort is this design platform-independant?
yes, it should work on BricsCAD as well

[gswifort]

1. Standard installation and .arx files in the project directory:
   (Failure)

   Polecenie: _APPLOAD
   Wczytano pomyślnie RxLoader24.1.arx.
   
   Polecenie:
   PyRxLoader::getPythonVenvPath
   PyRxLoader::thisModulePath d:\gswi\documents\pyrx\
   PyRxLoader::getInstallPath C:\Users\gswi\AppData\Local\Programs\PyRx
   PyRxLoader::getIniPath d:\gswi\documents\pyrx\PyRx.INI
   PyRxLoader::validateINIPythonInstallPath c:\users\gswi\appdata\local\programs\python\python312
   PyRxLoader::validateINIwxPythonPath c:\users\gswi\appdata\local\programs\python\python312\Lib\site-packages\wx
   
   Loading PyRx from ini condition
   PyRxLoader::PyRxLoader_loader Loading, C:\Users\gswi\AppData\Local\Programs\PyRx\Bin\PyRx24.1.arx  # !!!!!
   
   Finished PyRxLoader_loader
   
   PyRx version <1.3.011.20240809> loaded:
   Python Interpreter Loaded successfully!
   Określ przeciwny narożnik lub [KRAwędź/OWbok/Zwbok]:
   Polecenie: PYLOAD
   
   Success module DEMO is loaded:
   Polecenie:
   Polecenie: DOIT
   C:\Program Files\Autodesk\AutoCAD 2022\acad.exe
   

2. Changed name of the installation directory:
   (It works !)

   Polecenie: _APPLOAD
   Wczytano pomyślnie RxLoader24.1.arx.
   
   Polecenie:
   PyRxLoader::getPythonVenvPath
   PyRxLoader::thisModulePath d:\gswi\documents\pyrx\
   PyRxLoader::getInstallPath C:\Users\gswi\AppData\Local\Programs\PyRx
   PyRxLoader::getIniPath d:\gswi\documents\pyrx\PyRx.INI
   PyRxLoader::validateINIPythonInstallPath c:\users\gswi\appdata\local\programs\python\python312
   PyRxLoader::validateINIwxPythonPath c:\users\gswi\appdata\local\programs\python\python312\Lib\site-packages\wx
   
   Loading PyRx from ini condition
   PyRxLoader::PyRxLoader_loader Loading, d:\gswi\documents\pyrx\PyRx24.1.arx
   
   Finished PyRxLoader_loader
   
   PyRx version <1.3.011.20240809> loaded:
   
   *****Error importing the wxPython API!*****:
   Python Interpreter Loaded successfully!
   PYLOAD
   
   Success module DEMO is loaded:
   Polecenie: DOIT
   D:\GSWI\Documents\pyrx\venv\Scripts\python.exe
   # sys.path ↓
   c:\users\gswi\appdata\local\programs\python\python312\python312.zip
   c:\users\gswi\appdata\local\programs\python\python312\DLLs
   c:\users\gswi\appdata\local\programs\python\python312\Lib
   C:\Users\gswi\AppData\Local\Programs\Python\Python312
   D:\GSWI\Documents\pyrx\venv
   D:\GSWI\Documents\pyrx\venv\Lib\site-packages
   D:\GSWI\Documents\pyrx\venv\Lib\site-packages\pip\_internal
   C:\Users\gswi\AppData\Local\Programs\PyRx\Stubs
   d:\gswi\documents\pyrx\
   

This probably means that you first look for .ini in the installation directory and then in the local directory instead of the other way around.

3. Virtual environment activated and global loaders used:
   (It works !)

   PyRxLoader::getPythonVenvPath D:\GSWI\Documents\pyrx\venv
   PyRxLoader::thisModulePath c:\programdata\autodesk\applicationplugins\pyrx.bundle\contents\
   PyRxLoader::getInstallPath C:\Users\gswi\AppData\Local\Programs\PyRx
   PyRxLoader::getIniPath C:\Users\gswi\AppData\Local\Programs\PyRx\Bin\PyRx.INI
   
   Loading PyRx from venv condition
   D:\GSWI\Documents\pyrx\venv\Lib\site-packages\wx
   PyRxLoader::PyRxLoader_loader Loading, C:\Users\gswi\AppData\Local\Programs\PyRx\Bin\PyRx24.1.arx
   
   Finished PyRxLoader_loader
   
   PyRx version <1.3.011.20240809> loaded:
   
   *****Error importing the wxPython API!*****:
   Python Interpreter Loaded successfully!
   Określ przeciwny narożnik lub [KRAwędź/OWbok/Zwbok]:
   Polecenie: PYLOAD
   
   Success module DEMO is loaded:
   Polecenie: DOIT
   D:\GSWI\Documents\pyrx\venv\Scripts\python.exe
   c:\users\gswi\appdata\local\programs\python\python312\python312.zip
   c:\users\gswi\appdata\local\programs\python\python312\DLLs
   c:\users\gswi\appdata\local\programs\python\python312\Lib
   C:\Users\gswi\AppData\Local\Programs\Python\Python312
   D:\GSWI\Documents\pyrx\venv
   D:\GSWI\Documents\pyrx\venv\Lib\site-packages
   D:\GSWI\Documents\pyrx\venv\Lib\site-packages\pip\_internal
   C:\Users\gswi\AppData\Local\Programs\PyRx\Stubs
   c:\users\gswi\appdata\local\programs\pyrx\bin\
   D:\GSWI\Documents\pyrx\
   

[CEXT-Dan]

Yay Progress,

The search order is next to the module, but since there’s no .INI in the bundle, it looks to the install
You might delete the bundle and have a try to see the behavior, or put a .INI in the bundle

I’ll have to reconsider the paths in that case

[schoeller]

Sorry to bother. Not getting there. Have undertaken the following steps

1#Fresh install of Python 3.12.3 on Win11
2#Opened a folder in VS Code and added a virtual env in the terminal

python -m venv .venv
.\.venv\Scripts\activate
pip install debugpy
pip install wxpython

3#Altered PyRX.INI as follows

[PYRXSETTINGS]
PYTHONINSTALLEDPATH =c:\users\sebastian schoeller\appdata\local\programs\python\python312
WXPYTHONPATH =C:\Users\Sebastian Schoeller\Documents\GitHub\PyDevBox\.venv\Lib\site-packages\wx
PYRXSTUBPATH =C:\Users\Sebastian Schoeller\AppData\Local\Programs\PyRx\Stubs
PYTHONISOLATED=1
PYTHONEXECUTABLE =C:\Users\Sebastian Schoeller\Documents\GitHub\PyDevBox\.venv\Scripts\python.exe

4#Called (setenv "PYRX_LOG" "1") on BCAD
5#Restarting BCAD I get

PyRxLoader::getPythonVenvPath 
PyRxLoader::thisModulePath C:\Users\Sebastian Schoeller\AppData\Local\Programs\PyRx\Bin\
PyRxLoader::getInstallPath C:\Users\Sebastian Schoeller\AppData\Local\Programs\PyRx
PyRxLoader::getIniPath C:\Users\Sebastian Schoeller\AppData\Local\Programs\PyRx\Bin\PyRx.INI
PyRxLoader::validateINIPythonInstallPath c:\users\sebastian schoeller\appdata\local\programs\python\python312
PyRxLoader::validateINIwxPythonPath C:\Users\Sebastian Schoeller\Documents\GitHub\PyDevBox\.venv\Lib\site-packages\wx
Loading PyRx from ini condition
PyRxLoader::PyRxLoader_loader Loading, C:\Users\Sebastian Schoeller\AppData\Local\Programs\PyRx\Bin\PyRxV24.0.brx
Finished PyRxLoader_loader
PyRx version <1.3.015.20240826> loaded:
PYTHONEXECUTABLE failed initIsolated: 
initIsolated failed, trying Py_Initialize  WxRxApp::Init_wxPython: Python Interpreter Loaded successfully!

6#Path to python.exe exists
7#Below settings work

PYTHONINSTALLEDPATH =C:\Users\Sebastian Schoeller\Documents\GitHub\PyDevBox\.venv\Scripts
PYTHONISOLATED=0

[CEXT-Dan]

I’ll have to see if I can get an error message from Python as to what went wrong
Py_InitializeFromConfig returned the error

[CEXT-Dan]

I think I see the error, I will test this this afternoon

[CEXT-Dan]

Actually, turned out to be an issue with BRX, I’m testing a workaround, filed SR 176582

[gswifort]

When using a virtual environment, sys.prefix is set to the <venv> directory, I suggest changing it to sys.base_prefix, it will work in both cases.

PyRx/PyRxStubs/PyRxDebug.py

Line 27 in a461f00

PYTHON_PATH = sys.prefix + "\\python.exe"

[CEXT-Dan]

Thanks! I add that

[github-actions]

Stale issue message