helper for module not found errors for code written for venv environments

[johndpope]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

background - im using many libaries or codebases (frequently from japan) that use venv
eg. https://github.com/bmaltais/kohya_ss/
and I do prefer miniconda - so I persist to evade creating the venv.

i end up with following symptoms when I persists to use python code with minconda (even though I have all the dependencies already) and symptoms look like this

this code saves the day.

.zshrc

function venv_path(){
    source ~/pypath.sh
}

pypath.sh

#!/bin/bash
# This script adds the current working directory to PYTHONPATH

# Get the current working directory
PWD=$(pwd)

# Check if PYTHONPATH is already set
if [ -z "$PYTHONPATH" ]; then
    # If PYTHONPATH is not set, initialize it with the current directory
    export PYTHONPATH="$PWD"
else
    # If PYTHONPATH is already set, append the current directory to it
    export PYTHONPATH="$PYTHONPATH:$PWD"
fi

# Optionally, print the PYTHONPATH to verify it's correctly set
echo "PYTHONPATH is now set to: $PYTHONPATH"

Why is this needed?

so when I hit the module not found error
providing I execute the venv_path command in correct folder -
then I can get this code running.

What should happen?

this solution works - but should be helpful to everyone - out of the box.

Additional Context

No response

[travishathaway]

@johndpope,

Thanks for filing this feature request. I understand that this approach may be very helpful for you, but I don't believe this would be a reasonable default for all users of conda. This may work for you, but making this the default behavior for everyone would have unintended consequences and break our user's assumptions about how conda currently works.

I hope you understand. Please let me know if you would like further clarification.

[johndpope]

if someone attempts to run python code that's built for venv - it's impossible to get working without setting python path to the current directory - something that venv does out of the box. googling this error doesn't solve it - nor chatgpt so well. otherwise you have to update the python code to say sys.path.append etc.
there are so many libraries in stable diffusion world still using venv - which admitadly - they should be educated on benefits of conda and migrate over - but that's a different disucssion.

the other option is to somehow hook into the error from python - telemetry data etc - when this error fires....
"No module found library - but look we found a folder here called library here - so you should set the python path to current directory......"
if you can help people overcome that - then that is fruitful outcome.
maybe you need to reach out the python developers upstream.
then it that error explain how people can fix it - or automatically recover by appending the runpath.

n.b. my hack above - setting the pythonpath to current directory using that above helper - doesn't persist session to session. I would assert that this error is happening like a 10s of thousands a times a week.

Im not saying that this should execute anytime except when called right.

Consider a git repo - that
https://github.com/bmaltais/kohya_ss

challenge is - explain how to get this to work with conda

install dependencies - done.
you run the code - it errors.
you google
you chatgpt
you are stuck.
your solution that somehow should be straightforward but its ever so slightly esoteric for newbies.
EXPORT pythonpath=currentpath:pwd ???

my solution
just run
venv_path

Me setting the pythonpath manually (export PYTHONPATH=existingpath windows path?? / linux path???:$Pwd ?is this windows compatible?
if you ask a regular stable diffusion to set python path off the cuff - Im sure they would struggle to do this. I know because it's frustrating to me. there's wrong with the python code - it's a problem with conda.

Need to spoon feed people on how to recover prompting with the instructions on how to do it or just automatigically do it.

CONSIDER option c)
if there's a venv folder - and conda can detect - then under this condition - when we dont find library?? - then eagerly append the python path with the pwd. or maybe
option d)

• we couldn't find library module - though it's clearly a directory here - but hey we detected a venv path - do you want add this folder to pythonpath?
  gracefully recovering...

UPDATE
this is only a problem for conda - that's why I'm creating this ticket here. upstream can't help so much - unless they can give you telemetry - hooks for errors ???? idk.

option e)
a developer comes along - wants to code a modue and put them in folder called library
they put all the import statements -
from library import blabla

this works with venv - but breaks with conda unless - he explicitedly adds to the code

sys.path.append(library)

this boiler plate is nonsense - consider the sprawl across codebases - as more directories are there - and more files.

somehow - create a new file type / system instruction for conda
it's a hidden a file called
.venv_conda

• and then automagically - when this file exists - append the pythonpath recognizing its existance

so in future - there's a solution to reduce this friction.

[kenodegard]

The problem you are describing is not related to venv or conda, it's a design/security decision from Python.

If you wish to access Python libraries defined in your CWD (which have not been installed into site-packages) you should be calling something like python -m ... or python -c .... Both of these flags will add the CWD to the start of sys.path/$PYTHONPATH.

In your case, instead of running python finetune/make_captions.py ... you should be running python -m finetune.make_captions ...:

$ python finetune/make_captions.py --help
Traceback (most recent call last):
  File "/Users/kodegard/dev/kohya_ss/sd-scripts/finetune/make_captions.py", line 14, in <module>
    from library.device_utils import init_ipex, get_preferred_device
ModuleNotFoundError: No module named 'library'

$ python -m finetune.make_captions --help
rich is not installed, using basic logging
get_preferred_device() -> cpu
usage: make_captions.py [-h] [--caption_weights CAPTION_WEIGHTS] [--caption_extention CAPTION_EXTENTION] [--caption_extension CAPTION_EXTENSION] [--beam_search]
                        [--batch_size BATCH_SIZE] [--max_data_loader_n_workers MAX_DATA_LOADER_N_WORKERS] [--num_beams NUM_BEAMS] [--top_p TOP_P] [--max_length MAX_LENGTH]
                        [--min_length MIN_LENGTH] [--seed SEED] [--debug] [--recursive]
                        train_data_dir
...

[kenodegard]

Another option is to do an editable install (which is what kohya_ss does, see https://github.com/bmaltais/kohya_ss/blob/05cf164279d7f8675cd498373e64c0148057472f/requirements.txt#L34-L35).

So you'd do the following:

$ conda create -n kohya_ss ...
$ pip install -e ./sd-scripts
$ cd ./sd-scripts
$ python finetune/make_captions.py --help
rich is not installed, using basic logging
get_preferred_device() -> cpu
usage: make_captions.py [-h] [--caption_weights CAPTION_WEIGHTS] [--caption_extention CAPTION_EXTENTION] [--caption_extension CAPTION_EXTENSION] [--beam_search]
                        [--batch_size BATCH_SIZE] [--max_data_loader_n_workers MAX_DATA_LOADER_N_WORKERS] [--num_beams NUM_BEAMS] [--top_p TOP_P] [--max_length MAX_LENGTH]
                        [--min_length MIN_LENGTH] [--seed SEED] [--debug] [--recursive]
                        train_data_dir
...

[johndpope]

yep - youre right. thanks for taking time to answer this.
This error message upstream is thus erroneous. It needs to be more instructive with steps to recover.
ModuleNotFoundError: No module named 'library'
(if the directory exits and it has init.py in it) it should say

ModuleNotAccessibleError: Module named 'library' inassessible due to .... Python. (Run with -m or pip install -e editable to run run with modules )

I submit this idea to python - thanks again
https://discuss.python.org/t/improve-modulenotfounderror-to-cater-for-condition-when-directory-exists/51277

I beseech you to make this condition easier to recover from for newbies - it just bricks running code.
q) does conda have a telemetry opt in? im sure this is a widespread problem.