Installer rewrite in Python by ebr · Pull Request #2338 · invoke-ai/InvokeAI

ebr · 2023-01-16T08:16:34Z

Summary

This PR rewrites the core of the installer in Python for cross-platform compatibility. Filesystem path manipulation, platform/arch decisions and various edge cases are handled in a more convenient fashion. The original install.bat.in/install.sh.in scripts are kept as entrypoints for their respective OSs, but only serve as thin wrappers to the Python module.

TL;DR

Test the "happy path":

$ python3 installer/main.py --root <path_to_somewhere> --yes

Remove the --yes for interactive testing.

Details

Downloading of specific .zip bundles (refs) from Github is not yet implemented; this right now functions as a more polished "source" installer. The pip-based install and ability to place the virtualenv outside of the runtime dir are not yet implemented, may be added to this or a future PR.

Introduces a new ldm/invoke/config module and moves the configure_invokeai script into it. Note: this supports a separate set of changes I'm almost ready to push to the config module, which improve runtime directory management and validation. They are not a part of this PR yet, but will be soon™. Omitted for now for better testability and to keep rebasing to a minimum.

more testing

Because the installer isn't yet standalone and doesn't pull the source tree from Github, it needs to be tested from a cloned copy of this repo, with this branch checked out.

See TODO checklist in a comment below for things that are known broken . The "happy path" should be very reliable. Changing filesystem paths should also be reliable. Please test "weird" scenarios (delete the configs dir after installation? etc), in addition to the "happy path".

Test the shell script wrappers as well:

Copy or rename the installer/install.*.in (as appropriate for your OS) such that it doesn't have the .in extension
Run this install script. Try running it by absolute path after cd to somewhere else on your system. It should work regardless of your CWD.

The install script accepts the --root <path> argument to install in a location that is different from the default, and --yes for a hands-off install (this does not yet work on Windows!!).

The install script should go through the entire process of creating the venv, installing the dependencies and the app, and then handing over to the config script.

There are still many bugs! Checklist posted as a separate comment.

ebr · 2023-01-16T08:32:36Z

Task list (still to be done)

TODO (some blocking):

debt (non-blockers):

use console.print() for all standard printing
use prompt_toolkit for all freeform or multiple-choice selection prompts?
- or only where autocomplete is needed?
consider also replacing Console.ask() with prompt_toolkit? just so we have one approach
can we make models.yaml a list instead of a mapping? (should we?)

known bugs:

if hf token is already set, it can't be reset by rerunning the script, even if invalid
if configs dir does not exist, models.yaml is not created
if you skip model dl initially, models.yaml is not created (maybe fixed? confirm)
user is not given the opportunity to auth to HF if they skip model download
clip model gets downloaded to ~/.cache/clip on first run

future/feat:

feat: ability to reconfigure various aspects, like the outdir
feat: add HF_HOME to Paths
feat: there should be a way to skip all downloads
feat: config fallback (report_model_error, bail out from CLI failure) tries to run through the whole workflow for any issue (i.e. "missing initfile? download models!"). Currently refactoring it to fix only what needs fixin'.

decisions (fyi / to discuss):

keep using bash/cmd wrapper scripts (however now very thin) to maintain the same release process (will reinvent the wheel another time!)
use prompt_toolkit for interaction instead of readline/pyreadline3 - why?
- pure python (does not require readline)
- works great on Windows - a successor of pyreadline3
- unlike readline, doesn't break the console after formatted text was displayed (this is a known GNU readline bug!)
- has excellent autocompletion/validation capabilities
- formatting/layout tools are worse than rich
  - but they integrate nicely: prompt_toolkit for interaction, rich for presentation

WRITE TESTS 😭

when model download is skipped
- and config file doesn't exist
  - config file should be created
  - just with defaults
- and configs dir is missing
  - configs dir should be reinitialized
- and there are no models
  - do what?
when configs dir is missing
- create a new configs folder
- create a default models.yaml
- do not download models!
when models dir is missing
- offer to download models
- ussual flow
when init file is missing
- only ask enough to populate initfile contents
- skip downloads etc unless needed
when a model fails to load
- list currently installed models
- offer to download models (follow usual flow)
- this is already somewhat the case, but offers to run the config script

Might spin these out into separate issues......

hipsterusername · 2023-01-17T12:31:22Z

My experience with the installer on windows has been nothing short of pleasant. Extremely nice work.

I haven’t tried latest version - what feedback/input do you need?

lstein · 2023-01-17T14:59:29Z

I'm looking forward to testing this. Today I'm still ironing out bugs with diffusers, but I'll be back to evaluating the installation process on wednesday or Thursday, depending on how it goes.

…

On Tue, Jan 17, 2023 at 7:31 AM Kent Keirsey ***@***.***> wrote: My experience with the installer on windows has been nothing short of pleasant. Extremely nice work. I haven’t tried latest version - what feedback/input do you need? — Reply to this email directly, view it on GitHub <#2338 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3EVNRVY33WNFD33GU34DWS2GKLANCNFSM6AAAAAAT4ODDGE> . You are receiving this because your review was requested.Message ID: ***@***.***>

ebr · 2023-01-20T01:08:41Z

My experience with the installer on windows has been nothing short of pleasant. Extremely nice work.

I haven’t tried latest version - what feedback/input do you need?

Thank you <3

I'm interested to know if any path aside from the "happy path" (i.e. "yes to all" on a brand-new runtime dir) breaks in any way. For example, does selection of paths on a different drive work? Destination with spaces in the path name? those kind of edge cases. I tested a lot of it on Linux, but my Windows VM is way too slow (and lacks GPU) for any meaningful testing.

Thanks!

Lycantant · 2023-01-23T14:12:23Z

Downloading of specific .zip bundles (refs) from Github is not yet implemented; this right now functions as a more polished "source" installer. The pip-based install and ability to place the virtualenv outside of the runtime dir are not yet implemented, may be added to this or a future PR.

Still patiently waiting for this one 😁

…app venv

…er importing

There is a race condition affecting the 'tempfile' module on Windows. A PermissionsError is raised when cleaning up the temp dir Python3.10 introduced a flag to suppress this error. Windows + Python3.9 users will receive an unpleasant stack trace for now

This allows reliable distribution of the initial 'configs' directory with the Python package, and enables the configuration script to be running from anywhere, as long as the virtual environment is available on the sys.path

the 'setup.py install' method is deprecated in favour of a build-system independent format: https://peps.python.org/pep-0517/ this is needed to install dependencies that don't have a pyproject.toml file (only setup.py) in a forward-compatible way

if the config directory is missing, initialize it using the standard process of copying it over, instead of failing to create the config file this can happen if the user is re-running the config script in a directory which already has the init file, but no configs dir

…h experience on Linux/Mac - install.sh is now a thin wrapper around the pythonized install script - install.bat not done yet - to follow - user messaging is tailored to the current platform (paste shortcuts, file paths, etc) - emit invoke.sh/invoke.bat scripts to the runtime dir - improve launch scripts (add help option, etc) - only emit the platform-specific scripts

…all/config

- remove redundant options (unchanged from defaults) - don't test 3rd party code - omit fully covered files from coverage report - gitignore junit (xml) test output directory

…' dir

- add torch MPS fallback directly to CLI.py - rename CLI scripts with `invoke-...` prefix - delete long-deprecated scripts - add a missing package dependency - delete setup.py as obsolete

…stalling from a downloaded release .zip

… the wheel reduces wheel size to 3MB from 27MB

…stall from bundled .whl by default

…new invokeai-configure command

…er after configuration

ebr · 2023-01-28T22:41:01Z

The commit log for this has gotten ridiculous. Opening a new PR for sanity.

ebr requested a review from tildebyte as a code owner January 16, 2023 08:16

ebr mentioned this pull request Jan 16, 2023

[DRAFT] Installer python rewrite #2292

Closed

ebr marked this pull request as draft January 16, 2023 08:18

ebr requested review from hipsterusername and lstein January 16, 2023 08:18

ebr requested review from mauwii and netsvetaev January 16, 2023 16:53

ebr force-pushed the dev/installer branch from 95ebeaa to efad88e Compare January 17, 2023 00:28

ebr force-pushed the dev/installer branch 2 times, most recently from fdb516b to 2264f9c Compare January 19, 2023 20:34

ebr force-pushed the dev/installer branch 3 times, most recently from 95b849a to 1044fc5 Compare January 28, 2023 07:43

ebr added 12 commits January 28, 2023 17:39

(installer) initial work on the installer

2296f54

(installer) Windows fixes

d3962ab

(installer) add venv creation for the app

bf1996b

(installer) PoC to install the app (source installer style) into the …

f75b194

…app venv

(installer) use plumbum for better stdout streaming

c279314

(installer) use prompt-toolkit for directory picking instead of tkinter

802b95b

(installer) run configure_invokeai from within the installer

93129fd

(installer) add graphics accelerator selection

071a438

(config) move configure_invokeai script to the config module for easi…

88526b9

…er importing

(installer) ask the user for their GPU type; improve other messaging

b186965

(installer) install PyTorch from correct repositories

169c56e

ebr added 20 commits January 28, 2023 17:39

(config) moving the 'configs' dir into the 'config' module

9997fde

This allows reliable distribution of the initial 'configs' directory with the Python package, and enables the configuration script to be running from anywhere, as long as the virtual environment is available on the sys.path

(installer) work around Windows install issues

02c530e

(installer) install.bat wrapper for the python script

4b65998

(installer) add a --y[es[to_all]] argument for a fully hands-off inst…

242abac

…all/config

(installer) new torch index urls + support installation from PyPi

c5914ce

(parser) fix missing argument default in parse_legacy_blend

c5a9e70

(test) tweak pytest coverage options

b58a801

- remove redundant options (unchanged from defaults) - don't test 3rd party code - omit fully covered files from coverage report - gitignore junit (xml) test output directory

(package) move TI scripts into a module; update packaging of 'configs…

22c34aa

…' dir

(scripts) rename/reorganize CLI scripts

2ff47cd

- add torch MPS fallback directly to CLI.py - rename CLI scripts with `invoke-...` prefix - delete long-deprecated scripts - add a missing package dependency - delete setup.py as obsolete

(installer) support both pip and source install; no longer support in…

d209dab

…stalling from a downloaded release .zip

(installer) do not install if already in a venv

ebe6daa

(assets) move 'caution.png' to avoid including entire 'assets' dir in…

47dbe7b

… the wheel reduces wheel size to 3MB from 27MB

(installer) build .whl and distribute together with the installer; in…

f1f8ce6

…stall from bundled .whl by default

(scripts) improved script entrypoints

abe4e1e

(config) rename all mentions of scripts/configure_invokeai.py to the …

34e0d7a

…new invokeai-configure command

(docs) update install docs to refer to the platform-agnostic installer

8f86aa7

(config) ensure the correct 'invokeai' command is displayed to the us…

2c0bee2

…er after configuration

ebr force-pushed the dev/installer branch from 6f0fc9a to 2c0bee2 Compare January 28, 2023 22:39

ebr closed this Jan 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installer rewrite in Python#2338

Installer rewrite in Python#2338
ebr wants to merge 32 commits into
mainfrom
dev/installer

ebr commented Jan 16, 2023 •

edited

Loading

Uh oh!

ebr commented Jan 16, 2023 •

edited

Loading

Uh oh!

hipsterusername commented Jan 17, 2023

Uh oh!

lstein commented Jan 17, 2023 via email •

edited

Loading

Uh oh!

ebr commented Jan 20, 2023

Uh oh!

Lycantant commented Jan 23, 2023

Uh oh!

ebr commented Jan 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ebr commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

TL;DR

Details

more testing

Uh oh!

ebr commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Task list (still to be done)

TODO (some blocking):

debt (non-blockers):

known bugs:

future/feat:

decisions (fyi / to discuss):

WRITE TESTS 😭

Uh oh!

hipsterusername commented Jan 17, 2023

Uh oh!

lstein commented Jan 17, 2023 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ebr commented Jan 20, 2023

Uh oh!

Lycantant commented Jan 23, 2023

Uh oh!

ebr commented Jan 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ebr commented Jan 16, 2023 •

edited

Loading

ebr commented Jan 16, 2023 •

edited

Loading

lstein commented Jan 17, 2023 via email •

edited

Loading