-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
realesrgan on M1/M2 Macs #390
Comments
I absolutely plan on getting this to work because I want it! :D I just haven't had time to work on it. |
Thank you, works like a charm ! I do have a question though... for me it doesn't matter if i have -U 4 or -U 2, it always performs a 4 time upscale, is it working for you ? Also, in case somebody has issues with mac security settings with the realesrgan file, i want to recommend to following article to resolve this issue: https://macpaw.com/how-to/allow-apps-anywhere BR |
@thg-muc Thanks for catching that! I forgot about passing the option to realesrgan-ncnn-vulkan. These are the changes to get it to work.
For simplicity, here's the updated file |
If subprocess.run() works similarly on Windows and Linux, then this could be a whole framework for launching post-processing jobs, including ones that aren't native python. This would have the advantage of being both extensible and insulating us from changes in the GFPGAN package. I see that there are ESRGAN ncnn-vulkan releases for Mac, Ubuntu and Windows. Is there something similar for GFPGAN, or do users have to go through installation from source? Another question. The subprocess call is using a relative path to the executable. Does run() search the PATH, and is this portable between windows and mac/linux? |
I assume No executables for GFPGAN in Releases https://github.com/TencentARC/GFPGAN/releases About subprocess.run() and PATH, not sure. Someone smarter will know :=D
|
On You will have:
|
In addition, adding realesrgan in the root folder of this project is not the best. ex: subprocess.run(
['./realesrgan-ncnn-vulkan', '-i', 'esrgan_in.png', '-o', 'esrgan_out.png', '-s', scale],
stdout=subprocess.PIPE,
cwd="../realesrgan"
).stdout.decode('utf-8') |
Here an implementation that works on my side.
You can download with wget https://github.com/glonlas/Stable-Diffusion-Apple-Silicon-M1-Install/blob/main/patches/gfpgan_tools.py.patch
wget https://github.com/glonlas/Stable-Diffusion-Apple-Silicon-M1-Install/blob/main/patches/simplet2i.py.patch
git apply gfpgan_tools.py.patch
git apply simplet2i.py.patch Note: I will do a PR tomorrow, this will be cleaner. |
Face restoration alone (without upscaling) seems to fail as well. Are you using an Intel Mac? I'm on M1.
|
I did not get this issue at all. Are you on Main or Development branch? |
The issue is certainly because my script forces I could not reproduce the issue, but I will not be surprised it is the root cause for this issue. |
I'm using the main branch. |
Yes, 4x works. The default seems to be realesr-animevideov3 (not sure which is better, vs. realesrgan-x4plus)
|
The PR is coming. It will be easier for a review and to create a patch if needed. Much better than this 20 min hack. |
@glonlas do you also intend to update the macos README? it has no mention of realesrgan at the moment. Would be super nice! |
@Any-Winter-4079 the patch has been updated. It handles both x2 and x4. So if you are using the main branch please use the command in my previous comment (it is fixed and fully woking) |
Here the PR for the cc: @Any-Winter-4079, @lstein |
I don't have write access, but I hope @lstein or a contributor see the Pull request and accept it. From my end, I've tested it and added my review. Hope it gets available soon! |
So, I had a quick look at getting RealESRGAN working on MPS and it was only a two line change (and a pip install realesrgan)
|
That is interesting. Will try to test this later today. Certainly it is simple and requires very little code change. One side effect of For example, without Haven't tested if this version downgrade has performance implications though. |
I've have a peak see what version of numpy I've got installed when the M1 is finished running the preflight checks, I can't recall it downgrading it, but I probably wasn't watching it properly either. |
% pip list | grep real I've had a quick look as Real-ESRGAN's git repo and the requirement.txt just lists numpy with no version. |
It may not matter, but based on watching CPU and GPU usage while running this, I'm guessing this is falling back to CPU and not using MPS (assuming I'm understanding MPS correctly as an alternate PyTorch device that can use GPU cores on Apple Silicon.) I think there would need to be changes to GFPGAN and Real-ESRGAN to implement MPS as a backend device for them to run on the GPU. I installed GFPGAN and Real-ESRGAN by cloning their git repos and running the setup scripts inside the ldm conda environment. |
Perhaps working on MPS was not quite the way to but it but the code was activity blocking anything that wasn't CUDA before. Not idea if they are actually using the MPS pytorch back end but f the code inside GPFGAN / Real-ESRGAN themselves are neutrally coded it should by default. |
Unfortunately it doesn't look like they wrote Real-ESRGAN neutrally as its eating CPU at this moment. |
I'm seeing times of 49s to upscale x4 while the |
Interestingly its a simple hack to the Real-ESRGAN code to get Real-ESRGAN running on the GPU and it runs without an error, the results are wrong though |
FYI, I just merged in a PR for the "Embiggen" feature, which does arbitrary
upscaling by tiling the image, stretching the tiles and then merging them
back together using img2img. It does have a realesrgan step, but that can
be turned off and it seems to work pretty well nevertheless.
…On Mon, Sep 12, 2022 at 4:01 PM Vargol ***@***.***> wrote:
Interesting;y its a simple hack to the Real-ESRGAN to get Real-ESRGAN
running on the GPU and it runs without an error, the results are wrong
though
—
Reply to this email directly, view it on GitHub
<#390 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAA3EVPMQ6TZ6J5ZJSGM2ODV56D2DANCNFSM6AAAAAAQFESFPI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Are you referencing PR #474? Does this need to come out, or are you talking
about a modification?
|
@lstein I just tried a clean install.
|
Imo, the last step to do is to put this Since I didn't know when to do that download, I did it too early, so that the GFPGANv1.3.pth didnt go in the proper place (I think the |
Embiggen sounds fun from a technical point of view but to fess up there's a legally licensed copy of TopazLab's GigaPixel AI on my M1 so the resizing isn't a hot topic for actual use for me ;-) |
The PR's is a alternative where a process call to a Real-ESRGAN's binary is used rather than the PyPi or git code |
This was part of @blessedcoolant's original design. My understanding is that GFPGAN works better on larger faces, so the upscaling is done before the face reconstruction. It might be worth swapping the order of operations in the run_gfpgan() method to confirm this. |
Oh, I remember that one. The only thing I didn't like about it were concerns it might not be a portable solution across the platforms. Do you know if it runs on Linux/Windows? |
Yep. GFPGAN depends on facial landmarks being detectable which is a lot easier to do on larger resolution images. I've used GFPGAN and other face restoration tools for a very long time now and I have found it better to always perform scaling up and then face restoration. There are always cases where it is irredeemable but otherwise, it works great. |
GFPGAN might actually be a one line change for MPS use. It's a bit hard to tell as it runs to quickly for Activity Monitor to pick it up at 512x512 doing a 1024x1024 run to see if make it run for long enough. |
I ran GFPGAN on 2048x2048 and it's extremely fast. It does indeed gives very good results. |
It supports Windows/Linux/MacOS https://github.com/xinntao/Real-ESRGAN/releases The only thing is that GFPGAN doesn't have an executable. So we'd need to run Real-ESRGAN from executable and GFPGAN from source. Not a big deal, but it adds a bit more variety/complexity to the code. On the flip side, performance. |
Exactly. Same experience here. Personally, I have |
I'd say GFPGAN runs fast enough not to matter really, despite that I'm waiting see what the MPS change does (as far as I can tell it works, its enhancing faces, just can't tell if its actually using the GPU) |
Agree. By the way can't that change be applied to Real-ESRGAN as well? Haven't had much time to look into this part of the code. |
"This is not the image you're looking for...."
|
did you install |
@slk333 Download from here https://github.com/xinntao/Real-ESRGAN/releases To run |
Darn it GFPGAN most have some other 'cuda or cpu' code embedded rather than just guarding at the start like Real-ERGAN |
I moved this comment by @eduardoarinopelegrin from here. Original comment follows.
Kind of unrelated, but also kind of related (b/c it involves simplet2ti.py).
Has anyone got
realesrgan
working on M1/M2 Macs with this repo? If not, are there any plans or works in progress?I just got it to work with the
realesrgan-ncnn-vulkan
executable + updating the appropriate parts of simplet2ti.py. So if there were no plans to implement / not in the near future, maybe you could include this hack for us M1/M2 Mac users to get it to work.Steps are:
realesrgan-ncnn-vulkan-20220424-macos
) and moverealesrgan-ncnn-vulkan
insidestable-diffusion
(this project folder). Move the Real-ESRGAN model files fromrealesrgan-ncnn-vulkan-20220424-macos/models
intostable-diffusion/models
as well.chmod u+x realesrgan-ncnn-vulkan
to allow it to be run.self.device.type == 'mps'
The changes are:
import subprocess
Add a function to do the upscaling
Modify this part to call the function
This is the full file. This may be more appropriate as a pull request, but since I've never done one (so far!) and simplet2ti.py is being modified (from the version I implemented my changes on), maybe it's easier to add it to your version if you see it fit.
simplet2i.py.zip
Usage:
python3 ./scripts/dream.py
dream >
Anubis the Ancient Egyptian God of Death riding a motorbike in Grand Theft Auto V cover, with palm trees in the background, cover art by Stephen Bliss, artstation, high quality -U 4 -m ddim -S 1469565
Obtained/ expected result (4x upscale):
The text was updated successfully, but these errors were encountered: