Intel ARC support with IPEX

[Disty0]

Initial Intel ARC GPU support with IPEX on Linux / WSL.
This PR presents IPEX as CUDA.

Users needs to set these manually:
Mixed precision to BF16,
Attention to SDPA,
Optimizer to: AdamW (or any other non 8 bit one).

[kohya-ss]

Thank you for the excellent PR. It certainly looks valuable.

However, I regret to say that I don't possess an ARC GPU, which poses challenges in both testing and future maintenance should this PR be merged. Additionally, if I or other contributors make future modifications or add new features to the repository, it's conceivable that the scripts might not work in the ARC GPU environment.

If you're able and willing to continue submitting PRs in the future to address such issues, I would be more than happy to merge this PR. On the other hand, if that's a challenge, one possible solution might be to create a new branch from the current version of the repository and merge this PR there.

I would appreciate hearing your thoughts on this matter.

[Disty0]

I can maintain IPEX side. You can tag me with IPEX specific issues. That's what other repos do most of the time.

For new scripts, adding these lines after importing torch should enable IPEX support and you can continue with using torch.cuda:

try:
    import intel_extension_for_pytorch as ipex
    if torch.xpu.is_available():
        from library.ipex import ipex_init
        ipex_init()
except Exception:
    pass

I went with converting torch.xpu to torch.cuda route because directly using torch.xpu needs a lot of maintenance on my end.
I also manage my IPEX library externally here:
https://github.com/Disty0/ipex_to_cuda

[kohya-ss]

Thank you for your detailed response and willingness to maintain the IPEX side. It's great to know that I can tag you for any IPEX-specific issues.

The code snippet you've provided for integrating IPEX support in new scripts seems clear and straightforward. I'll make sure to keep it in mind when I work on future updates.

I apologize for the delay in getting back to you.

I'd like to merge your PR, perhaps in this weekend. I truly appreciate your efforts and collaboration.

[kohya-ss]

Thank you for this! I will merge to sdxl branch sooner.

[FurkanGozukara]

when installing this via bmaltais GUI don't select this option if you don't have it and know what you are doing

I have 13900 K and it caused me to not to be able to do any training

[Disty0]

Because IPEX is trying to use the iGPU and IPEX currently doesn't work properly with iGPUs.
This PR shouldn't have any affect if you don't install IPEX.