New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DRAFT] GAN plugin #60
Conversation
Note: Usage is the same as for other versions @Apollo122 this is the new version. It is the same as the previous but it is seprated from other more generic modifications |
@Clorr got it. Will test it today. |
6007c10
to
db902a2
Compare
I wrote a FaceSwapGAN class. Edit: Oops, I didn't notice that Clorr is already working on adding GAN as a plugin. Not intended to disrupt your flow (really appreciate your work). If this class didn't seem to be of help, just ignore it. |
Hi @shaoanlu and thanks for coming, I'm just taking your code, you still remain the one who owns it ;-) The approach I took for that, is a plugin architecture. The advantage of it is that many concurrent script can live together. What would be awesome is if you stick to the formalism of the plugins. If you have a look, you will see that for GAN plugin there are three classes: Model, Trainer and "Trainable" (maybe the name is not great). If you want, I can fork your repo and reshape your class so that it gets compatible. That way we could update this repo and you could make your code evolve without any burden. What do you think? |
@Clorr
|
Aw, it is from my last rebase. Remove the |
@Clorr
|
Ah, this is because the Comment out the And delete empty files |
@Clorr now this. Starting. Press "Enter" to stop training and save model |
Ah... (Should really have tested my rebase before pushing, sorry for that...)
You'll notice the new args param, passed to read_image |
its alright we are testing it :) |
now its says "Working" but i dont see the preview screen |
worked a little then gave this error File K:\faceswap-dev-gan-plugin\models\decoder_A_GAN.h5 does not exist, creating... |
removed the comments from load still the same error. [22:41:47] [51/150][50] Loss_DA: 0.193126 Loss_DB: 0.189106 Loss_GA: 0.245307 Loss_GB: 0.259314 |
About the preview, there are options to add at command line look inside Otherwise it starts being specific to the GAN code. If you are really motivated, you can look at the GAN repo. Otherwise, I will have a look tomorrow hopefully, and maybe, as I said to @shaoanlu , I'll rework the latest code from there Anyhow thanks for the feedback! |
im afraid its beyond me right now. |
Ah, I just had a closer look at the trace. The bug is in the preview, so the training certainly can go on without this. If you disable the preview, it may still be training... (In the trace, it says |
Without preview I would be flying blind so the speak. |
plugins/Model_GAN/Model.py
Outdated
return lambda img : converter.swap_face(img) | ||
|
||
def load(self, swapped): | ||
self.encoder.load_weights(ensure_file_exists(self.model_dir, encoderH5)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wrap all of the methods that load weights in a try/except
so that it will attempt to load, but still continue and create the models if they do not exist.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep, I did this when integrating another PR, but I did this a bit brainlessly ;-) I'll revert...
lib/training_data.py
Outdated
target_images[i] = target_img | ||
|
||
return warped_images, target_images | ||
def read_image(fn, random_transform_args=random_transform_args): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the default random_transform_args is not accessible from this context and is not passed from the method that is calling it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I moved the args in the model as the GAN has other params. But I messed up things when rebasing...
lib/training_data.py
Outdated
return warped_img, target_img | ||
|
||
# A generator function that yields epoch, batchsize of warped_img and batchsize of target_img | ||
def minibatch(data, batchsize): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing args
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as above
plugins/Model_GAN/Trainer.py
Outdated
], axis=1 ) | ||
|
||
figure = np.concatenate([figure_A, figure_B], axis=0 ) | ||
figure = figure.reshape((4,7) + figure.shape[1:]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Throws exception regarding invalid arguments.
lib/ModelAE.py
Outdated
], axis=1) | ||
|
||
figure = numpy.concatenate([figure_A, figure_B], axis=0) | ||
figure = figure.reshape((4, 7) + figure.shape[1:]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as other instance.
plugins/Model_GAN/Trainer.py
Outdated
|
||
figure = np.concatenate([figure_A, figure_B], axis=0 ) | ||
figure = figure.reshape((4,7) + figure.shape[1:]) | ||
figure = stack_images(figure) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
stack_images
is not accessible as it is not imported.
plugins/Model_GAN/Trainer.py
Outdated
|
||
figure = np.concatenate([figure_A, figure_B], axis=0 ) | ||
figure = figure.reshape((4,7) + figure.shape[1:]) | ||
figure = stack_images(figure) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's quite a few things broken. Neither the preview and the sample it writes work.
Using this code to train a model from scratch on GPU now. Will report back. |
Thanks! Anyhow I did not focus on that today. Today I did split the bugs + the models plugins from the GAN itself. Yesterday it was all done in one big commit. I'll work on the GAN version tomorrow if I can |
If you can have a look at the #59 , it is much more about new features and bug corrections... |
sometimes, I hate myself ;-) Look at Model_GEN/Model.py and rename the |
:( another |
Indeed I really hate myself ;-) I took the convert method from the old version of the gan plugin, but I changed it on the master btw... I pushed a new non tested version, I'll check it on my side (just I have not enough ram to have firefox and docker at the same time running, so it is a pain each time to switch...) |
:) |
Ok, I pushed a new version I was able to "test". I put quotes on it as I don't have a proper model to test, but at least it outputs something... Beside the converter method I did not update, there was a problem I missed. The GAN predictor outputs 2 images: a mask and an image. For now, I just use the image in the convert. I'll try to see later the GAN code to include the masking and all the nice features, but for now, I'm done, I won't have more time to work on it today... I hope it works for you! |
Thanks for your efforts @Clorr. I'll be testing it today. |
Hmm, no it is still a bit of work before using the mask. What has to be done, is to take the convert code from the @shaoanlu repo and put it into plugins/convert_GAN.py And also I have to see how I can have the converter return the mask without breaking other plugins. For now the Model_GAN/GANModel class returns only the image (the [1] in the converter lambda). It is not a big problem but I have to take care of it. However note that the model you train contains already the mask, so you won't have to retrain |
I trained it for 14000 iterations. [Image Removed] |
Tested on linux, similar results to Apollo, I only have 2 preview windows instead of 3, RAW and Mask. Both are reported by the DE as unresponsive windows. |
I'll have a look at the preview thing hopefully tomorrow. meanwhile, you can use the file preview and see if it gives correct results. For the convert, can anyone send me trained models? I won't be able to train on my actual machine... |
File preview writes 3 images, they are attached. I'm not sure if this is what you're looking for. They look fairly similar to the GAN scripts you forked from - however I never got those working correctly either. Models are at 15k epochs : Loss_DA: 0.264415 Loss_DB: 0.235073 Loss_GA: 0.228625 Loss_GB: 0.353471 [images removed] |
Thanks for the feedback. I don't remember of the blueish look when I played with the jupyter notebook version. Maybe @shaoanlu can confirm this is correct or not? |
We got these guest from Na'vi because of missing |
@Apollo122 thanks! You did a trump/cage model on GAN? |
@Clorr https://mega.nz/#!0h4FXKwT!SIf-YAhkgnBcJmVUxkWjf8vl5-0HVOTY95_estUdQ2M |
@Apollo122 those are file output jpegs, not preview screenshots. |
Ok. Here is an update. I took the advice from @shaoanlu and added in the cv2.cvtcolor() function and now my preview output has proper colors. However, even after 18k iterations the images are still pretty "messed" (too many drugs or something). Not sure if this has to do with image scaling from -1 to +1 instead of 0 to 1. I couldn't figure out where to change / add that in. I don't know any python at all, and googling this didn't help. Where do we go next? |
Hi guys, sorry but I'm in the middle of a process to move to another city, so that does not let me as much time as wanted. Also I only have an old imac at hand and it's a bit tedious to work on it... I'll try to see what I can do, but I can't guarantee anything. If you have patches, feel free to put them on my repo, and I'll add them to this PR @nkfs maybe you can modify the file directly in my repo, it will do a pull request I think For the +1 / -1 thing, if you can pinpoint the problem, that would be of great help. For the preview thing, I'll try to make something as it is more in the area of my habitual programming skills (but if you have propositions, feel free) |
@Clorr good luck with your move, this is a side project for all of us anyways. I'll try to figure out how to modify the file in your repo, I'm new to this whole dev thing. Been researching the +1 / -1 thing, I don't think it's affecting us? I feel like the current extract is already doing it, or maybe it's how the files get loaded in... I'm not sure. If you have a moment, could you explain how the 'plugin' system works? I'm summarizing that you're trying to make all the different components (align, merge, convert, etc.) into different files and then calling them in as plugins? The preview issue is of minimal value imo, the file output works. What needs to be done on the convert side to get the mask + img combined? Can we utilize shaoanlu's code for the inline cnn video convert? One issue I do see about it though is that it is very limited by a single CPU thread, it consumes all my vram, but the GPU isn't doing any heavy lifting at all. |
I think the extract is ok, I think the +1 -1 thing is more in the image load before training, but maybe @shaoanlu can point this in his own code. Plugins are just basic classes. The potentially tedious part is that a part of the code can be in a base class like "Model_Original inherits from ModelAE" so you have to open both classes to see the whole code. Also you should open the corresponding script to see where the code begins to run. The only strange looking part of the plugin system is the PluginLoader, but you shouldn't have to look at it. Just know that the plugin loader selects the good class for you depending on the string you give as load function argument. For the convert part, I'm not sure of what is up to date or not. In the https://github.com/shaoanlu/faceswap-GAN/blob/master/FaceSwap_GAN_v2_test_img.ipynb file, the model does not seem to correspond to what we have in the GAN model class. Also, this script shows an image and a mask separately. I think the https://github.com/shaoanlu/faceswap-GAN/blob/master/FaceSwap_GAN_v2_test_video.ipynb file is much more complete but has to be reworked to be applied to one image only. However, one good improvment of the video part is that it has a bounding box smoothed over multiple frames, and then it makes more sense to do a direct video convert. That's the meaning of my PR #54 |
I checked the code again and I think the input is already -1 ~ +1. There is a piece of code for video making of GAN class in this notebook. Modify the following line in video making cell to make it compatible with the plugins: result = gan.netGA.predict(np.array([ae_input])) # Change path_A/path_B here Change the above line to something like (not sure about the exact variable name used in the plugin): result = trainer.gan.netGA.predict(np.array([ae_input])) The output of netGA is a list of As nkfs had mentioned, it seems like there are some people encountering problems that dlib not using GPU for CNN acceleration. I don't know the cause of this issue. Perhaps it has something to do with installing dlib, e.g., not compiling dlib with CUDA support. |
The video script works well. CNN acceleration is hit and miss, but I get basically the same results with hog instead, so no loss there. |
Update GAN plugin to latest official version Correction for "cannot reshape" error on display_fig returning True in model for convert.py Adding named previews convert correction convert correction convert corrections gan convert correction
Closing this, to reopen a new one ;-) |
link to new one: |
This PR is specific to the GAN plugin that you can test and play with. Note that this is a Work in Progress and that latest features from the GAN repo are not totally integrated