Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] GAN plugin #60

Closed
wants to merge 1 commit into from
Closed

[DRAFT] GAN plugin #60

wants to merge 1 commit into from

Conversation

Clorr
Copy link
Contributor

@Clorr Clorr commented Jan 8, 2018

This PR is specific to the GAN plugin that you can test and play with. Note that this is a Work in Progress and that latest features from the GAN repo are not totally integrated

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

Note: Usage is the same as for other versions

@Apollo122 this is the new version. It is the same as the previous but it is seprated from other more generic modifications

@Jack29913
Copy link
Contributor

@Clorr got it. Will test it today.

@Clorr Clorr force-pushed the dev/gan-plugin branch 3 times, most recently from 6007c10 to db902a2 Compare January 8, 2018 17:26
@shaoanlu
Copy link

shaoanlu commented Jan 8, 2018

I wrote a FaceSwapGAN class. model.train_on_batch() and model.predict() are used so it looks like keras code now. But I haven't fully checked if it performs the same with original one (Running it for 30 mins, everything seems fine). I will report the results later.

Edit: Oops, I didn't notice that Clorr is already working on adding GAN as a plugin. Not intended to disrupt your flow (really appreciate your work). If this class didn't seem to be of help, just ignore it.

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

Hi @shaoanlu and thanks for coming,

I'm just taking your code, you still remain the one who owns it ;-)

The approach I took for that, is a plugin architecture. The advantage of it is that many concurrent script can live together. What would be awesome is if you stick to the formalism of the plugins.

If you have a look, you will see that for GAN plugin there are three classes: Model, Trainer and "Trainable" (maybe the name is not great). If you want, I can fork your repo and reshape your class so that it gets compatible. That way we could update this repo and you could make your code evolve without any burden.

What do you think?

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 8, 2018

@Clorr
when i tried to train i got this exception

Traceback (most recent call last): File "C:\IntelPython3\lib\threading.py", line 916, in _bootstrap_inner self.run() File "C:\IntelPython3\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "K:\faceswap-dev-gan-plugin\scripts\train.py", line 95, in processThread model = PluginLoader.get_model(variant)(self.arguments.model_dir) File "K:\faceswap-dev-gan-plugin\plugins\PluginLoader.py", line 13, in get_model return PluginLoader._import("Model", "Model_{0}".format(name)) File "K:\faceswap-dev-gan-plugin\plugins\PluginLoader.py", line 22, in _import module = __import__(name, globals(), locals(), [], 1) File "K:\faceswap-dev-gan-plugin\plugins\Model_GAN\__init__.py", line 7, in <module> from .Trainer import Trainer File "K:\faceswap-dev-gan-plugin\plugins\Model_GAN\Trainer.py", line 5, in <module> from lib.training_data import minibatchAB File "K:\faceswap-dev-gan-plugin\lib\training_data.py", line 7, in <module> def read_image(fn, random_transform_args=random_transform_args): NameError: name 'random_transform_args' is not defined

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

Aw, it is from my last rebase. Remove the =random_transform_args and that will work. I'll push a correction ASAP

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 8, 2018

@Clorr
now another error.

File K:\faceswap-dev-gan-plugin\models\encoder_GAN.h5 does not exist, creating... Exception in thread Thread-1: Traceback (most recent call last): File "C:\IntelPython3\lib\threading.py", line 916, in _bootstrap_inner self.run() File "C:\IntelPython3\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "K:\faceswap-dev-gan-plugin\scripts\train.py", line 96, in processThread model.load(swapped=False) File "K:\faceswap-dev-gan-plugin\plugins\Model_GAN\Model.py", line 159, in load self.encoder.load_weights(ensure_file_exists(self.model_dir, encoderH5)) File "C:\IntelPython3\lib\site-packages\keras\engine\topology.py", line 2616, in load_weights f = h5py.File(filepath, mode='r') File "C:\IntelPython3\lib\site-packages\h5py\_hl\files.py", line 269, in __init__ fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr) File "C:\IntelPython3\lib\site-packages\h5py\_hl\files.py", line 99, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py\h5f.pyx", line 78, in h5py.h5f.open OSError: Unable to open file (file signature not found)

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

Ah, this is because the ensure_file_exists creates empty files that cannot be read. Someone pointed this out, but I forgot to correct.

Comment out the load_weights lines, and train until files are saved once

And delete empty files

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 8, 2018

@Clorr
i commented like this in Model.py under Model_GAN
def load(self, swapped):
#self.encoder.load_weights(ensure_file_exists(self.model_dir, encoderH5))
#self.decoder_A.load_weights(ensure_file_exists(self.model_dir, decoder_AH5))
#self.decoder_B.load_weights(ensure_file_exists(self.model_dir, decoder_BH5))
#self.netDA.load_weights(ensure_file_exists(self.model_dir, netDAH5))
#self.netDB.load_weights(ensure_file_exists(self.model_dir, netDBH5))
print ("model loaded.")

now this.

Starting. Press "Enter" to stop training and save model
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\IntelPython3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\IntelPython3\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "K:\faceswap-dev-gan-plugin\scripts\train.py", line 107, in processThread
sample_gen = trainer.train_one_step(epoch)
File "K:\faceswap-dev-gan-plugin\plugins\Model_GAN\Trainer.py", line 33, in train_one_step
epoch, warped_A, target_A = next(self.images_A)
File "K:\faceswap-dev-gan-plugin\lib\training_data.py", line 31, in minibatchAB
batch = BackgroundGenerator(minibatch(images, batchsize, args))
TypeError: minibatch() takes 2 positional arguments but 3 were given

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

Ah... (Should really have tested my rebase before pushing, sorry for that...)
replace minibatch by

def minibatch(data, batchsize, args):
    length = len(data)
    epoch = i = 0
    shuffle(data)
    while True:
        size = batchsize
        if i+size > length:
            shuffle(data)
            i = 0
            epoch+=1        
        rtn = numpy.float32([read_image(data[j], args) for j in range(i,i+size)])
        i+=size
        yield epoch, rtn[:,0,:,:,:], rtn[:,1,:,:,:]       

You'll notice the new args param, passed to read_image

@Jack29913
Copy link
Contributor

its alright we are testing it :)

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 8, 2018

now its says "Working" but i dont see the preview screen
i used -p

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 8, 2018

worked a little then gave this error

File K:\faceswap-dev-gan-plugin\models\decoder_A_GAN.h5 does not exist, creating...
File K:\faceswap-dev-gan-plugin\models\decoder_B_GAN.h5 does not exist, creating...
File K:\faceswap-dev-gan-plugin\models\netDA_GAN.h5 does not exist, creating...
File K:\faceswap-dev-gan-plugin\models\netDB_GAN.h5 does not exist, creating...
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\IntelPython3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\IntelPython3\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "K:\faceswap-dev-gan-plugin\scripts\train.py", line 111, in processThread
self.show(sample_gen)
File "K:\faceswap-dev-gan-plugin\scripts\train.py", line 126, in show
cv2.imshow('', image_gen())
File "K:\faceswap-dev-gan-plugin\plugins\Model_GAN\Trainer.py", line 58, in
return lambda: self.show_sample()
File "K:\faceswap-dev-gan-plugin\plugins\Model_GAN\Trainer.py", line 64, in show_sample
self.showG(tA, tB, self.trainer_A.path, self.trainer_B.path)
File "K:\faceswap-dev-gan-plugin\plugins\Model_GAN\Trainer.py", line 80, in showG
figure = figure.reshape((4,7) + figure.shape[1:])
ValueError: cannot reshape array of size 2359296 into shape (4,7,3,64,64,3)

@Jack29913
Copy link
Contributor

removed the comments from load still the same error.
last print:

[22:41:47] [51/150][50] Loss_DA: 0.193126 Loss_DB: 0.189106 Loss_GA: 0.245307 Loss_GB: 0.259314
[22:41:50] Working...
[22:41:53] Working...
[22:41:55] Working...
[22:41:58] Working...
[22:42:01] Working...
[22:42:03] Working...
[22:42:06] Working...
[22:42:09] Working...
[22:42:11] Working...
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\IntelPython3\lib\threading.py", line 916, in _bootstrap_inner
..........

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

About the preview, there are options to add at command line look inside scripts/train.py if the config is correct

Otherwise it starts being specific to the GAN code. If you are really motivated, you can look at the GAN repo. Otherwise, I will have a look tomorrow hopefully, and maybe, as I said to @shaoanlu , I'll rework the latest code from there

Anyhow thanks for the feedback!

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 8, 2018

im afraid its beyond me right now.
no problem. thanks for putting your time into this

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

Ah, I just had a closer look at the trace. The bug is in the preview, so the training certainly can go on without this. If you disable the preview, it may still be training...

(In the trace, it says File "K:\faceswap-dev-gan-plugin\scripts\train.py", line 126, in show cv2.imshow('', image_gen()) which means the bug relies in the preview display...)

@Jack29913
Copy link
Contributor

Without preview I would be flying blind so the speak.
I can take a look again when it's ready

return lambda img : converter.swap_face(img)

def load(self, swapped):
self.encoder.load_weights(ensure_file_exists(self.model_dir, encoderH5))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap all of the methods that load weights in a try/except so that it will attempt to load, but still continue and create the models if they do not exist.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, I did this when integrating another PR, but I did this a bit brainlessly ;-) I'll revert...

target_images[i] = target_img

return warped_images, target_images
def read_image(fn, random_transform_args=random_transform_args):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the default random_transform_args is not accessible from this context and is not passed from the method that is calling it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I moved the args in the model as the GAN has other params. But I messed up things when rebasing...

return warped_img, target_img

# A generator function that yields epoch, batchsize of warped_img and batchsize of target_img
def minibatch(data, batchsize):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing args

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

], axis=1 )

figure = np.concatenate([figure_A, figure_B], axis=0 )
figure = figure.reshape((4,7) + figure.shape[1:])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Throws exception regarding invalid arguments.

lib/ModelAE.py Outdated
], axis=1)

figure = numpy.concatenate([figure_A, figure_B], axis=0)
figure = figure.reshape((4, 7) + figure.shape[1:])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as other instance.


figure = np.concatenate([figure_A, figure_B], axis=0 )
figure = figure.reshape((4,7) + figure.shape[1:])
figure = stack_images(figure)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stack_images is not accessible as it is not imported.


figure = np.concatenate([figure_A, figure_B], axis=0 )
figure = figure.reshape((4,7) + figure.shape[1:])
figure = stack_images(figure)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Copy link
Contributor

@Ganonmaster Ganonmaster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's quite a few things broken. Neither the preview and the sample it writes work.

@Ganonmaster
Copy link
Contributor

Using this code to train a model from scratch on GPU now. Will report back.

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

Thanks! Anyhow I did not focus on that today. Today I did split the bugs + the models plugins from the GAN itself. Yesterday it was all done in one big commit.

I'll work on the GAN version tomorrow if I can

@Clorr
Copy link
Contributor Author

Clorr commented Jan 8, 2018

If you can have a look at the #59 , it is much more about new features and bug corrections...

@Clorr
Copy link
Contributor Author

Clorr commented Jan 13, 2018

sometimes, I hate myself ;-)

Look at Model_GEN/Model.py and rename the def convert(self): into def converter(self):

@Jack29913
Copy link
Contributor

:( another
Loading Convert from Convert_GAN plugin...
Traceback (most recent call last):
File "faceswap.py", line 18, in
arguments.func(arguments)
File "K:\faceswap-dev-gan-plugin\lib\cli.py", line 52, in process_arguments
self.process()
File "K:\faceswap-dev-gan-plugin\scripts\convert.py", line 50, in process
converter = PluginLoader.get_converter(conv_name)(model.converter(False))
TypeError: converter() takes 1 positional argument but 2 were given

@Clorr
Copy link
Contributor Author

Clorr commented Jan 13, 2018

Indeed I really hate myself ;-)

I took the convert method from the old version of the gan plugin, but I changed it on the master btw... I pushed a new non tested version, I'll check it on my side (just I have not enough ram to have firefox and docker at the same time running, so it is a pain each time to switch...)

@Jack29913
Copy link
Contributor

:)
let me know when its ok to test.

@Clorr
Copy link
Contributor Author

Clorr commented Jan 13, 2018

Ok, I pushed a new version I was able to "test". I put quotes on it as I don't have a proper model to test, but at least it outputs something...

Beside the converter method I did not update, there was a problem I missed. The GAN predictor outputs 2 images: a mask and an image. For now, I just use the image in the convert. I'll try to see later the GAN code to include the masking and all the nice features, but for now, I'm done, I won't have more time to work on it today...

I hope it works for you!

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 13, 2018

Thanks for your efforts @Clorr. I'll be testing it today.
As for the mask, for me its the only reason why this method matters.
We can use a a command line parameter to tell convert class which image should it output and would be great if we do the same thing for preview screen.
But for now if its not that hard you can tell me where to change within code so i can try masked output as well later.

@Clorr
Copy link
Contributor Author

Clorr commented Jan 13, 2018

Hmm, no it is still a bit of work before using the mask.

What has to be done, is to take the convert code from the @shaoanlu repo and put it into plugins/convert_GAN.py

And also I have to see how I can have the converter return the mask without breaking other plugins. For now the Model_GAN/GANModel class returns only the image (the [1] in the converter lambda). It is not a big problem but I have to take care of it.

However note that the model you train contains already the mask, so you won't have to retrain

@Jack29913
Copy link
Contributor

So i started training. Now i see 3 preview windows but they are not working (Not responding).
Is it a Win10 problem or a general one?

a

@Jack29913
Copy link
Contributor

Jack29913 commented Jan 13, 2018

I trained it for 14000 iterations.
When i started to convert it I got this for all faces it merges:

[Image Removed]

@nkfs
Copy link

nkfs commented Jan 13, 2018

Tested on linux, similar results to Apollo, I only have 2 preview windows instead of 3, RAW and Mask. Both are reported by the DE as unresponsive windows.

@Clorr
Copy link
Contributor Author

Clorr commented Jan 13, 2018

I'll have a look at the preview thing hopefully tomorrow. meanwhile, you can use the file preview and see if it gives correct results.

For the convert, can anyone send me trained models? I won't be able to train on my actual machine...

@nkfs
Copy link

nkfs commented Jan 14, 2018

File preview writes 3 images, they are attached. I'm not sure if this is what you're looking for. They look fairly similar to the GAN scripts you forked from - however I never got those working correctly either. Models are at 15k epochs : Loss_DA: 0.264415 Loss_DB: 0.235073 Loss_GA: 0.228625 Loss_GB: 0.353471

[images removed]

@Clorr
Copy link
Contributor Author

Clorr commented Jan 14, 2018

Thanks for the feedback. I don't remember of the blueish look when I played with the jupyter notebook version. Maybe @shaoanlu can confirm this is correct or not?

@shaoanlu
Copy link

shaoanlu commented Jan 14, 2018

We got these guest from Na'vi because of missing cv2.cvtColor(...) in display function. The GAN model expect input as 64x64 BGR channel image, and it should be scaled to -1 to +1, not 0 to +1. The output of netGA and netGB are also 64x64 BGR image in range of -1 to +1.

@Jack29913
Copy link
Contributor

@nkfs you wrote you also got unresponsive preview windows before. What did you do differently to have preview windows working?

@Clorr I'll send you trump-cage model later today

@Clorr
Copy link
Contributor Author

Clorr commented Jan 14, 2018

@Apollo122 thanks! You did a trump/cage model on GAN?

@Jack29913
Copy link
Contributor

@Clorr
yeah. last print before save:
[468/num_epochs][13000] Loss_DA: 0.141957 Loss_DB: 0.211055 Loss_GA: 0.259931 Loss_GB: 0.237249

https://mega.nz/#!0h4FXKwT!SIf-YAhkgnBcJmVUxkWjf8vl5-0HVOTY95_estUdQ2M
includes photos as well.

@nkfs
Copy link

nkfs commented Jan 14, 2018

@Apollo122 those are file output jpegs, not preview screenshots.
@Clorr I didn't get blue output in the notebook version either. But I think the solution is already posted.

@nkfs
Copy link

nkfs commented Jan 16, 2018

Ok. Here is an update. I took the advice from @shaoanlu and added in the cv2.cvtcolor() function and now my preview output has proper colors. However, even after 18k iterations the images are still pretty "messed" (too many drugs or something). Not sure if this has to do with image scaling from -1 to +1 instead of 0 to 1. I couldn't figure out where to change / add that in. I don't know any python at all, and googling this didn't help. Where do we go next?

@Clorr
Copy link
Contributor Author

Clorr commented Jan 16, 2018

Hi guys, sorry but I'm in the middle of a process to move to another city, so that does not let me as much time as wanted. Also I only have an old imac at hand and it's a bit tedious to work on it... I'll try to see what I can do, but I can't guarantee anything. If you have patches, feel free to put them on my repo, and I'll add them to this PR

@nkfs maybe you can modify the file directly in my repo, it will do a pull request I think

For the +1 / -1 thing, if you can pinpoint the problem, that would be of great help.

For the preview thing, I'll try to make something as it is more in the area of my habitual programming skills (but if you have propositions, feel free)

@nkfs
Copy link

nkfs commented Jan 16, 2018

@Clorr good luck with your move, this is a side project for all of us anyways.

I'll try to figure out how to modify the file in your repo, I'm new to this whole dev thing. Been researching the +1 / -1 thing, I don't think it's affecting us? I feel like the current extract is already doing it, or maybe it's how the files get loaded in... I'm not sure.

If you have a moment, could you explain how the 'plugin' system works? I'm summarizing that you're trying to make all the different components (align, merge, convert, etc.) into different files and then calling them in as plugins?

The preview issue is of minimal value imo, the file output works. What needs to be done on the convert side to get the mask + img combined? Can we utilize shaoanlu's code for the inline cnn video convert? One issue I do see about it though is that it is very limited by a single CPU thread, it consumes all my vram, but the GPU isn't doing any heavy lifting at all.

@Clorr
Copy link
Contributor Author

Clorr commented Jan 16, 2018

I think the extract is ok, I think the +1 -1 thing is more in the image load before training, but maybe @shaoanlu can point this in his own code.


Plugins are just basic classes. The potentially tedious part is that a part of the code can be in a base class like "Model_Original inherits from ModelAE" so you have to open both classes to see the whole code. Also you should open the corresponding script to see where the code begins to run.

The only strange looking part of the plugin system is the PluginLoader, but you shouldn't have to look at it. Just know that the plugin loader selects the good class for you depending on the string you give as load function argument.


For the convert part, I'm not sure of what is up to date or not. In the https://github.com/shaoanlu/faceswap-GAN/blob/master/FaceSwap_GAN_v2_test_img.ipynb file, the model does not seem to correspond to what we have in the GAN model class. Also, this script shows an image and a mask separately. I think the https://github.com/shaoanlu/faceswap-GAN/blob/master/FaceSwap_GAN_v2_test_video.ipynb file is much more complete but has to be reworked to be applied to one image only. However, one good improvment of the video part is that it has a bounding box smoothed over multiple frames, and then it makes more sense to do a direct video convert. That's the meaning of my PR #54

@shaoanlu
Copy link

I checked the code again and I think the input is already -1 ~ +1.

There is a piece of code for video making of GAN class in this notebook. Modify the following line in video making cell to make it compatible with the plugins:

result = gan.netGA.predict(np.array([ae_input])) # Change path_A/path_B here

Change the above line to something like (not sure about the exact variable name used in the plugin):

result = trainer.gan.netGA.predict(np.array([ae_input])) 

The output of netGA is a list of [alpha_mask, bgr_image] where alpha_mask has shape (64, 64, 1) and bgr_image (64, 64, 3)

As nkfs had mentioned, it seems like there are some people encountering problems that dlib not using GPU for CNN acceleration. I don't know the cause of this issue. Perhaps it has something to do with installing dlib, e.g., not compiling dlib with CUDA support.

@nkfs
Copy link

nkfs commented Jan 18, 2018

The video script works well. CNN acceleration is hit and miss, but I get basically the same results with hog instead, so no loss there.

Update GAN plugin to latest official version

Correction for "cannot reshape" error on display_fig

returning True in model for convert.py

Adding named previews

convert correction

convert correction

convert corrections

gan convert correction
@Clorr
Copy link
Contributor Author

Clorr commented Feb 2, 2018

Closing this, to reopen a new one ;-)

@Clorr Clorr closed this Feb 2, 2018
@gdunstone
Copy link
Contributor

link to new one:
#102

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants