[Linux] Fail to run on Linux without reverting linux fix

[DieselWoodard]

If I try run after straight checkout i get 'TypeError: can't pickle weakref objects'

I saw there was commit for fixin linux, but it seems to be opposite, if I remove 'multiprocessing.set_start_method("spawn")' again from main.py it works.

so unless 296e8a1 is necessary for windows I think it should be reverted to make it work on windows

[iperov]

windows works with and without
multiprocessing.set_start_method("spawn")

@lbfs

[lbfs]

I don't understand why this would break it, let me look into it because I haven't been able to have this issue on my system.

Can you tell me what operating system, python version, you're using and how you've installed DeepFaceLab deps

What part of deepfacelab are you having issues with exactly; if you could post the command you're using to run that'd be appreciative.

[lbfs]

Also I'm slightly confused by your issue, you state at the end of your post that you need to revert it to make it work on Windows, but in your title it says Fails to Run on Linux?

[lbfs]

Check to see if you're using Python 3.7, apparently this is a common error with a lot of libraries on Python 3.7?

Also might be a good idea to reinstall the requirements-cuda.txt to make sure your on correct versions.

I don't think this multiprocessing fix has anything to do with this issue, everything works fine on my end.

[lbfs]

@DieselWoodard Hello?

[TheFeelTrain]

I have the same issue. Removing that line fixed it for me as well.

[iperov]

ROFL
i can remove
multiprocessing.set_start_method("spawn")
this never affect windows.

[iperov]

But I will not write python code for different versions of linux. This breaks the paradigm of python.

[lbfs]

Can you please tell me what version of Python you're using, what operating system you are using, and the way you are launching DeepFaceLab, and current dependencies. @TheFeelTrain . I can't suddenly have a magical fix without knowing additional information.

I would like to see a traceback too. There is no reason this fix shouldn't work on other Linux distros.

[lbfs]

The only reason I think this could fail is if you're launching it from another python application which is already setting a start mode of fork.

But this fix is required for the extractor to work, as detailed here https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods which shows that Linux and Windows share different default multiprocess logic so it needs to be defined as which one is being used.

All Linux distros should default to fork, but be set to spawn by the fix to match Windows behavior, which is why I know this shouldn't have anything to do with the issues encountered.

[TheFeelTrain]

Yeah the extractor fails without the line now. I'm not home but off the top of my head it's Arch Linux, all python dependencies are the same version as in the requirements-cuda.txt, and it's python 3.7.2 from the official Arch repos.

[lbfs]

I have a feeling this is related to Python 3.7.2 because I've read about similar errors caused in Python 3.7.

Although I'm surprised you even got tensorflow-gpu to install because its not a satisfiable requirement of 3.7.2, unless you're using Tensorflow 1.13

[TheFeelTrain]

I will try downgrading to 3.6 when I get back home and I'll let you know how it goes.

[lbfs]

Can confirm that using the Anaconda installation instructions work on Arch Linux after testing, you will need to make sure your newest video driver is installed and that you've installed ffmpeg, cmake, git, and probably gcc. https://github.com/lbfs/DeepFaceLab_Linux/blob/master/README.md

I'm not having luck setting up a stable Python3.7 environment, but I'll look more into it later this weekend when I have time.

[lbfs]

Full traceback on Python 3.7

Traceback (most recent call last):
  File "/mnt/secondary/DeepFaceLab_Linux/mainscripts/Trainer.py", line 78, in trainerThread
    loss_string = model.train_one_iter()
  File "/mnt/secondary/DeepFaceLab_Linux/models/ModelBase.py", line 360, in train_one_iter
    sample = self.generate_next_sample()
  File "/mnt/secondary/DeepFaceLab_Linux/models/ModelBase.py", line 357, in generate_next_sample
    return [next(generator) for generator in self.generator_list]
  File "/mnt/secondary/DeepFaceLab_Linux/models/ModelBase.py", line 357, in <listcomp>
    return [next(generator) for generator in self.generator_list]
  File "/mnt/secondary/DeepFaceLab_Linux/samples/SampleGeneratorFace.py", line 58, in __next__
    return next(generator)
  File "/mnt/secondary/DeepFaceLab_Linux/utils/iter_utils.py", line 55, in __next__
    self.p.start()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle weakref objects
Done.

I imagine this issue would happen on Windows as well with Python 3.7 regardless of the multiprocessing fix or not.

Does not occur with Python 3.6

https://bugs.python.org/issue34034