Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Crash][Reproducible] EOFError: Ran out of input when import yapf with multiprocess #1204

Closed
whlook opened this issue Feb 27, 2024 · 1 comment · Fixed by #1243
Closed

Comments

@whlook
Copy link

whlook commented Feb 27, 2024

previous similar issue #1164

Problems

  • When i use yapf (just import it) in multiprocess program, it will crash with 'EOFError: Ran out of inpu',log info below:

image

Reproduction

  1. yapf==0.40.2
  2. rm -r ~/.cache/YAPF
  3. run this python code:
 import multiprocessing
 
 def proc():
     import yapf
 
 if __name__ == "__main__":
     pp=[]
     for i in range(100):
         p = multiprocessing.Process(target=proc)
         pp.append(p)
 
     for p in pp:
         p.start()
 
     for p in pp:
         p.join()
                                                   

Reason

the problem code is here:

When in multi process enviroment,some process will create grammar cache(when ~/.cache/YAPF not exists), before the process write to the cache file, other process will see the file exist and load it, but the cache file is empty right now , so EOFError will raise and the program crashed.

How to fix it?

@whlook whlook changed the title [Bug] [Crash] EOFError: Ran out of input when import yapf with multiprocess [Bug] [Crash][Reproducible] EOFError: Ran out of input when import yapf with multiprocess Feb 27, 2024
@hartwork
Copy link
Contributor

hartwork commented Oct 3, 2024

@whlook that's a great report, in particular the reproducer is very handy — thank you!

I have a pull request coming up for this — should be auto-linked below in a minute or two —, and I would like to share my extended version of your reproducer below:

import multiprocessing

def cleanup():
    import os
    import yapf
    from yapf_third_party._ylib2to3.pgen2.driver import _generate_pickle_name
    for i in ("Grammar.txt", "PatternGrammar.txt"):
        filename = _generate_pickle_name(i)
        print(f"  Removing {filename}...")
        os.remove(_generate_pickle_name(i))

def proc():
    import yapf

if __name__ == "__main__":
    max_parallelity = 30
    for parallelity in range(2, max_parallelity + 1):
        print("Cleaning up...")
        cleanup_p = multiprocessing.Process(target=cleanup)
        cleanup_p.start()
        cleanup_p.join()

        print(f"Testing for the race condition with {parallelity} processes...")
        pp = []
        for i in range(parallelity):
            p = multiprocessing.Process(target=proc)
            pp.append(p)

        for p in pp:
            p.start()

        for p in pp:
            p.join()

        if any(p.exitcode != 0 for p in pp):
            print(f"Done, race condition proven above, took {parallelity} processes.")
            break
    else:
        print("Done, no luck crashing this time.")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants