Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProcessExitedException when using @showprogress @distributed with file operations #242

Closed
mattwigway opened this issue Jul 6, 2022 · 1 comment · Fixed by #295
Closed

Comments

@mattwigway
Copy link

The documentation says that @distributed for without a reducer implies @sync but when I run a loop that contains a file operation as the last step in my program, I get the following error:

Unhandled Task ERROR: ProcessExitedException(4)

...and 5 more exceptions.

Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:381
 [2] (::Distributed.var"#177#179"{var"#3#8"{RemoteChannel{Channel{Bool}}}, UnitRange{Int64}})()
   @ Distributed ./task.jl:400

As far as I can tell, the tasks actually completed correctly. It seems to be related to workers getting shut down too quickly. If I add sleep(10) at the end of the program the error goes away. It also goes away if I replace @showprogress with @sync.

Julia 1.7.3 x86 build, running on an Apple M1, with ProgressMeter 1.7.2. MWE below.

using Distributed

addprocs(exeflags="--project=$(Base.active_project())")

@everywhere using ProgressMeter

function main()
    fname = ARGS[1]
    @showprogress @distributed for i in 1:10
        open(fname) do str
            open("/tmp/$i.txt", "w") do out
                write(out, read(str))
            end
        end
    end
end

main()
@MarcMush
Copy link
Collaborator

fixed by #295

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants