remove unsafe use of dictionary in multithreading

[OkonSamuel]

closes #48
I also reverted to using channels for Multithreading Threading reason being julia Ctrl + C cancellation doesn't yet work well for multithreading so i have to use channels for now to work around this.

[OkonSamuel]

Due to chunks iterator introduced and _process_accel_settings method, this PR requires a new tagged release of MLJBase.jl

[ablaom]

@OkonSamuel

Not had a chance to review but I see CI is failing on julia 1.0 where there is no multi-threading. Something to do with reversion to Channels?

[ablaom]

Due to chunks iterator introduced and _process_accel_settings method, this PR requires a new tagged release of MLJBase.jl

This should be live now but will also require you to wait for #50 because the new MLJBase requires MLJModelInterface 0.3. Aaarrrgghhh....

[ablaom]

Okay dev now has extended MLJModelInterface [compat]. Retriggering CI:

[ablaom]

@OkonSamuel Just a note that it is easier for me if you push your PR branches directly to github rather than basing the PR on a fork. You should have permission to do that.

[OkonSamuel]

Just a note that it is easier for me if you push your PR branches directly to github rather than basing the PR on a fork. You should have permission to do that.

Noted

[ablaom]

@OkonSamuel Many thanks for this!

Merge when ready.

We can tag a release but the previous PR to General is stalled for some reason.

[OkonSamuel]

@ablaom before i merge let me drawn your attention to an issue i have been researching for some time now. This unfortunately showed up in the tests

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

      From worker 2:	error in running finalizer: ErrorException("concurrency violation detected")

see it here.
I don't know if youve come across this while using MLJ on your Mac.??
I have been able to reproduce it on my PC(This was done using a Linux PC). But still this rarely occur. It only shows in some cases when i run evaluate using CPUThreads for the first time after MLJ is loaded(Maybe it has something to do with compilation??)
Subsequent runs doesn't show this warning no matter what i do.
The weird thing is that it never halts the program or affects any results in any way.

[ablaom]

@OkonSamuel Thank you very much indeed for looking into this. Unfortunately, I'm not sure what to add except that, as far as I know, I have not observed this and cannot reproduce the CI "failure"

julia> versioninfo()
Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.0.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_PATH = /Applications/Julia-1.3.app/Contents/Resources/julia/bin/julia
  JULIA_NUM_THREADS = 5

Had you observed this prior in commits prior to this PR?

[OkonSamuel]

Had you observed this prior in commits prior to this PR?

@ablaom. No. I Observed this sometime ago on my PC.
The following code reproduces it.

julia> using Distributed

julia> addprocs(2)

julia> @everywhere using MLJ
#The module must be precompiled for error to occur below
julia> @everywhere logic = @load LogisticClassifier pkg=MLJLinearModels
julia> X = table(rand(10000,3));

julia> y = categorical(repeat(["up","no","yes","down"],outer=2500));

julia> mach = machine(logic, X, y)

julia> evaluate!(mach, acceleration = CPUThreads(), repeats=10, verbosity=1);
#At this point the warning should popup but won't stop the evaluation

To get the warning to show up is tricky and you have to edit MLJBase locally at your .julia folder. (You can simple add blank spaces. This is to force MLJ to precompile again)
PS
If you can't reproduce this on Mac then it means it only occurs on a Linux

[OkonSamuel]

@ablaom. I think it's coming from get_measurements method in evaluate method from MLJBase. ( It might be due to precompilation??. Since it only occurs once after compilation and never shows up again).
I may have a fix (This basically makes a single threaded call to get_measurements once before calling it in other threads.) for this but am still testing to see if it's fixes it.(So far i haven't seen the error)

[ablaom]

Happy to wait for your experiements. Many thanks!

[OkonSamuel]

@ablaom. I've implemented a fix here.
This has the consequence of being slower than the current implementation for computationally intensive algorithms (still faster that CPU1 option though).
If you ask me i prefer the current implementation( since these errors only pops up sometimes when MLJ is precompiled).
What's your take on this?

[ablaom]

I'm still happy for you to merge when ready. See my comment here: JuliaAI/MLJBase.jl#338 (comment)