Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

foldxd fails over Iterators.Product #552

Closed
sethaxen opened this issue Mar 3, 2023 · 1 comment
Closed

foldxd fails over Iterators.Product #552

sethaxen opened this issue Mar 3, 2023 · 1 comment

Comments

@sethaxen
Copy link

sethaxen commented Mar 3, 2023

I first came across this trying to use FLoops to parallelize execution over an Iterators.Product. It worked fine for sequential or multithreaded execution but not for distributed execution:

julia> using Transducers

julia> trans = Iterators.product(1:10, 1:10) |> Transducers.Map(sum);

julia> foldl(+, trans)
1100

julia> foldxt(+, trans)
1100

julia> foldxd(+, trans)
ERROR: MethodError: no method matching firstindex(::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}})

Closest candidates are:
  firstindex(::Any, ::Any)
   @ Base abstractarray.jl:443
  firstindex(::Union{Tables.AbstractColumns, Tables.AbstractRow})
   @ Tables ~/.julia/packages/Tables/T7rHm/src/Tables.jl:182
  firstindex(::AbstractChar)
   @ Base char.jl:200
  ...

Stacktrace:
 [1] dtransduce(xform::Map{typeof(sum)}, step::Function, init::Transducers.InitOf{Transducers.DefaultInitOf}, coll0::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}; simd::Val{false}, basesize::Nothing, threads_basesize::Nothing, pool::Distributed.WorkerPool, _remote_reduce::Function)
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:78
 [2] dtransduce(xform::Map{typeof(sum)}, step::Function, init::Transducers.InitOf{Transducers.DefaultInitOf}, coll0::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:50
 [3] foldxd(step::Function, xform::Map{typeof(sum)}, itr::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}; init::Transducers.InitOf{Transducers.DefaultInitOf}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:40
 [4] foldxd(step::Function, xform::Map{typeof(sum)}, itr::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:40
 [5] foldxd(step::typeof(+), foldable::Transducers.Eduction{Transducers.Reduction{Map{typeof(sum)}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:42
 [6] foldxd(step::typeof(+), foldable::Transducers.Eduction{Transducers.Reduction{Map{typeof(sum)}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:42
 [7] top-level scope
   @ REPL[36]:1
MasonProtter added a commit that referenced this issue May 5, 2023
@MasonProtter
Copy link
Member

Should be fixed in 2fe7849, will be available once #553 merges, though I'm not super happy with the implementation since it uses Iterators.partition instead of halve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants