How to make zip work with more than 6 elements?

[hnorkowski]

I want to use zip (or azip!) with 7 producers but its only implemented for max 6 producers. Is there a convenient way to also implement it for more producers?

Minimal Example Code:

let v1: Array1<i32> = Array1::zeros(3);
let v2: Array1<i32> = Array1::zeros(3);
let v3: Array1<i32> = Array1::zeros(3);
let v4: Array1<i32> = Array1::zeros(3);
let v5: Array1<i32> = Array1::zeros(3);
let v6: Array1<i32> = Array1::zeros(3);
let v7: Array1<i32> = Array1::zeros(3);
Zip::from(&v1)
	.and(&v2)
	.and(&v3)
	.and(&v4)
	.and(&v5)
	.and(&v6)
	.and(&v7)
	.for_each(|v1, v2, v3, v4, v5, v6, v7| {
		println!("{}, {}, {}, {}, {}, {}, {}", v1, v2, v3, v4, v5, v6, v7);
	});

Error:

error[E0599]: no method named and found for struct ndarray::Zip<(ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>), Dim<[usize; 1]>> in the current scope
--> src\main.rs:47:14
|
47 | .and(&v7)
| ^^^ method not found in ndarray::Zip<(ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>, ArrayBase<ViewRepr<&i32>, Dim<[usize; 1]>>), Dim<[usize; 1]>>
|
= note: the method was found for
- ndarray::Zip<(P1,), D>
- ndarray::Zip<(P1, P2), D>
- ndarray::Zip<(P1, P2, P3), D>
- ndarray::Zip<(P1, P2, P3, P4), D>
- ndarray::Zip<(P1, P2, P3, P4, P5), D>

[nilgoyette]

It's not currently possible with ndarray0.15. It might be possible some day with the new features Rust keeps adding, I don't know.

What I do when I need to Zip on "too much" elements is

• Try to refactor. I don't know your precise problem but it's sometimes possible to
  • reduce the number of arrays. Maybe by merging some of them Array1<(f32, f32)>. This is sometimes possible if you're are creating them just before the Zip.
  • separate your work into 2 blocks and loop 2 times instead of 1.
• Use indexed and use the index to get the missing elements.
• Prepare an iterator beforehand and call .next() at the start of the Zip.

Of course, those are all hacks, but we must live with the limit. If they make it possible to work with 7 arrays, someone will ask for 8 :)

[cuihantao]

Coming from Python and Julia in my area of simulation, the expressions come from SymPy and can often involve many variables. I find the limitation on the number of produces in Zip a bit unnatural.

I started this discussion #1240 for figuring out the correct way to vectorize (SIMD) element-wise operation. A separate but possible related question is, how to get optimized performance for element-wise operations that involve more than six arrays?

[felsenhower]

I agree that support for more than 6 producers would make some patterns way more comfortable to implement. For example: Iterating over a matrix using Zip with a "stencil pattern" may involve more than 6 iterators.

I guess the workarounds from @nilgoyette work, but I don't find them very satisfying.

It's not currently possible with ndarray0.15. It might be possible some day with the new features Rust keeps adding, I don't know.

Could you elaborate on that a bit? Is there any feature missing in Rust that keeps you from implementing this?

Of course, those are all hacks, but we must live with the limit. If they make it possible to work with 7 arrays, someone will ask for 8 :)

Why? Obviously the optimal solution would be an arbitrary number of producers, but I think we can all agree that we can settle for more than anybody could ever need?

I implemented Zip for up to 9 producers here (I only chose 9 to keep the parameter names short (P9 but not P10), but I guess we could also have more.

felsenhower@b302ab3

I hope I updated all relevant parts in the source code. Shall I create a PR?

[nilgoyette]

Could you elaborate on that a bit?

Not much :) With or without new features, the maintainers picked the design we have and iiuc we would have to refactor the project to have a generic way.

Since it's so easy to add more, maybe we should add more. But please add at least one test with a Zip over 9 arrays. I don't know why they picked a limit of 6. It might have to do with the memory cache. Maybe iterating over 20 arrays kills the performance.

[felsenhower]

Since it's so easy to add more, maybe we should add more. But please add at least one test with a Zip over 9 arrays. I don't know why they picked a limit of 6. It might have to do with the memory cache. Maybe iterating over 20 arrays kills the performance.

I wrote some tests for azip! and par_azip! with 9 producers here: 7758ffb

I can't say anything about the performance. But if a Zip with 9 producers delivers a bad performance, I still would argue that it should be possible, at least for prototyping and testing purposes.

If the changes are approved, I'll create a PR.

[nilgoyette]

Yes, I'd approve them.