Skip to content

Speeding up onehotbatch by creating OneHotArray directly #1888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Mar 4, 2022
Merged

Speeding up onehotbatch by creating OneHotArray directly #1888

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
8db7e2a
Speeding up `onehotbatch` by constructing array directly
TLipede Feb 22, 2022
af976e0
fixing error cases
TLipede Feb 22, 2022
0e7b608
can't replace in-place
TLipede Feb 22, 2022
5ed74b6
Update src/onehot.jl
TLipede Feb 22, 2022
5041a27
Update src/onehot.jl
TLipede Feb 22, 2022
2fd1515
Update src/onehot.jl
TLipede Feb 22, 2022
d9eae39
changed broadcasts to map; new variable name for replaced indices
TLipede Feb 22, 2022
421b9b3
Added a method for strings
TLipede Feb 23, 2022
d041014
Error should be given on default value, not default index
TLipede Feb 23, 2022
eb15e85
Removed string method; changed `map`s to a comprehensions
TLipede Feb 23, 2022
5e5dc7d
fixed doctests
TLipede Feb 23, 2022
55572ab
using something instead of calling findval twice; fixing case where f…
TLipede Feb 24, 2022
6ad6e97
additional test probably isn't necessary
TLipede Feb 24, 2022
783fa7d
Make error messaging for unexpected values simpler
TLipede Feb 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions src/onehot.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,9 +183,24 @@ julia> reshape(1:15, 3, 5) * oh # this matrix multiplication is done efficientl
3 6 15 3 9 3 12 3 6 15 3
```
"""
onehotbatch(ls, labels, default...) = _onehotbatch(ls, length(labels) < 32 ? Tuple(labels) : labels, default...)
# NB function barier:
_onehotbatch(ls, labels, default...) = batch([onehot(l, labels, default...) for l in ls])
onehotbatch(data, labels, default...) = _onehotbatch(data, length(labels) < 32 ? Tuple(labels) : labels, default...)

function _onehotbatch(data, labels)
indices = UInt32[something(_findval(i, labels), 0) for i in data]
if 0 in indices
for x in data
isnothing(_findval(x, labels)) && error("Value $x not found in labels")
end
end
return OneHotArray(indices, length(labels))
end

function _onehotbatch(data, labels, default)
default_index = _findval(default, labels)
isnothing(default_index) && error("Default value $default is not in labels")
indices = UInt32[something(_findval(i, labels), default_index) for i in data]
return OneHotArray(indices, length(labels))
end

"""
onecold(y::AbstractArray, labels = 1:size(y,1))
Expand Down
2 changes: 2 additions & 0 deletions test/onehot.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ using Test
@test onehotbatch("abc", 'a':'c') == Bool[1 0 0; 0 1 0; 0 0 1]
@test onehotbatch("zbc", ('a', 'b', 'c'), 'a') == Bool[1 0 0; 0 1 0; 0 0 1]

@test onehotbatch([10, 20], [30, 40, 50], 30) == Bool[1 1; 0 0; 0 0]

@test_throws Exception onehotbatch([:a, :d], [:a, :b, :c])
@test_throws Exception onehotbatch([:a, :d], (:a, :b, :c))
@test_throws Exception onehotbatch([:a, :d], [:a, :b, :c], :e)
Expand Down