Add BART model #6
Conversation
seanmor5
force-pushed
the
sm-bart
branch
2 times, most recently
from
b178f6d
to
2c9cd29
Compare
| offset_position_ids = | ||
| Axon.nx(input_embeds, fn embeds -> | ||
| seq_len = Nx.axis_size(embeds, 1) | ||
| positions = Nx.iota({seq_len}) |
There was a problem hiding this comment.
Flax supports position_ids as an input on top of that, although pt and tf don't. For BERT however, all of them accept position_ids.
On a related note, flax doesn't accept inputs_embeds as an input, neither for BERT nor BART, perhaps to make things simpler? Is it common to use that?
There may be some inconsistencies between different models/implementations support, so I'm just pointing those out so we can try to make our implementations consistent.
There was a problem hiding this comment.
Yeah I noticed that as well, I think we should definitely try to standardize around something and support the same inputs per class of model. I think probably more flexibility is better? By accepting inputs_embeds you can choose possibly a different embedding e.g. specify a different initial state, but I don't know how common it is. Accepting position_ids is probably okay too, though I also don't know how common it is
Co-authored-by: Sean Moriarity <smoriarity.5@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
Co-authored-by: Jonatan Kłosko <jonatanklosko@gmail.com>
No description provided.