Dummy dataloader v2#534
Conversation
|
@sg-wbi looks like this is fine; to run, I did the following: The generator is very clever. Can we add in the
this is a lot of work so i dont mind if its a TODO but this is good to merge as is imo |
|
@sg-wbi are we merging this? |
|
If you think it looks good we can merge it!
You are right for completeness we should have these too but it will have to wait for now I guess... |
|
I am merging this as a first building block. I created this #665 so we do not forget about this. |
Supersedes #524
It adds a
DummyKBExampleto reduce boilerplate and sets examples per split, this way we can test the bypass args here #533.An example is provided with dummy_kb_dataloader.py which implements an example which fails the "global_unque_ids" test.