Skip to content

Dummy dataloader v2#534

Merged
sg-wbi merged 3 commits intobigscience-workshop:masterfrom
sg-wbi:dummy_dataloader_v2
May 31, 2022
Merged

Dummy dataloader v2#534
sg-wbi merged 3 commits intobigscience-workshop:masterfrom
sg-wbi:dummy_dataloader_v2

Conversation

@sg-wbi
Copy link
Copy Markdown
Collaborator

@sg-wbi sg-wbi commented May 2, 2022

Supersedes #524

It adds a DummyKBExample to reduce boilerplate and sets examples per split, this way we can test the bypass args here #533.

An example is provided with dummy_kb_dataloader.py which implements an example which fails the "global_unque_ids" test.

@sg-wbi
Copy link
Copy Markdown
Collaborator Author

sg-wbi commented May 3, 2022

@hakunanatasha @galtay

@hakunanatasha hakunanatasha self-assigned this May 4, 2022
@hakunanatasha
Copy link
Copy Markdown
Collaborator

@sg-wbi looks like this is fine; to run, I did the following:
python -m tests.test_bigbio tests/dummy/dummy_kb_dataloader.py and saw a failure.

The generator is very clever. Can we add in the tests case one for every sub function? This should be

  • passage offsets
  • entity offsets
  • event offsets
  • coreference
  • q+a
  • text2text
  • text labels

this is a lot of work so i dont mind if its a TODO but this is good to merge as is imo

@hakunanatasha
Copy link
Copy Markdown
Collaborator

@sg-wbi are we merging this?

@sg-wbi
Copy link
Copy Markdown
Collaborator Author

sg-wbi commented May 23, 2022

If you think it looks good we can merge it!

Can we add in the tests case one for every sub function? This should be

passage offsets
entity offsets
event offsets
coreference
q+a
text2text
text labels

this is a lot of work so i dont mind if its a TODO but this is good to merge as is imo

You are right for completeness we should have these too but it will have to wait for now I guess...

@sg-wbi sg-wbi mentioned this pull request May 31, 2022
@sg-wbi
Copy link
Copy Markdown
Collaborator Author

sg-wbi commented May 31, 2022

I am merging this as a first building block. I created this #665 so we do not forget about this.

@sg-wbi sg-wbi merged commit 12188f3 into bigscience-workshop:master May 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants