Bbaw egyptian

[phiwi]

This is the "hieroglyph corpus" that I could unfortunately not contribute during the marathon. I re-extracted it again now, so that it is in the state as used in my paper (seee documentation). I hope it satiesfies your requirements and wish every scientist out their loads of fun deciphering a 5.000 years old language :-)

[albertvillanova]

Hi @phiwi,

Thanks for contributing this nice dataset. If you have any blocking problem or question, do not hesitate to ask here. We are pleased to help you.

Could you please first synchronize with our master branch? From your branch bbaw_egyptian, type:

git fetch upstream master
git merge upstream/master

[albertvillanova]

You should also remove the file datasets/dummy/0.0.0/dummy_data.zip, because you have already attached the dummy data in datasets/bbaw_egyptian/dummy/0.0.0/dummy_data.zip.

[lhoestq]

Thanks for adding this one :)

I left a few comments
Also could you remove the file at datasets/dummy/0.0.0/dummy_data.zip please ?

[lhoestq]

specialized egyptologists is not a valid annotations_creators tag. You can use this instead:

annotations_creators:
- expert-generated

There is a tool to create those tags here.

For the languages, there should be one language per line:

- de
- en
- eg

Finally the task_ids tags are missing:

task_categories:
- conditional-text-generation
task_ids:
- machine-translation

[lhoestq]

Suggested change

	### Contributions
	### Contributions
	Thanks to [@phiwi](https://github.com/phiwi) for adding this dataset.

[lhoestq]

There's no extraction

Suggested change

	data_dir = dl_manager.download_and_extract(my_urls)
	data_dir = dl_manager.download(my_urls)

[lhoestq]

Thanks ! Can you check that you have black==21.4b0 and run make style again ? This should fix the "check_code_quality" CI issue

[phiwi]

Reformatted with black.

[albertvillanova]

Hi @phiwi, there are still some minor problems in relation with the tags you used in the dataset card (README.md).

Here you can find the output of the metadata validator:

WARNING:root:❌ Failed to validate 'datasets/bbaw_egyptian/README.md':
Could not validate the metada, found the following errors:
* field 'size_categories':
	['100K<n<1000K'] are not registered tags for 'size_categories', reference at https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources/size_categories.json
* field 'task_ids':
	['machine translation'] are not registered tags for 'task_ids', reference at https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources/tasks.json
* field 'languages':
	['eg'] are not registered tags for 'languages', reference at https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources/languages.json

[phiwi]

@albertvillanova corrected :-)

[albertvillanova]

Thanks, @phiwi. Now all tests should pass green.

However, I think there is still an issue with the language code:

• the code for the Ancient Egyptian is not ar-EG
• there is no ISO 639-1 code for the Ancient Egyptian
• there is an ISO 639-2 code: egy; but this code will not pass the validation test because it is not in the list of valid codes

I am not sure what to do in this case... Maybe @lhoestq has an idea? Maybe adding the code to the list? https://github.com/huggingface/datasets/blob/master/src/datasets/utils/resources/languages.json

[albertvillanova]

I have just checked that in the list of valid codes there are already ISO 639-2 codes. Therefore, I would suggest you to add it to the list:

"egy": "Egyptian (Ancient)",

and change it in the dataset card.

[phiwi]

Done.

[phiwi]

Hope, everything is okay right now.

[albertvillanova]

It looks good to me. Let's see if @lhoestq has any other suggestions before merging it to master.

[lhoestq]

Looks all good now thanks !