Deprecate Beam API and download from HF GCS bucket

[mariosasko]

Deprecate the Beam API and download from the HF GCS bucked.

TODO:

• Convert the Beam-based wikipedia to an Arrow-based dataset (Hub PR)
• Make natural_questions a no-code dataset (Hub PR)
• Make wiki40b a no-code dataset (Hub PR)
• Make wiki_dpr an Arrow-based dataset (Hub PR)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

Awesome ! Bye bye beam 😄

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005280 / 0.011353 (-0.006073)	0.003770 / 0.011008 (-0.007238)	0.064320 / 0.038508 (0.025812)	0.031250 / 0.023109 (0.008141)	0.245113 / 0.275898 (-0.030785)	0.268646 / 0.323480 (-0.054834)	0.003147 / 0.007986 (-0.004839)	0.002736 / 0.004328 (-0.001592)	0.050802 / 0.004250 (0.046551)	0.044539 / 0.037052 (0.007487)	0.261702 / 0.258489 (0.003213)	0.304696 / 0.293841 (0.010855)	0.028601 / 0.128546 (-0.099945)	0.010731 / 0.075646 (-0.064915)	0.208183 / 0.419271 (-0.211089)	0.036597 / 0.043533 (-0.006936)	0.245107 / 0.255139 (-0.010032)	0.265936 / 0.283200 (-0.017264)	0.019430 / 0.141683 (-0.122253)	1.184697 / 1.452155 (-0.267458)	1.206073 / 1.492716 (-0.286643)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.110242 / 0.018006 (0.092236)	0.304185 / 0.000490 (0.303695)	0.000220 / 0.000200 (0.000020)	0.000047 / 0.000054 (-0.000008)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018839 / 0.037411 (-0.018573)	0.062840 / 0.014526 (0.048314)	0.075849 / 0.176557 (-0.100708)	0.122983 / 0.737135 (-0.614153)	0.075352 / 0.296338 (-0.220987)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.283617 / 0.215209 (0.068408)	2.801903 / 2.077655 (0.724248)	1.447678 / 1.504120 (-0.056442)	1.327561 / 1.541195 (-0.213633)	1.348714 / 1.468490 (-0.119776)	0.574283 / 4.584777 (-4.010493)	2.449396 / 3.745712 (-1.296316)	2.908659 / 5.269862 (-2.361203)	1.813935 / 4.565676 (-2.751742)	0.063141 / 0.424275 (-0.361134)	0.005024 / 0.007607 (-0.002583)	0.338854 / 0.226044 (0.112810)	3.346680 / 2.268929 (1.077752)	1.833792 / 55.444624 (-53.610832)	1.553680 / 6.876477 (-5.322796)	1.590403 / 2.142072 (-0.551670)	0.657442 / 4.805227 (-4.147785)	0.119961 / 6.500664 (-6.380703)	0.042602 / 0.075469 (-0.032867)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.977510 / 1.841788 (-0.864278)	12.105448 / 8.074308 (4.031140)	9.635282 / 10.191392 (-0.556110)	0.132453 / 0.680424 (-0.547971)	0.014225 / 0.534201 (-0.519976)	0.292847 / 0.579283 (-0.286436)	0.271498 / 0.434364 (-0.162866)	0.329785 / 0.540337 (-0.210552)	0.435058 / 1.386936 (-0.951879)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005656 / 0.011353 (-0.005697)	0.003903 / 0.011008 (-0.007106)	0.050670 / 0.038508 (0.012162)	0.032405 / 0.023109 (0.009296)	0.271443 / 0.275898 (-0.004455)	0.298989 / 0.323480 (-0.024491)	0.004397 / 0.007986 (-0.003589)	0.003027 / 0.004328 (-0.001301)	0.050227 / 0.004250 (0.045977)	0.047798 / 0.037052 (0.010745)	0.287610 / 0.258489 (0.029120)	0.314084 / 0.293841 (0.020243)	0.030406 / 0.128546 (-0.098140)	0.011159 / 0.075646 (-0.064487)	0.059510 / 0.419271 (-0.359762)	0.056842 / 0.043533 (0.013309)	0.274653 / 0.255139 (0.019514)	0.290175 / 0.283200 (0.006976)	0.019822 / 0.141683 (-0.121861)	1.168894 / 1.452155 (-0.283261)	1.205867 / 1.492716 (-0.286850)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094907 / 0.018006 (0.076901)	0.304920 / 0.000490 (0.304431)	0.000218 / 0.000200 (0.000018)	0.000048 / 0.000054 (-0.000006)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023015 / 0.037411 (-0.014396)	0.076868 / 0.014526 (0.062342)	0.089099 / 0.176557 (-0.087458)	0.128786 / 0.737135 (-0.608350)	0.090836 / 0.296338 (-0.205503)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.284552 / 0.215209 (0.069343)	2.800870 / 2.077655 (0.723215)	1.577167 / 1.504120 (0.073047)	1.448788 / 1.541195 (-0.092406)	1.475493 / 1.468490 (0.007003)	0.566639 / 4.584777 (-4.018138)	2.503671 / 3.745712 (-1.242041)	2.796769 / 5.269862 (-2.473092)	1.829344 / 4.565676 (-2.736332)	0.064200 / 0.424275 (-0.360075)	0.005066 / 0.007607 (-0.002541)	0.335390 / 0.226044 (0.109346)	3.421265 / 2.268929 (1.152336)	1.942245 / 55.444624 (-53.502379)	1.641401 / 6.876477 (-5.235076)	1.824574 / 2.142072 (-0.317499)	0.661379 / 4.805227 (-4.143849)	0.117207 / 6.500664 (-6.383457)	0.041630 / 0.075469 (-0.033839)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.009636 / 1.841788 (-0.832152)	12.998379 / 8.074308 (4.924071)	10.609013 / 10.191392 (0.417621)	0.142794 / 0.680424 (-0.537630)	0.015648 / 0.534201 (-0.518553)	0.292825 / 0.579283 (-0.286458)	0.280790 / 0.434364 (-0.153574)	0.329016 / 0.540337 (-0.211322)	0.442440 / 1.386936 (-0.944496)