Release: 2.12.0

[lhoestq]

No description provided.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008303 / 0.011353 (-0.003050)	0.005681 / 0.011008 (-0.005327)	0.111830 / 0.038508 (0.073322)	0.039222 / 0.023109 (0.016112)	0.336773 / 0.275898 (0.060875)	0.376673 / 0.323480 (0.053193)	0.006756 / 0.007986 (-0.001230)	0.006078 / 0.004328 (0.001749)	0.083552 / 0.004250 (0.079301)	0.054430 / 0.037052 (0.017377)	0.337310 / 0.258489 (0.078821)	0.386138 / 0.293841 (0.092297)	0.040068 / 0.128546 (-0.088478)	0.013895 / 0.075646 (-0.061751)	0.384174 / 0.419271 (-0.035097)	0.058244 / 0.043533 (0.014711)	0.342410 / 0.255139 (0.087271)	0.362417 / 0.283200 (0.079217)	0.123470 / 0.141683 (-0.018213)	1.662938 / 1.452155 (0.210784)	1.786488 / 1.492716 (0.293771)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.232629 / 0.018006 (0.214622)	0.478252 / 0.000490 (0.477762)	0.008519 / 0.000200 (0.008319)	0.000111 / 0.000054 (0.000057)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.031222 / 0.037411 (-0.006190)	0.125875 / 0.014526 (0.111350)	0.138995 / 0.176557 (-0.037562)	0.213073 / 0.737135 (-0.524062)	0.141848 / 0.296338 (-0.154490)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.463648 / 0.215209 (0.248439)	4.582969 / 2.077655 (2.505314)	2.104622 / 1.504120 (0.600502)	1.887697 / 1.541195 (0.346502)	1.946096 / 1.468490 (0.477606)	0.809008 / 4.584777 (-3.775769)	4.527871 / 3.745712 (0.782159)	4.862721 / 5.269862 (-0.407141)	2.423257 / 4.565676 (-2.142419)	0.101080 / 0.424275 (-0.323196)	0.014767 / 0.007607 (0.007160)	0.574471 / 0.226044 (0.348427)	5.746445 / 2.268929 (3.477516)	2.682584 / 55.444624 (-52.762040)	2.320113 / 6.876477 (-4.556364)	2.474530 / 2.142072 (0.332458)	0.992979 / 4.805227 (-3.812249)	0.200812 / 6.500664 (-6.299852)	0.076291 / 0.075469 (0.000822)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.395533 / 1.841788 (-0.446254)	17.418803 / 8.074308 (9.344495)	16.584875 / 10.191392 (6.393483)	0.167739 / 0.680424 (-0.512685)	0.020923 / 0.534201 (-0.513278)	0.500788 / 0.579283 (-0.078496)	0.510270 / 0.434364 (0.075906)	0.589608 / 0.540337 (0.049270)	0.694233 / 1.386936 (-0.692703)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008440 / 0.011353 (-0.002913)	0.005871 / 0.011008 (-0.005137)	0.085805 / 0.038508 (0.047297)	0.039324 / 0.023109 (0.016215)	0.400587 / 0.275898 (0.124689)	0.431729 / 0.323480 (0.108249)	0.006557 / 0.007986 (-0.001429)	0.005778 / 0.004328 (0.001450)	0.084394 / 0.004250 (0.080144)	0.055274 / 0.037052 (0.018222)	0.410568 / 0.258489 (0.152079)	0.439952 / 0.293841 (0.146111)	0.040335 / 0.128546 (-0.088211)	0.013968 / 0.075646 (-0.061679)	0.098765 / 0.419271 (-0.320507)	0.055897 / 0.043533 (0.012364)	0.387584 / 0.255139 (0.132445)	0.412568 / 0.283200 (0.129368)	0.120393 / 0.141683 (-0.021290)	1.730996 / 1.452155 (0.278841)	1.821538 / 1.492716 (0.328822)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.245688 / 0.018006 (0.227682)	0.484888 / 0.000490 (0.484398)	0.000485 / 0.000200 (0.000285)	0.000068 / 0.000054 (0.000013)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032340 / 0.037411 (-0.005072)	0.130819 / 0.014526 (0.116293)	0.138491 / 0.176557 (-0.038065)	0.196902 / 0.737135 (-0.540233)	0.145404 / 0.296338 (-0.150935)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.487643 / 0.215209 (0.272434)	4.818956 / 2.077655 (2.741301)	2.332316 / 1.504120 (0.828196)	2.102018 / 1.541195 (0.560823)	2.156743 / 1.468490 (0.688253)	0.803365 / 4.584777 (-3.781412)	4.308561 / 3.745712 (0.562849)	2.373331 / 5.269862 (-2.896530)	1.539474 / 4.565676 (-3.026202)	0.099081 / 0.424275 (-0.325194)	0.014627 / 0.007607 (0.007020)	0.609883 / 0.226044 (0.383838)	6.092402 / 2.268929 (3.823474)	2.858137 / 55.444624 (-52.586488)	2.463256 / 6.876477 (-4.413220)	2.637048 / 2.142072 (0.494976)	0.959552 / 4.805227 (-3.845676)	0.194170 / 6.500664 (-6.306495)	0.075231 / 0.075469 (-0.000238)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.516502 / 1.841788 (-0.325285)	18.077893 / 8.074308 (10.003585)	16.507961 / 10.191392 (6.316569)	0.171643 / 0.680424 (-0.508780)	0.020378 / 0.534201 (-0.513823)	0.491508 / 0.579283 (-0.087775)	0.492136 / 0.434364 (0.057772)	0.602258 / 0.540337 (0.061920)	0.719882 / 1.386936 (-0.667054)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006572 / 0.011353 (-0.004781)	0.004647 / 0.011008 (-0.006362)	0.098277 / 0.038508 (0.059769)	0.027937 / 0.023109 (0.004828)	0.339833 / 0.275898 (0.063935)	0.398305 / 0.323480 (0.074825)	0.005093 / 0.007986 (-0.002893)	0.003374 / 0.004328 (-0.000954)	0.075287 / 0.004250 (0.071037)	0.037355 / 0.037052 (0.000303)	0.339779 / 0.258489 (0.081290)	0.403756 / 0.293841 (0.109915)	0.030705 / 0.128546 (-0.097841)	0.011596 / 0.075646 (-0.064050)	0.323809 / 0.419271 (-0.095463)	0.043357 / 0.043533 (-0.000176)	0.342817 / 0.255139 (0.087678)	0.386330 / 0.283200 (0.103130)	0.088229 / 0.141683 (-0.053454)	1.466017 / 1.452155 (0.013862)	1.566551 / 1.492716 (0.073835)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.196276 / 0.018006 (0.178269)	0.420321 / 0.000490 (0.419831)	0.002234 / 0.000200 (0.002034)	0.000071 / 0.000054 (0.000016)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023999 / 0.037411 (-0.013412)	0.095117 / 0.014526 (0.080592)	0.102544 / 0.176557 (-0.074013)	0.164796 / 0.737135 (-0.572340)	0.107030 / 0.296338 (-0.189309)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.429299 / 0.215209 (0.214089)	4.272503 / 2.077655 (2.194849)	2.101890 / 1.504120 (0.597771)	1.978907 / 1.541195 (0.437713)	2.008993 / 1.468490 (0.540503)	0.695171 / 4.584777 (-3.889606)	3.427050 / 3.745712 (-0.318662)	1.892945 / 5.269862 (-3.376917)	1.247156 / 4.565676 (-3.318521)	0.082576 / 0.424275 (-0.341699)	0.012526 / 0.007607 (0.004918)	0.526338 / 0.226044 (0.300293)	5.313855 / 2.268929 (3.044927)	2.421134 / 55.444624 (-53.023490)	2.072026 / 6.876477 (-4.804451)	2.159846 / 2.142072 (0.017773)	0.800753 / 4.805227 (-4.004474)	0.150507 / 6.500664 (-6.350157)	0.066378 / 0.075469 (-0.009091)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.218709 / 1.841788 (-0.623079)	13.649239 / 8.074308 (5.574931)	13.952762 / 10.191392 (3.761370)	0.141967 / 0.680424 (-0.538457)	0.016443 / 0.534201 (-0.517758)	0.380408 / 0.579283 (-0.198875)	0.377693 / 0.434364 (-0.056671)	0.439819 / 0.540337 (-0.100518)	0.529667 / 1.386936 (-0.857269)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006722 / 0.011353 (-0.004630)	0.004495 / 0.011008 (-0.006513)	0.075459 / 0.038508 (0.036951)	0.028135 / 0.023109 (0.005026)	0.349904 / 0.275898 (0.074006)	0.390620 / 0.323480 (0.067140)	0.005175 / 0.007986 (-0.002810)	0.004720 / 0.004328 (0.000392)	0.074243 / 0.004250 (0.069993)	0.039084 / 0.037052 (0.002032)	0.352486 / 0.258489 (0.093997)	0.397549 / 0.293841 (0.103708)	0.030596 / 0.128546 (-0.097950)	0.011627 / 0.075646 (-0.064020)	0.083394 / 0.419271 (-0.335878)	0.042155 / 0.043533 (-0.001378)	0.345668 / 0.255139 (0.090529)	0.383474 / 0.283200 (0.100275)	0.096530 / 0.141683 (-0.045153)	1.493360 / 1.452155 (0.041206)	1.572259 / 1.492716 (0.079543)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.162605 / 0.018006 (0.144599)	0.409513 / 0.000490 (0.409023)	0.002029 / 0.000200 (0.001829)	0.000069 / 0.000054 (0.000015)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.025824 / 0.037411 (-0.011588)	0.102439 / 0.014526 (0.087913)	0.109515 / 0.176557 (-0.067041)	0.160650 / 0.737135 (-0.576486)	0.112971 / 0.296338 (-0.183367)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.433293 / 0.215209 (0.218084)	4.340286 / 2.077655 (2.262631)	2.055857 / 1.504120 (0.551737)	1.854451 / 1.541195 (0.313256)	1.912752 / 1.468490 (0.444261)	0.700076 / 4.584777 (-3.884701)	3.361542 / 3.745712 (-0.384170)	2.760204 / 5.269862 (-2.509658)	1.477395 / 4.565676 (-3.088282)	0.082868 / 0.424275 (-0.341407)	0.012479 / 0.007607 (0.004872)	0.532749 / 0.226044 (0.306704)	5.323701 / 2.268929 (3.054772)	2.509524 / 55.444624 (-52.935100)	2.168668 / 6.876477 (-4.707809)	2.259112 / 2.142072 (0.117040)	0.806686 / 4.805227 (-3.998542)	0.154620 / 6.500664 (-6.346044)	0.068348 / 0.075469 (-0.007121)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.316512 / 1.841788 (-0.525276)	14.158143 / 8.074308 (6.083835)	14.110643 / 10.191392 (3.919251)	0.143760 / 0.680424 (-0.536664)	0.016851 / 0.534201 (-0.517350)	0.376594 / 0.579283 (-0.202689)	0.386957 / 0.434364 (-0.047407)	0.466185 / 0.540337 (-0.074152)	0.550269 / 1.386936 (-0.836667)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009457 / 0.011353 (-0.001896)	0.006453 / 0.011008 (-0.004555)	0.136392 / 0.038508 (0.097884)	0.038378 / 0.023109 (0.015269)	0.413171 / 0.275898 (0.137273)	0.451605 / 0.323480 (0.128126)	0.007123 / 0.007986 (-0.000863)	0.006316 / 0.004328 (0.001987)	0.103009 / 0.004250 (0.098758)	0.049182 / 0.037052 (0.012130)	0.398635 / 0.258489 (0.140146)	0.463146 / 0.293841 (0.169305)	0.056247 / 0.128546 (-0.072299)	0.019589 / 0.075646 (-0.056058)	0.475882 / 0.419271 (0.056610)	0.094918 / 0.043533 (0.051385)	0.416502 / 0.255139 (0.161363)	0.447129 / 0.283200 (0.163929)	0.133314 / 0.141683 (-0.008369)	2.132888 / 1.452155 (0.680733)	2.073383 / 1.492716 (0.580667)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.273037 / 0.018006 (0.255030)	0.625675 / 0.000490 (0.625185)	0.003449 / 0.000200 (0.003249)	0.000185 / 0.000054 (0.000130)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.031889 / 0.037411 (-0.005523)	0.131673 / 0.014526 (0.117148)	0.141575 / 0.176557 (-0.034982)	0.214978 / 0.737135 (-0.522158)	0.145586 / 0.296338 (-0.150752)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.711135 / 0.215209 (0.495926)	7.162492 / 2.077655 (5.084837)	2.906028 / 1.504120 (1.401908)	2.488855 / 1.541195 (0.947660)	2.574628 / 1.468490 (1.106138)	1.587824 / 4.584777 (-2.996953)	6.332962 / 3.745712 (2.587250)	5.419578 / 5.269862 (0.149717)	2.935413 / 4.565676 (-1.630263)	0.169159 / 0.424275 (-0.255116)	0.015358 / 0.007607 (0.007751)	0.862036 / 0.226044 (0.635992)	8.559256 / 2.268929 (6.290328)	3.530756 / 55.444624 (-51.913868)	2.626288 / 6.876477 (-4.250188)	2.770063 / 2.142072 (0.627990)	1.500116 / 4.805227 (-3.305112)	0.265109 / 6.500664 (-6.235555)	0.084944 / 0.075469 (0.009475)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.631060 / 1.841788 (-0.210728)	19.022827 / 8.074308 (10.948519)	22.973632 / 10.191392 (12.782240)	0.296265 / 0.680424 (-0.384158)	0.032317 / 0.534201 (-0.501884)	0.624171 / 0.579283 (0.044888)	0.690643 / 0.434364 (0.256279)	0.691206 / 0.540337 (0.150869)	0.758855 / 1.386936 (-0.628081)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009441 / 0.011353 (-0.001912)	0.006270 / 0.011008 (-0.004739)	0.110284 / 0.038508 (0.071776)	0.035952 / 0.023109 (0.012842)	0.521894 / 0.275898 (0.245996)	0.582624 / 0.323480 (0.259144)	0.011400 / 0.007986 (0.003414)	0.004677 / 0.004328 (0.000348)	0.115721 / 0.004250 (0.111470)	0.048521 / 0.037052 (0.011469)	0.497142 / 0.258489 (0.238653)	0.573733 / 0.293841 (0.279892)	0.055788 / 0.128546 (-0.072759)	0.020949 / 0.075646 (-0.054697)	0.132968 / 0.419271 (-0.286303)	0.063045 / 0.043533 (0.019512)	0.537769 / 0.255139 (0.282630)	0.527560 / 0.283200 (0.244361)	0.123756 / 0.141683 (-0.017927)	1.994111 / 1.452155 (0.541956)	2.104623 / 1.492716 (0.611907)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.279057 / 0.018006 (0.261051)	0.537342 / 0.000490 (0.536852)	0.007782 / 0.000200 (0.007582)	0.000115 / 0.000054 (0.000060)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032018 / 0.037411 (-0.005394)	0.133456 / 0.014526 (0.118930)	0.142039 / 0.176557 (-0.034517)	0.213769 / 0.737135 (-0.523366)	0.143811 / 0.296338 (-0.152527)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.680142 / 0.215209 (0.464933)	6.450439 / 2.077655 (4.372784)	2.820724 / 1.504120 (1.316604)	2.520407 / 1.541195 (0.979212)	2.568972 / 1.468490 (1.100482)	1.250584 / 4.584777 (-3.334193)	6.108222 / 3.745712 (2.362509)	3.065965 / 5.269862 (-2.203897)	2.108675 / 4.565676 (-2.457002)	0.167870 / 0.424275 (-0.256405)	0.015127 / 0.007607 (0.007520)	0.849645 / 0.226044 (0.623600)	8.508727 / 2.268929 (6.239799)	3.707897 / 55.444624 (-51.736727)	3.009279 / 6.876477 (-3.867198)	3.067179 / 2.142072 (0.925106)	1.516370 / 4.805227 (-3.288858)	0.264845 / 6.500664 (-6.235819)	0.095137 / 0.075469 (0.019668)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.826306 / 1.841788 (-0.015481)	20.119641 / 8.074308 (12.045333)	21.532158 / 10.191392 (11.340766)	0.278631 / 0.680424 (-0.401793)	0.029494 / 0.534201 (-0.504707)	0.621887 / 0.579283 (0.042604)	0.686864 / 0.434364 (0.252500)	0.695412 / 0.540337 (0.155074)	0.864829 / 1.386936 (-0.522108)