Calculate TranssformersTokenizer.__hash__ Once to Improve CFGFSM performance

[lapp0]

Improve performance of CFGFSM performance test lark_lark_self_grammar.lark.test-True.

Second run performance goes from 63 tokens / second -> 311 tokens / second

(Profiled with #587)

Fixes #620

TODO:

• awaiting Use FastInteractiveParser Subclass to Improve CFGFSM Performance #622

Solution

Hashing TransformersTokenizer is expensive, so we only do it once.

New Benchmarks

Additional Benchmark Details:
tests/benchmark/test_benchmark_cfg_generation.py::test_benchmark_cfg_generation[lark_lark_self_grammar.lark.test-True]:
	Tokens / Second: 310.627
	(Num Tokens: 2771, Time: 8.921 seconds)


--------------------------------------------------------------------- benchmark: 1 tests --------------------------------------------------------------------
Name (time in s)                                                            Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_cfg_generation[lark_lark_self_grammar.lark.test-True]     8.9207  8.9207  8.9207  0.0000  8.9207  0.0000       0;0  0.1121       1           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------

Profile:

tests/benchmark/test_benchmark_cfg_generation.py::test_benchmark_cfg_generation[lark_lark_self_grammar.lark.test-True]
ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
1	0.0001	0.0001	17.5441	17.5441	outlines/tests/benchmark/test_benchmark_cfg_generation.py:130(<lambda>)
1	2.5908	2.5908	17.5440	17.5440	outlines/tests/benchmark/test_benchmark_cfg_generation.py:51(run_until_eos)
2772	0.8284	0.0003	14.5217	0.0052	outlines/outlines/fsm/fsm.py:224(allowed_token_ids)
2455	0.0373	0.0000	5.1374	0.0021	outlines/.myenv/lib/python3.11/site-packages/lark/parsers/lalr_interactive_parser.py:47(exhaust_lexer)
194248	0.1169	0.0000	5.1001	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parsers/lalr_interactive_parser.py:35(iter_parse)
2454	4.3396	0.0018	4.3396	0.0018	outlines/outlines/fsm/fsm.py:274(<listcomp>)
2455	0.0622	0.0000	3.2306	0.0013	outlines/.myenv/lib/python3.11/site-packages/lark/parsers/lalr_interactive_parser.py:103(accepts)
149119/26899	0.1766	0.0000	2.8608	0.0001	/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/copy.py:66(copy)
216237	0.1096	0.0000	2.8426	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parsers/lalr_interactive_parser.py:28(feed_token)
24444	0.0284	0.0000	2.7801	0.0001	outlines/outlines/fsm/fast_lark.py:58(__copy__)
216237	1.6619	0.0000	2.7330	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parsers/lalr_parser_state.py:67(feed_token)
194248	0.1426	0.0000	2.3986	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:661(lex)
24444	0.1599	0.0000	2.3657	0.0001	outlines/outlines/fsm/fast_lark.py:19(__copy__)
194248	0.5871	0.0000	2.2248	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:590(next_token)
367363/9046	0.6133	0.0001	2.1497	0.0002	/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/copy.py:128(deepcopy)
136438/4515	0.1141	0.0000	2.1048	0.0005	outlines/.myenv/lib/python3.11/site-packages/lark/tree.py:206(__deepcopy__)
136438/4515	0.6656	0.0001	2.0830	0.0005	/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/copy.py:201(_deepcopy_list)
304840	0.1775	0.0000	1.0246	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:587(match)
304840	0.3277	0.0000	0.7915	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:387(match)
5226	0.5057	0.0001	0.5080	0.0001	outlines/outlines/fsm/fsm.py:130(allowed_token_ids)
310724	0.1799	0.0000	0.4816	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:202(__new__)
2771	0.2313	0.0001	0.4199	0.0002	outlines/outlines/fsm/fsm.py:309(next_state)
305361	0.4099	0.0000	0.4099	0.0000	~:0(<method 'match' of '_regex.Pattern' objects>)
159870	0.2709	0.0000	0.3581	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parse_tree_builder.py:145(__call__)
310724	0.2378	0.0000	0.3017	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:213(_future_new)
339	0.2485	0.0007	0.2485	0.0007	outlines/outlines/fsm/fsm.py:305(<listcomp>)
921854	0.2219	0.0000	0.2219	0.0000	~:0(<method 'get' of 'dict' objects>)
304840	0.2060	0.0000	0.2693	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:292(feed)
407204	0.1567	0.0000	0.1950	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:265(__eq__)
367363	0.1436	0.0000	0.1963	0.0000	/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/copy.py:243(_keep_alive)
2341037	0.1430	0.0000	0.1430	0.0000	~:0(<method 'append' of 'list' objects>)
555623	0.1360	0.0000	0.1874	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/grammar.py:124(__eq__)
1462197	0.1241	0.0000	0.1241	0.0000	~:0(<built-in method builtins.len>)
1230260	0.1254	0.0000	0.1323	0.0000	~:0(<built-in method builtins.isinstance>)
1112576	0.0928	0.0000	0.0928	0.0000	~:0(<built-in method builtins.id>)
489362	0.0569	0.0000	0.0569	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/grammar.py:121(__hash__)
412091	0.0891	0.0000	0.0891	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/tree.py:61(__init__)
393617	0.0604	0.0000	0.0604	0.0000	~:0(<built-in method builtins.getattr>)
355600	0.0339	0.0000	0.0339	0.0000	~:0(<built-in method builtins.issubclass>)
337623	0.0675	0.0000	0.0675	0.0000	~:0(<built-in method __new__ of type object at 0x7f15a5fb1ba0>)
304840	0.0543	0.0000	0.0543	0.0000	~:0(<method 'group' of '_regex.Match' objects>)
304840	0.0485	0.0000	0.0555	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:581(scanner)
255255	0.1231	0.0000	0.1486	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parse_tree_builder.py:20(__call__)
94487	0.0656	0.0000	0.2214	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:262(__deepcopy__)
8834	0.0015	0.0000	0.0015	0.0000	~:0(<method 'keys' of 'dict' objects>)
71	0.0001	0.0000	0.0002	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:81(to_regexp)
71	0.0000	0.0000	0.0000	0.0000	~:0(<method 'translate' of 'str' objects>)
71	0.0000	0.0000	0.0001	0.0000	/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/re/__init__.py:253(escape)
695	0.0001	0.0000	0.0001	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:70(_get_flags)
624	0.0002	0.0000	0.0003	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:98(to_regexp)
60769	0.0115	0.0000	0.0115	0.0000	~:0(<method 'rindex' of 'str' objects>)
538	0.0015	0.0000	0.0043	0.0000	outlines/.myenv/lib/python3.11/site-packages/regex/regex.py:449(_compile)
538	0.0003	0.0000	0.0004	0.0000	outlines/.myenv/lib/python3.11/site-packages/regex/regex.py:471(complain_unused_args)
538	0.0003	0.0000	0.0004	0.0000	<frozen importlib._bootstrap>:1207(_handle_fromlist)
538	0.0001	0.0000	0.0001	0.0000	outlines/.myenv/lib/python3.11/site-packages/regex/regex.py:476(<setcomp>)
53798	0.0170	0.0000	0.0170	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parsers/lalr_parser_state.py:40(__init__)
53600	0.0067	0.0000	0.0067	0.0000	~:0(<method 'isupper' of 'str' objects>)
521	0.0005	0.0000	0.0049	0.0000	outlines/.myenv/lib/python3.11/site-packages/regex/regex.py:249(match)
521	0.0002	0.0000	0.0051	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:329(_get_match)
3706	0.0098	0.0000	0.0110	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/exceptions.py:230(__init__)
3706	0.0070	0.0000	0.0093	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parsers/lalr_parser_state.py:79(<setcomp>)
17	0.0005	0.0000	0.0062	0.0004	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:334(_create_unless)
2771	0.0335	0.0000	0.0335	0.0000	~:0(<method 'decode' of 'tokenizers.Tokenizer' objects>)
17	0.0001	0.0000	0.0002	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/utils.py:23(classify)
2771	0.0236	0.0000	0.0982	0.0000	outlines/.myenv/lib/python3.11/site-packages/transformers/utils/generic.py:232(to_py_obj)
2455	0.0203	0.0000	0.0398	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/parser_frontends.py:96(_make_lexer_thread)
17	0.0001	0.0000	0.0071	0.0004	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:568(_build_scanner)
2771	0.0131	0.0000	0.0336	0.0000	outlines/.myenv/lib/python3.11/site-packages/transformers/utils/generic.py:90(_get_frameworks_and_test_func)
17	0.0001	0.0000	0.0007	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:368(_build_mres)
2771	0.0125	0.0000	0.0552	0.0000	outlines/.myenv/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py:614(_decode)
26800	0.1180	0.0000	0.1180	0.0000	~:0(<method 'copy' of 'list' objects>)
2771	0.0065	0.0000	0.1804	0.0001	outlines/outlines/models/transformers.py:158(decode)
2771	0.0044	0.0000	0.1739	0.0001	outlines/.myenv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:3686(batch_decode)
2771	0.0072	0.0000	0.1695	0.0001	outlines/.myenv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:3710(<listcomp>)
2771	0.0089	0.0000	0.1623	0.0001	outlines/.myenv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:3720(decode)
17	0.0000	0.0000	0.0007	0.0000	outlines/.myenv/lib/python3.11/site-packages/lark/lexer.py:357(__init__)
2455	0.0091	0.0000	0.0918	0.0000	outlines/outlines/fsm/fast_lark.py:71(parse_interactive)
17	0.0000	0.0000	0.0000	0.0000	~:0(<method 'values' of 'dict' objects>)
13855	0.0050	0.0000	0.0050	0.0000	~:0(<method 'startswith' of 'str' objects>)
27710	0.0025	0.0000	0.0025	0.0000	~:0(<method 'replace' of 'str' objects>)
2116	0.0004	0.0000	0.0004	0.0000	~:0(<method 'remove' of 'set' objects>)
2771	0.0006	0.0000	0.0006	0.0000	~:0(<method 'pop' of 'dict' objects>)
2472	0.0013	0.0000	0.0016	0.0000	~:0(<method 'join' of 'str' objects>)
29687	0.0038	0.0000	0.0038	0.0000	~:0(<method 'items' of 'dict' objects>)
2771	0.0004	0.0000	0.0004	0.0000	~:0(<method 'extend' of 'list' objects>)
1	0.0000	0.0000	0.0000	0.0000	~:0(<method 'disable' of '_lsprof.Profiler' objects>)
125296	0.0271	0.0000	0.0271	0.0000	~:0(<method 'count' of 'str' objects>)
22854	0.0039	0.0000	0.0039	0.0000	~:0(<method 'add' of 'set' objects>)
26899	0.0327	0.0000	0.0327	0.0000	~:0(<method '__reduce_ex__' of 'object' objects>)
2771	0.0078	0.0000	0.0078	0.0000	~:0(<built-in method builtins.sorted>)
136950	0.0115	0.0000	0.0115	0.0000	~:0(<built-in method builtins.setattr>)

[rlouf]

Closing as inactive. Feel free to reopen later.