jinja2._stringdefs uses much RAM

[methane]

jinja2._stringdefs makes large strings for each unicode category.
They eat 100KB+ RAM. But none of them are used.

Only xid_start and xid_continue are used by jinja2.lexer.
Would you remove category strings?

Your Environment

• Python version: Python 3.6
• Jinja version: 2.9.4

[ThiefMaster]

100kb is not really "much ram". However, the module could probably be removed from sys.modules and the lexer.py module scope after this code has been executed so it can be garbage-collected...

try:
    compile('föö', '<unknown>', 'eval')
except SyntaxError:
    name_re = re.compile(r'\b[a-zA-Z_][a-zA-Z0-9_]*\b')
else:
    from jinja2 import _stringdefs
    name_re = re.compile(r'[%s][%s]*' % (_stringdefs.xid_start,
                                         _stringdefs.xid_continue))

Either way, the regex probably uses not much less memory since it contains all these chars.

---

IMO using non-ascii chars in identifiers is an awful idea, but I guess you can always find someone who doesn't code in english and Jinja is too mature to remove something like this anyway...

[davidism]

Solved more by #731.