Lazy-initialize the global reference pool to reduce its overhead when unused

Lazy-initialize the global reference pool to reduce its overhead when unused #4550