Section 4.7 in SP800-63C Rev4 draft states:
The IdP SHALL communicate to the RP any information the IdP has regarding the time of the subscriber’s latest authentication event at the IdP, and the RP MAY use this information in making authorization and access decisions.
IPSIE SL1 should make the auth_time claim required in the id token to resolve this gap.