Size of the feature maps at later layers

[weihan1]

How do you deal with feature maps comparison in later layers if you don't upsample the original input images? Wouldn't the feature maps at the last convolutional layers be too small?