podman image build does not hash bind mounts correctly

[elbehery95]

I recently noticed a behavior different than docker when podman builds an image.

Scenario

mkdir my_src && echo "hello" > my_src/test.txt
echo "
FROM alpine:latest
RUN --mount=type=bind,src=./my_src,dst=/my_src cat /my_src/test.txt
RUN echo last layer
" > Containerfile

podman image build .

# now try-rebuilding again, expecting all layers to be re-used from cache
podman image build .

In the above scenario it is expected that the second image build command would be re-using the previous build layer, and output would be like the following

STEP 1/3: FROM alpine:latest
STEP 2/3: RUN --mount=type=bind,src=./my_src,dst=/my_src cat /my_src/test.txt
--> Using cache 1b838c0f375b0c30928cb1f5ee0928605ac1f144919d2b3a2f324ab76c46caad
--> 1b838c0f375b
STEP 3/3: RUN echo last layer
--> Using cache c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36
--> c8d55ac6afeb
c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36

The strange thing happen if any of the content of the bind mount are changed for example

echo "world" >> my_src/test.txt
podman image build .

The above command should invalidate the second image layer, in podman it does not do this. Basically podman will re-use a stale layer in this case. And output will be as the following

STEP 1/3: FROM alpine:latest
STEP 2/3: RUN --mount=type=bind,src=./my_src,dst=/my_src cat /my_src/test.txt
--> Using cache 1b838c0f375b0c30928cb1f5ee0928605ac1f144919d2b3a2f324ab76c46caad
--> 1b838c0f375b
STEP 3/3: RUN echo last layer
--> Using cache c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36
--> c8d55ac6afeb
c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36

In docker this behavior works as expected and layer would be invalid in this case given that the content of one of the files inside the bind mount source are changed.

My question is, how does one use bind mounts with podman while at the same time ensures it handles layer caching as expected?

Thank you

Originally posted by @elbehery95 in #21859

[baude]

@nalind @TomSweeneyRedHat can either of you comment on this ?