mcopy: avoid use of HEAD to determine filename since some object storage returns 403 #549
Description
[init] Running as uid=1000 gid=1000 with /data as 'drwxr-x--- 2 1000 1000 4096 Apr 12 18:47 /data'
[init] Image info: buildtime=2025-04-19T02:44:43.813Z,version=java21,revision=0ec908b243d1ab435823f653b3777b408aea9e2f
[init] Autostop functionality enabled
[init] Resolving type given PAPER
2025/04/24 14:08:08 INFO SSH listening listen_address=[fdaa:9:4edc:a7b:3f9:a025:23bf:2]:22
[mc-image-helper] 14:08:14.148 INFO : Downloaded /data/paper-1.21.4-226.jar
[mc-image-helper] 14:08:16.333 ERROR : 'mcopy' command failed. Version is 1.41.7
me.itzg.helpers.http.FailedRequestException: HTTP request of https://example.com/staging/builds/4e8f7980-3955-48c4-9ff9-9a8dbedcbf41/test-plugin-1.0-SNAPSHOT.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=6OL4X4U2SMEH6AKWA71D%2F20250424%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20250424T134349Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b37c7258d121448b019e0523edfde74d173a24c011750b5b110a8c73794da260 failed with 403 Forbidden: Extracting filename
at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________checkpoint ⇢ Fetch HEAD of requested file
|_ checkpoint(Retrieving https://example.com/staging/builds/4e8f7980-3955-48c4-9ff9-9a8dbedcbf41/test-plugin-1.0-SNAPSHOT.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=6OL4X4U2SMEH6AKWA71D%2F20250424%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20250424T134349Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b37c7258d121448b019e0523edfde74d173a24c011750b5b110a8c73794da260) ⇢ at me.itzg.helpers.sync.MulitCopyCommand.processRemoteSource(MulitCopyCommand.java:248)
Original Stack Trace:
at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)
2025-04-24T14:08:16.3at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2097)
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onComplete(FluxDefaultIfEmpty.java:134)
at reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:223)
at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)
at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)
at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:211)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2097)
at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:118)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:455)
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:509)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:824)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
2025-04-at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
at reactor.core.publisher.Mono.block(Mono.java:1779)
at me.itzg.helpers.sync.MulitCopyCommand.call(MulitCopyCommand.java:83)
at me.itzg.helpers.sync.MulitCopyCommand.call(MulitCopyCommand.java:33)
at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
at picocli.CommandLine.execute(CommandLine.java:2174)
at me.itzg.helpers.McImageHelper.main(McImageHelper.java:178)
INFO Main child exited normally with code: 1
INFO Starting clean up.
curl seems to use a much simpler heuristic for --remote-name and when using -v I determined it indeed doesn't use a HEAD operation:
-O, --remote-name
Write output to a local file named like the remote file we get. (Only the file part of the remote
file is used, the path is cut off.)
The file will be saved in the current working directory. If you want the file saved in a different
directory, make sure you change the current working directory before invoking curl with this
option.
The remote file name to use for saving is extracted from the given URL, nothing else, and if it
already exists it will be overwritten. If you want the server to be able to choose the file name
refer to -J, --remote-header-name which can be used in addition to this option. If the server
chooses a file name and that name already exists it will not be overwritten.
There is no URL decoding done on the file name. If it has %20 or other URL encoded parts of the
name, they will end up as-is as file name.
You may use this option as many times as the number of URLs you have.
Example:
curl -O https://example.com/filename
See also --remote-name-all.