[downloads] elasticsearch-6.8.5.tar.gz incompatible with other versions

[missinglink]

Hi,

Your current download of elasticsearch-6.8.5.tar.gz with the shasum of a66773585c2195e22114922534f65069d669c774 has been generated in such a way that makes it incompatible with scripts which work with other versions in the 6 and 7 series.

More specifically it was previously possible to use the --strip-components=1 argument for tar in order to remove the leading directory (which is versioned), an example of such a command would be:

curl -Ls "https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.5.tar.gz" \
    | tar xvz --directory=/tmp/elasticsearch --strip-components=1;

You can see in the output below that the 6.8.5 release has an inconsistency which wasn't present in the 6.8.4 release and is not present in the 7.4.2 release:

note the leading ./

tar tfz elasticsearch-6.8.4.tar.gz | grep 'elasticsearch-plugin$'
elasticsearch-6.8.4/bin/elasticsearch-plugin

tar tfz elasticsearch-6.8.5.tar.gz | grep 'elasticsearch-plugin$'
./elasticsearch-6.8.5/bin/elasticsearch-plugin

tar tfz elasticsearch-7.4.2-linux-x86_64.tar.gz | grep 'elasticsearch-plugin$'
elasticsearch-7.4.2/bin/elasticsearch-plugin

I suspect you have made a change to your build script which is no longer setting the -C, --directory=DIR flag correctly.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[droberts195]

@Conky5 this is most likely related to the way that the ES 6.8.5 .tar.gz artifact had to be manually run through macOS notarization for 6.8.5. Will the automated notarization script leave the paths as they were prior to notarization?

[Conky5]

@droberts195 correct, future releases will not include the leading ./. #49417 (comment) includes some discussion on this.

[missinglink]

So is the plan to fix the 6.8.5 tar file or to release a 6.8.6 version?

[Conky5]

The 6.8.5 tar file will not be modified. For handling it you can use --strip-components=2 to extract the files to the expected location.

[missinglink]

I understand that it's a difficult position to be in but I still think it's important to fix the error.

It's a public API in which compatibility has been broken within a minor release which goes against your End of Life policy?

We know that not all users upgrade as quickly as we release. For these users, we maintain the last minor of the prior major release series. For example, with Elasticsearch 6.x, we are maintaining the 6.8.x series. This allows these users to obtain fixes while making only minor changes to their running software.

It's also worth considering the potentially thousands of developers who will waste countless hours of their time debugging this issue.

[missinglink]

Sorry to lobby you so hard on this :)

I just want to clarify that the issue here is that 'oneline bash installs' are no longer possible and there is no way to configure tar appropriately in a stream.

These oneliner bash installs are fairly popular to avoid writing the tar file to disk before extracting, they are also useful when the cwd is not writable or you're in docker and you want to create a minimal image.

What I'm trying to highlight is that for all the Stackoverflow posts, CI environments and bash scripts which use this approach will require a human to manually add an 'if version eq' line to change the 1 to a 2.

I understand that releases should be immutable but in this case you can simply fix the file and it'll still work as expected for oneliners and twoliners alike?

[rjernst]

@missinglink We appreciate the feedback and understand the situation you are in. At the same time, as you pointed out, releases should be immutable. Modifying a file in place means also changing checksums, which can have even broader impact on others automations since the artifact in question would no longer be the same that they previously downloaded.

The end of life policy for Elastic is around minor releases, not specific bugfix releases. While we strive to maintain no changes to behavior, bugs do happen, and we apologize here for that. There will likely be a 6.8.6, and this bug should be fixed there, but we can't give any timeline when that will happen. In the meantime, we recommend using the workaround @Conky5 mentioned above with --strip-components=2.