Add support for .tar archives as downloaded from dev.mysql.com #165

utdrmac · 2023-06-20T13:49:51Z

I'm trying to add the latest mysql to the downloads list.

https://dev.mysql.com/downloads/mysql/
'Linux - Generic' -> 'Linux - Generic (glibc 2.17) (x86, 64-bit)'

Results in this option (with others):
https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar

$ dbdeployer downloads get https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar
$ dbdeployer unpack mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar
error unpacking tarball mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar: tarball extension must be either '.tar.gz' or '.tar.xz'

Alternatively,

$ dbdeployer downloads add mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar --arch x86_64 --OS Linux --flavor mysql --minimal --url https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar

Tarball below added to /home/xxxx/.dbdeployer/tarball-list.json
Name:          mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-x86_64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar
Checksum:      SHA512:023425f498f8dfc04165afeb874a5711ea6c7804c6c34616776b1956809bf1d76ed8023099f6b6fa54058398f4279b8feeee6d445d4899be66c240e93d3720c6
Size:          426 MB
Notes:         added with version 1.72.0
Added on:      2023-06-20 13:40

$ dbdeployer downloads get-unpack mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar
...
Downloading mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar
File /home/xxxx/dbdeployer/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar downloaded
Checksum matches
################################################################################
WARNING: current architecture is amd64, but the tarball architecture is x86_64
################################################################################
Error: tarball extension must be either '.tar.gz' or '.tar.xz'

If .tar is not valid, please either A) add support for .tar or B) disallow the ability to add .tar archives.

The text was updated successfully, but these errors were encountered:

datacharmer · 2023-06-20T14:01:51Z

The file you are trying to add is an uncompressed archive containing more compressed archives.
As the error message says, dbdeployer only deals with .tar.gz or .tar.xz, not .tar
To add a new tarball, don't choose files manually, but do the following:

dbdeployer downloads add-remote mysql 8.0 Linux --minimal

Example result:

Tarball below added to $HOME/.dbdeployer/tarball-list.json
Name:          mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-amd64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Checksum:      MD5:377b6ae9c88a698438a19d67efa03252
Size:          60 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59
Name:          mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-aarch64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Checksum:      MD5:f5e01242975453ec6db02727848975c9
Size:          58 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59

datacharmer · 2023-06-20T14:05:07Z

BTW: the problem is not "lack of support for .tar", but that the MySQL binaries are always in compressed files. Support for plain .tar would not accomplish anything

utdrmac · 2023-06-20T14:06:01Z

If .tar is not supported, please don't allow .tar to be added in the first place.

The file you are trying to add is an uncompressed archive containing more compressed archives.

I'm not sure how I was supposed to know that.

I originally tried as you suggested, which failed:

~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0.32 linux
error getting remote tarball description: version '8.0.32' is not accepted for tarball type mysql: it must be one of [[5.7 8.0]]
~/dbdeployer$
~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 linux
error getting remote tarball description: no mysql tarballs found for 8.0
~/dbdeployer$
~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 Linux
error getting remote tarball description: no mysql tarballs found for 8.0
~/dbdeployer$
~/dbdeployer$ dbdeployer update
......  7.0 MB
1
  97 B
checksum for dbdeployer-1.72.0.linux.tar.gz matches
dbdeployer version 1.72.0
~/dbdeployer$
~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 Linux
Provided defaults are for version 1.62.0. Current version is 1.70.0
********************************************************************************
Defaults file /home/xxxxxx/.dbdeployer/config.json not validated.
Loading internal defaults
********************************************************************************

Provided defaults are for version 1.62.0. Current version is 1.70.0
error writing tarball list: [write tarball file info] tarball list check failed : tarball with OS Linux-amd64, flavor mysql, version 8.0.33, and minimal false listed more than once

utdrmac · 2023-06-20T14:08:48Z

Then I'm trying this:

$ dbdeployer downloads add mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz \
  --arch x86_64 \
  --OS Linux --flavor mysql \
  --minimal \
  --url https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
file '/home/xxxxx/dbdeployer/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz' not found

Of course it is not found, it hasn't been downloaded yet. I expect dbdeployer to add it (as the command states) then I can get-unpack (?) after it is added.

datacharmer · 2023-06-20T14:15:47Z

dbdeployer downloads add requires a local file, which means you need to download it first.
Using dbdeployer downloads add-remote will free you from that burden.

Regarding your failure:

dbdeployer downloads add-remote mysql 8.0.32 linux

You need to use 8.0, and dbdeployer will find the newest one.

utdrmac · 2023-06-20T14:18:24Z

You need to use 8.0, and dbdeployer will find the newest one.

If you look at the last example in my comment above, I did try exactly that, which produced the error:

~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 Linux
Provided defaults are for version 1.62.0. Current version is 1.70.0
********************************************************************************
Defaults file /home/xxxxxx/.dbdeployer/config.json not validated.
Loading internal defaults
********************************************************************************

Provided defaults are for version 1.62.0. Current version is 1.70.0
error writing tarball list: [write tarball file info] tarball list check failed : tarball with OS Linux-amd64, flavor mysql, version 8.0.33, and minimal false listed more than once

utdrmac · 2023-06-20T14:24:37Z

dbdeployer downloads add requires a local file, which means you need to download it first.

The help for add does not indicate anywhere that it must already be downloaded. As I said above, I'm expecting add to simply add it to the list and then I can download from that list later. Some additional messaging might be helpful here. I've spent the last hour trying to simply add the latest mysql to dbdeployer and it's very frustrating.

I would not expect a tool that has download abilities to require me to first manually download what I want to add before it can be added.

$ dbdeployer downloads add --help
Adds a tarball to the list

Usage:
  dbdeployer downloads add tarball_name [flags]

datacharmer · 2023-06-20T14:33:51Z

Since you have add and add-remote, I think the meaning of add is clear.
The purpose of add is to add to the download list, using a local file, while add-remote does the same, but from remote information.

Anyway, dbdeployer can download things for you. If you want to do things with add instead of add-remote, you can download the file first

dbdeployer downloads  get https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz

and then use dbdeployer downloads add ...

utdrmac · 2023-06-20T14:37:04Z

How about changing the help to look like this, which would avoid this confusion:

Available Commands:
  add            Adds a local tarball to the list
  add-remote     Adds a tarball to the list, by searching MySQL downloads site

And also:
if(substring(FILENAME, -3) == ".tar") {
error(".tar is not allowed. Please provide the .tar.gz or .tar.xz")
}

httpete · 2023-06-28T15:58:19Z

I agree @utdrmac - I spent the last few hours fishing around trying to just add the latest version. asdf, a very popular utility relies on the fantastic dbdeployer and we are all blocked with that 8.0.33 in the list. A one command way to do the update and open a PR and we can help out.

datacharmer · 2023-06-28T16:04:50Z

@httpete ,
To add a new tarball, do the following:

$ dbdeployer downloads add-remote mysql 8.0 Linux --minimal
Example result:

Tarball below added to $HOME/.dbdeployer/tarball-list.json
Name:          mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-amd64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Checksum:      MD5:377b6ae9c88a698438a19d67efa03252
Size:          60 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59
Name:          mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-aarch64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Checksum:      MD5:f5e01242975453ec6db02727848975c9
Size:          58 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59

Repeat the procedure for all flavours and operating systems:

dbdeployer downloads add-remote shell 8.0 Linux 
dbdeployer downloads add-remote cluster 8.0 Linux
dbdeployer downloads add-remote mysql 8.0 macOS
dbdeployer downloads add-remote cluster 8.0 macOS
dbdeployer downloads add-remote shell 8.0 macOS

The result will be in $HOME/.dbdeployer/tarball-list.json
That will make the downloads usable in your environment.

If you want to create a pull request for inclusion, run this command from the root of dbdeployer repo code

jq . < $HOME/.dbdeployer/tarball-list.json > downloads/tarball_list.json

Note jq is an open source JSON processor

httpete · 2023-06-28T17:54:24Z

done and PR updated.

httpete · 2023-06-28T21:09:54Z

OK something didn't work right downstream, I don't see any entry for the NON minimal mysql for linux and I Can't rerun the script. You had me use the --minimal flag for mysql and now I don't see it. Lil help?

httpete · 2023-06-28T21:15:46Z

@datacharmer - stuck cold. I thought overwrite flag would work but it keeps telling me it's already listed.

[dbdeployer]$ ./dbdeployer downloads add-remote mysql 8.0 Linux
error writing tarball list: [write tarball file info] tarball list check failed : tarball with OS Linux-amd64, flavor mysql, version 8.0.33, and minimal false listed more than once

httpete · 2023-06-30T20:34:35Z

Let's be sure for posterity to have this be the command set for updating to the latest MySql and publishing the tarballs:

First, run the installation script

dbdeployer downloads add-remote mysql 8.0 Linux
dbdeployer downloads add-remote shell 8.0 Linux 
dbdeployer downloads add-remote cluster 8.0 Linux
dbdeployer downloads add-remote mysql 8.0 macOS
dbdeployer downloads add-remote cluster 8.0 macOS
dbdeployer downloads add-remote shell 8.0 macOS

The result will be in $HOME/.dbdeployer/tarball-list.json
That will make the downloads usable in your environment.

To create a pull request for inclusion, run this command from the root of dbdeployer repo code

jq . < $HOME/.dbdeployer/tarball-list.json > downloads/tarball_list.json

datacharmer · 2023-07-09T19:46:04Z

List of downloads updated in release 1.73.0

datacharmer added the not-a-bug label Jun 20, 2023

datacharmer mentioned this issue Jun 28, 2023

Adding 8.0.33 to the list #167

Merged

httpete mentioned this issue Jun 29, 2023

Tests are failing after merge #168

Closed

datacharmer closed this as completed Jul 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for .tar archives as downloaded from dev.mysql.com #165

Add support for .tar archives as downloaded from dev.mysql.com #165

utdrmac commented Jun 20, 2023 •

edited

Loading

datacharmer commented Jun 20, 2023

datacharmer commented Jun 20, 2023

utdrmac commented Jun 20, 2023

utdrmac commented Jun 20, 2023 •

edited

Loading

datacharmer commented Jun 20, 2023

utdrmac commented Jun 20, 2023

utdrmac commented Jun 20, 2023 •

edited

Loading

datacharmer commented Jun 20, 2023

utdrmac commented Jun 20, 2023 •

edited

Loading

httpete commented Jun 28, 2023

datacharmer commented Jun 28, 2023

httpete commented Jun 28, 2023

httpete commented Jun 28, 2023

httpete commented Jun 28, 2023 •

edited

Loading

httpete commented Jun 30, 2023 •

edited

Loading

datacharmer commented Jul 9, 2023

Add support for .tar archives as downloaded from dev.mysql.com #165

Add support for .tar archives as downloaded from dev.mysql.com #165

Comments

utdrmac commented Jun 20, 2023 • edited Loading

datacharmer commented Jun 20, 2023

datacharmer commented Jun 20, 2023

utdrmac commented Jun 20, 2023

utdrmac commented Jun 20, 2023 • edited Loading

datacharmer commented Jun 20, 2023

utdrmac commented Jun 20, 2023

utdrmac commented Jun 20, 2023 • edited Loading

datacharmer commented Jun 20, 2023

utdrmac commented Jun 20, 2023 • edited Loading

httpete commented Jun 28, 2023

datacharmer commented Jun 28, 2023

httpete commented Jun 28, 2023

httpete commented Jun 28, 2023

httpete commented Jun 28, 2023 • edited Loading

httpete commented Jun 30, 2023 • edited Loading

datacharmer commented Jul 9, 2023

utdrmac commented Jun 20, 2023 •

edited

Loading

utdrmac commented Jun 20, 2023 •

edited

Loading

utdrmac commented Jun 20, 2023 •

edited

Loading

utdrmac commented Jun 20, 2023 •

edited

Loading

httpete commented Jun 28, 2023 •

edited

Loading

httpete commented Jun 30, 2023 •

edited

Loading