Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python-dill][hailtop] update past the bad version of dill #13536

Merged
merged 2 commits into from
Sep 1, 2023

Conversation

danking
Copy link
Contributor

@danking danking commented Sep 1, 2023

Fixes #13535.

All the new hailgenetics/dill images are now up: https://hub.docker.com/r/hailgenetics/python-dill/tags

@jigold
Copy link
Contributor

jigold commented Sep 1, 2023

Can you remind me how these new images get into the hail-vdc artifact registry and on Azure as well?

@danking
Copy link
Contributor Author

danking commented Sep 1, 2023

Sure! There's a CI build step that copies everything from hailgenetics into the local repo. mirror_hailgenetics_images. Then, in the worker, we notice a limited set of hailgenetics images and, instead of getting them from docker hub, we get them from our internal repositories.

@jigold
Copy link
Contributor

jigold commented Sep 1, 2023

We only copy if the image isn't already present. I think this code is not what you want in mirror images. We should copy if the contents change as well.

copy_if_not_present() {
    src_image=$1
    dest_image=$2
    if ! skopeo inspect "docker://docker.io/$1";
    then
        echo "$1 does not exist yet, doing nothing"
    elif skopeo inspect "docker://$2";
    then
      echo "$2 already exists, doing nothing"
    else
      echo "$2 does not exist, copying $1 to $2"
      copy_image $1 $2
    fi
}

@danking
Copy link
Contributor Author

danking commented Sep 1, 2023

ugh, that used to just use copy_image and skopeo did The Right Thing (i.e. copy if it the tag had been updated).

@danking
Copy link
Contributor Author

danking commented Sep 1, 2023

I'll just manually do this right now.

@danking
Copy link
Contributor Author

danking commented Sep 1, 2023

Done.

danking pushed a commit to danking/hail that referenced this pull request Sep 1, 2023
This inspect command prevents us from updating a tag, for example, if we need to replace an
image with a security problem or if there is a bug like the one fixed by hail-is#13536.
@danking
Copy link
Contributor Author

danking commented Sep 1, 2023

And a PR to fix the root cause #13538.

@danking danking merged commit 344b9e3 into hail-is:main Sep 1, 2023
danking added a commit that referenced this pull request Sep 1, 2023
This inspect command prevents us from updating a tag, for example, if we
need to replace an image with a security problem or if there is a bug
like the one fixed by #13536.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[query] Hail Python jobs do not work when Python is version greater than 3.9.
2 participants