Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildah pull from registry.fedoraproject.org fails with error tls: unexpected message #1504

Closed
sinnykumari opened this issue Apr 10, 2019 · 16 comments

Comments

@sinnykumari
Copy link

Description
buildah pull fails while fetching image from registry.fedoraproject.org

$ $ buildah  pull  registry.fedoraproject.org/fedora
Pulling docker://registry.fedoraproject.org/fedora
1 error occurred:
        * Error determining manifest MIME type for docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message

buildah version - bbuildah-1.7-17.dev.git146a0fc.fc30.x86_64
System detail - Fedora 30 Silverblue

Full log:

$ buildah --debug  pull registry.fedoraproject.org/fedora
DEBU[0000] running [buildah-in-a-user-namespace --debug pull registry.fedoraproject.org/fedora] with environment [SHELL=/bin/bash SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/1984,unix/unix:/tmp/.ICE-unix/1984 WINDOWID=23068679 COLORTERM=truecolor HISTCONTROL=ignoredups XDG_MENU_PREFIX=gnome- HOSTNAME=sinny-laptop HISTSIZE=-1 LANGUAGE= SSH_AUTH_SOCK=/run/user/1000/keyring/ssh HISTTIMEFORMAT=[%F %T]  SHELL_SESSION_ID=f4d382f1b3c14e9b923b3837d1668ea5 AWS_DEFAULT_REGION=ap-southeast-1 XMODIFIERS=@im=ibus DESKTOP_SESSION=gnome XDG_SEAT=seat0 ENV=/usr/share/Modules/init/profile.sh PWD=/home/skumari XDG_SESSION_DESKTOP=gnome LOGNAME=skumari XDG_SESSION_TYPE=wayland MODULESHOME=/usr/share/Modules MANPATH=: GDM_LANG=en_US.UTF-8 HOME=/home/skumari USERNAME=skumari SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass LANG=en_US.UTF-8 LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45: XDG_CURRENT_DESKTOP=GNOME KONSOLE_DBUS_SERVICE=:1.63 AWS_DEFAULT_OUTPUT=text WAYLAND_DISPLAY=wayland-0 KONSOLE_DBUS_SESSION=/Sessions/2 PROFILEHOME= AWS_SECRET_ACCESS_KEY=XHVN9NTCn4XotPq/+J8ILcsBJM7lHBNdHHrH9cqQ KONSOLE_VERSION=181202 MALLOC_PERTURB_=138 MODULEPATH_modshare=/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1 XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=||/usr/bin/lesspipe.sh %s USER=skumari COLORFGBG=15;0 MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LOADEDMODULES= DISPLAY=:0 SHLVL=1 BASH_ENV=/usr/share/Modules/init/bash AWS_ACCESS_KEY_ID=AKIAISWPEMTT3JAZ3TIA QT_IM_MODULE=ibus XDG_VTNR=2 DESKTOP_AUTOSTART_ID=1061fced81e302558b15549064562792900000019840018 XDG_SESSION_ID=2 LD_LIBRARY_PATH=:/home/skumari/work/dev/libabigail/build/src/.libs:/home/skumari/work/dev/fork/ostree/build/.libs/:/home/skumari/work/dev/libabigail/build/src/.libs:/home/skumari/work/dev/fork/ostree/build/.libs/ XDG_RUNTIME_DIR=/run/user/1000 KDEDIRS=/usr XDG_DATA_DIRS=/home/skumari/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/ PATH=/home/skumari/work/dev/libabigail/build/tools/:/home/skumari/work/dev/fork/ostree/build/.libs/:/home/skumari/go/bin/:/home/skumari/work/dev/libabigail/build/tools/:/home/skumari/work/dev/fork/ostree/build/.libs/:/home/skumari/go/bin/:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles KONSOLE_PROFILE_NAME=Profile 1 GDMSESSION=gnome HISTFILESIZE=-1 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus MAIL=/var/spool/mail/skumari GIO_LAUNCHED_DESKTOP_FILE_PID=2571 GIO_LAUNCHED_DESKTOP_FILE=/home/skumari/.config/autostart/org.kde.yakuake.desktop MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/share/Modules/libexec/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/share/Modules/libexec/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/Modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
} BASH_FUNC_module%%=() {  _module_raw "$@" 2>&1
} BASH_FUNC_scl%%=() {  if [ "$1" = "load" -o "$1" = "unload" ]; then
 eval "module $@";
 else
 /usr/bin/scl "$@";
 fi
} BASH_FUNC__module_raw%%=() {  unset _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 unset _mlre _mlIFS;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 eval `eval ${_mlre}/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash '"$@"'`;
 else
 eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`;
 fi;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 unset _mlre _mlv _mlrv _mlIFS;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlshdbg;
 return $_mlstatus
} _=/usr/bin/buildah _BUILDAH_STARTED_IN_USERNS=1 BUILDAH_ISOLATION=rootless], UID map [{HostID:1000 ContainerID:0 Size:1} {HostID:100000 ContainerID:1 Size:65536}], and GID map [{HostID:1000 ContainerID:0 Size:1} {HostID:100000 ContainerID:1 Size:65536}] 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] error parsing image name "registry.fedoraproject.org/fedora", trying with transport "docker://": Invalid image name "registry.fedoraproject.org/fedora", expected colon-separated transport:reference 
Pulling docker://registry.fedoraproject.org/fedora
DEBU[0000] parsed image name "docker://registry.fedoraproject.org/fedora" 
DEBU[0000] registry "registry.fedoraproject.org" is not marked as blocked in registries configuration "/etc/containers/registries.conf" 
DEBU[0000] parsed reference into "[overlay@/var/home/skumari/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]registry.fedoraproject.org/fedora:latest" 
DEBU[0000] parsed reference into "[overlay@/var/home/skumari/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]registry.fedoraproject.org/fedora:latest" 
DEBU[0000] copying "docker://registry.fedoraproject.org/fedora" to "registry.fedoraproject.org/fedora:latest" 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.fedoraproject.org/fedora:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fedoraproject.org 
DEBU[0000] Error creating parent directories for blob-info-cache-v1.boltdb, using a memory-only cache: mkdir /var/lib/containers/cache: permission denied 
DEBU[0000] GET https://registry.fedoraproject.org/v2/   
DEBU[0000] Ping https://registry.fedoraproject.org/v2/ err Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message (&url.Error{Op:"Get", URL:"https://registry.fedoraproject.org/v2/", Err:(*net.OpError)(0xc00053d4a0)}) 
DEBU[0000] GET https://registry.fedoraproject.org/v1/_ping 
DEBU[0001] Ping https://registry.fedoraproject.org/v1/_ping err Get https://registry.fedoraproject.org/v1/_ping: local error: tls: unexpected message (&url.Error{Op:"Get", URL:"https://registry.fedoraproject.org/v1/_ping", Err:(*net.OpError)(0xc00053d720)}) 
DEBU[0001] error copying src image ["docker://registry.fedoraproject.org/fedora"] to dest image ["registry.fedoraproject.org/fedora:latest"] err: Error determining manifest MIME type for docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message 
1 error occurred:
        * Error determining manifest MIME type for docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message


ERRO[0001] exit status 1

Additional info:
Note that podman pull registry.fedoraproject.org/fedora and buildah pull docker.io/fedora seems to work fine

@grdryn
Copy link

grdryn commented Apr 10, 2019

I was hitting this same problem earlier, but now it seems to work fine again. Could it have been an issue on the registry side that may have since been fixed?

@TomSweeneyRedHat
Copy link
Member

It smells like it. @mtrmac thoughts/opinions?

@sinnykumari
Copy link
Author

Now, it is working for me as well without making any changes. Not sure where the problem was though ...

Log from successful run:

$  buildah --debug  pull registry.fedoraproject.org/fedora
DEBU[0000] running [buildah-in-a-user-namespace --debug pull registry.fedoraproject.org/fedora] with environment [SHELL=/bin/bash SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/1789,unix/unix:/tmp/.ICE-unix/1789 WINDOWID=23068679 COLORTERM=truecolor HISTCONTROL=ignoredups XDG_MENU_PREFIX=gnome- HOSTNAME=sinny-laptop HISTSIZE=-1 LANGUAGE= SSH_AUTH_SOCK=/run/user/1000/keyring/ssh HISTTIMEFORMAT=[%F %T]  SHELL_SESSION_ID=a88aeeee581f480dbd160da76794ce07 AWS_DEFAULT_REGION=ap-southeast-1 XMODIFIERS=@im=ibus DESKTOP_SESSION=gnome XDG_SEAT=seat0 ENV=/usr/share/Modules/init/profile.sh PWD=/home/skumari XDG_SESSION_DESKTOP=gnome LOGNAME=skumari XDG_SESSION_TYPE=wayland MODULESHOME=/usr/share/Modules MANPATH=: GDM_LANG=en_US.UTF-8 HOME=/home/skumari USERNAME=skumari SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass LANG=en_US.UTF-8 LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45: XDG_CURRENT_DESKTOP=GNOME KONSOLE_DBUS_SERVICE=:1.62 AWS_DEFAULT_OUTPUT=text WAYLAND_DISPLAY=wayland-0 KONSOLE_DBUS_SESSION=/Sessions/2 PROFILEHOME= AWS_SECRET_ACCESS_KEY=XHVN9NTCn4XotPq/+J8ILcsBJM7lHBNdHHrH9cqQ KONSOLE_VERSION=181202 MALLOC_PERTURB_=119 MODULEPATH_modshare=/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1 XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=||/usr/bin/lesspipe.sh %s USER=skumari COLORFGBG=15;0 MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LOADEDMODULES= DISPLAY=:0 SHLVL=1 BASH_ENV=/usr/share/Modules/init/bash AWS_ACCESS_KEY_ID=AKIAISWPEMTT3JAZ3TIA QT_IM_MODULE=ibus XDG_VTNR=2 DESKTOP_AUTOSTART_ID=10b84f20d91c1dd4e3155495792414887100000017890018 XDG_SESSION_ID=2 LD_LIBRARY_PATH=:/home/skumari/work/dev/libabigail/build/src/.libs:/home/skumari/work/dev/fork/ostree/build/.libs/:/home/skumari/work/dev/libabigail/build/src/.libs:/home/skumari/work/dev/fork/ostree/build/.libs/ XDG_RUNTIME_DIR=/run/user/1000 KDEDIRS=/usr XDG_DATA_DIRS=/home/skumari/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/ PATH=/home/skumari/work/dev/libabigail/build/tools/:/home/skumari/work/dev/fork/ostree/build/.libs/:/home/skumari/go/bin/:/home/skumari/work/dev/libabigail/build/tools/:/home/skumari/work/dev/fork/ostree/build/.libs/:/home/skumari/go/bin/:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles KONSOLE_PROFILE_NAME=Profile 1 GDMSESSION=gnome HISTFILESIZE=-1 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus MAIL=/var/spool/mail/skumari GIO_LAUNCHED_DESKTOP_FILE_PID=2371 GIO_LAUNCHED_DESKTOP_FILE=/home/skumari/.config/autostart/org.kde.yakuake.desktop MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/share/Modules/libexec/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/share/Modules/libexec/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/Modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
} BASH_FUNC_module%%=() {  _module_raw "$@" 2>&1
} BASH_FUNC_scl%%=() {  if [ "$1" = "load" -o "$1" = "unload" ]; then
 eval "module $@";
 else
 /usr/bin/scl "$@";
 fi
} BASH_FUNC__module_raw%%=() {  unset _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in 
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 unset _mlre _mlIFS;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 eval `eval ${_mlre}/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash '"$@"'`;
 else
 eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`;
 fi;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 unset _mlre _mlv _mlrv _mlIFS;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlshdbg;
 return $_mlstatus
} _=/usr/bin/buildah _BUILDAH_STARTED_IN_USERNS=1 BUILDAH_ISOLATION=rootless], UID map [{HostID:1000 ContainerID:0 Size:1} {HostID:100000 ContainerID:1 Size:65536}], and GID map [{HostID:1000 ContainerID:0 Size:1} {HostID:100000 ContainerID:1 Size:65536}] 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] error parsing image name "registry.fedoraproject.org/fedora", trying with transport "docker://": Invalid image name "registry.fedoraproject.org/fedora", expected colon-separated transport:reference 
Pulling docker://registry.fedoraproject.org/fedora
DEBU[0000] parsed image name "docker://registry.fedoraproject.org/fedora" 
DEBU[0000] registry "registry.fedoraproject.org" is not marked as blocked in registries configuration "/etc/containers/registries.conf" 
DEBU[0000] parsed reference into "[overlay@/var/home/skumari/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]registry.fedoraproject.org/fedora:latest" 
DEBU[0000] parsed reference into "[overlay@/var/home/skumari/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]registry.fedoraproject.org/fedora:latest" 
DEBU[0000] copying "docker://registry.fedoraproject.org/fedora" to "registry.fedoraproject.org/fedora:latest" 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.fedoraproject.org/fedora:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fedoraproject.org 
DEBU[0000] Error creating parent directories for blob-info-cache-v1.boltdb, using a memory-only cache: mkdir /var/lib/containers/cache: permission denied 
DEBU[0000] GET https://registry.fedoraproject.org/v2/   
DEBU[0001] Ping https://registry.fedoraproject.org/v2/ status 200 
DEBU[0001] GET https://registry.fedoraproject.org/v2/fedora/manifests/latest 
DEBU[0002] Source is a manifest list; copying (only) instance sha256:faa4581556515da9f3621cde484db2439563cda9cdc8fbd8db2837999598428a 
DEBU[0002] GET https://registry.fedoraproject.org/v2/fedora/manifests/sha256:faa4581556515da9f3621cde484db2439563cda9cdc8fbd8db2837999598428a 
DEBU[0004] IsRunningImageAllowed for image docker:registry.fedoraproject.org/fedora:latest 
DEBU[0004]  Using default policy section                
DEBU[0004]  Requirement 0: allowed                      
DEBU[0004] Overall: allowed                             
DEBU[0004] Downloading /v2/fedora/blobs/sha256:81174df11aeb5baf4ecf69d936b72c329d2ec3d3becec52f07caeeb414cab3f2 
DEBU[0004] GET https://registry.fedoraproject.org/v2/fedora/blobs/sha256:81174df11aeb5baf4ecf69d936b72c329d2ec3d3becec52f07caeeb414cab3f2 
Getting image source signatures
DEBU[0008] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0008] ... will first try using the original manifest unmodified 
DEBU[0008] Downloading /v2/fedora/blobs/sha256:8dba660c242f3b7cfc0b888cc71510cebeb733286ae7e20283825acaaec72e06 
DEBU[0008] GET https://registry.fedoraproject.org/v2/fedora/blobs/sha256:8dba660c242f3b7cfc0b888cc71510cebeb733286ae7e20283825acaaec72e06 
Copying blob 8dba660c242f: 0 B / 92.52 MiB [-----------------------------------]
DEBU[0009] Detected compression format gzip             -----------------------]
Copying blob 8dba660c242f: 91.39 MiB / 92.52 MiB [=============================]
Copying blob 8dba660c242f: 92.52 MiB / 92.52 MiB [==========================] 9s
Copying config 81174df11aeb: 0 B / 1.27 KiB [----------------------------------]DEBU[0017] No compression detected                      
DEBU[0017] Using original blob without modification     
Copying config 81174df11aeb: 1.27 KiB / 1.27 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
DEBU[0017] Applying tar in /var/home/skumari/.local/share/containers/storage/overlay/da22bd5bfb28630a6045ab8caa0770c672f2878f10a314a3cec19c66b17b7449/diff 
DEBU[0019] setting image creation date to 2019-02-18 06:48:39 +0000 UTC 
DEBU[0019] created new image ID "81174df11aeb5baf4ecf69d936b72c329d2ec3d3becec52f07caeeb414cab3f2" 
DEBU[0019] set names of image "81174df11aeb5baf4ecf69d936b72c329d2ec3d3becec52f07caeeb414cab3f2" to [registry.fedoraproject.org/fedora:latest] 
DEBU[0019] saved image metadata "{}"                    
81174df11aeb5baf4ecf69d936b72c329d2ec3d3becec52f07caeeb414cab3f2

Closing the issue becasue I no longer see this.

@fooishbar
Copy link

I'm still seeing this, on a F30 Silverblue host. I wonder if it's something to do with Go and TLS certificate handling? curl https://registry.fedoraproject.org/v2/ works fine, whereas it fails with unexpected message from Buildah.

@grdryn
Copy link

grdryn commented Apr 15, 2019

Hmm, yeah, I can confirm that buildah pull registry.fedoraproject.org/fedora doesn't work again, but as before, podman pull registry.fedoraproject.org/fedora still works. 🤔

@TomSweeneyRedHat
Copy link
Member

Per the above and IRC pings, I'm going to reopen.

@TomSweeneyRedHat
Copy link
Member

TomSweeneyRedHat commented Apr 16, 2019

Just adding a IRC chat line that might help with debugging from rishi:

rishi> Apparently 'podman pull registry.fedoraproject.org/fedora' and 
'buildah pull pull docker.io/fedora' work fine. 
Haven't had any problems on F29.

@debarshiray
Copy link
Member

Just adding a IRC chat line that might help with debugging from rishi:

It was @sinnykumari who discovered that originally. It's part of her initial report but got a bit lost in the text. :)

@tosmi
Copy link

tosmi commented Apr 18, 2019

i'm having the exact same issue under silverblue 30. podman pull works, buildah pull fails with tls: unexpected message. discovered yesterday (tried to create a fc 30 container with toolbox). still happening today.

@rhatdan
Copy link
Member

rhatdan commented Apr 18, 2019

Can you try with the buildah from master?

@tosmi
Copy link

tosmi commented Apr 20, 2019

sorry for the late response, i was fighting with my local installation of silverblue 30...

pinhead@steve: buildah  [master] $ ./buildah version
Version:         1.8-dev
Go Version:      go1.12.2
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-rc2
Git Commit:      2f0179fa
Built:           Sat Apr 20 16:49:02 2019
OS/Arch:         linux/amd64
pinhead@steve: buildah  [master] $ ./buildah pull registry.fedoraproject.org/f29/fedora-toolbox:29
Getting image source signatures
Copying blob 6799e006b89c done
Copying blob 0b134f8f4722 done
Copying config 53700f9999 done
Writing manifest to image destination
Storing signatures
53700f999992175f9201af8e3672beafb93b1d39eed3da7ee002404e9b390373

so the latest master works for me. here is the output of buildah that comes currently with silverblue 30:

pinhead@steve: buildah  [master] $ buildah version
Version:         1.7-dev
Go Version:      go1.12beta2
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Jan  1 01:00:00 1970
OS/Arch:         linux/amd64
pinhead@steve: buildah  [master] $ buildah pull registry.fedoraproject.org/f29/fedora-toolbox:29
Pulling docker://registry.fedoraproject.org/f29/fedora-toolbox:29
1 error occurred:
	* Error determining manifest MIME type for docker://registry.fedoraproject.org/f29/fedora-toolbox:29: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message


ERRO[0000] exit status 1   

thanks
toni

@carlwgeorge
Copy link

I'm also seeing this error with buildah pulling from registry.fedoraproject.org on Fedora 30. It works if I downgrade, and it works if I upgrade to the buildah from rawhide.

  • buildah-1.7-3.git873f001.fc30.x86_64 works
  • buildah-1.7-17.dev.git146a0fc.fc30.x86_64 doesn't work
  • buildah-1.8-39.dev.git2f0179f.fc31.x86_64 works

@debarshiray
Copy link
Member

Thanks for testing @carlwgeorge

* `buildah-1.7-3.git873f001.fc30.x86_64` works

* `buildah-1.7-17.dev.git146a0fc.fc30.x86_64` doesn't work

* `buildah-1.8-39.dev.git2f0179f.fc31.x86_64` works

If that's true, then it seems like a Fedora packaging bug to me:

$ rpmdev-vercmp 1.7-17.dev.git146a0fc 1.7-3.git873f001
1.7-17.dev.git146a0fc > 1.7-3.git873f001

buildah-1.7-17.dev.git146a0fc.fc30.x86_64 is a pre-release snapshot leading up to 1.7, and should have been replaced with buildah-1.7-3.git873f001.fc30.x86_64. It would have worked if the snapshot had used 0.17, not 17, as the release.

@carlwgeorge
Copy link

Ah, I see.

This is an RPM ordering issue. The short term fix is to bump the F30 package to release 18 or higher. Long term, your suggestion of using 0.X for pre-release snapshots in rawhide would prevent this from happening again.

@debarshiray
Copy link
Member

There's an new build that unbreaks the update path:
https://bodhi.fedoraproject.org/updates/FEDORA-2019-a329c204c3

This issue can be closed now.

@sinnykumari
Copy link
Author

This issue is fixed with buildah-1.7-18.git873f001.fc30.x86_64 in Fedora 30

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants