Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow go get from https://*.onion with self-signed certificates #41076

Closed
AnimusPEXUS opened this issue Aug 27, 2020 · 8 comments
Closed

Allow go get from https://*.onion with self-signed certificates #41076

AnimusPEXUS opened this issue Aug 27, 2020 · 8 comments

Comments

@AnimusPEXUS
Copy link

@AnimusPEXUS AnimusPEXUS commented Aug 27, 2020

For some reasons I can not use clearnet for some of My golang packages.
So I'm trying to host my packages using gitea and tor hidden services.

calling

export all_proxy="socks5://127.0.0.1:9050"
export http_proxy=$all_proxy
export https_proxy=$all_proxy
go get -u -v 'onionsite.onion/project/package'

results in x509: certificate signed by unknown authority
server hosts self-signed certificate

probably it is ok to allow serf-signed .onion certificates for golang packages hosts

also, trying to use --insecure flag, result in Could not resolve host error

@AnimusPEXUS AnimusPEXUS changed the title Allow go get from https:// with self-signed certificates Allow go get from https://*.onion with self-signed certificates Aug 27, 2020
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Aug 27, 2020

Using GOINSECURE (which unlike --insecure lets you specify that you only want to skip checks for those domains) and GOPRIVATE (to bypass proxy.golang.org and sumdb) should work. If it doesn't and you think it's a Go issue, please provide more details of how you used those settings and the result, including environment and output.

@AnimusPEXUS
Copy link
Author

@AnimusPEXUS AnimusPEXUS commented Aug 27, 2020

Looks like something strange happening:
looks like GOINSECURE and GOPRIVATE doesn't change go command behavior.

now I've modified above script so:

export all_proxy="socks5://127.0.0.1:9050"
export http_proxy=$all_proxy
export https_proxy=$all_proxy

ONION_DOMAIN='someonion.onion'

export GOINSECURE=$ONION_DOMAIN
#export GOPRIVATE=$ONION_DOMAIN
#export GONOPROXY=$ONION_DOMAIN
#export GONOSUMDB=$ONION_DOMAIN
"$@"

I've saved it to ./soxy file and I'm trying to get repository using following command: ./soxy go get -u -v 'onionsite.onion/project/package'
only adding --insecure flag make some change and I getting Could not resolve host with it and x509: certificate signed by unknown authority without it.

as You asked about environment:

  • my go version is go version go1.14.6 linux/amd64
  • my environment is 64-bit Fedora 32
  • also I use this docker-compose to setup hidden gitea instance https://github.com/AnimusPEXUS/hiddengitea
  • following is my environment variables after above script:
GOINSECURE=someonion.onion
GONOPROXY=someonion.onion
GONOSUMDB=someonion.onion
GOPATH=/home/animuspexus/gopath_main:/home/animuspexus/gopath_work
GOPRIVATE=someonion.onion
GOROOT=/usr/lib/golang
all_proxy=socks5://127.0.0.1:9050
http_proxy=socks5://127.0.0.1:9050
https_proxy=socks5://127.0.0.1:9050
  • some standard Fedora 32 variables
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Aug 27, 2020

What's the output of go env and what is the full go get output?

@AnimusPEXUS
Copy link
Author

@AnimusPEXUS AnimusPEXUS commented Aug 27, 2020

BASH=/bin/bash
BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=([0]="0")
BASH_SOURCE=([0]="/home/animuspexus/if/soxy")
BASH_VERSINFO=([0]="5" [1]="0" [2]="17" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='5.0.17(1)-release'
COLORTERM=truecolor
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DESKTOP_SESSION=gnome
DIRSTACK=()
DISPLAY=:1
EUID=1000
GDMSESSION=gnome
GDM_LANG=en_US.UTF-8
GJS_DEBUG_OUTPUT=stderr
GJS_DEBUG_TOPICS='JS ERROR;JS LOG'
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/459e68c4_36b2_4f2b_a140_bb42256e8444
GNOME_TERMINAL_SERVICE=:1.78
GOINSECURE=someonion.onion
GONOPROXY=someonion.onion
GONOSUMDB=someonion.onion
GOPATH=/home/animuspexus/gopath_main:/home/animuspexus/gopath_work
GOPRIVATE=someonion.onion
GOROOT=/usr/lib/golang
GROUPS=()
HISTCONTROL=ignoreboth
HISTSIZE=1000
HOME=/home/animuspexus
HOSTNAME=localhost.localdomain
HOSTTYPE=x86_64
IFS=$' \t\n'
INVOCATION_ID=126bf759904c448698c391253a06f5fb
JOURNAL_STREAM=8:48861
LANG=en_US.UTF-8
LESSOPEN='||/usr/bin/lesspipe.sh %s'
LOADEDMODULES=
LOGNAME=animuspexus
LS_COLORS='rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.webp=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
MACHTYPE=x86_64-redhat-linux-gnu
MAIL=/var/spool/mail/animuspexus
MANAGERPID=1660
MANPATH=:
MC_SID=38677
MC_TMPDIR=/var/tmp/mc-animuspexus
MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles
MODULEPATH_modshare=/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1
MODULESHOME=/usr/share/Modules
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH
OLDPWD=/home/animuspexus/tmp
ONION_DOMAIN=someonion.onion
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/home/animuspexus/.local/bin:/home/animuspexus/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/animuspexus/gopath_main/bin:/home/animuspexus/gopath_work/bin:/home/animuspexus/gopath_main/bin:/home/animuspexus/gopath_work/bin
PIPESTATUS=([0]="0")
PPID=38677
PS4='+ '
PWD=/home/animuspexus/tmp/go3
QT_IM_MODULE=ibus
SDL_VIDEO_MINIMIZE_ON_FOCUS_LOSS=0
SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/1792,unix/unix:/tmp/.ICE-unix/1792
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=3
SSH_AGENT_PID=1747
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
STEAM_FRAME_FORCE_CLOSE=1
TERM=xterm-256color
UID=1000
USER=animuspexus
USERNAME=animuspexus
VTE_VERSION=6003
WINDOWPATH=2
WISECONFIGDIR=/usr/share/wise2/
XAUTHORITY=/run/user/1000/gdm/Xauthority
XDG_CURRENT_DESKTOP=GNOME
XDG_DATA_DIRS=/home/animuspexus/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/
XDG_MENU_PREFIX=gnome-
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_CLASS=user
XDG_SESSION_DESKTOP=gnome
XDG_SESSION_TYPE=x11
XMODIFIERS=@im=ibus
_=GONOSUMDB=someonion.onion
all_proxy=socks5://127.0.0.1:9050
http_proxy=socks5://127.0.0.1:9050
https_proxy=socks5://127.0.0.1:9050
_module_raw () 
{ 
    unset _mlshdbg;
    if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
        case "$-" in 
            *v*x*)
                set +vx;
                _mlshdbg='vx'
            ;;
            *v*)
                set +v;
                _mlshdbg='v'
            ;;
            *x*)
                set +x;
                _mlshdbg='x'
            ;;
            *)
                _mlshdbg=''
            ;;
        esac;
    fi;
    unset _mlre _mlIFS;
    if [ -n "${IFS+x}" ]; then
        _mlIFS=$IFS;
    fi;
    IFS=' ';
    for _mlv in ${MODULES_RUN_QUARANTINE:-};
    do
        if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
            if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
                _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
            fi;
            _mlrv="MODULES_RUNENV_${_mlv}";
            _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
        fi;
    done;
    if [ -n "${_mlre:-}" ]; then
        eval `eval ${_mlre}/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash '"$@"'`;
    else
        eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`;
    fi;
    _mlstatus=$?;
    if [ -n "${_mlIFS+x}" ]; then
        IFS=$_mlIFS;
    else
        unset IFS;
    fi;
    unset _mlre _mlv _mlrv _mlIFS;
    if [ -n "${_mlshdbg:-}" ]; then
        set -$_mlshdbg;
    fi;
    unset _mlshdbg;
    return $_mlstatus
}
module () 
{ 
    _module_raw "$@" 2>&1
}
scl () 
{ 
    if [ "$1" = "load" -o "$1" = "unload" ]; then
        eval "module $@";
    else
        /usr/bin/scl "$@";
    fi
}
switchml () 
{ 
    typeset swfound=1;
    if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
        typeset swname='main';
        if [ -e /usr/share/Modules/libexec/modulecmd.tcl ]; then
            typeset swfound=0;
            unset MODULES_USE_COMPAT_VERSION;
        fi;
    else
        typeset swname='compatibility';
        if [ -e /usr/share/Modules/libexec/modulecmd-compat ]; then
            typeset swfound=0;
            MODULES_USE_COMPAT_VERSION=1;
            export MODULES_USE_COMPAT_VERSION;
        fi;
    fi;
    if [ $swfound -eq 0 ]; then
        echo "Switching to Modules $swname version";
        source /usr/share/Modules/init/bash;
    else
        echo "Cannot switch to Modules $swname version, command not found";
        return 1;
    fi
}

without --insecure

~/if/soxy go get -u -v 'someonion.onion/name/repo'
unrecognized import path "someonion.onion/name/repo": https fetch: Get "https://someonion.onion/name/repo?go-get=1": x509: certificate signed by unknown authority

with --insecure

~/if/soxy go get -u -v --insecure 'someonion.onion/name/repo'
get "someonion.onion/name/repo": found meta tag get.metaImport{Prefix:"someonion.onion/name/repo", VCS:"git", RepoRoot:"https://someonion.onion/name/repo.git"} at //someonion.onion/name/repo?go-get=1
someonion.onion/name/repo (download)
# cd .; git clone -- https://someonion.onion/name/repo.git /home/animuspexus/gopath_main/src/someonion.onion/name/repo
Cloning into '/home/animuspexus/gopath_main/src/someonion.onion/name/repo'...
fatal: unable to access 'https://someonion.onion/name/repo.git/': Could not resolve host: someonion.onion
package someonion.onion/name/repo: exit status 128
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Aug 27, 2020

This is not the go tool erroring out, the go tool can reach the HTML page, which points it to https://someonion.onion/name/repo.git. That's passed to git clone, which fails because presumably it's not setup to work with the SOCKS5 proxy.

Not sure why GOINSECURE isn't working though. Can you post the output of go env (not env)?

@AnimusPEXUS
Copy link
Author

@AnimusPEXUS AnimusPEXUS commented Aug 27, 2020

presumably it's not setup to work with the SOCKS5 proxy

thanks, I'll look into this.

~/if/soxy go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/animuspexus/.cache/go-build"
GOENV="/home/animuspexus/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE="someonion.onion"
GONOPROXY="someonion.onion"
GONOSUMDB="someonion.onion"
GOOS="linux"
GOPATH="/home/animuspexus/gopath_main:/home/animuspexus/gopath_work"
GOPRIVATE="someonion.onion"
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build302437077=/tmp/go-build -gno-record-gcc-switches"
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Aug 27, 2020

Ah, GOINSECURE only works in module mode, while you might still be using GOPATH here. I think the go tool is working as intended, and you need to resolve your issue with git.

@AnimusPEXUS
Copy link
Author

@AnimusPEXUS AnimusPEXUS commented Aug 28, 2020

If somebody interested, I've managed this working

git simply requires this in config:

[http "https://*.onion"]
   sslVerify = false
   proxy = "socks5h://127.0.0.1:9050"

also I used following script to run go programs over tor (note: GOINSECURE and GOPRIVATE not needed if go modules not used):

#!/bin/bash

export all_proxy="socks5://127.0.0.1:9050"
export http_proxy=$all_proxy
export https_proxy=$all_proxy

"$@"

also I used ~/.netrc file to provide http username:password

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.