UnicodeDecodeError exception in clone command

[johannesgajdosik]

I try to use hg-git for cloning a large git repo.
After running 28 hours it gives me a UnicodeDecodeError.
I would rather like that the cloning command could finish successfully.
In my opinion the cloning should not be impossible just because some file in the repo contains a non-ascii character.
There is not even a hint which file has the non-ascii char.

I have hg-git 0.8.11 and Mercurial 4.5.2, gentoo Linux.

Please have a look at this and tell me how to proceed.
Thanks in advance!

gaj@gajdosik /sda3 $ time hg clone git+ssh://xxxxx.xxx.xxx.xx:22/xxx/DefaultCollection/_git/XXXXXX
destination directory: XXXXXX
importing git objects into hg
** unknown exception encountered, please report by visiting
** https://mercurial-scm.org/wiki/BugTracker
** Python 2.7.14 (default, Jun 8 2018, 19:03:03) [GCC 6.4.0]
** Mercurial Distributed SCM (version 4.5.2)
** Extensions loaded: hgk, hggit
Traceback (most recent call last):
File "/usr/lib/python-exec/python2.7/hg", line 41, in
dispatch.run()
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 88, in run
status = (dispatch(req) or 0) & 255
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 183, in dispatch
ret = _runcatch(req)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 324, in _runcatch
return _callcatch(ui, _runcatchfunc)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 332, in _callcatch
return scmutil.callcatch(ui, func)
File "/usr/lib64/python2.7/site-packages/mercurial/scmutil.py", line 154, in callcatch
return func()
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 314, in _runcatchfunc
return _dispatch(req)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 918, in _dispatch
cmdpats, cmdoptions)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 673, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 926, in _runcommand
return cmdfunc()
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 915, in
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 1195, in check
return func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/mercurial/commands.py", line 1449, in clone
shareopts=opts.get('shareopts'))
File "/usr/lib64/python2.7/site-packages/mercurial/hg.py", line 661, in clone
streamclonerequested=stream)
File "/usr/lib64/python2.7/site-packages/hggit/util.py", line 56, in inner
return f(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/hggit/init.py", line 354, in exchangepull
pullop.cgresult = repo.githandler.fetch(remote.path, heads)
File "/usr/lib64/python2.7/site-packages/hggit/git_handler.py", line 300, in fetch
self.update_remote_branches(remote_name, result.refs)
File "/usr/lib64/python2.7/site-packages/hggit/git_handler.py", line 1467, in update_remote_branches
self.git.refs[ref_name] = sha
File "/usr/lib64/python2.7/site-packages/dulwich/refs.py", line 290, in setitem
self.set_if_equals(name, None, ref)
File "/usr/lib64/python2.7/site-packages/dulwich/refs.py", line 625, in set_if_equals
realnames, _ = self.follow(name)
File "/usr/lib64/python2.7/site-packages/dulwich/refs.py", line 226, in follow
contents = self.read_ref(refname)
File "/usr/lib64/python2.7/site-packages/dulwich/refs.py", line 200, in read_ref
contents = self.read_loose_ref(refname)
File "/usr/lib64/python2.7/site-packages/dulwich/refs.py", line 556, in read_loose_ref
filename = self.refpath(name)
File "/usr/lib64/python2.7/site-packages/dulwich/refs.py", line 482, in refpath
name = name.decode(sys.getfilesystemencoding())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 20: ordinal not in range(128)

real 1731m21.487s
user 1043m52.445s
sys 91m17.568s

[tamsky]

I've hit this error when tags or named refs contain unicode.
One fix would which will avoid cloning any tags or refs, would be to:

• first use git to clone the upstream repo as a git repo to your local machine:
  git clone git+ssh://xxxxx.xxx.xxx.xx:22/xxx/DefaultCollection/_git/XXXXXX /path/to/local/XXXXXX
• cd /path/to/local/XXXXXX
  then remove all git tags and refs (left as an exercise for the reader)
• cd ..
• hg clone XXXXXX new-hg-git-repo

which should let you succeed in cloning.

good luck, hopefully this helps

[johannesgajdosik]

Thanks for your reply and suggestion!
The repository is huge and constantly used by many coworkers. My plan was not only to clone the repo with hg-git but to actually work with it every day. Removing the git tags and refs all over again each time I fetch or pull is not an option.
Can you instead lead me to some peace of code that I can tweak so that the cloning can continue despite of having unicode tags and refs?
Thanks in advance!