makecatalogs.munki fails with UnicodeEncodeError

[workflowsguy]

Describe the problem
Since the upgrade to Munki 4, the makecatalogs.munki recipe keeps failing with the following error:

Processing makecatalogs.munki...
WARNING: makecatalogs.munki is missing trust info and FAIL_RECIPES_WITHOUT_TRUST_INFO is not set. Proceeding...
MakeCatalogsProcessor
makecatalogs failed: 
Traceback (most recent call last):
  File "/usr/local/munki/makecatalogs", line 109, in <module>
    main()
  File "/usr/local/munki/makecatalogs", line 98, in main
    errors = makecatalogslib.makecatalogs(repo, options, output_fn=print)
  File "/usr/local/munki/munkilib/admin/makecatalogslib.py", line 258, in makecatalogs
    repo, options, output_fn=output_fn)
  File "/usr/local/munki/munkilib/admin/makecatalogslib.py", line 231, in process_pkgsinfo
    output_fn("Adding %s to %s..." % (pkginfo_ref, catalogname))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0301' in position 25: ordinal not in range(128)

Failed.
Receipt written to /Users/stefan/Library/AutoPkg/Cache/com.github.autopkg.munki.makecatalogs/receipts/makecatalogs-receipt-20191219-172916.plist

The following recipes failed:
    makecatalogs.munki
        Error in com.github.autopkg.munki.makecatalogs: Processor: MakeCatalogsProcessor: Error: makecatalogs failed: 
        Traceback (most recent call last):
          File "/usr/local/munki/makecatalogs", line 109, in <module>
            main()
          File "/usr/local/munki/makecatalogs", line 98, in main
            errors = makecatalogslib.makecatalogs(repo, options, output_fn=print)
          File "/usr/local/munki/munkilib/admin/makecatalogslib.py", line 258, in makecatalogs
            repo, options, output_fn=output_fn)
          File "/usr/local/munki/munkilib/admin/makecatalogslib.py", line 231, in process_pkgsinfo
            output_fn("Adding %s to %s..." % (pkginfo_ref, catalogname))
        UnicodeEncodeError: 'ascii' codec can't encode character u'\u0301' in position 25: ordinal not in range(128)

Version (please complete the following information):

• OS version: 10.13.6
• AutoPkg Version: 1.4.1

[homebysix]

Hi @workflowsguy - \u0301 is a "combining acute accent" character. Does that character exist in any of your AutoPkg recipes, or in the names of any catalogs?

[hjuutilainen]

Does it help if you set the environment variable PYTHONIOENCODING to utf-8 when running AutoPkg (run export PYTHONIOENCODING=utf-8)?

Long story short, I was debugging a similar issue with MunkiAdmin where makecatalogs would run perfectly when in Terminal but would fail when being called by external process. Python 3 seems to treat terminal output differently than piping the output to another command. Not really knowing enough about this I’d appreciate if someone with more Python 3 experience could explain why this is happening and what would be the best solution.

[mikaellofgren]

Similar problem here:
autopkg run -v local.munki.MakeCatalogs -k force_rebuild=YES Processing local.munki.MakeCatalogs... MakeCatalogsProcessor makecatalogs failed: Traceback (most recent call last): File "/usr/local/munki/makecatalogs", line 109, in <module> main() File "/usr/local/munki/makecatalogs", line 98, in main errors = makecatalogslib.makecatalogs(repo, options, output_fn=print) File "/usr/local/munki/munkilib/admin/makecatalogslib.py", line 258, in makecatalogs repo, options, output_fn=output_fn) File "/usr/local/munki/munkilib/admin/makecatalogslib.py", line 231, in process_pkgsinfo output_fn("Adding %s to %s..." % (pkginfo_ref, catalogname)) UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 47: ordinal not in range(128)

OS version: Mojave 10.14.6
Autopkg: 1.4.1
Munkitools: 4.0.0.3881
When running makecatalogs in terminal it seems to work fine

[mikaellofgren]

Did a update to Munki 4.0.1.3899, seems to get rid of the error

[homebysix]

I am able to reproduce this on Munki tools 4.0.1 with either AutoPkg 1.4.1 or AutoPkg 2.0 RC1, if there's a non-ASCII file/folder path in the Munki repo. (Verbose log: https://gist.github.com/homebysix/aac010cce244d4e12a7bad38a1417665)

@workflowsguy, could you run this on your Munki repo and see whether any files appear?

find /path/to/munki_repo -name '*[! -~]*'

[workflowsguy]

@homebysix , apologies for the late reply.

Running find /Volumes/Web/Softwareverteilung -name '*[! -~]*' gives

/Volumes/Web/Softwareverteilung/pkgsinfo/apps/Dag Ågren
/Volumes/Web/Softwareverteilung/pkgsinfo/apps/Carsten Blüm
/Volumes/Web/Softwareverteilung/pkgsinfo/apps/Stéphane Sudre
/Volumes/Web/Softwareverteilung/pkgsinfo/apps/René Mach
/Volumes/Web/Softwareverteilung/pkgsinfo/apps/Sveinbjörn Þórðarson
/Volumes/Web/Softwareverteilung/pkgs/apps/Dag Ågren
/Volumes/Web/Softwareverteilung/pkgs/apps/Stéphane Sudre
/Volumes/Web/Softwareverteilung/pkgs/apps/Carsten Blüm
/Volumes/Web/Softwareverteilung/pkgs/apps/René Mach
/Volumes/Web/Softwareverteilung/pkgs/apps/Sveinbjörn Þórðarson
/Volumes/Web/Softwareverteilung/Recipe-Repositories/com.github.autopkg.workflowsguy-recipes/Dag Ågren
/Volumes/Web/Softwareverteilung/Recipe-Repositories/com.github.autopkg.workflowsguy-recipes/Carsten Blüm
/Volumes/Web/Softwareverteilung/Recipe-Repositories/com.github.autopkg.workflowsguy-recipes/René Mach

[mikaellofgren]

I get this output, the first one seems reporting right, but the second "Skolkorridoren Kyocera P6035.plist" When I added underscore instead "Skolkorridoren_Kyocera_P6035.plist" the file doesn't show up when searching. Thanks!

/Volumes/repo/pkgs/Apps/•/STARTA BLÅ BOKEN.fmp12.pkg
/Volumes/repo/pkgsinfo/Apps/•/Skolkorridoren Kyocera P6035.plist

• Stripped out, but same name as another files in same path

[workflowsguy]

autopkg 2.1:
The issue still exists

[workflowsguy]

What are the chances of this issue getting addressed anytime soon?

I have more and more packages that I cannot install with Munki because they contain Umlauts or accented characters in the package or developer name.

[nmcspadden]

Does this repro with any recipes right now in North American English locale? Or if you can provide an exact repro case, that would be helpful.

[gregneagle]

If it's only MakeCatalogs.munki that is failing, a workaround would be to not run that recipe as part of a recipe list, and to run makecatalogs separately after a recipe list is run, perhaps in a wrapper script.

It's kind of over-stating things to claim "I have more and more packages that I cannot install with Munki because they contain Umlauts or accented characters in the package or developer name." because of this issue, since the workarounds are easy and straight-forward.

[workflowsguy]

@nmcspadden

Does this repro with any recipes right now in North American English locale? Or if you can provide an exact repro case, that would be helpful.

if you need any additional information besides what i have already given in previous posts, please tell me what you need.

[workflowsguy]

@gregneagle

It's kind of over-stating things to claim "I have more and more packages that I cannot install with Munki because they contain Umlauts or accented characters in the package or developer name." because of this issue, since the workarounds are easy and straight-forward.

It is not my main job to be a system admin, i have to do this in addition to my regular work. So a workaround to an issue that might be "easy and straight-forward" to you as the creator/maintainer of Munki is not to me.

[gregneagle]

Fair enough. We’re all volunteers here, working on things in our spare time. I’m just suggesting alternative solutions you could explore to at least address it for yourself.