Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

outcontainer unicodeencodeerror #600

Closed
woshimanong1990 opened this issue Jan 22, 2020 · 8 comments
Closed

outcontainer unicodeencodeerror #600

woshimanong1990 opened this issue Jan 22, 2020 · 8 comments
Labels
bug

Comments

@woshimanong1990
Copy link

IMPORTANT: Be sure to replace all template sections {{ like this }} or your issue may be discarded.

Overview

the out file path contain chinese character, it report unicodeencodeerror

I have fix the bug. But I don't know how to sumbit a PR

`

    diff --git a/av/container/output.pyx b/av/container/output.pyx
    index 8781958..daefd1c 100644
    --- a/av/container/output.pyx
    +++ b/av/container/output.pyx
    @@ -1,5 +1,6 @@
    from fractions import Fraction
    import logging
    +import sys

    from av.codec.codec cimport Codec
    from av.container.streams cimport StreamContainer
    @@ -11,6 +12,14 @@ from av.utils cimport dict_to_avdict

    from av.dictionary import Dictionary

    +try:
    +    from os import fsencode
    +except ImportError:
    +    _fsencoding = sys.getfilesystemencoding()
    +
    +    def fsencode(s):
    +        return s.encode(_fsencoding)
    +

    log = logging.getLogger(__name__)

    @@ -162,7 +171,9 @@ cdef class OutputContainer(Container):
                stream._finalize_for_output()

            # Open the output file, if needed.
    -        cdef char *name = "" if self.file is not None else self.name
    +        file_name = "" if self.file is not None else self.name
    +        cdef bytes name_obj = fsencode(file_name) if isinstance(file_name, unicode) else file_name
    +        cdef char *name = name_obj
            if self.ptr.pb == NULL and not self.ptr.oformat.flags & lib.AVFMT_NOFILE:
                err_check(lib.avio_open(&self.ptr.pb, name, lib.AVIO_FLAG_WRITE))

`
@mikeboers
Copy link
Member

Thanks! I'll make this change at some point. =]

@necan
Copy link

necan commented Mar 17, 2020

seems not fixed yet

@jlaine
Copy link
Collaborator

jlaine commented Apr 29, 2020

Hi @necan could you give me an example of a filename which triggers this bug please so that we can capture it in a unit test?

Now that we only support Python 3.5+ we can unconditionally use os.fsencode()

@necan
Copy link

necan commented Apr 29, 2020 •
edited

@jlaine it appears in Windows with Simplified Chinese Locale which uses GB18030 as the default character encoding of cmd or Powershelll.

jlaine added a commit to jlaine/PyAV that referenced this issue Apr 29, 2020
@jlaine
Use os.fsencode for both input and output file names (fixes: PyAV-Org…
352ccb1 
…#600)

It's safe to call os.fsencode() regardless of whether the filename was
passed as `bytes` or `str`. Remove shim for `os.fsencode()`, it has been
available since Python 3.2.
jlaine added a commit to jlaine/PyAV that referenced this issue Apr 29, 2020
@jlaine
Use os.fsencode for both input and output file names (fixes: PyAV-Org…
0ee38f6 
…#600)

It's safe to call os.fsencode() regardless of whether the filename was
passed as `bytes` or `str`. Remove shim for `os.fsencode()`, it has been
available since Python 3.2.
jlaine added a commit to jlaine/PyAV that referenced this issue Apr 29, 2020
@jlaine
Use os.fsencode for both input and output file names (fixes: PyAV-Org…
067b0d6 
…#600)

It's safe to call os.fsencode() regardless of whether the filename was
passed as `bytes` or `str`. Remove shim for `os.fsencode()`, it has been
available since Python 3.2.
@jlaine
Copy link
Collaborator

jlaine commented Apr 29, 2020

Could you give PR #643 a try and let me know if it fixes the issue for you?

@necan
Copy link

necan commented Apr 29, 2020

@jlaine build error

    Running setup.py install for av ... error
    ERROR: Command errored out with exit status 1:
     command: 'd:\desktop\新建文件夹\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\cheng\\AppData\\Local\\Temp\\pip-req-build-ah33rmuy\\setup.py'"'"'; __file__='"'"'C:\\Users\\cheng\\AppData\\Local\\Temp\\pip-req-build-ah33rmuy\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\cheng\AppData\Local\Temp\pip-record-w2ujctdp\install-record.txt' --single-version-externally-managed --compile --install-headers 'd:\desktop\新建文件夹\venv\include\site\python3.8\av'
         cwd: C:\Users\cheng\AppData\Local\Temp\pip-req-build-ah33rmuy\
    Complete output (132 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.8
    creating build\lib.win-amd64-3.8\av
    copying av\datasets.py -> build\lib.win-amd64-3.8\av
    copying av\deprecation.py -> build\lib.win-amd64-3.8\av
    copying av\__init__.py -> build\lib.win-amd64-3.8\av
    copying av\__main__.py -> build\lib.win-amd64-3.8\av
    creating build\lib.win-amd64-3.8\av\audio
    copying av\audio\__init__.py -> build\lib.win-amd64-3.8\av\audio
    creating build\lib.win-amd64-3.8\av\codec
    copying av\codec\__init__.py -> build\lib.win-amd64-3.8\av\codec
    creating build\lib.win-amd64-3.8\av\container
    copying av\container\__init__.py -> build\lib.win-amd64-3.8\av\container
    creating build\lib.win-amd64-3.8\av\data
    copying av\data\__init__.py -> build\lib.win-amd64-3.8\av\data
    creating build\lib.win-amd64-3.8\av\filter
    copying av\filter\__init__.py -> build\lib.win-amd64-3.8\av\filter
    creating build\lib.win-amd64-3.8\av\sidedata
    copying av\sidedata\__init__.py -> build\lib.win-amd64-3.8\av\sidedata
    creating build\lib.win-amd64-3.8\av\subtitles
    copying av\subtitles\__init__.py -> build\lib.win-amd64-3.8\av\subtitles
    creating build\lib.win-amd64-3.8\av\video
    copying av\video\__init__.py -> build\lib.win-amd64-3.8\av\video
    running build_ext
    running config
    writing build\temp.win-amd64-3.8\Release\include\pyav\config.h
    running cythonize
    Compiling av\buffer.pyx because it changed.
    [1/1] Cythonizing av\buffer.pyx
    Compiling av\bytesource.pyx because it changed.
    [1/1] Cythonizing av\bytesource.pyx
    Compiling av\descriptor.pyx because it changed.
    [1/1] Cythonizing av\descriptor.pyx
    Compiling av\dictionary.pyx because it changed.
    [1/1] Cythonizing av\dictionary.pyx
    Compiling av\enum.pyx because it changed.
    [1/1] Cythonizing av\enum.pyx
    Compiling av\error.pyx because it changed.
    [1/1] Cythonizing av\error.pyx
    Compiling av\format.pyx because it changed.
    [1/1] Cythonizing av\format.pyx
    Compiling av\frame.pyx because it changed.
    [1/1] Cythonizing av\frame.pyx
    Compiling av\logging.pyx because it changed.
    [1/1] Cythonizing av\logging.pyx
    Compiling av\option.pyx because it changed.
    [1/1] Cythonizing av\option.pyx
    Compiling av\packet.pyx because it changed.
    [1/1] Cythonizing av\packet.pyx
    Compiling av\plane.pyx because it changed.
    [1/1] Cythonizing av\plane.pyx
    Compiling av\stream.pyx because it changed.
    [1/1] Cythonizing av\stream.pyx
    Compiling av\utils.pyx because it changed.
    [1/1] Cythonizing av\utils.pyx
    Compiling av\_core.pyx because it changed.
    [1/1] Cythonizing av\_core.pyx
    Compiling av\audio\codeccontext.pyx because it changed.
    [1/1] Cythonizing av\audio\codeccontext.pyx
    Compiling av\audio\fifo.pyx because it changed.
    [1/1] Cythonizing av\audio\fifo.pyx
    Compiling av\audio\format.pyx because it changed.
    [1/1] Cythonizing av\audio\format.pyx
    Compiling av\audio\frame.pyx because it changed.
    [1/1] Cythonizing av\audio\frame.pyx
    Compiling av\audio\layout.pyx because it changed.
    [1/1] Cythonizing av\audio\layout.pyx
    Compiling av\audio\plane.pyx because it changed.
    [1/1] Cythonizing av\audio\plane.pyx
    Compiling av\audio\resampler.pyx because it changed.
    [1/1] Cythonizing av\audio\resampler.pyx
    Compiling av\audio\stream.pyx because it changed.
    [1/1] Cythonizing av\audio\stream.pyx
    Compiling av\codec\codec.pyx because it changed.
    [1/1] Cythonizing av\codec\codec.pyx
    Compiling av\codec\context.pyx because it changed.
    [1/1] Cythonizing av\codec\context.pyx
    Compiling av\container\core.pyx because it changed.
    [1/1] Cythonizing av\container\core.pyx
    Compiling av\container\input.pyx because it changed.
    [1/1] Cythonizing av\container\input.pyx
    Compiling av\container\output.pyx because it changed.
    [1/1] Cythonizing av\container\output.pyx
    Compiling av\container\pyio.pyx because it changed.
    [1/1] Cythonizing av\container\pyio.pyx
    Compiling av\container\streams.pyx because it changed.
    [1/1] Cythonizing av\container\streams.pyx
    Compiling av\data\stream.pyx because it changed.
    [1/1] Cythonizing av\data\stream.pyx
    Compiling av\filter\context.pyx because it changed.
    [1/1] Cythonizing av\filter\context.pyx
    Compiling av\filter\filter.pyx because it changed.
    [1/1] Cythonizing av\filter\filter.pyx
    Compiling av\filter\graph.pyx because it changed.
    [1/1] Cythonizing av\filter\graph.pyx
    Compiling av\filter\link.pyx because it changed.
    [1/1] Cythonizing av\filter\link.pyx
    Compiling av\filter\pad.pyx because it changed.
    [1/1] Cythonizing av\filter\pad.pyx
    Compiling av\sidedata\motionvectors.pyx because it changed.
    [1/1] Cythonizing av\sidedata\motionvectors.pyx
    Compiling av\sidedata\sidedata.pyx because it changed.
    [1/1] Cythonizing av\sidedata\sidedata.pyx
    Compiling av\subtitles\codeccontext.pyx because it changed.
    [1/1] Cythonizing av\subtitles\codeccontext.pyx
    Compiling av\subtitles\stream.pyx because it changed.
    [1/1] Cythonizing av\subtitles\stream.pyx
    Compiling av\subtitles\subtitle.pyx because it changed.
    [1/1] Cythonizing av\subtitles\subtitle.pyx
    Compiling av\video\codeccontext.pyx because it changed.
    [1/1] Cythonizing av\video\codeccontext.pyx
    Compiling av\video\format.pyx because it changed.
    [1/1] Cythonizing av\video\format.pyx
    Compiling av\video\frame.pyx because it changed.
    [1/1] Cythonizing av\video\frame.pyx
    Compiling av\video\plane.pyx because it changed.
    [1/1] Cythonizing av\video\plane.pyx
    Compiling av\video\reformatter.pyx because it changed.
    [1/1] Cythonizing av\video\reformatter.pyx
    Compiling av\video\stream.pyx because it changed.
    [1/1] Cythonizing av\video\stream.pyx
    building 'av.buffer' extension
    creating build\temp.win-amd64-3.8\Release\src
    creating build\temp.win-amd64-3.8\Release\src\av
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Dinline=__inline -Ibuild\temp.win-amd64-3.8\Release\include -Id:\desktop\新建文件夹\venv\include -ID:\Program\Files\Python\3.8.1\include -Iinclude -Id:\desktop\新建文件夹\venv\include -ID:\Program\Files\Python\3.8.1\include -ID:\Program\Files\Python\3.8.1\include -Ibuild\temp.win-amd64-3.8\Release\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcsrc\av\buffer.c /Fobuild\temp.win-amd64-3.8\Release\src\av\buffer.obj
    buffer.c
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:D:\Program\Files\Python\3.8.1\libs /LIBPATH:D:\Program\Files\Python\3.8.1 /LIBPATH:d:\desktop\新建文件夹\venv\libs /LIBPATH:d:\desktop\新建文件夹\venv\PCbuild\amd64 /LIBPATH:d:\desktop\新建文件夹\venv\libs /LIBPATH:D:\Program\Files\Python\3.8.1\libs /LIBPATH:D:\Program\Files\Python\3.8.1 /LIBPATH:d:\desktop\新建文件夹\venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" swresample.lib avcodec.lib avformat.lib avfilter.lib avdevice.lib avutil.lib swscale.lib /EXPORT:PyInit_buffer build\temp.win-amd64-3.8\Release\src\av\buffer.obj /OUT:build\lib.win-amd64-3.8\av\buffer.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\src\av\buffer.cp38-win_amd64.lib /OPT:NOREF
    LINK : fatal error LNK1181: 无法打开输入文件“swresample.lib”
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.24.28314\\bin\\HostX86\\x64\\link.exe' failed with exit status 1181
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'd:\desktop\新建文件夹\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\cheng\\AppData\\Local\\Temp\\pip-req-build-ah33rmuy\\setup.py'"'"'; __file__='"'"'C:\\Users\\cheng\\AppData\\Local\\Temp\\pip-req-build-ah33rmuy\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\cheng\AppData\Local\Temp\pip-record-w2ujctdp\install-record.txt' --single-version-externally-managed --compile --install-headers 'd:\desktop\新建文件夹\venv\include\site\python3.8\av' Check the logs for full command output.

@jlaine
Copy link
Collaborator

jlaine commented Apr 29, 2020

That error looks unrelated to my change, I don't understand the error message but it refers to swresample.lib so I suggest you check the FFmpeg libraries are somewhere where the compiler can find them:

CL="/IC:\path\to\ffmpeg\include" LINK="/LIBPATH:C:\path\to\ffmpeg\lib"

@necan
Copy link

necan commented Apr 29, 2020

@jlaine Ok, it works now.

@jlaine jlaine closed this as completed in c4b8237 Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mikeboers @jlaine @woshimanong1990 @necan