Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vcpkg freeze after sucessfully building ffmpeg #5476

Closed
shurunxuan opened this issue Feb 27, 2019 · 9 comments
Closed

vcpkg freeze after sucessfully building ffmpeg #5476

shurunxuan opened this issue Feb 27, 2019 · 9 comments
Labels
requires:repro The issue is not currently repro-able

Comments

@shurunxuan
Copy link

Platform: Windows
Package: ffmpeg:x64-windows & ffmpeg:x86-windows
Vcpkg version: 2018.11.23-nohash
Description:

I was trying to install ffmpeg (4.1.0) with vcpkg on Windows 10. The build seems to succeed, but after that vcpkg freezes and won't quit. I can still use ffmpeg after I force it to quit. While this is not quite a problem on my local machine, the CI builds will be blocked by this and won't continue.

Command Line:
vcpkg install ffmpeg:x86-windows

Command Line Output:

  The following packages will be built and installed:
      ffmpeg[core]:x86-windows
  Starting package 1/1: ffmpeg:x86-windows
  Building package ffmpeg[core]:x86-windows... 
  A suitable version of git was not found (required v2.20.0). Downloading portable git v2.20.0...
  Downloading git...
    https://github.com/git-for-windows/git/releases/download/v2.20.0.windows.1/PortableGit-2.20.0-32-bit.7z.exe -> C:\Tools\vcpkg\downloads\PortableGit-2.20.0-32-bit.7z.exe
  Extracting git...
  A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
  Downloading 7zip...
    https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Tools\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
  Extracting 7zip...
  -- Downloading http://ffmpeg.org/releases/ffmpeg-4.1.tar.bz2...
  -- Extracting source C:/Tools/vcpkg/downloads/ffmpeg-4.1.tar.bz2
  -- Applying patch C:/Tools/vcpkg/ports/ffmpeg/create-lib-libraries.patch
  -- Applying patch C:/Tools/vcpkg/ports/ffmpeg/detect-openssl.patch
  -- Applying patch failed. This is expected if this patch was previously applied.
  -- Applying patch C:/Tools/vcpkg/ports/ffmpeg/configure_opencv.patch
  -- Applying patch failed. This is expected if this patch was previously applied.
  -- Downloading https://www.tortall.net/projects/yasm/snapshots/v1.3.0.6.g1962/yasm-1.3.0.6.g1962.exe...
  -- Installing: C:/Tools/vcpkg/downloads/tools/yasm/1.3.0.6/yasm.exe
  -- Acquiring MSYS2...
  -- Downloading http://repo.msys2.org/distrib/x86_64/msys2-base-x86_64-20181211.tar.xz...
  gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
  gpg: no ultimately trusted keys found
  gpg: starting migration from earlier GnuPG versions
  gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
  gpg: migration succeeded
  gpg: Generating pacman keyring master key...
  gpg: key A39EF6BBA7C712F1 marked as ultimately trusted
  gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
  gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/612A00C94159FBB38363F9E4A39EF6BBA7C712F1.rev'
  gpg: Done
  ==> Updating trust database...
  gpg: marginals needed: 3  completes needed: 1  trust model: pgp
  gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
  ==> Appending keys from msys2.gpg...
  ==> Locally signing trusted keys in keyring...
    -> Locally signing key D55E7A6D7CE9BA1587C0ACACF40D263ECA25678A...
    -> Locally signing key 123D4D51A1793859C2BE916BBBE514E53E0D0813...
    -> Locally signing key B91BCF3303284BF90CC043CA9F418C233E652008...
    -> Locally signing key 9DD0D4217D75A33B896159E6DA7EF2ABAEEA755C...
  ==> Importing owner trust values...
  gpg: setting ownertrust to 4
  gpg: setting ownertrust to 4
  gpg: setting ownertrust to 4
  gpg: inserting ownertrust of 4
  ==> Updating trust database...
  gpg: marginals needed: 3  completes needed: 1  trust model: pgp
  gpg: depth: 0  valid:   1  signed:   4  trust: 0-, 0q, 0n, 0m, 0f, 1u
  gpg: depth: 1  valid:   4  signed:   3  trust: 0-, 0q, 0n, 4m, 0f, 0u
  gpg: depth: 2  valid:   3  signed:   0  trust: 3-, 0q, 0n, 0m, 0f, 0u
  :: Synchronizing package databases...
  downloading mingw32.db...
  downloading mingw32.db.sig...
  downloading mingw64.db...
  downloading mingw64.db.sig...
  downloading msys.db...
  downloading msys.db.sig...
  :: Starting core system upgrade...
  resolving dependencies...
EXEC : warning : terminate other MSYS2 programs before proceeding [C:\projects\igme750-dsengine\src\DSEngine\DSEngineFramework\DSEngineFramework.vcxproj]
  looking for conflicting packages...
  
  Packages (2) filesystem-2018.12-1  pacman-5.1.2-2
  
  Total Download Size:   14.34 MiB
  Total Installed Size:  46.14 MiB
  Net Upgrade Size:       1.13 MiB
  
  :: Proceed with installation? [Y/n] 
  :: Retrieving packages...
  downloading filesystem-2018.12-1-x86_64.pkg.tar.xz...
  downloading pacman-5.1.2-2-x86_64.pkg.tar.xz...
  checking keyring...
  checking package integrity...
  loading package files...
  checking for file conflicts...
  checking available disk space...
  :: Processing package changes...
  upgrading filesystem...
  upgrading pacman...
EXEC : warning : terminate MSYS2 without returning to shell and check for updates again [C:\projects\igme750-dsengine\src\DSEngine\DSEngineFramework\DSEngineFramework.vcxproj]
  
  -- Acquiring MSYS2... OK
  -- Acquiring MSYS Packages...
  -- Acquiring MSYS Packages... OK
  -- Building Options: --enable-asm --enable-yasm --disable-doc --enable-debug --disable-ffmpeg --enable-runtime-cpudetect --disable-openssl --disable-ffplay --disable-ffprobe --disable-libx264 --disable-opencl --disable-lzma --extra-cflags=-DHAVE_UNISTD_H=0 --disable-static --enable-shared
  -- Building ffmpeg for Release
  -- Building ffmpeg for Debug
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\debug\lib\avcodec.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\debug\lib\avdevice.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\debug\lib\avfilter.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\debug\lib\avformat.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\debug\lib\avutil.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\debug\lib\swresample.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\debug\lib\swscale.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\lib\avcodec.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\lib\avdevice.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\lib\avfilter.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\lib\avformat.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\lib\avutil.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\lib\swresample.lib
  -- Generating C:\Tools\vcpkg\packages\ffmpeg_x86-windows\lib\swscale.lib
  -- Performing post-build validation
  -- Performing post-build validation done
  Building package ffmpeg[core]:x86-windows... done
  Installing package ffmpeg[core]:x86-windows... 
  Installing package ffmpeg[core]:x86-windows... done
  Elapsed time for package ffmpeg:x86-windows: 32.08 min
  
  Total elapsed time: 32.08 min

After that vcpkg freezes. The build was running for two hours when I copied the log.

@LilyWangL LilyWangL added the requires:repro The issue is not currently repro-able label Mar 5, 2019
@TheCycoONE
Copy link
Contributor

I'm also hitting this. I've been trying to track it down for a couple days now. It happens with both upstream, and my own trimmed down ffmpeg port. Once I've isolated the change I'll report more. You can sorta follow my (very messy) diagnosis so far at: CorsixTH/CorsixTH#1515

@TheCycoONE
Copy link
Contributor

Weirdly, even though the problem only happens when building ffmpeg, it still happens when reverting the ffmpeg port to a known good version (while keeping the rest of vcpkg on the latest version). This (combined with the behaviour of just freezing) makes the issue rather hard to diagnose.

@shurunxuan
Copy link
Author

Weirdly, even though the problem only happens when building ffmpeg, it still happens when reverting the ffmpeg port to a known good version (while keeping the rest of vcpkg on the latest version). This (combined with the behaviour of just freezing) makes the issue rather hard to diagnose.

Thank you for this! I already gave up since this is beyond my knowledge.

Also, for your information, if this is helpful, my Appveyor CI build will always freeze, but not my Azure Pipelines, so I stopped using Appveyor temporarily. You can check my Azure Pipelines log here.

@ghost
Copy link

ghost commented Apr 10, 2019

Some ideas - let me know and I'll delete this comment if I'm way off the mark.

  • Try the build again on AppVeyor and log into the build box to debug via https://www.appveyor.com/docs/how-to/rdp-to-build-worker/

  • Change the build config temporarily to make it build the bare minimum ffmpeg to see if the freeze is resolved, or at least make the build much faster to debug

@shurunxuan are you able to share your appveyor.yml or any scripts you're using? I'd like to replicate the build and see if it freezes for me as well.

@shurunxuan
Copy link
Author

@voltagex Sure.

version: 1.0.{build}
image: Visual Studio 2017
configuration: 
  - Debug
  - Release
platform: 
  - x86
  - x64
before_build:
  - nuget restore src/DSEngine/DSEngine.sln
install:
  - cd C:\Tools\vcpkg
  - git pull
  - .\bootstrap-vcpkg.bat
  - cd %APPVEYOR_BUILD_FOLDER%
  - vcpkg integrate install
  - vcpkg update
  - vcpkg upgrade --no-dry-run
  - cmd: if "%platform%"=="Win32" set VCPKG_ARCH=x86-windows
  - cmd: if "%platform%"=="x86"   set VCPKG_ARCH=x86-windows
  - cmd: if "%platform%"=="x64"   set VCPKG_ARCH=x64-windows
  - vcpkg install boost:%VCPKG_ARCH%
  - vcpkg install ffmpeg:%VCPKG_ARCH%
build:
  project: src/DSEngine/DSEngine.sln
  verbosity: minimal
cache: c:\tools\vcpkg\installed\

@TheCycoONE
Copy link
Contributor

I've switched my appveyor to just cloning vcpkg, installing, and building ffmpeg in a powershell script. With my own minimal ffmpeg that means I can reproduce in ~7min, but for bisecting I'd rather just use the the upstream port.

As far as RDPing into Appveyor, I tried that but I didn't see anything interesting. I saved a dump from process explorer while it was stuck. Nothing stands out to me, but maybe someone here sees something? If there's something else in particular I should look at I'll run RDP again; otherwise I'll continue with my bisect.

Process	CPU	Private Bytes	Working Set	PID	Description	Company Name
System Idle Process	96.97	0 K	4 K	0		
System	0.03	128 K	116 K	4		
 Interrupts	0.36	0 K	0 K	n/a	Hardware Interrupts and DPCs	
 Secure System	Suspended	0 K	2,572 K	268		
 smss.exe		408 K	1,196 K	272	Windows Session Manager	Microsoft Corporation
csrss.exe		1,536 K	4,136 K	376	Client Server Runtime Process	Microsoft Corporation
csrss.exe	0.14	2,660 K	9,128 K	444	Client Server Runtime Process	Microsoft Corporation
wininit.exe		992 K	5,208 K	464	Windows Start-Up Application	Microsoft Corporation
 services.exe		3,788 K	7,524 K	572	Services and Controller app	Microsoft Corporation
  svchost.exe		5,988 K	19,676 K	664	Host Process for Windows Services	Microsoft Corporation
   RuntimeBroker.exe		7,572 K	22,376 K	2860	Runtime Broker	Microsoft Corporation
   ShellExperienceHost.exe	Suspended	33,712 K	76,900 K	3148	Windows Shell Experience Host	Microsoft Corporation
   SearchUI.exe	Suspended	11,992 K	45,380 K	3224	Search and Cortana application	Microsoft Corporation
   WmiPrvSE.exe	< 0.01	2,056 K	8,200 K	3648	WMI Provider Host	Microsoft Corporation
  svchost.exe	0.01	4,388 K	9,572 K	720	Host Process for Windows Services	Microsoft Corporation
  svchost.exe	0.65	76,120 K	94,816 K	856	Host Process for Windows Services	Microsoft Corporation
   rdpclip.exe	0.04	2,224 K	10,028 K	2704	RDP Clipboard Monitor	Microsoft Corporation
  svchost.exe		24,648 K	56,904 K	864	Host Process for Windows Services	Microsoft Corporation
   sihost.exe		4,028 K	18,776 K	2832	Shell Infrastructure Host	Microsoft Corporation
   taskhostw.exe		10,292 K	18,832 K	2932	Host Process for Windows Tasks	Microsoft Corporation
  svchost.exe	< 0.01	16,048 K	24,872 K	936	Host Process for Windows Services	Microsoft Corporation
  svchost.exe		14,884 K	23,552 K	968	Host Process for Windows Services	Microsoft Corporation
  svchost.exe		10,180 K	18,136 K	304	Host Process for Windows Services	Microsoft Corporation
  svchost.exe	0.02	3,044 K	9,528 K	340	Host Process for Windows Services	Microsoft Corporation
  svchost.exe	0.01	8,024 K	20,868 K	368	Host Process for Windows Services	Microsoft Corporation
  svchost.exe		1,876 K	7,288 K	1168	Host Process for Windows Services	Microsoft Corporation
  svchost.exe		10,740 K	22,776 K	1180	Host Process for Windows Services	Microsoft Corporation
  VSSVC.exe		1,548 K	7,680 K	1204	Microsoft® Volume Shadow Copy Service	Microsoft Corporation
  svchost.exe		3,680 K	10,356 K	1752	Host Process for Windows Services	Microsoft Corporation
  com.docker.service		54,908 K	54,764 K	1760	Docker.Service	Docker Inc.
   com.docker.proxy.exe		8,784 K	17,792 K	3744		
    conhost.exe		4,920 K	9,364 K	3720	Console Window Host	Microsoft Corporation
   Docker.Watchguard.exe		368 K	2,216 K	1984		
    conhost.exe		1,180 K	5,260 K	2628	Console Window Host	Microsoft Corporation
   com.docker.localhost-forwarder.exe		7,716 K	9,180 K	3956		
    conhost.exe		1,192 K	5,412 K	3876	Console Window Host	Microsoft Corporation
  svchost.exe		7,664 K	21,784 K	1788	Host Process for Windows Services	Microsoft Corporation
  svchost.exe		6,556 K	17,100 K	1824	Host Process for Windows Services	Microsoft Corporation
  svchost.exe		2,092 K	8,096 K	1908	Host Process for Windows Services	Microsoft Corporation
  vmms.exe		8,600 K	21,396 K	1296	Virtual Machine Management Service	Microsoft Corporation
  svchost.exe		1,760 K	6,996 K	2316	Host Process for Windows Services	Microsoft Corporation
  vmcompute.exe	< 0.01	1,704 K	7,640 K	2588	Hyper-V Host Compute Service	Microsoft Corporation
  svchost.exe		4,224 K	19,508 K	2812	Host Process for Windows Services	Microsoft Corporation
  dockerd.exe	0.01	19,424 K	30,988 K	3820		
  svchost.exe		1,488 K	6,256 K	4316	Host Process for Windows Services	Microsoft Corporation
  msdtc.exe		2,744 K	9,684 K	3356	Microsoft Distributed Transaction Coordinator Service	Microsoft Corporation
 lsass.exe	< 0.01	6,368 K	15,912 K	588	Local Security Authority Process	Microsoft Corporation
winlogon.exe		2,140 K	10,772 K	504	Windows Logon Application	Microsoft Corporation
 dwm.exe	0.54	30,776 K	78,872 K	832	Desktop Window Manager	Microsoft Corporation
 fontdrvhost.exe		776 K	3,080 K	4824	Usermode Font Driver Host	Microsoft Corporation
explorer.exe	0.04	28,084 K	91,052 K	2448	Windows Explorer	Microsoft Corporation
 Docker for Windows.exe		59,800 K	72,696 K	4032	Docker for Windows	Docker Inc.
 procexp64.exe	1.16	54,904 K	88,916 K	3144	Sysinternals Process Explorer	Sysinternals - www.sysinternals.com
Appveyor.BuildAgent.Interactive.exe	0.03	111,396 K	136,356 K	4084	AppVeyor Build Agent Interactive	Appveyor Systems Inc.
 conhost.exe		5,332 K	11,216 K	3728	Console Window Host	Microsoft Corporation
csrss.exe		1,312 K	4,032 K	4736	Client Server Runtime Process	Microsoft Corporation
winlogon.exe		1,836 K	6,824 K	4784	Windows Logon Application	Microsoft Corporation
 LogonUI.exe		9,184 K	42,080 K	5108	Windows Logon User Interface Host	Microsoft Corporation
 dwm.exe		13,200 K	29,664 K	5084	Desktop Window Manager	Microsoft Corporation
conhost.exe		5,172 K	10,652 K	4696	Console Window Host	Microsoft Corporation
gpg-agent.exe		5,352 K	5,848 K	4952		

@TheCycoONE
Copy link
Contributor

The commit that triggered the issue is 0d1229d by @cenit
Which would appear to punt the problem down to msys2.

@TheCycoONE
Copy link
Contributor

@wangli28 there's enough information now to remove the needs repo label no?

@mugmuggy
Copy link

As the TheCycoOne points out msys2 change, appears to introduce gpg 2.1 which launches gpg-agent daemon process. The appveyor builds appear to be lingering awaiting that process to die. Our script for CorsixTH, I altered with a Stop-Process -Force -Name "gpg-agent" for a quick test just before it exits and the process continued as expected.

Though running locally I didn't see the same as @shurunxuan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
requires:repro The issue is not currently repro-able
Projects
None yet
Development

No branches or pull requests

4 participants