Windows support

[you-win]

Hi! I saw your project from a This week in Godot post. I'm working on a project that relies on face tracking data, and was planning on adding mediapipe support.

What are the difficulties in adding Windows support? I can take a look at tackling that.

[j20001970]

Hi, thanks for showing interest for the project. I think there are two difficulties in adding support on Windows:

1. Using Bazel on Windows, you can check mediapipe docs on how to install and use it. Note that although there are third_party/godot_windows.BUILD in GDMP, I never checked if it ever work, since my primary OS is not Windows and currently our primary platform target is Android (with Linux for desktop testing).

2. MediaPipe GPU support on Windows is lacking, there are attempt on using ANGLE to add GPU support on Windows (see this issue on mediapipe repo), but not sure if it can work. Or, if running on CPU is acceptable, we could add the CPU code path and make sure the current GDNative code can run on Windows.

[you-win]

1. It looks like the third_party directory is omitted from this repository (GDMP) and I don't see the referenced third_party/godot_windows.BUILD in the main mediapipe repository. Is that something that you can share?

2. Running from CPU is probably good enough for my use case but I can see if I can apply the ANGLE patches and build on Windows for better performance. The upside to applying the ANGLE patches is that the current gdmp.cc implementation probably won't need to be modified too much.

[j20001970]

1. It looks like the `third_party` directory is omitted from this repository (GDMP) and I don't see the referenced `third_party/godot_windows.BUILD` in the main mediapipe repository. Is that something that you can share?

Ah sorry forgot to mention that third_party/godot_windows.BUILD is located in the mediapipe directory after applied mediapipe_setup.diff as stated in README

2. Running from CPU is probably good enough for my use case but I can see if I can apply the ANGLE patches and build on Windows for better performance. The upside to applying the ANGLE patches is that the current gdmp.cc implementation probably won't need to be modified too much.

It would be great if we have GPU support on every supported platform, I'm looking forward to your progress on bringing GPU support on Windows.

[you-win]

That makes sense, I'll apply the patch and take a look.

Regarding Windows GPU support, the originally author of the Windows ANGLE patches has an unmaintained repo with a few additional changes. I've forked it and will take a look at updating it for mediapipe master.

[j20001970]

Hi @you-win , do you have a moment to try out if current code can work on Windows since 5242a1d? I have updated the build script and documentation that it should build successfully without GPU support, but haven't tried if the library actually worked.

[you-win]

@j20001970 I have successfully compiled GDMP using commit fbf3158

Build env specs:

• OS: Windows 10
• Compiler: Clang + LLVM(?)
  • I think it was using Clang because of this output cl : Command line warning D9025 : overriding '/w' with '/W3'
• Git Bash via MSYS2
• Calculator: //mediapipe/graphs/pose_tracking:pose_tracking_cpu_deps

Note: I had to do some tweaking since I already have OpenCV 4 installed and didn't want to break my current environment, but that just involved changing a few versions and changing the path to OpenCV in the WORKSPACE file.

EDIT: I'm dumb. I modified the wrong calculators.bzl file, will report back

Currently, I am working off of your example but am running into this error

This was for holistic tracking, ignore this for now since I also modified the wrong calculators.bzl file

[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:335] Error parsing text-format mediapipe.CalculatorGraphConfig: 29:66: Could not find type "type.googleapis.com/mediapipe.FlowLimiterCalculatorOptions" stored in google.protobuf.Any.
ERROR: Condition '!parse_graph_config' is true.
   at: initialize (mediapipe/GDMP/framework/graph.cc:52)

When running the pose tracking graph from a pbtext file. Project files mimic the demo project.

ERROR: Condition '!add_packet.ok()' is true.
   at: add_packet (mediapipe/GDMP/framework/graph.cc:151)
ERROR: INVALID_ARGUMENT: Graph has errors:
Calculator::Open() for node "poselandmarkcpu__poselandmarkbyroicpu__poselandmarkmodelloader__LocalFileContentsCalculator" failed: Failed to get resource contents: 7
Calculator::Open() for node "poselandmarkcpu__posedetectioncpu__inferencecalculator__poselandmarkcpu__posedetectioncpu__InferenceCalculator" failed: Failed to get resource contents: 7
   at: add_packet (mediapipe/GDMP/framework/graph.cc:151)

I am looking into this. If that fails, I guess I'll try recompiling using MSVC + Powershell as recommended on the official Bazel docs

[you-win]

Update! Moving the graphs/modules from res://assets/mediapipe -> res://mediapipe allows me to get output.

I didn't see any references to hardcoded paths, so I guess this requirement is on the mediapipe-side?

After running for about 1 second, the application stops processing data and starts spamming these errors:

... position data ...
x:0.571171, y:3.021481, z:-0.055348
x:0.381952, y:3.003469, z:-0.402263
ERROR: Condition '!add_packet.ok()' is true.
   at: add_packet (mediapipe/GDMP/framework/graph.cc:151)
ERROR: INTERNAL: Graph has errors:
Calculator::Process() for node "poserenderercpu__RecolorCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/image/recolor_calculator.cc:241) input_mat.channels() == 3
   at: add_packet (mediapipe/GDMP/framework/graph.cc:151)
... repeated, no more position data ...

Seems like the error might be coming from camera_helper_cv, investigating now

[you-win]

After referencing recolor_calculator.cc, I found that cpu tracking requires RGB.

RET_CHECK(input_mat.channels() == 3);  // RGB only.

I modified the start method in camera_helper_cv.cc to convert the video frames to RGB instead of RGBA and things seem to work!

cv::cvtColor(video_frame, video_frame, cv::COLOR_BGR2RGB);
auto input_frame = std::make_unique<mediapipe::ImageFrame>(
	mediapipe::ImageFormat::SRGB, video_frame.cols, video_frame.rows,
	mediapipe::ImageFrame::kGlDefaultAlignmentBoundary);

Seems like making this a configurable flag might make sense. I'll start putting something together

[j20001970]

I didn't see any references to hardcoded paths, so I guess this requirement is on the mediapipe-side?

Yes, the hardcoded path is in the calculator configs. For example when loading TFLite model for CPU pose tracking, they use a calculator called PoseLandmarkModelLoader(in mediapipe/mediapipe/modules/pose_landmark/pose_landmark_model_loader.pbtxt) to select model path. When the path string reach LocalFileContentsCalculator, it will use GetResourceContents (which calls DefaultGetResourceContents, which is implemented by resource_util_godot.cc) to load the file.

[j20001970]

After referencing recolor_calculator.cc, I found that cpu tracking requires RGB.

RET_CHECK(input_mat.channels() == 3);  // RGB only.

I modified the start method in camera_helper_cv.cc to convert the video frames to RGB instead of RGBA and things seem to work!

cv::cvtColor(video_frame, video_frame, cv::COLOR_BGR2RGB);
auto input_frame = std::make_unique<mediapipe::ImageFrame>(
	mediapipe::ImageFormat::SRGB, video_frame.cols, video_frame.rows,
	mediapipe::ImageFrame::kGlDefaultAlignmentBoundary);

Seems like making this a configurable flag might make sense. I'll start putting something together

I think converting video frames depending on MEDIAPIPE_DISABLE_GPU would work.

[j20001970]

@j20001970

Compiling for Window is still a major problem

It would be more helpful if you could provide the information about the error you encounter while building for Windows, like what does the output show when Bazel failing to build. It might be some setup for Windows is missing, or the code syntax somewhere works on other compilers but MSVC doesn't like it (yeah that happened once before) or something else.

[j20001970]

Information discussed here is confusing.

* is it necessary to compile with  `Clang + LLVM`

If there is no Official Release for Windows and There is no GDMP Demo for Windows,

perhaps it is NOT YET possible NOW with Windows

GDMP library on Windows is actually compiled with MSVC. Bazel will need to be configured to locate the toolchain with BAZEL_VC environment variable if needed.

[you-win]

@GeorgeS2019 The error is saying that your PYTHON_BIN_PATH is invalid.

This is the build command I used before. Note: this is just an example and will need to be adjusted for your system. I ran this command via git bash. I have not built this project recently, so something might have changed.

PYTHON_BIN_PATH=/c/Users/theaz/scoop/apps/python/current/python BAZEL_LLVM=/c/Users/theaz/scoop/apps/llvm/current/bin python build.py desktop

[j20001970]

@GeorgeS2019

Still, most like could be the way I have installed MSYS, still having problems with the Windows build

What @you-win said is true, the path to python executable is not valid to Bazel. Can you check that there actually is python.exe on your setup's C:\SomeDirectory\envs\SomeEnvironment\?

If you are building the library since tag v0.3, the build script build.py should provide a valid python path to Bazel, so you don't have to set environment variable PYTHON_BIN_PATH manually. If you still have python-related problem building GDMP after v0.3, please provide the error log relevant to PYTHON_BIN_PATH, maybe I've missed some corner cases regarding path convention.

As for precompiled library, I plan to provide Windows version on GDMP Demo after migrating several examples to newer MediaPipe Task API.

[j20001970]

Cannot open include file: 'windows.h': No such file or directory

Looks like Windows SDK is not installed on your setup, please read the updated README to make sure you've install the required stuffs.

If you don't want to install the whole Visual Studio, you can use Visual Studio Build Tools to install cli-only packages.
Then you can install Windows SDK under SDK category in individual components.

E.g.

[j20001970]

Git bash fails to recognize winsdk

I tried to reproduce your situation in a Windows 10 VM but with no success. Is your Windows SDK installed on C:\Program Files (x86)\Windows Kits? In your case of windows.h is missing, it should be on somewhere like C:\Program Files (x86)\Windows Kits\10\Include\10.0.20348.0\um\Windows.h if you install on default location, can you check that it actually exist?

[you-win]

Windows support has been present for a while, I think this issue can be closed.