-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Calling gRPC Client VIs from TestStand hang at "Wait for Occurrence" while using LabVIEW RTE #330
Comments
@kt-jplotzke - You mentioned in the other thread that you were using local loopback. Are you running local loopback in this case as well, or a device across the network? If it's loopback, does the problem replicate when the client is run from a different device? |
@AndrewHeim - Yes, I initially tested just running the server on the same system as the TestStand client using the local loopback connection. I just tested running the gRPC server on a different host and I get the same results, regardless of using localhost or a separate machine. |
I was able to determine the root cause of this. The issue is is that TestStand can launch multiple versions of the LV RTE in the background -- For example, even though I'm using LV 2021 SP1 and TS 2021, TestStand uses both the LV2021 and LV2023 RTEs in the background - I can see this in a process explorer: However, when a grpc client or server is created in the grpc-labview DLL, the DLL dynamically gets a handle to the
The issue is that the To resolve this, I built a new DLL which provides an exported function to define a path to a specific LVRT module. I call this function in my code prior to calling any other gRPC functions. To this function, I pass in the path of the LV2021 RTE: Using this method, my issue is resolved and the occurrence fires every time. While maybe not the most elegant solution, I am submitting a push request of these change. In my mind, the 'Set LVRT Module Path' function is not used by the generated code at all. However, this function can be available to a user if they run into the same situation I am. |
Good work! So as-is, is this library not thread-safe? (Thinking out loud as I work through this...) Well, sort of. If I had two clients on different ports... as long as I had one instance of LabVIEW, it would be ok as they would reference the same DLL. If I had two separate exes, each would have its own execution environment and each would reference the DLL, and it would be ok. At least as long as they're pointed at the same version, that is. I can't say beyond that. Interesting that to me this seems to be mostly specific to running multiple runtime engines (that will open different copies of the DLL) in the same application. Which we are unlikely to see outside of TestStand. Or... would they collide if they were in separate applications? I would need to do more homework to say for sure. |
I don't see them colliding if they are in different applications. Each copy of the gRPC DLL would have its own memory space. The call to In my opinion, this edge case would only manifest itself in TestStand, where a single process can call multiple RTEs at once. |
This would likely be resolved by setting the "Version Independence" flag in TestStand for the LabVIEW RTE Adapter. Then TestStand will only use one LabVIEW Version for RTE (the latest you have installed), instead of multiple. |
@ShockHouse5 Potentially if only running from the TestStand development environment, but potentially not if running from a custom LabVIEW operator interface. In my case, my custom OI is built with LabVIEW 2021, and thus uses that RTE. When running from an OI, the LabVIEW RTEs that are called by TestStand run under the OI's process. So, setting the "Version Independence" flag in my setup caused both the LV2021 RTE (for the OI) and the LV2023 RTE (for the version independent teststand calls) under a single OI process, which still caused the DLL callback issues. |
@kt-jplotzke Yeah correct. Unless your EXE is also built with "Allow future versions to run this". If that is checked in your exe, and "Version Independence" in TestStand, they will always match because they will both use the latest version. |
Released as part of https://github.com/ni/grpc-labview/releases/tag/v1.2.1.1 |
I've found a condition where calling gRPC client VIs from TestStand, while using the LabVIEW Run Time Engine adapter, causes the "Client Unary Call" VI to hang at "Wait on Occurrence". I can confirm that the gRPC request does get sent from the client prior to hanging and that the server does respond to the request. This issue does not occur when running using the LabVIEW Development Environment adapter.
Steps that I've used to create a reproducible example are below. I've also uploaded all of this code to https://github.com/kt-jplotzke/grpc-labview-teststand-hang.
I've created a small gRPC service with a single function, "Wait". This function simply wraps the LabVIEW Wait (ms) function. I then used grpc-labview 1.0.1.1 to generate the server and client code for this service.
For the server, I simply wrapped the Wait (ms) function in the "Start Sync" function. This is the only gRPC method. I then built
![image](https://private-user-images.githubusercontent.com/88543727/288796219-d818f840-1cdc-46b8-92a5-f0f368175a9d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MzgzMDgsIm5iZiI6MTcyNDUzODAwOCwicGF0aCI6Ii84ODU0MzcyNy8yODg3OTYyMTktZDgxOGY4NDAtMWNkYy00NmI4LTkyYTUtZjBmMzY4MTc1YTlkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDIyMjAwOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU1YTc3MzlmMzZiMTZmMzJjOGNjNjQxN2VkNzA2ZTI0ZTNlMDM1MzQwMDM5Yzc3Y2E5OTBjM2RmZWRiMWMyZmMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.4eaZ7OLNOSd7HfDETaJtiatKymPj8-tEs_UbJMYl0Ss)
Run Service.vi
as an EXE to allow this to run in the background.For the client, I simply wrapped my gRPC client function into a VI which calls
![image](https://private-user-images.githubusercontent.com/88543727/288796761-9cb47b73-ced7-44fe-8a49-ff7dce1f0111.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MzgzMDgsIm5iZiI6MTcyNDUzODAwOCwicGF0aCI6Ii84ODU0MzcyNy8yODg3OTY3NjEtOWNiNDdiNzMtY2VkNy00NGZlLThhNDktZmY3ZGNlMWYwMTExLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDIyMjAwOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNiMjU2NjBjMzJhY2I2MTI2ZjViMDZkMzdiMDhiOThiNjBkNTE1YmViOGJhODEzNzY0YjJlOGY2Njk0YWZmOWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.cUeCDkC6VMPHDmmk-AOmkWZW-RZDsReFRCQw5NLxsSo)
Client Create
, the gRPC client unary function, and thenClient Destroy
. I build this function into a Packed Project Library.I can confirm that both the native wrapper VI and the build PPL wrapper VI function as expected when executed from the LabVIEW Dev Environment. Both send a request and receive a response from the built gRPC server.
Then, I created a TestStand sequence with a single step -- to call this gRPC client wrapper VI.
![image](https://private-user-images.githubusercontent.com/88543727/288797972-14ff4335-962d-4458-a316-c4877b59e097.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MzgzMDgsIm5iZiI6MTcyNDUzODAwOCwicGF0aCI6Ii84ODU0MzcyNy8yODg3OTc5NzItMTRmZjQzMzUtOTYyZC00NDU4LWEzMTYtYzQ4NzdiNTllMDk3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDIyMjAwOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWIxYmZmMTU4Y2RlNTRjN2Y0ZmJmMDZkNDc1MThjYzU3Y2Q0YjQwNmJmMGM1YWY0MGI0NTVmMzliNTEwNWZiZTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.THdZZGebda2Kz6pEaJWwE_ukU6ZpgB3ut9rEjDLYfyw)
When this sequence is executed using the LabVIEW Development Environment as the LabVIEW adapter, the step executes successfully and returns the response. However, when this sequence is executed using the LabVIEW Run Time Engine, the gRPC step hangs indefinitely. Adding debug code into the
Client Unary Call.vim
VI, I have been able to prove that the hang is in theWait On Occurence.vi
. When I implement the patch mentioned in #193 (to connect the "timeout (ms)" control to the "Wait on Occurence" VI), the call no longer hangs in TestStand, but an error (-1004) is always generated and no data is returned.I have also proven that the gRPC request is sent out by the client and that a response is sent back from the server. It appears that either the client is not appropriately triggering the occurrence or receiving the response successfully. However, given that this call is 100% successful when running using the LabVIEW development environment adapter, I feel like it is more likely the former than the latter.
I also believe that this is not related to the specific issue described in #193 as that issue refers to a potential race condition when responses are received quickly. In my case, I am passing in a 1000 msec delay into my gRPC function to purposely slow down the response.
This is currently a blocking issue for me. Please let me know any other information you need to help reproduce / help diagnose. Thank you!
AB#2598217
The text was updated successfully, but these errors were encountered: