The client doesn't return correct value when used with unleash-edge

[JonasBak]

Describe the bug
We have a toggle in three environments (dev, test, prod). The toggle is turned on (100%) in dev, and off (0%) in test and prod.

We use a token with access to the dev environment. When we configure the client to use unleash directly it works as expected and the toggle evaluates as true, but if we change the url to our unleash-edge instance, it evaluates as false.

I have recreated it with the "Synchronous startup" example. I thinks this is an issue with the dotnet client, as when i test the golang package with the same token and url using the SimpleUsage example here, it works as expected.

I have also tried calling /api/client/features on the unleash-edge instance directly with the same token, and it returns the flag with "enabled": true.

To Reproduce
Steps to reproduce the behavior:

1. Create a toggle in three environments
2. Enable the toggle in one environment
3. Create a token for that environment
4. Deploy an unleash-edge instance
5. Configure the client to use the token and unleash-edge, check if the feature is enabled using unleash.IsEnabled
6. Change the url to use unleash without unleash-edge and try again

Expected behavior
I expect the client to return the same value as configured upstream and as returned by other clients.

Screenshots

Additional context
This is using version 3.3.3 of the client and v13.0.0 of unleash-edge.

The unleash-edge deployment basically looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: unleash-edge
  namespace: unleash
spec:
  selector:
    matchLabels:
      app: unleash-edge
  template:
    metadata:
      labels:
        app: unleash-edge
    spec:
      containers:
      - name: unleash-edge
        image: docker.io/unleashorg/unleash-edge:v13.0.0
        args:
        - edge
        env:
        - name: UPSTREAM_URL
          value: "https://eu.app.unleash-hosted.com/OUR_UPSTREAM_URL"
        ports:
        - containerPort: 3063
          name: http

This is tested both with and without setting a SERVICE_ACCOUNT_TOKEN environment variable.

[andreas-unleash]

Hi @JonasBak , thanks for bringing this to our attention. I will try to reproduce, investigate and get back to you

[JonasBak]

Thanks for looking into it, I tried comparing the data returned by edge and "not-edge", and these are the differences (diff edge.json not-edge.json):

8,9d7
<       "stale": false,
<       "impressionData": false,
10a9
>       "stale": false,
23c22,24
<       "variants": []
---
>       "variants": [],
>       "description": null,
>       "impressionData": false
26d26
<   "segments": [],
28c28
<     "projects": [
---
>     "project": [
32c32,37
<     "inlineSegmentConstraints": false
---
>     "inlineSegmentConstraints": true
>   },
>   "meta": {
>     "revisionId": 424,
>     "etag": "\"181720d4:424\"",
>     "queryHash": "181720d4"

There are some fields that are added/removed, as well as projects vs project.

[daveleek]

Hello @JonasBak, thank you for bringing this up.
I had a look at your repro steps and failed to reproduce this locally. I've also had a conversation with one of the edge devs.

I have:

• A feature flag defined with 3 environments, enabled for only one of them
• A client token for that environment
• Edge running against Unleash
• SDK set up with an example app running against Edge

From what I can tell edge works as it should. I also took a look at your update with the diff in the returned JSON, but there's nothing there that stands out. I think a good next step is to validate any constraints that may have been put on your strategy, appname for instance, against what is being defined in your apps.

Also:

I have also tried calling /api/client/features on the unleash-edge instance directly with the same token, and it returns the flag with "enabled": true.

The "enabled" property here does not reflect the result of feature flag evaluation, which happens inside the Unleash .NET Client SDK based on the defined parameters, constraints, strategies, and provided context from host application

[JonasBak]

It seems like the issue might be with our setup, possibly related to our ingress/nginx/cloudflare setup. When I use the client to connect directly to an edge pod or service using kubectl port-forward, it works as expected, but when going through the ingress the "normal route" it doesn't (but I see that the call to /api/client/features returns 200). It's weird how both the go client and frontends still handle going through the ingress. I'll look more into it.

[daveleek]

Thank you @JonasBak, that's interesting. If you can, please keep us posted on your findings!

[JonasBak]

I've been able to reproduce the issue locally using caddy and edge.

docker run --rm -d -p 3063:3063 -e "UPSTREAM_URL=https://eu.app.unleash-hosted.com/OUR_URL" -e RUST_LOG=DEBUG docker.io/unleashorg/unleash-edge:v13.0.0 edge
caddy reverse-proxy --from http://localhost:8080 --to http://localhost:3063

When using http://localhost:3063/api/ (edge directly) it works, but http://localhost:8080/api/ (through caddy) it doesn't. (both work with golang)

My best guess atm would be that it's somehow related to the headers UNLEASH-APPNAME and UNLEASH-INSTANCEID. It seems like the dotnet client uses the headers in all caps, but caddy and nginx rewrites them to Unleash-Appname and Unleash-Instanceid.

[daveleek]

Hello @JonasBak, thank you for these repro steps. We were able to reproduce and found that edge does not send back etag when content is gzipped. We have a PR on the way that should fix the issue

[JonasBak]

Thanks @daveleek 🙌

I've not been able to test v13.1.0 as when i start the container it exits with the following message:

/unleash-edge: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory

[daveleek]

Thank you @JonasBak - we're looking into it and will get back to you when we have a fix

[sighphyre]

Hey @JonasBak, we had an issue in our build which produced a broken docker image. We've uploaded a new 13.1.0 image over the broken one, which should fix this. Apologies for the trouble here!

[JonasBak]

Nice, I've checked that the new version works in our setup, so I'll close the issue 🎉