Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Private link support for Workload Profiles #867

Open
torosent opened this issue Aug 9, 2023 · 41 comments
Open

Private link support for Workload Profiles #867

torosent opened this issue Aug 9, 2023 · 41 comments
Labels
Networking Related to ACA networking roadmap This feature is on the roadmap

Comments

@torosent
Copy link
Member

torosent commented Aug 9, 2023

No description provided.

@microsoft-github-policy-service microsoft-github-policy-service bot added the Needs: triage 🔍 Pending a first pass to read, tag, and assign label Aug 9, 2023
@torosent torosent added roadmap This feature is on the roadmap Networking Related to ACA networking and removed Needs: triage 🔍 Pending a first pass to read, tag, and assign labels Aug 9, 2023
@mortenf1984
Copy link

Any new information on this? It is needed for FrontDoor support with Consumption + Dedicated and internal vNet only

@nad-au nad-au mentioned this issue Sep 20, 2023
Closed
@trylvis
Copy link

trylvis commented Oct 23, 2023

Any estimates on when this will become available?

@wsloth
Copy link

wsloth commented Nov 6, 2023

Looking for a status update on this as well. I'm aiming to run Azure Container Apps behind an Azure Firewall with UDR's, which leaves me no choice but to use Workload Profiles (serverless only would be fine, but it doesn't support UDR's according to this doc).

Workload profiles luckily also supports a serverless variation, but now I'm struggling on how to set up some form of inbound internet connectivity, ideally using Azure Frontdoor. The one guide from Microsoft which details on how to set this up assumes the exposed loadbalancer is called kubernetes-internal, which is not applicable to the Workload Profiles load balancer as it's called capp-svc-lb and is an IP-based backend pool, which is not supported by the Private Link Service.

image

@mortenf1984
Copy link

Can we please get a update on this?

@mortenf1984
Copy link

Are there any new information on the progress to get PLS support to CA with internal LB?

@tasdflkjweio
Copy link

Can we get an update on this? This seems like basic functionality that's missing.

@alhardy
Copy link

alhardy commented Jan 23, 2024

I am wanting to migrate from consumption only to dedicated since I need:

  • UDR support
  • increased Cores per replica

I don't think there are plans for a migration from consumption only to dedicated?

I am using front door for ingress to my consumption only plan, I cannot migrate due to lack of support for private links i.e. connecting front door where apis are on the same domain

@jjindrich
Copy link

My customer has same problem. Any estimates ?

@MrImpossibru
Copy link

+1 waiting for the implementation.
P.S. There're some workarounds if needed: 1) use an Application Gateway, which supports a private endpoint (expensive), 2) custom forwarder based on VMSS and a HAProxy + a Standard Load Balancer + Private Link Service (requires some work to do)

@mitchdowd
Copy link

I'd like to use workload profiles to have NAT gateway support, but then I can't have the environments private which is a big downside.

@rambo108
Copy link

Any update on when the fix will be ready for this?

@jackcaplin
Copy link

Is there an update or ETA?

@jsheetzmt
Copy link

Also looking for an update

@falsadoo
Copy link

Hi all can we get an eta for this?

@mortenf1984 mortenf1984 mentioned this issue Mar 21, 2024
Open
@cachai2
Copy link
Collaborator

cachai2 commented Mar 21, 2024

Hi folks, we are targeting end of Q2 CY2024 for this feature.

@KaiWalter
Copy link

KaiWalter commented Mar 22, 2024 •
edited
Loading

So CY = Calendar Year, hence Apr-Jun I assume. And when @cachai2 @paulyuk do you think, we have this for Azure China / 21vianet? We cannot proceed with our Service Fabric migration to ACA as we will wait for a clean Private Link solution in China.

@ardeshir
Copy link

This is the error we get when attempting to build one directly:

Image

@NemSimpraga
Copy link

It's mid May, any update on this?

@anthonychu anthonychu mentioned this issue May 31, 2024
Open
@chinadragon0515
Copy link
Member

chinadragon0515 commented May 31, 2024 •
edited
Loading

We are ready for private preview, if you are interested in private preview, can you help to fill this form https://forms.office.com/r/fuhKa4UMy6
We will whitelist your sub for private preview.
Public preview still need some time.

@vandanchev
Copy link

We are ready for private preview, if you are interested in private preview, please send you subscription id to acasupport(at)microsoft(dot)com and we can whitelist your sub for private preview. Public preview still need some time.

Would this be available for private preview in China also?

@ardeshir
Copy link

ardeshir commented May 31, 2024 via email

@chinadragon0515
Copy link
Member

@vandanchev we do not open Azure China for private preview now, thanks

@mortenf1984
Copy link

I have sent a E-Mail and used the form, we are ready for Private Preview testing of this functionality

@NemSimpraga
Copy link

I've tested out the functionality, but either I am missing something or this does not address the issue at hand:

  • The added functionality enables creating a private endpoint for the Container App Environment in Workload Profiles mode
  • As far as I can see, this does not enable the private link service support that's needed to route traffic from Front Door to the Container Apps in the Container App environment

The Container Apps need to be added as origins to Front Door via a private link service, which is still not doable due to the original issue of Workload Profiles Container App Environment's managed Load Balancers using the IP based backend pool.

@jackcaplin
Copy link

jackcaplin commented Jun 3, 2024 via email

@NSimpragaVolur
Copy link

NSimpragaVolur commented Jun 3, 2024 •
edited
Loading

Is there any documentation? I cant see how to add a private endpoint on my workload profile ACA

You need to apply for the private preview, and you'll be able to create a private endpoint for your Container App Environment. The problem is that it doesn't really seem to solve anything, as you just get a private endpoint to your Environment in a subnet of your choosing. The same can be accomplished by simply creating an Environment with internalOnly mode, where you get an L4 LB with a private IP.

That, or I am just missing something :)

@jackcaplin
Copy link

jackcaplin commented Jun 3, 2024 via email •
edited
Loading

@NSimpragaVolur
Copy link

You don't need to enable anything, you're just able to create a private endpoint to your ACA Environment, as I said. That's about it.

@jackcaplin
Copy link

Ok, I have created the private endpoint for the workload profile.

I have added this endpoint as an origin group in Azure Front Door.

Looks good so far

@otuerker
Copy link

otuerker commented Jun 3, 2024

Hi! We need this Feature also whitelisted for our Subscription. Can you please Whitelist us?
Subscription: 4adbcf85-ddca-4471-aaf1-0ee7488ae1b9

@microsoft microsoft deleted a comment from chinadragon0515 Jun 3, 2024
@chinadragon0515
Copy link
Member

@ardeshir @otuerker can you help to fill the form here so I can send you the instructions how to set up private endpoint with ACA.

@NSimpragaVolur
Copy link

NSimpragaVolur commented Jun 14, 2024 •
edited
Loading

I have been playing with this functionality, and I have to say there are some very weird points to consider about the implementation around this feature.

Primarily, how the public availability of the application is influenced by the creation of a private endpoint.

Example:

  • Created a private endpoint in my VNet for my Container App Environment
  • Connected via VNet Gateway into the same VNet where the private endpoint is
  • I can see that when calling the FQDN of my Container App deployed to that Environment, it resolves to the private IP of my environment's private endpoint and I can successfully connect to my application
  • When not connected to the VPN, the FQDN of my Container App resolves to a random Azure public IP
    • importantly, not the actual public IP of the deployed Container App Environment!
  • this means my application is not publically available over its FQDN, if I am not connected to my VPN (and configured private DNS resolution)

Here's the kicker: I can still reach the Container App over the public IP of the Container App environment!
All I have to do is issue an HTTP request directly to the public IP of the Container App Environment (instead of the FQDN of the Container App) while setting the "Host" header to the FQDN of my deployed Container App, and I will be able to successfully reach the Container App.

This seems like a very weird setup as the application is not completely private (which we can see from the fact that it's available over its public IP), but it is also not available over the public internet when trying to resolve its actual FQDN.

What I expected to get after creating a private endpoint for the Container App Environment:

  • a way into the Container Apps of the Environment via the private network the private endpoint is in
  • by default, the Container Apps would stay publically available over their FQDN if the access was not specifically restricted to a private IP range via the Container Apps ingress IP restriction

What we get is a partial solution: public access is restricted only on the DNS level for the Container Apps since their FQDNs won't resolve to the public IP of the Container App Environment.
But, crucially, they will stay available by directly accessing the public IP with a properly set "Host" header.

This does not make sense if you ask me, hope it helps in figuring our which direction this feature needs to go.

@jackcaplin
Copy link

jackcaplin commented Jun 14, 2024 via email •
edited
Loading

@cachai2
Copy link
Collaborator

cachai2 commented Jun 14, 2024

Hi @NSimpragaVolur, thank you for the feedback and for the detailed writeup with repro steps. We'll investigate this to verify the behavior and whether we need to make updates to align with expectations for the feature.

@NemSimpraga
Copy link

NemSimpraga commented Jun 15, 2024 •
edited
Loading

Doesn't this depend on how you created the Container App Environment? Do you have a Workload Profile with ingress set to internal only? Sent from Outlook for Androidhttps://aka.ms/AAb9ysg
…
________________________________ From: Nemanja Simpraga @.> Sent: Friday, June 14, 2024 5:47:00 PM To: microsoft/azure-container-apps @.> Cc: Jack Caplin @.>; Comment @.> Subject: Re: [microsoft/azure-container-apps] Private link support for Workload Profiles (Issue #867) NOTICE! This email was sent from outside Paxton. Please do not click links or open attachments unless you recognise the sender and know the content is safe. I have been playing with this functionality, and I have to say there are some very weird points to consider about the implementation around this feature. Primarily, how the public availability of the application is influenced by the creation of a private endpoint. Example: * Created a private endpoint in my VNet for my Container App Environment * Connected via VNet Gateway into the same VNet where the private endpoint is * I can see that when calling the FQDN of my Container App deployed to that Environment, it resolves to the private IP of my environment's private endpoint and I can successfully connect to my application * When not connected to the VPN, the FQDN of my Container App resolves to a random Azure public IP * importantly, not the actual public IP of the deployed Container App Environment! * this means my application is not available by resolving its FQDN, if I am not connected to my VPN (and configured private DNS resolution) Here's the kicker: I can still reach the Container App over the public IP of the Container App environment! All I have to do is issue an HTTP request directly to the public IP (not the FQDN) of the Container App while setting the "Host" header to the FQDN of my deployed Container App, and I will be able to successfully reach the Container App. This seems like a very weird setup as the application is not completely private (which we can see from the fact that it's available over its public IP), but it is also not available over the public internet when trying to resolve its actual FQDN. What I expected to get after creating a private endpoint for the Container App Environment: * a way into the Container Apps of the Environment via the private network the private endpoint is in * by default, the Container Apps would stay publically available over their FQDN if the access was not specifically restricted to a private IP range via the Container Apps ingress IP restriction What we get is a partial solution: public access is restricted only on the DNS level for the Container Apps since their FQDNs won't resolve to the public IP of the Container App Environment. But, crucially, they will stay available by directly accessing the public IP with a properly set "Host" header. This does not make sense if you ask me, hope it helps in figuring our which direction this feature needs to go. — Reply to this email directly, view it on GitHub<#867 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2AWLZQZBZDXCUNGZERXMOTZHMNAJAVCNFSM6AAAAAA3KJBMG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRYGQYTCMRXHA. You are receiving this because you commented.Message ID: @.> Jack Caplin Chief IT Architect [https://public.paxton-access.co.uk/public/emailfooterimages/image001.png]https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature Phone: 01273 811011 Mobile: 07534563400 Web: www.paxton-access.com<https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature> Email: @.@.> Paxton House, Brighton, BN1 9HU Find us on the maphttps://goo.gl/maps/9vNLzgN1qR1zNg9A9 [https://public.paxton-access.co.uk/public/emailfooterimages/image002.png] https://www.facebook.com/PaxtonAccessLtd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image003.png] https://www.linkedin.com/company/paxton-access-ltd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image004.png] https://twitter.com/paxtonaccess [https://public.paxton-access.co.uk/public/emailfooterimages/image005.png] @./> [https://www.paxton-access.com/wp-content/uploads/2024/04/Tech-Tour-2024-%E2%80%93-Email-Signature-1.png]https://www.paxton-access.com/tech-tour/?utm_source=emailsig&utm_medium=email&utm_campaign=techtour&utm_id=techtour

That is true, if you create a Container App Environment with properties.vnetConfiguration.internal as true, the L4 LB you get with the environment has a private IP, and there's no way to reach it directly from the public. But in such Container App Environments, you mostly don't really have a need to create a private endpoint. Perhaps to expose your Container Apps privately to VNets other than the one you integrated the Container App Environment with.

My issue is primarily with ACA Envs with public IPs (i.e. properties.vnetConfiguration.internal = false). If creating a private endpoint will make the FQDNs non-routable publically, then the public IP should not accept internet traffic at all, not let in traffic if someone is pointing calls to the public IP of you environment directly.

The better option, if you ask me (and no one is : ]), is leaving the internet route intact even with a created private endpoint. That is, the FQDNs resolve to the actual public IP of the environment when called from the internet, but when the FQDNs resolve from a VNet, they will resolve to the private IP of the endpoint. This is similar to other private endpoint implementations.

Example: the FQDN of an Azure Flex Postgres server will resolve to the private IP of a private endpoint when the DNS resolution call comes from inside a VNet, but will resolve to the public IP of the server when it comes from the internet. Then on the server itself you can configure firewall rules for the public IP, i.e. let in whatever you want from the public, while private access is handled by the private endpoint.

Container Apps themselves have IP restriction rules, which might be used similarly as a Postgres Flex server firewall rule.

@jackcaplin
Copy link

jackcaplin commented Jun 15, 2024 via email

@NSimpragaVolur
Copy link

NSimpragaVolur commented Jun 15, 2024 •
edited
Loading

I've been using Application Gateway as interim hops.

But if you're using AppGw for interim hops between Front Door and your internal Container App environment, then you don't need the private endpoint. You just put your AppGw in the same VNet as your Container App Environment and route traffic to its IP, which is already private anyways.

Only if your AppGw is integrated into a VNet other than the one your Container App Environment is, private endpoint becomes useful. Which is what I meant by:

Perhaps to expose your Container Apps privately to VNets other than the one you integrated the Container App Environment with.

@jackcaplin
Copy link

jackcaplin commented Jun 15, 2024 via email

@NSimpragaVolur
Copy link

But with the private endpoints, I no longer need the app gateways. Until private endpoint support for workload profiles, they were the only method (apart from app proxies) to integrate front door. I have now removed the app gateways. One per region. Less infrastructure to maintain and less cost. Sent from Outlook for Androidhttps://aka.ms/AAb9ysg
…
________________________________ From: Nemanja Simpraga @.> Sent: Saturday, June 15, 2024 3:12:58 PM To: microsoft/azure-container-apps @.> Cc: Jack Caplin @.>; Comment @.> Subject: Re: [microsoft/azure-container-apps] Private link support for Workload Profiles (Issue #867) NOTICE! This email was sent from outside Paxton. Please do not click links or open attachments unless you recognise the sender and know the content is safe. I've been using Application Gateway as interim hops. But if you're using AppGw for interim hops between Front Door and your internal Container App environment, then you don't need the private endpoint. You just put your AppGw in the same VNet as your Container App Environment and route traffic to its IP, which is already private anyways. Only if your AppGw is integrated into a VNet other than the one your Container App Environment is, private endpoint becomes useful. Which is what I meant by: Perhaps to expose your Container Apps privately to VNets other than the one you integrated the Container App Environment with. — Reply to this email directly, view it on GitHub<#867 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2AWLZVCW2RDTJRALDPRICLZHRDWVAVCNFSM6AAAAAA3KJBMG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRZG4ZTMMZSG4. You are receiving this because you commented.Message ID: @.> Jack Caplin Chief IT Architect [https://public.paxton-access.co.uk/public/emailfooterimages/image001.png]https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature Phone: 01273 811011 Mobile: 07534563400 Web: www.paxton-access.com<https://www.paxton-access.com/?utm_source=staff-signature&utm_medium=email&utm_campaign=staff-signature> Email: @.@.> Paxton House, Brighton, BN1 9HU Find us on the maphttps://goo.gl/maps/9vNLzgN1qR1zNg9A9 [https://public.paxton-access.co.uk/public/emailfooterimages/image002.png] https://www.facebook.com/PaxtonAccessLtd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image003.png] https://www.linkedin.com/company/paxton-access-ltd/ [https://public.paxton-access.co.uk/public/emailfooterimages/image004.png] https://twitter.com/paxtonaccess [https://public.paxton-access.co.uk/public/emailfooterimages/image005.png] @./> [https://www.paxton-access.com/wp-content/uploads/2024/04/Tech-Tour-2024-%E2%80%93-Email-Signature-1.png]https://www.paxton-access.com/tech-tour/?utm_source=emailsig&utm_medium=email&utm_campaign=techtour&utm_id=techtour

Really? I am really curious how you send traffic from Front Door to the private endpoint directly. You usually do it via Private Link Service and the Container App's L4 load balancer with a non IP-based backend pool, which is only applicable for Consumption-only environments. Workload profiles still have the IP-based backend pool L4 LB, so I don't understand how a private endpoint helps overcome this issue?

@dyamo
Copy link

dyamo commented Jul 10, 2024

Hi folks, we are targeting end of Q2 CY2024 for this feature.

Any updates?

@hbuckle
Copy link

hbuckle commented Jul 11, 2024

We've been enabled for the preview and I can confirm that this does let you create a private endpoint attached to the container app environment (so you don't have to create a private link service on the internal load balancer).

However there is still the missing piece that Front Door does not support adding a private link origin attached to the container app environment, which I think is what most people here are after.

@chinadragon0515 - is Front Door going to be updated with origin support for direct private endpoint connectivity to a container app environment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Networking Related to ACA networking roadmap This feature is on the roadmap
Projects
Azure Container Apps Roadmap
Status: In-Progress (Development)
Milestone
No milestone
Development

No branches or pull requests