Cannot deploy apps on Azure when mTLS is enabled on the cell(s) #300
Comments
|
We have created an issue in Pivotal Tracker to manage this: https://www.pivotaltracker.com/story/show/143940609 The labels on this github issue will be updated when the story is started. |
|
More logs from the rep.stdout.log |
|
Thanks for the report, @paolostivanin. We'll prioritize this for the Diego team to investigate. In the meantime, is it correct that your Azure deployment is configured to use Azure Blob Storage for the CC blobstore, and that the staging task contains a CC URL for the droplet that redirects to the Azure URL? That's my impression from the From the Oh, it might also be informative to connect to that Azure URL with the same client credentials using Best, |
|
@ematpl we got an answer from MS about the problem:
but if I got it right, the mTLS is between the cell and the cc_bridge, am I right? Then the cloud controller will communicate with the blobstore using "standard" TLS. |
|
|
@ematpl what do you mean with "... with the same client creds using ..."? The cell is using the rep server cert not the client one (as written here) |
|
Just for my understanding, Is the connection flow Thanks :) |
|
strangely enough, I was looking at the traffic on the cc_bridge on our openstack dev landscape and, while here everything is working fine, the traffic still goes through port 9090 instead of 9091: |
|
Most blobstores don't support client-side cert validation so in this case the Fog library handles interactions with the Azure blobstore (likely just standard TLS)
Something more like Right now it would seem that the rep is failing to download from the blobstore, but CC has successfully uploaded to the blobstore. It's hard to say if this is a mutual TLS issue. |
|
@paolostivanin specifically with regard to the data flow to download from blobstore, the flow would be like the following after the cell rep has the information from the bbs about the lrp, |
|
I found the issue!
This is a dump of the traffic on port 80 and port 443 from within the cell. During the ssl handshake the client is asking for the above 2 ciphers but...
Basically, the SSL connection cannot be established because of missing ciphers support from the blobstore. PS when will this feature become mandatory? |
|
Summary: The ssl handshake then fails because the blobstore doesn't support the aforementioned ciphers. The server then answers with the hello and chooses: May I ask you when are you planning to make this option mandatory? Do you already have an ETA? |
|
@paolostivanin great find! @ematpl might have a better idea of when we plan to make it the default. If this is the case we can potentially push that out until the blobstore supports it. |
|
@caod123 thanks :) we are also in touch with MSFT, let's see what they will answer! |
|
Thanks for tracking that issue down, @paolostivanin! There has recently been an effort to make TLS clients and servers throughout Cloud Foundry more restrictive in the cipher suites and TLS versions they accept. It looks like when the client credentials are configured, the TLS configuration for the HTTP client that the rep uses in uploads and downloads also happens to include this more restricted set of cipher suites (from https://github.com/cloudfoundry/cfhttp/blob/012f40c12b36dd67274e611f4f44ed932ae155d5/cf_http.go#L17-L20) instead of the larger default set configured in Golang (https://golang.org/pkg/crypto/tls/#pkg-constants). Even if Microsoft enables those two GCM cipher suites on the Azure Blob Storage service relatively soon, there are potentially other blob stores or asset servers that wouldn't support them. I'll check with other security-minded folks on CF, but I think we'll probably configure this particular HTTP client to support a broader set of cipher suites for TLS 1.2 communication. @zrob and the CAPI team have been driving this particular set of TLS changes in order to finish securing communication between Cloud Controller, the CC-Bridge, and Diego, so they would be most authoritative about when these changes will be required. In any case, I'm hoping to get the cipher-suite relaxation into Diego before that happens. Thanks again, |
|
@paolostivanin The CAPI is working on a document to describe how operators can opt in to the more secure TLS communications b/t CC and Diego. The code should be available as of either CF 257 or 258. Once the guide is available, we will wait around a month before removing the opt-in option and making it mandatory. |
|
@paolostivanin That cipher-suite configuration is on the Diego side, in the cell rep. I've added https://www.pivotaltracker.com/story/show/144119911 for that work, and the Diego team will likely get to it later this week. Best, |
|
For what it's worth, this is also breaking the TLS handshake to our Swift endpoints. |
|
Thanks, @amhuber, I just finished acceptance on https://www.pivotaltracker.com/story/show/144119911 and will be releasing Diego v1.15.0 with the correct behavior shortly. Best, |
cf-gitbot
added
in progress
delivered
accepted
and removed
unscheduled
in progress
delivered
labels
|
Closing, as Diego v1.15.0 resolves the issue. Please let us know if you encounter any further problems! Thanks, |
|
Confirmed fixed with 1.15.0, thanks! |
|
Fantastic, thanks for letting us know, @amhuber ! |
|
I can also confirm that Diego 1.15.x solved the problem! |
Submodule src/github.com/onsi/ginkgo eea6ad008..d90e0dcda: > Add integration test > Fix coverage files combining > A new CLI option: -ginkgo.reportFile <file path> (#601) > v1.10.2 > speed up table entry generateIt() (#609) > Fix. Write errors to stderr instead of stdout (#610) > v1.10.1 > stack backtrace: fix skipping (#600) > v1.10.0 > stack backtrace: fix alignment and skipping > fix typo in documentation > v1.9.0 > Fixed typos in comments > gofmt code > Simplify code > Simplify concatenation, incrementation and function assignment > Avoid unnecessary conversions > JUnit: include more detailed information about panic > Option to print output into report, when tests have passed > Print help to stdout when the user asks for help Submodule src/github.com/onsi/gomega 41673fd8f..bdebf9e0e: > v1.7.0 > export format property variables (#347) > minor fix in the documentation of ExpectWithOffset (#358) > v1.6.0 > Remove duplication in XML matcher tests > Remove unnecessary conversions (#357) > Fixed import order (#353) > Added missing error handling in test (#355) > Simplify code (#356) > Simplify code (#354) > Fixed typos (#352) > Add failure message tests to BeElementOf matcher > Update go-testcov untested sections > mark all uncovered files so go-testcov ./... works > display special chars on error > Reenable gotip in travis > Add BeElementOf matcher > Fix the typo of comment (#345) > Optimize contain_element_matcher > v1.5.0 > add analogous tests to fields_test.go > added tests for error messages > remove redundant validity check > ensure error messages refer to the underlying interface, not the `reflect.Value` > Run CI on go 1.12 and not 1.9 > Turn off gotip on travis > Mark session_test.go with build tag (#324) > more matcher tests > try to appease linters > MatchKeys > Clarify message for unexpected errors > make it easy to run everythin that travis would do > remove go 1.6, 1.7, 1.8 from the build matrix, and add tip > fix lint errors and use type aliases to remove stuttering > remove duplication of code and add docs to public function > v1.4.3 > test yaml parser errors and better comment on why panics are there > add test for not matcher with errors > add test for or matcher with errors > add tests for positive failure messages of Panic > ensure file name and line numbers are correctly reported > Fail the build if gofmt detects any changes required > Make changes to satisfy gofmt > Move GO111MODULE into more suitable env key in Travis > Add coverage for BeZero and Succeed matchers (#307) > Fixed matcher for content-type (#305) > v1.4.2 > Refactor goexec build test > Updates for Go v1.11 official release > Add go.mod and go.sum files to define the gomega go module > Work around go vet issue with Go v1.11 (#300) > Better output when using with go XUnit-style tests, fixes #255 (#297) > Fix MatchJSON fail to parse json.RawMessage (#298) > Move funcs shared by MatchJSON and MatchYAML to a dedicated support file > Added first failure path matching for YAML (#279) > show threshold in failure message of BeNumericallyMatcher (#293) > v1.4.1 > Update documentation (#289)
When the mTLS is enabled (https://github.com/cloudfoundry/diego-release/blob/develop/manifest-generation/diego.yml#L240) it's not possible to deploy apps on Azure. On AWS and Openstack everything works fine though.
Diego manifest:
CF manifest:
Info:
CF 257
Diego 1.13.0
Bosh 261.4
Stemcell 3363.15
The text was updated successfully, but these errors were encountered: