created IMemoryStore interface from existing methods.

[siennathesane]

in order to use pebble as an in-process integration test suite, being able to
implement the memory store by hand or with mocks is very helpful. this
change only extracts existing methods and updates the methods which
relied on the original MemoryStore receiver.

Signed-off-by: Mike Lloyd mike.lloyd@gsa.gov

[jsha]

Hi @mxplusb ! Thanks for the contribution. We've discussed Pebble as an in-memory test suite before, and our conclusion was that we don't want to support that use case. We'd like to support integration at the ACME and HTTP layers, with Pebble always considered a standalone process. This gives us more flexibility to refactor internals in the future with breaking integrations that reference specific symbols. Sorry about that!

[siennathesane]

We've discussed Pebble as an in-memory test suite before, and our conclusion was that we don't want to support that use case.

@jsha can you provide some thoughts or reading as to how you arrived at that decision? The project description says it's a test server.

We'd like to support integration at the ACME and HTTP layers

This is what I'm actually trying to test. My specific use case is that I have a service broker which will broker LE cert creation.

Here is the workflow of how the broker works, for reference:

• Customer creates an instance of the brokered service with a domain name.
• The service broker creates a DNS-01-based CSR with LE and we store the challenge information.
  • The service broker uses a client with a custom DNS-01 challenge provider so we have more flexibility on handling the challenge without direct interactions from the customers.
• The customer checks the brokered service instance for the challenge information so they can add it to their DNS provider.
• The customer uploads the challenge information to their DNS provider.
• We wait on the DNS-01 challenge to complete, and then we attach their SSL certificate to upstream networking devices.

I'm very specifically trying to test the DNS-01 challenge logic, so I can ensure that our code that talks to the production LE deployment isn't buggy or constantly getting rate-limited. The only way I can think to go through and properly test our ACME client's custom DNS-01 challenge provider is hook into the database, get the target challenge information, add it to the integration DNS server, and then let the ACME challenge continue.

Do you have any recommended projects/solutions for testing a custom DNS-01 challenge provider with an ACME client?

[jsha]

I'm very specifically trying to test the DNS-01 challenge logic, so I can ensure that our code that talks to the production LE deployment isn't buggy or constantly getting rate-limited.

Thanks! We definitely appreciate that.

test our ACME client's custom DNS-01 challenge provider is hook into the database, get the target challenge information, add it to the integration DNS server, and then let the ACME challenge continue.

I'm not sure I understand. In this situation I would set up the ACME client to retrieve the challenge information via ACME, the same way as it would do in a production setting. What am I missing?

The customer checks the brokered service instance for the challenge information so they can add it to their DNS provider.

By the way, I strongly recommend you design your service so that the main use case has fully automated renewals. This increases reliability significantly. For instance, during the last shutdown, a number of government websites went down because their operators were not allowed to install renewed certificates. However, those that were running automated renewals were able to keep going through the shutdown.

The acme-dns project has an example of how you can use CNAMEs to fully automated DNS-01-based renewals even if you are serving customers with disparate DNS providers.

[jsha]

Ah, whoops I read your linked document and I see you are already using CNAMEs to support automated renewal. Great! Sorry for not reading that first. :-)

BTW I think you have a typo: _acme-my.example.gov. should probably be _acme-challenge.my.example.gov.

[siennathesane]

BTW I think you have a typo

Thanks! See cloud-gov/cg-site#1436

What am I missing?

Nothing, I think I explained it wrong, apologies.

Here is my ACME client workflow, I'm using github.com/go-acme/lego as the base library:

• Instantiate ACME client
• Register a custom ValidateFunc (uses custom resolvers due to networking designs, but pointed at 1.1.1.1 and 8.8.8.8 *when not pointed at the DNS integration test server)
• Register a custom DNS-01 challenge provider (so we can store the challenge elsewhere for later),
• Register the user
• Obtain certificate
  • The client then goes to order a new certificate.

It's my understanding that certificate issuance is blocked until both Pebble and the ValidateFunc implementation can resolve the challenge. I am in the process of finalising internal unit tests on the custom DNS-01 challenge provider, so I will be able to separately validate my DNS-01 challenge provider does what it's supposed to.

I would like a way to simulate the user experience (and ensure it all works together), which means I have to be able to have a record appear in the DNS integration server so both Pebble and the ValidateFunc can resolve the challenge as if the user had entered the data in outside the system. If hooking into the Pebble DB and updating the DNS integration server to update it's TXT records isn't the right way to simulate that in a test, and you know of a better way, I'm more than happy to explore that path too. :)

Edit: clarified the custom resolver

[siennathesane]

We've discussed Pebble as an in-memory test suite before, and our conclusion was that we don't want to support that use case.

@jsha thinking more about this, I'd be very interested in starting and incubating an in-memory test suite project if there is a clear path to having potential support by the LE project. This is definitely something that can be very useful to the community and help add some maturity to the client side of things by giving folks a test suite they can consume. I've already pulled out some chunks of the core code from Pebble, so it wouldn't be very hard to just finish that...is there a way to test for RFC 8555 compliance?

[jsha]

It sounds like the best way to get your test suite to behave the way you want is to make your ACME client do resolutions against the same challtestsrv that Pebble uses. The client can update DNS via HTTP POST requests.

[siennathesane]

@jsha I ended up creating an integration environment for my use case from this PR. Happy to for any feedback you have on it.

https://github.com/18F/gravel