Make storage an interface

[Gufran]

This patch makes WebFrontEndImpl compatible with an interface type for storage.
The abstraction will let people write their own storage driver when using pebble as a library.

[Gufran]

@cpu Yours thoughts are appreciated here.

[cpu]

The abstraction will let people write their own storage driver when using pebble as a library.

@Gufran Can you describe more why you would want to use Pebble as a library?

My initial reaction is that this isn't an appropriate change for Pebble. We specifically chose an in-memory storage approach to discourage using Pebble except as a server for integration testing. I'm worried adding a new storage abstraction will encourage uses beyond this purpose. In general it isn't meant to be a flexible solution but a purpose built test rig that does everything possible to ensure you won't use it in another context.

[Gufran]

@cpu Pebble is an excellent implementation of ACME and it is only going to improve so I'd prefer to use it instead of writing my own server. If I can use it as a library I can write a server to simulate different test scenarios .

The driving use case for me right now is data persistence. I want to be able to shut down the server and resume at a later time. This will let us bring the server up with pre-populated data for testing.

[cpu]

Hi @Gufran, apologies for the delay in responding. Busy week!

What kind of test scenarios are you imagining?

The driving use case for me right now is data persistence. I want to be able to shut down the server and resume at a later time. This will let us bring the server up with pre-populated data for testing.

Pebble is very explicit in trying to avoid data persistence. I think it might be a better fit to try and rework your testing such that you can replay requests that create the data instead of prepopulating pebble artificially. Would that be feasible?

E.g. if your tests require 10 registrations with a handful of authzs exist, then create a script that starts up pebble and creates 10 new registrations and associated authz's and then proceeds with the tests.

[jsha]

One the main goals of Pebble is to make testing of ACME clients easier, with lots of hooks and ways to trigger various failures. Perhaps you can make a proposal about adding HTTP-based hooks in Pebble for prepopulating the data you need? For instance, you could have an endpoint that creates a previously-validated order for a given CSR, expiring at a date you provide. That would potentially benefit both your tests, and other tests people might want to run against Pebble.

[Gufran]

Apologies for the long delay.
Here is a very specific example. In our product every customer is given a subdomain by default and they are also allowed to bring their own domains if they wish so. We are still dealing with the legacy code which lets a user upload their own SSL certificate for all of their domains and there are so many cases that we'd like to test against a couple thousand records in every test run. A pre-populated pebble storage which is shared with the product will help us massively reduce the time it takes to set up the testing ground while we work on slowly moving over to Let's Encrypt completely.

I understand that an ephemeral storage is chosen and enforced to eradicate a whole class of corner cases that can creep up as the storage is mutated by tests over time, and I also understand that my particular use can may not be adequate to warrant an abstraction over data storage, but I believe that the ability to swap out the storage will prove helpful to many others in similar situation as me.

[cpu]

Hi again @Gufran

A pre-populated pebble storage which is shared with the product will help us massively reduce the time it takes to set up the testing ground while we work on slowly moving over to Let's Encrypt completely.

Can you elaborate on what data you pre-populate in this scenario? Accounts? Authorizations? Orders?
All of the above?

Thanks!

[cpu]

I think the consensus here is that we don't want to make the storage generic through an interface and would instead prefer to understand what test data needs to be pre-populated and to offer HTTP hooks to do so.

@Gufran If you can elaborate on your needs in a new issue that would be great. Thanks!