Skip to content

feat: local backend#20

Closed
flavio wants to merge 2 commits intoFiloSottile:mainfrom
flavio:local-backend
Closed

feat: local backend#20
flavio wants to merge 2 commits intoFiloSottile:mainfrom
flavio:local-backend

Conversation

@flavio
Copy link
Copy Markdown
Contributor

@flavio flavio commented Jun 28, 2024

Note: this PR builds on top of #19

I really love how easy it is to sunlight for testing/playing purposes. I've noticed however that sunlight requires access to a S3-compatible server to store the logs data. I've added the possibility to the local filesystem instead of relying on a S3 bucket.
This is not meant to be used in production, it just simplifies the setup for testing purposes.

I hope you like this change. Thanks again for the great tool!

Just for reference, this is the configuration file I used to spin up a local testing instance:

listen: ":8080"

checkpoints: checkpoints.db

logs:
  - name: sunlight.127.0.0.1.sslip.io/2024h1
    shortname: bergamo2024h1
    inception: 2024-06-28
    httpprefix: /2024h1
    roots: ./chain.pem
    key: ./key.pem
    cache: bergamo2024h1.db
    poolsize: 750
    notafterstart: 2024-06-26T00:00:00Z
    notafterlimit: 2024-07-01T00:00:00Z
    localbackend: local_backend

flavio added 2 commits June 28, 2024 14:09
Using sqlite to save checkpoints failed at runtime because:

* Internal Mutex was a nil pointer
* The help messages explaining how to create the sqlite3 database had
  the wrong CREATE statement
* The implementation of the `LockBackend` interface had the `Fetch`
  function return a wrong object (a `dynamoCheckpoint` instead of a
  `sqliteCheckpoint`)

Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Prior to this commit, sunlight required access to a S3-compatible server
to send backend data.

Now there's the possibility to store this data on the local filesystem.
This is not meant to be used in production, but it hugely simplifies
testing and local development.

Signed-off-by: Flavio Castelli <fcastelli@suse.com>
@hslatman
Copy link
Copy Markdown

hslatman commented Mar 14, 2025

Haven't submitted a log yet, but seems to start as expected locally.

I tried using MinIO first, but was getting malformed XML errors when putting the checkpoint:

{"time":"2025-03-14T01:30:40.441419+01:00","level":"DEBUG","source":{"function":"filippo.io/sunlight/internal/ctlog.(*S3Backend).Upload","file":"/Users/herman/Development/example-sunlight-deploy/sunlight-src/internal/ctlog/s3.go","line":172},"msg":"S3 PUT","log":"minio12025q1","key":"checkpoint","size":439,"compress":false,"type":"text/plain; charset=utf-8","immutable":false,"elapsed":5722041,"err":"operation error S3: PutObject, https response error StatusCode: 400, RequestID: 182C83F421156513, HostID: e0c385c033c4356721cc9121d3109c9b9bfdefb22fd2747078acd22328799e36, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema."}
time=2025-03-14T01:30:40.441+01:00 level=DEBUG msg="S3 PUT" log=minio12025q1 key=checkpoint size=439 compress=false type="text/plain; charset=utf-8" immutable=false elapsed=5.722041ms err="operation error S3: PutObject, https response error StatusCode: 400, RequestID: 182C83F421156513, HostID: e0c385c033c4356721cc9121d3109c9b9bfdefb22fd2747078acd22328799e36, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema."
{"time":"2025-03-14T01:30:40.441588+01:00","level":"ERROR","source":{"function":"main.fatalError","file":"/Users/herman/Development/example-sunlight-deploy/sunlight-src/cmd/sunlight/main.go","line":597},"msg":"failed to create log","log":"minio12025q1","err":"couldn't upload checkpoint: failed to upload \"checkpoint\" to S3: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 182C83F421156513, HostID: e0c385c033c4356721cc9121d3109c9b9bfdefb22fd2747078acd22328799e36, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema."}
time=2025-03-14T01:30:40.441+01:00 level=ERROR msg="failed to create log" log=minio12025q1 err="couldn't upload checkpoint: failed to upload \"checkpoint\" to S3: operation error S3: PutObject, https response error StatusCode
[core]
: 400, RequestID: 182C83F421156513, HostID: e0c385c033c4356721cc9121d3109c9b9bfdefb22fd2747078acd22328799e36, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema.

Most likely something specific to MinIO, but haven't dug deeper into the traffic yet. Just found #22, which seems to use MinIO, so maybe I just need to figure out some more config to make that part work. After some more fiddling with the configuration of both sunlight and MinIO it now seems to run, and can be restarted.

With your localbackend changes the log gets created successfully:

time=2025-03-14T11:02:01.004+01:00 level=INFO msg="today is the Inception date, creating log" log=minio12025q1
{"time":"2025-03-14T11:02:01.020348+01:00","level":"INFO","source":{"function":"filippo.io/sunlight/internal/ctlog.CreateLog","file":"/Users/herman/Development/example-sunlight-deploy/sunlight-src/internal/ctlog/ctlog.go","line":150},"msg":"created log","log":"minio12025q1","timestamp":1741946521006,"logID":"iaRIf7fNQ5MdBGhBNcxtJnObRefgJ+1NeJYgAq1thq8="}
time=2025-03-14T11:02:01.020+01:00 level=INFO msg="created log" log=minio12025q1 timestamp=1741946521006 logID="iaRIf7fNQ5MdBGhBNcxtJnObRefgJ+1NeJYgAq1thq8="
{"time":"2025-03-14T11:02:01.021641+01:00","level":"INFO","source":{"function":"filippo.io/sunlight/internal/ctlog.LoadLog","file":"/Users/herman/Development/example-sunlight-deploy/sunlight-src/internal/ctlog/ctlog.go","line":278},"msg":"loaded log","log":"minio12025q1","logID":"iaRIf7fNQ5MdBGhBNcxtJnObRefgJ+1NeJYgAq1thq8=","size":0,"timestamp":1741946521006}

@hslatman
Copy link
Copy Markdown

@flavio when restarting it fails to retrieve the checkpoint from the monitoring endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants