Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add t0124 with ipns fixtures #37

Merged
merged 21 commits into from
May 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
27 changes: 23 additions & 4 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,33 +7,52 @@ on:
jobs:
test:
runs-on: 'ubuntu-latest'
strategy:
fail-fast: false
matrix:
target: ['latest', 'master']
defaults:
run:
shell: bash
steps:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: 1.19.1
go-version: 1.20.4
Comment on lines 18 to +21
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't actually use Go here right now, do we?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, but we will with the matrix!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the catch; probably a leftover from before the containerization 👍

Note I'll merge the two tests.yml, which means we'll need go because we want to call:

- run: go install github.com/ipfs/kubo/cmd/ipfs@${{ matrix.target }}

In place of

# only work with released binaries
- uses: ipfs/download-ipfs-distribution-action@v1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably could get away with using the default from the machine for this but it's fine to leave it in.

- uses: actions/checkout@v3
with:
path: 'gateway-conformance'
- name: Extract fixtures
uses: ./gateway-conformance/.github/actions/extract-fixtures
with:
output: fixtures
- uses: ipfs/download-ipfs-distribution-action@v1
- uses: protocol/cache-go-action@v1
- run: go install github.com/ipfs/kubo/cmd/ipfs@${{ matrix.target }}
shell: bash
- name: Configure Kubo Gateway
run: |
ipfs init;
source ./gateway-conformance/kubo-config.example.sh;
IPFS_NS_MAP=$(cat ./fixtures/dnslinks.json | jq -r 'to_entries | map("\(.key).example.com:\(.value)") | join(",")')
echo "IPFS_NS_MAP=${IPFS_NS_MAP}" >> $GITHUB_ENV
# note: the IPFS_NS_MAP set above will be passed the daemon:
# note: the IPFS_NS_MAP set above will be passed the daemon
- uses: ipfs/start-ipfs-daemon-action@v1
- name: Provision Kubo Gateway
run: |
find ./fixtures -name '*.car' -exec ipfs dag import {} \;
# Import car files
cars=$(find ./fixtures -name '*.car')
for car in $cars
do
ipfs dag import "$car"
done

# Import ipns records
records=$(find ./fixtures -name '*.ipns-record')
for record in $records
do
key=$(basename -s .ipns-record "$record" | cut -d'_' -f1)
ipfs routing put --allow-offline "/ipns/$key" "$record"
Copy link
Member

@lidel lidel May 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 (not a blocker, just a note for the future) when ipfs/specs#369 is figured out, we will be able to remove our reliance on ipfs routing put

done
- name: Run the tests
uses: ./gateway-conformance/.github/actions/test
with:
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ provision-cargateway: ./fixtures.car

provision-kubo: fixtures.car
find ./fixtures -name '*.car' -exec ipfs dag import {} \;
find ./fixtures -name '*.ipns-record' -exec sh -c 'ipfs routing put --allow-offline /ipns/$$(basename -s .ipns-record "{}") "{}"' \;

fixtures.car: gateway-conformance
./gateway-conformance extract-fixtures --merged=true --dir=.
Expand Down
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,13 @@
- [Commands](#commands)
- [test](#test)
- [Inputs](#inputs)
- [Subdomain Testing and `subdomain-url`](#subdomain-testing-and-subdomain-url)
- [Usage](#usage)
- [extract-fixtures](##extract-fixtures)
- [extract-fixtures](#extract-fixtures)
- [Inputs](#inputs-1)
- [Usage](#usage-1)
- [Testing your gateway](#testing-your-gateway)
- [Provisioning the Gateway](#provisioning-the-gateway)
- [Examples](#examples)
- [FAQ](#faq)
- [In Development](#in-development)
Expand Down Expand Up @@ -111,6 +114,24 @@ The `extract-fixtures` command is used to extract the test fixtures from the `ga
docker run -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance extract-fixtures --output fixtures --merged false
```

## Testing Your Gateway

You can find the workflow that runs the gateway conformance test suite against Kubo in the file [.github/workflows/test.yml](.github/workflows/test.yml). This can serve as a good starting point when setting up your own test suite.

We've also aimed to keep the [kubo-config.example.sh](kubo-config.example.sh) script and the [Makefile](Makefile) as straightforward as possible to provide useful examples to get started.

### Provisioning the Gateway

We make minimal assumptions about the capabilities of the gateway being tested. Which means that we don't require nor expect the gateway to be writable. Therefore, you need to provision the gateway with test fixtures before running the test suite.

These fixtures are located in the `./fixtures` folders. We distribute tools for extracting them. Refer to the documentation for the `extract-fixtures` command for more details.

**Fixtures:**

- Blocks & Dags: These are served as [CAR](https://ipld.io/specs/transport/car/) file(s).
- IPNS Records: These are distributed as files containing [IPNS Record](https://specs.ipfs.tech/ipns/ipns-record/#ipns-record) [serialized as protobuf](https://specs.ipfs.tech/ipns/ipns-record/#record-serialization-format). The file name includes the Multihash of the public key ([IPNS Name](https://specs.ipfs.tech/ipns/ipns-record/#ipns-name)) in this format: `pubkey(_optional_suffix)?.ipns-record`. We may decide to [share CAR files](https://github.com/ipfs/specs/issues/369) in the future.
- DNS Links: These are distributed as `yml` configurations. You can use the `--merge` option to generate a consolidated `.json` file, which can be more convenient for use in a shell script.

## Examples

The examples are going to use `gateway-conformance` as a wrapper over `docker run -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance` for simplicity.
Expand Down
25 changes: 23 additions & 2 deletions cmd/gateway-conformance/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,22 @@ func copyFiles(inputPaths []string, outputDirectoryPath string) error {
if err != nil {
return err
}
for _, inputPath := range inputPaths {
outputPath := filepath.Join(outputDirectoryPath, filepath.Base(inputPath))
for i, inputPath := range inputPaths {
src, err := os.Open(inputPath)
if err != nil {
return err
}
defer src.Close()

// Separate the base name and extension
base := filepath.Base(inputPath)
ext := filepath.Ext(inputPath)
name := base[0 : len(base)-len(ext)]

// Generate the new filename
newName := fmt.Sprintf("%s_%d%s", name, i, ext)

outputPath := filepath.Join(outputDirectoryPath, newName)
dst, err := os.Create(outputPath)
if err != nil {
return err
Expand Down Expand Up @@ -190,6 +199,13 @@ func main() {
if err != nil {
return err
}

// TODO: when https://github.com/ipfs/specs/issues/369 has been completed,
// merge the IPNS records into a car file.
err = copyFiles(fxs.IPNSRecords, directory)
if err != nil {
return err
}
} else {
err = copyFiles(fxs.CarFiles, directory)
if err != nil {
Expand All @@ -201,6 +217,11 @@ func main() {
return err
}

err = copyFiles(fxs.IPNSRecords, directory)
if err != nil {
return err
}

err := dnslink.Merge(fxs.ConfigFiles, filepath.Join(directory, "dnslinks.json"))
if err != nil {
return err
Expand Down
File renamed without changes.
Binary file not shown.
Binary file added fixtures/t0124/fixtures.car
Binary file not shown.
Binary file not shown.
12 changes: 9 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ module github.com/ipfs/gateway-conformance
go 1.20

require (
github.com/ipfs/boxo v0.8.1
github.com/ipfs/boxo v0.8.2-0.20230510114019-33e3f0cd052b
github.com/ipfs/go-cid v0.4.1
github.com/ipld/go-ipld-prime v0.20.0
github.com/libp2p/go-libp2p v0.26.3
github.com/stretchr/testify v1.8.2
github.com/urfave/cli/v2 v2.25.3
gopkg.in/yaml.v3 v3.0.1
Expand All @@ -16,18 +17,23 @@ require (
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
github.com/ipld/go-codec-dagpb v1.6.0 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-libp2p-record v0.2.0 // indirect
github.com/multiformats/go-multiaddr v0.8.0 // indirect
github.com/multiformats/go-multistream v0.4.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
go.uber.org/goleak v1.1.12 // indirect
golang.org/x/sync v0.1.0 // indirect
)

require (
github.com/go-logr/logr v1.2.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/gogo/protobuf v1.3.2
github.com/google/uuid v1.3.0 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
Expand Down
13 changes: 10 additions & 3 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ github.com/cskr/pubsub v1.0.2 h1:vlOzMhl6PFn60gRlTQQsIfVwaPB/B/8MziK8FhEPt/0=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/decred/dcrd/crypto/blake256 v1.0.0 h1:/8DMNYp9SGi5f0w7uCm6d6M4OU2rGFK09Y2A4Xv7EE0=
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 h1:HbphB4TFFXpv7MNrT52FGrrgVXF1owhMVTHFZIlnvd4=
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0/go.mod h1:DZGJHZMqrU4JJqFAWUS2UO1+lbSKsdiOoYi9Zzey7Fc=
github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
Expand Down Expand Up @@ -45,8 +47,8 @@ github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uG
github.com/huin/goupnp v1.0.3 h1:N8No57ls+MnjlB+JPiCVSOyy/ot7MJTqlo7rn+NYSqQ=
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
github.com/ipfs/boxo v0.8.1 h1:3DkKBCK+3rdEB5t77WDShUXXhktYwH99mkAsgajsKrU=
github.com/ipfs/boxo v0.8.1/go.mod h1:xJ2hVb4La5WyD7GvKYE0lq2g1rmQZoCD2K4WNrV6aZI=
github.com/ipfs/boxo v0.8.2-0.20230510114019-33e3f0cd052b h1:6EVpfwbBgwhfZOA19i55jOGokKOy+OaQAm1dg4RbXmc=
github.com/ipfs/boxo v0.8.2-0.20230510114019-33e3f0cd052b/go.mod h1:Ej2r08Z4VIaFKqY08UXMNhwcLf6VekHhK8c+KqA1B9Y=
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
github.com/ipfs/go-block-format v0.0.2/go.mod h1:AWR46JfpcObNfg3ok2JHDUfdiHRgWhJgCQF+KIgOPJY=
Expand Down Expand Up @@ -118,10 +120,13 @@ github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM=
github.com/libp2p/go-buffer-pool v0.1.0 h1:oK4mSFcQz7cTQIfqbe4MIj9gLW+mnanjyFtc6cdF0Y8=
github.com/libp2p/go-buffer-pool v0.1.0/go.mod h1:N+vh8gMqimBzdKkSMVuydVDq+UV5QTWy5HSiZacSbPg=
github.com/libp2p/go-cidranger v1.1.0 h1:ewPN8EZ0dd1LSnrtuwd4709PXVcITVeuwbag38yPW7c=
github.com/libp2p/go-libp2p v0.26.3 h1:6g/psubqwdaBqNNoidbRKSTBEYgaOuKBhHl8Q5tO+PM=
github.com/libp2p/go-libp2p v0.26.3/go.mod h1:x75BN32YbwuY0Awm2Uix4d4KOz+/4piInkp4Wr3yOo8=
github.com/libp2p/go-libp2p-asn-util v0.2.0 h1:rg3+Os8jbnO5DxkC7K/Utdi+DkY3q/d1/1q+8WeNAsw=
github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0=
github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk=
github.com/libp2p/go-libp2p-testing v0.12.0 h1:EPvBb4kKMWO29qP4mZGyhVzUyR25dvfUIK5WDu6iPUA=
github.com/libp2p/go-msgio v0.3.0 h1:mf3Z8B1xcFN314sWX+2vOTShIE0Mmn2TXn3YCUQGNj0=
github.com/libp2p/go-nat v0.1.0 h1:MfVsH6DLcpa04Xr+p8hmVRG4juse0s3J8HyNWYHffXg=
Expand All @@ -148,6 +153,7 @@ github.com/multiformats/go-base36 v0.1.0/go.mod h1:kFGE83c6s80PklsHO9sRn2NCoffoR
github.com/multiformats/go-base36 v0.2.0 h1:lFsAbNOGeKtuKozrtBsAkSVhv1p9D0/qedU9rQyccr0=
github.com/multiformats/go-base36 v0.2.0/go.mod h1:qvnKE++v+2MWCfePClUEjE78Z7P2a1UV0xHgWc0hkp4=
github.com/multiformats/go-multiaddr v0.8.0 h1:aqjksEcqK+iD/Foe1RRFsGZh8+XFiGo7FgUCZlpv3LU=
github.com/multiformats/go-multiaddr v0.8.0/go.mod h1:Fs50eBDWvZu+l3/9S6xAE7ZYj6yhxlvaVZjakWN7xRs=
github.com/multiformats/go-multiaddr-dns v0.3.1 h1:QgQgR+LQVt3NPTjbrLLpsaT2ufAA2y0Mkk+QRVJbW3A=
github.com/multiformats/go-multiaddr-fmt v0.1.0 h1:WLEFClPycPkp4fnIzoFoV9FVd49/eQsuaL3/CWe167E=
github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs=
Expand All @@ -164,6 +170,7 @@ github.com/multiformats/go-multihash v0.0.15/go.mod h1:D6aZrWNLFTV/ynMpKsNtB40mJ
github.com/multiformats/go-multihash v0.2.1 h1:aem8ZT0VA2nCHHk7bPJ1BjUbHNciqZC/d16Vve9l108=
github.com/multiformats/go-multihash v0.2.1/go.mod h1:WxoMcYG85AZVQUyRyo9s4wULvW5qrI9vb2Lt6evduFc=
github.com/multiformats/go-multistream v0.4.1 h1:rFy0Iiyn3YT0asivDUIR05leAdwZq3de4741sbiSdfo=
github.com/multiformats/go-multistream v0.4.1/go.mod h1:Mz5eykRVAjJWckE2U78c6xqdtyNUEhKSM0Lwar2p77Q=
github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/multiformats/go-varint v0.0.7 h1:sWSGR+f/eu5ABZA2ZpYKBILXTTs9JWpdEM/nEGOHFS8=
Expand All @@ -174,6 +181,7 @@ github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 h1:1/WtZae0yGtPq+TI6+
github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9/go.mod h1:x3N5drFsm2uilKKuuYo6LdyD8vZAW55sH/9w+pbo1sw=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/polydawn/refmt v0.0.0-20190221155625-df39d6c2d992/go.mod h1:uIp+gprXxxrWSjjklXD+mN4wed/tMfjMMmN/9+JsA9o=
Expand Down Expand Up @@ -238,7 +246,6 @@ go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ=
go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.1.11-0.20210813005559-691160354723/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
go.uber.org/goleak v1.1.12 h1:gZAh5/EyT/HQwlpkCy6wTpqfH9H8Lz8zbm3dZh+OyzA=
go.uber.org/goleak v1.1.12/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
Expand Down