k6 0.32 requires xk6-output-kafka that requires 0.31

[rspendl]

go: go.k6.io/k6@v0.32.0 requires
github.com/k6io/xk6-output-kafka@v0.2.0 requires
go.k6.io/k6@v0.31.2-0.20210511090412-61f464b99a2d requires
github.com/k6io/xk6-output-kafka@v0.1.2-0.20210510135110-a159d7c8c171: invalid version: unknown revision a159d7c

[mstoykov]

Hi @rspendl,

We recently (the last two days) changed the path module of k6 as part of moving it this, unfortunately, broke in practice all extensions, but xk6-output-kafka has been updated as it is still part of k6 and we import it from this repo for now.

xk6 build --with github.com/k6io/xk6-output-kafka@latest works, I think you might have not updated xk6 to v0.4.1 or are not having @latest at the end which will mean you might get some cached version.

Can you please share some more information about your setup:
what version are you using, how were they installed, OS, what commands you run, etc.

[rspendl]

Hi, these 2 lines from my go.mod are probably relevant:
go.k6.io/k6 v0.32.0
github.com/k6io/xk6 v0.4.1 // indirect

Even this fails:
${GOPATH}/bin/xk6 build --with github.com/k6io/xk6-output-kafka@latest --with go.k6.io/k6@latest
go get: go.k6.io/k6@v0.27.1 updating to
go.k6.io/k6@v0.32.0 requires
github.com/k6io/xk6-output-kafka@v0.2.0 requires
go.k6.io/k6@v0.31.2-0.20210511090412-61f464b99a2d requires
github.com/k6io/xk6-output-kafka@v0.1.2-0.20210510135110-a159d7c8c171: invalid version: unknown revision a159d7c

[rspendl]

xk6 is installed using go get -u github.com/k6io/xk6/cmd/xk6
and the system is Linux, Fedora 34, go version 1.16.3

I've created a minimal go.mod with only "go.k6.io/k6 v0.32.0" in the require block and "go mod vendor" still fails.

[mstoykov]

Please update/install xk6 with go install github.com/k6io/xk6/cmd/xk6@latest or go install github.com/k6io/xk6/cmd/xk6@v0.4.1

I don't understand why you are creating a go.mod file this is what xk6 does for you, so you just need to run xk6 build --github.com/k6io/xk6-output-kafka@latest and that should be sufficient.

[rspendl]

OK, now I logged into a totally empty machine and ran this in an empty folder:
go get github.com/k6io/xk6/cmd/xk6@latest
${GOPATH}/bin/xk6 build --with github.com/k6io/xk6-output-kafka@latest
... and after a lot of "getting", the same error:
go get: go.k6.io/k6@v0.32.0 requires
github.com/k6io/xk6-output-kafka@v0.2.0 requires
go.k6.io/k6@v0.31.2-0.20210511090412-61f464b99a2d requires
github.com/k6io/xk6-output-kafka@v0.1.2-0.20210510135110-a159d7c8c171: invalid version: unknown revision a159d7c

[mstoykov]

I just ran in a completely new docker

(and much later)

so it does work. Are you by any chance having a custom golang cache server or something ?

[mstoykov]

I also just ran with 1.15.11 and 1.14.11 and it did work as well so ... I have no other ideas :(

[rspendl]

Thx, I ran the commands in docker and they work fine, like you've shown. Will see what's off on other machines, thx very much for your help!

[rspendl]

I have the strangest experience with xk6-output-kafka. When I run
go get -v github.com/k6io/xk6-output-kafka@v0.1.2-0.20210510135110-a159d7c8c171
in Docker (image golang:latest, go version 1.15.5) it works

Though: I don't know why it works, as there's not such tag on xk6-output-kafka on github.com

When I run the same command on any machine, not inside docker container, go get fails:
go get -v github.com/k6io/xk6-output-kafka@v0.1.2-0.20210510135110-a159d7c8c171
go: finding github.com/k6io/xk6-output-kafka v0.1.2-0.20210510135110-a159d7c8c171
go get github.com/k6io/xk6-output-kafka@v0.1.2-0.20210510135110-a159d7c8c171: github.com/k6io/xk6-output-kafka@v0.1.2-0.20210510135110-a159d7c8c171: invalid version: unknown revision a159d7c

So, can you create these tags on github.com repository or explain what magic does docker use to get the repository with non-existing tags?

[mstoykov]

Though: I don't know why it works, as there's not such tag on xk6-output-kafka on github.com

This is pseudo-version. As explained there we care about the previous minor version v0.1.1 to exist, to be the ancestor of the commit a159d7c and the timestamp to be of the commit.

It seems though that I managed to reproduce it by setting GONOPROXY=github.com/k6io/xk6-output-kafka.

But I have no fix for that, and I don't know why this is a problem for you? You should be able to build the latest v0.32.0 of k6 with the latest of xk6-output-kafka, and even currently, you don't need to as we still included it in k6. Although maybe we will drop it this release, maybe the next one, so thank you for trying to move 🤗 .

[rspendl]

It looks like my problem was that GOPROXY was not set. Once I've set GOPROXY=proxy.golang.org , k6 finally got all repositories!

[56KBs]

This issue is also affecting us on our internal athens proxy - It is unable to get this version in order to cache it. From checking a number of Go proxies publicly available, only a few of them have the pseudo-version cached (proxy.golang.org being one of them).

Having looked at commit a159d7c8c1713651724621c66c788f7327ba80f3, GitHub is reporting 'This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.'. I have also attempted to check out this SHA locally but I get fatal: reference is not a tree: a159d7c8c1713651724621c66c788f7327ba80f3, this likely is the reason why using no proxy causes it to fail.

As this is a blocker to us updating to the latest k6 version we're having to look into configuring our athens proxy to use proxy.golang.org as an upstream Go module proxy.

@mstoykov is there any likelihood for this to be fixed? - As others could be affected by this too in future.

[mstoykov]

@56KBs looking at the dependency graph this will require us to release a version of k6 after we release yet another version xk6-output-kafka.

I definitely don't want to release a point release of k6 just for this so this particular fix will need to wait until at least v0.33.0 is released at the end of the month.

I do think though that if we ... "ressurect" the commit and put it in a branch this should be fixed?

WDYT @imiric @na--

[mstoykov]

also, I forgot to mention this ... but we might just as well drop the kafka output in v0.33.0 as we are technically planning to drop it either way in the future.

But regardless of that, we should probably update xk6-output-kafka to depend on a version of k6:

1. probably before we release v0.33.0, regardless of the other solution
2. and additionally when we drop it from k6 so it will loose the "circular" dependancy

[na--]

I think we should give at least 2 versions notice before we drop something, so the built-in kafka output in k6 should be dropped in k6 v0.34.0.

Until then, 👍 for any solution that doesn't require a new k6 point version. I'm fine with adding this dangling commit to a branch, but to me, releasing a new xk6-output-kafka version that depends exactly on k6 v0.32.0 seems the simplest?

[mstoykov]

k6 still will depend on a version of k6-output-kafka from before that.

More importantly, v0.32.0 depends on the version of k6-output-kafka that depends on the version of k6 that depends on the commit that is dangling.

So until v0.32.0 will not be used the only ... way to get it done is to undangle the commit. I think ... obviously I could've missed something.

[na--]

Ugh, ok, I didn't quite realize what has happened here... 😱

I wonder, can some xk6 magic with replace directives fix this? I feel like xk6 build v<k6-version> --with github.com/some/xk6-extension@<extension-version>, we should always use only k6-version as the k6 dependency, regardless of what the go.mod file in extension-version says.

Edit: and I guess only extension-version of github.com/some/xk6-extension, regardless of what it says in k6's go.mod file...

[mstoykov]

we should always use only k6-version as the k6 dependency, regardless of what the go.mod file in extension-version says.

but the problem is that in this case xk6-output-kafka is used by k6 internally and the specific version that is said is required (even if then we use one that is "higher") so in the end go needs to still resolve the dependencies down the line.

[mstoykov]

I have undangled it in https://github.com/k6io/xk6-output-kafka/tree/dontDeleteHereForIssue6

@56KBs can you confirm that this now works for you?

[56KBs]

@mstoykov I've just checked and our Athens proxy is now serving the module data correctly - Thanks very much :)

[mstoykov]

Thanks for reporting and checking and I hope we won't have to re-re-revisit this 🤞