Error when running livekit-load-tester on v0.8.0

[stevenhansel]

I was trying to run this command:

livekit-load-tester \
       --url wss://wss.example.com \
       --api-key my_api_key \
       --api-secret my_api_secret \
       --duration 1m \
       --publishers 1 \
       --subscribers 5

Then, I was getting this error:

Starting load test with 1 publishers, 5 subscribers, room:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x6 pc=0xa8172e]

goroutine 572 [running]:
github.com/livekit/livekit-cli/pkg/provider.(*H264VideoLooper).nextSample(0xc000335b00, 0x1)
        /root/go/pkg/mod/github.com/livekit/livekit-cli@v0.8.0/pkg/provider/h264looper.go:77 +0x38e
github.com/livekit/livekit-cli/pkg/provider.(*H264VideoLooper).NextSample(0xc000335b00?)
        /root/go/pkg/mod/github.com/livekit/livekit-cli@v0.8.0/pkg/provider/h264looper.go:54 +0x48
github.com/livekit/server-sdk-go.(*LocalSampleTrack).writeWorker(0xc0003cfad0, {0x7fcfd9be30c0, 0xc000335b00}, 0x0)
        /root/go/pkg/mod/github.com/livekit/server-sdk-go@v0.10.0/localsampletrack.go:395 +0x1c7
created by github.com/livekit/server-sdk-go.(*LocalSampleTrack).Bind
        /root/go/pkg/mod/github.com/livekit/server-sdk-go@v0.10.0/localsampletrack.go:185 +0x57b

My current solution is to just downgrade to v0.7.2
go install github.com/livekit/livekit-cli/cmd/livekit-load-tester@v0.7.2

[davidzhao]

@stevenhansel

I believe this is due to interaction between go install and Git LFS. For now, can you try the following:

1. ensure git lfs is installed
2. clone the repo
3. run make install

[stevenhansel]

I just tried installing git lfs, and running make install but it seems that the error still persists. I actually tried debugging the CLI and here's what I found:

I was curious to see what the error contains in h264looper.go

func (l *H264VideoLooper) nextSample(rewindEOF bool) (media.Sample, error) {
	sample := media.Sample{}
	if l.reader == nil {
		var err error
		l.reader, err = h264reader.NewReader(bytes.NewReader(l.buffer))
		if err != nil {
			return sample, err
		}
	}
	nal, err := l.reader.NextNAL()
	fmt.Println("err: ", err)
	if err == io.EOF && rewindEOF {
		l.reader = nil
		return l.nextSample(false)
	}

So i added a simple println log, and the error was: err: data is not a H264 bitstream.

I pinpoint the location of the code for method NextNAL(), which is here, and it seems the error happens when it's calling bitStreamStartsWithH264Prefix(), and the bitstream prefix is not 3 or 4.

@davidzhao Do you mind if i take on this issue and investigate further and hopefully creating a PR to fix this? I would be really happy to contribute to this project!

[davidzhao]

Absolutely! a PR sounds great!

Could you check how large the files are inside pkg/provider/resources. LFS might place reference files in place of actual video files, which would cause H264 parser to fail.

[stevenhansel]

Hello @davidzhao , it seems that the problem comes from LFS itself. After doing git-lfs pull on the repository, then the reference files inside pkg/provider/resources are replaced immediately with the real one.

I think to fix the issue in v0.8.0, what we need to do is that in .github/workflows/release.yml, we have to dereference the lfs files so in the golang build it includes the real files. In the github actions checkout docs, there is a parameter to exactly do that:

    # Whether to download Git-LFS files
    # Default: false
    lfs: ''

But, there's also an ongoing issue, where you actually have to also run the pull command afterwards:

                  - name: checkout
                    uses: actions/checkout@v2
                    with:
                        lfs: 'true'
                  - name: checkoutLFS
                    uses: actions/checkout@v2
                  - run: git lfs pull

What do you think? If you think it's okay, I will try to test the CI first and then open a PR once I get a working example

[davidzhao]

this looks good! I think go install would still fail the same way? I'll need to verify that path also works.

[stevenhansel]

this looks good! I think go install would still fail the same way? I'll need to verify that path also works.

If the person is running go install locally and haven't run git-lfs pull yet, I think it will still fail. We can always add a documentation to tell people who are going to run this repo locally to run git-lfs pull, or we can also automate this by either creating a git hook or creating a script that runs before building or installing the cmd

[davidzhao]

Released 0.8.1 and updated readme to manually download repo instead of relying on go install. thank you for fixing it