how to work with sonar data in sl2 file

[NicholasBallard]

Hi Peter, thanks for your work on SL2 files! Your work has helped me so much. I am working on mapping lakes in New York State with a Lowrance Elite 7 Ti fish finder. Using your library am able to decode the binary for the records like waterDepth, latitude, longitude, etc. My question is, how do I work with the actual imaging data, not just the log but the sonar it pings back. I want to work with the data in Python to determine underwater plant height and density.

Have been unable to find an answer where to start with this. Your library has been so helpful I figure you are my best chance!

[kmpm]

The image data is in there somehow but I haven't figured out how it's encoded.

That project of yours sounds interesting, any source code?
I haven't been able to find a good process to generate the maps, have you?

[NicholasBallard]

I have figured out how to decode the binary data. Each echogram can be a different number of bytes, the log says how large each datagram is. Still working on the echogram data, going to try Fourier Transformations and radial intensity mapping. When I have working code I will share it. Right now I have a matplotlib graph of the echograms but need a 3D image. Next step is to use numpy.memmap for I hope a faster way to decode the SL2 files.

[Seva-coder]

Hi Nicholas! I have used Lowrance elite 4 for recording logs, it hasn't StructureScan feature (unlike Elite 7) but otherwise they are similar (frequencies, DownScan Imaging). I think this difference may help to determine functions of parts in binary sl2.
I can give an example of such log if you need.

ps. interesting project!

[NicholasBallard]

Hi Nikita! Yes please send over the file (is it SLG?), and I'll parse it and give it back. There's some fields I'm not sure the value of. Maybe we can puzzle it out together.

[Seva-coder]

https://drive.google.com/file/d/1VRQPXB_x7oVtFTP38mxU5_XZd9Vr7eMO/view?usp=sharing
~20 MB
I have checked it (opened in SonarViewer from lowrance) - looks normal.
In addition to the coordinates and "image" log contain temperature value (at least DrDepth 4 can show it).
Say if you need more data (such tracks have rather big size, up to 500 MB)

[mbenson182]

Hi all,

I don't know if anyone will ever look at this or find it useful, but I've been working on my own C++ implementation of this parsing method and wanted to chip in for anyone that might be looking to parse the sounding data for themselves. But first, let me thank kmpm for his work on this project, as he did all the heavy lifting finding the offsets, I merely adapted the work to C++ and did some extra work to get the sounding data.

This probably varies based on model and channel, but at least for the sidescan data (channel = 5) on my unit, the data in each block is a collection of packetsize (i.e. data at 0x34 in each block) bytes, in order. Each byte corresponds to one sonar return, mapped to intensity values with 255=maximum intensity and 0=minimum intensity. In order to use the data, all you have to do is read the continuous chunk of memory (which, in the case of the mixed sidescan data, is 2800 bytes, 1400 from each side), then cast it to whatever type is necessary for each byte (in my case, this was a conversion to a uchar, which was fed into an OpenCV Mat to generate the echogram).

I haven't looked at the other channels as, quite frankly, they're not of much interest to the work I'm doing, but from looking at the raw data it seems likely the data from all channels are bytepacked in a similar way.

Your mileage may vary depending on what model/channel/firmware you're using, but I hope this might help someone!

[Seva-coder]

@mbenson182 thanks, good job! it can really help

[kmpm]

Hi all.... I'v set up an organization in github called https://github.com/opensounder where I aim to collect all this kind of information, tools libraries as well as links to various other implementations.
Not much there at the moment but please add to it and I would really welcome some additional admins.

[NicholasBallard]

Looks good, I'd like to contribute code for decoding Lowrance binaries.

…

On Tue, Jun 18, 2019 at 11:47 AM Peter Magnusson ***@***.***> wrote: Hi all.... I'v set up an organization in github called https://github.com/opensounder where I aim to collect all this kind of information, tools libraries as well as links to various other implementations. Not much there at the moment but please add to it and I would really welcome some additional admins. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3?email_source=notifications&email_token=ABDAIOEQA3HV27YFHLSIKVDP3D7Q3A5CNFSM4G3KNJF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX7CPMI#issuecomment-503195569>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDAIOBVJOCJ3GVLUHJGTLTP3D7Q3ANCNFSM4G3KNJFQ> .

[kmpm]

@NicholasBallard, you already have contributed :) I linked to your code.
But I will eventually remove the documentation from this repo and link to the new one and hopefully have people contribute to the documentation there.

[Trout55]

Hello to all,

At the beginning, thank you all for work and sharing information related to the sl2 and sl3 logs.
I have a question to you, if there will be someone who can do a guide for a complete layman, how to use Python and libraries to extract coordinates from the sl2/sl3 logs? . I am an angler and land surveyor, I have no problem with creating a 3D bottom image but unfortunately I have no idea about programming. I can't dismantle Lowrance logs to get coordinates from them. I am even willing to pay for such a guide, of course my request is not commercial, only for my own fishing use.

Thank you in advance
Best regards
Slawek from Poland

[NicholasBallard]

Hi Slawek!

If you are looking to just map the coordinates, you're in luck. A Lowrance unit I believe can output just the coordinates to a csv file alongside the SL2/SL3 binary file.

Check this gist here: https://gist.github.com/NicholasBallard/8022ff8f3894c674873953b50e0f979c

[NicholasBallard]

Slawek, if you need to read the SL2 or SL3 binary files t0o, I can help with that. Have mapped the frames and headers of the binary file (for the most part) and definitely know how to get the coordinates. It is quite technical dealing with binary. I have some functions built to traverse the SL2 file, but haven't packaged it into an easy-to-use program.

The main thing is sorting out the sounding data which has a variable byte length, from the sensor data (like the GPS coordinates).

From there can map the coordinates to a satellite map. I like using smopy to lay the sounder's coordinates over an OpenStreetMap image.

[Trout55]

Hi Nicholas!

thank you very much for the information and willingness to help. As you know, the Lowrance .sl2 file contains GPS coordinates and the depth only under the transducer, only in one line, point by point. I am looking for the ability to read coordinates from .sl3 where the StructureScan 3D module was used, there are probably saved coordinates not only under the transducer, but also from the Side Imaging profiles. The HDS Live fishfinder can draw a 3D bottom image based on the .sl3 file (even from single track). Unfortunately, this can only be done in the fishfinder, there is no viewer or external program to do this. I am looking for a way to extract StructureScan 3D measurement data from .sl3 and draw the 3D bottom surfaceI and draw the 3D bottom surface in an external application (for example Surfer Goldensoftware ).

Thank you very much
Slawek

[NicholasBallard]

Okay so there's a difference between mapping the GPS coordinates (which you are right, are sent in each frame of the binary file for all pieces of hardware on the equipment -- the side scans left and right and the downscan sounder).

If you are looking to build a 3D mapping of the bottom, for that you will need something to build the image from the ping data packaged in each frame of the binary for each sounder (down/sidescan). There are software products (paid) like ReefMaster which will work with Lowrance file types. Companies like BioBase are doing the same thing and have a decent web app. But again, they charge money, and you probably want to do do something with the mapping data in your own application.

What it's doing is inferring a cone image with some signal processing algorithms. This is beyond what I've successfully done with an SL3 file. There are a few wizards out there who may be able to point you in the right direction as to libraries you can use (they will be mostly written in C or C++).

Kurt Schwehr (or his GitHub) is the number one expert who comes to mind for handling sonar data.

[kmpm]

@Trout55 Hi,
and hi all others as well.

There is a python library in the works at https://github.com/opensounder/python-sllib but it's far from complete. This will help you to extract the information from the file but as @NicholasBallard mentions, it might be a lot harder than that.
I do have it on my wish list, to create a decent 3D map using downscan at least but "life" tends to get in the way :)

[stephenj079]

Hello,
Many thanks to the above authors for their code to read sl2 files. I have been trying to extract the data from the packet, to produce a cross-section of the echogram. Using sllib, I have modified the example from the python-sllib GitHub to output the echogram as a numpy array:

import struct
with open(filename, 'rb') as f:
    reader = sllib.Reader(f)
    header = reader.header
    print(header.format)
    for frame in reader:
        x = struct.unpack("i" * (frame.packetsize // 4), frame.packet)
        x = np.array(x)
        try:
            data = np.vstack((data, x))
        except NameError:
            data = x

Then the section can be displayed using:

import matplotlib.pyplot as plt
plt.imshow(data.T)

However, this is pretty much a hack, I have no idea what is really going on here. The 'amplitude' values that come out are huge (on order of 1 x 10e9). Also, I am unsure what the vertical axis represents. I presume it is some factor of sampling rate in two-way-time, but I don't know for sure. Does anyone have any idea how to convert the vertical axis to depth? I presume there is a factor based on an assumed water velocity.