Kongsberg M3 Support

[rkeen9]

The Kongsberg M3 MBES exports data in the same .ALL format used by the EM series sonars (.ALL format for M3: https://www.mesotech.online/manuals/92220007051_m3_all_data_format_us_en.pdf). When importing data using intel_process, I've been getting
ValueError: Unable to find sonar startbyte, is this sonar supported?
Would Kluster be able to support this sonar?

Sample .ALL file: https://drive.google.com/file/d/179bnsFsUsKEgaB5TCQZsxWJm-agzk2wx/view?usp=sharing

[ericgyounkin]

Have two PRs that should resolve this:
#117
noaa-ocs-hydrography/drivers#19

I will say that the sample data you have here did not look great in Kluster. I saw missing navigation for most of the file, I believe because there was only a single nav record in the file. Hopefully that is not a result of our file reading.

[rkeen9]

Have two PRs that should resolve this: #117 noaa-ocs-hydrography/drivers#19

I will say that the sample data you have here did not look great in Kluster. I saw missing navigation for most of the file, I believe because there was only a single nav record in the file. Hopefully that is not a result of our file reading.

Thanks for the updates! I tried out the add_m3_support and all_m3_support branches and the data imports doesn't give the same startbyte error but as you mentioned, there is a lot of missing data. The .all file was exported while the sonar was indoors on a bench facing the ceiling so it makes sense the data isn't the best quality. I will have to retest once field data is available.

[rkeen9]

I've been trying to extract point cloud data from the sample .all file using Kluster but no luck so far. To see whether the .all file has the necessary data, I put it into Eiva's NaviModel Viewer and was able to get this image:

From this, it looks like the file has the data necessary to generate a pointcloud but I have yet to be able to reproduce the same results using Kluster.

My attempts at extracting data have been to do the following:

from HSTB.kluster.fqpr_intelligence import intel_process
_, fq = intel_process(r'C:\KML\M3_V0254\Recordings\2024_Jan_10_19-56-12.all')
fq = fq[0]

And this results in the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 2
      1 from HSTB.kluster.fqpr_intelligence import intel_process
----> 2 _, fq = intel_process(r'C:\KML\M3_V0254\Recordings\2024_Jan_10_19-56-12.all')
      3 fq = fq[0]

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_intelligence.py:1996, in intel_process(filname, outfold, coord_system, epsg, use_epsg, vert_ref, parallel_write, vdatum_directory, force_coordinate_system, cast_selection_method, designated_surface, process_mode, logger, client, skip_dask)
   1994             print(e)
   1995 while intel.has_actions:
-> 1996     intel.execute_action()
   1997 return intel, list(intel.project.fqpr_instances.values())

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_intelligence.py:1163, in FqprIntel.execute_action(self, idx)
   1161     self.parent.kluster_execute_action(self.action_container, 0)
   1162 else:
-> 1163     output = self.action_container.execute_action(idx)
   1164     if isinstance(output, Fqpr):  # if the output is fqpr data
   1165         self.project.add_fqpr(output)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_actions.py:276, in FqprActionContainer.execute_action(self, idx)
    274 self._update_actions()
    275 if action:
--> 276     action.execute()
    277     output = action.output
    278     return output

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_actions.py:52, in FqprAction.execute(self)
     50 self.is_running = True
     51 if self.kwargs:
---> 52     self.output = self.function(*self.args, **self.kwargs)
     53 else:
     54     self.output = self.function(*self.args)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_convenience.py:369, in process_multibeam(fqpr_inst, run_orientation, orientation_initial_interpolation, run_beam_vec, run_svcorr, run_georef, run_tpu, add_cast_files, input_datum, use_epsg, use_coord, epsg, coord_system, vert_ref, vdatum_directory, cast_selection_method, only_this_line, only_these_times)
    367     fqpr_inst.get_beam_pointing_vectors(subset_time=subset_time)
    368 if run_svcorr:
--> 369     fqpr_inst.sv_correct(add_cast_files=add_cast_files, cast_selection_method=cast_selection_method, subset_time=subset_time)
    370 if run_georef:
    371     fqpr_inst.georef_xyz(vdatum_directory=vdatum_directory, subset_time=subset_time)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:2884, in Fqpr.sv_correct(self, add_cast_files, cast_selection_method, subset_time, dump_data)
   2882 idx_by_chunk = self.return_chunk_indices(applicable_index, pings_per_chunk)
   2883 if len(idx_by_chunk[0]):  # if there are pings in this system that align with this installation parameter record
-> 2884     self._submit_data_to_cluster(ra, 'sv_corr', idx_by_chunk, max_chunks_at_a_time,
   2885                                  timestmp, prefixes, dump_data=dump_data, skip_dask=skip_dask,
   2886                                  cast_selection_method=cast_selection_method)
   2887 else:
   2888     if dump_data:

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:3276, in Fqpr._submit_data_to_cluster(self, rawping, mode, idx_by_chunk, max_chunks_at_a_time, timestmp, prefixes, dump_data, skip_dask, prefer_pp_nav, vdatum_directory, cast_selection_method, backscatter_settings)
   3274 if self.show_progress and rn != 0:  # first run we skip progress as it prints out the run info
   3275     print_progress_bar(rn + 1, tot_runs, prefix=f'Loading chunk    {rn + 1}/{tot_runs}:')
-> 3276 data_for_workers = chunk_function(*chunkargs, silent=silent)
   3277 try:
   3278     self.debug_print(f'Running {mode} process...', logging.INFO)

File ~\mambaforge\envs\m3_test\lib\site-packages\HSTB\kluster\fqpr_generation.py:1594, in Fqpr._generate_chunks_svcorr(self, ra, cast_chunks, casts, prefixes, timestmp, addtl_offsets, run_index, silent)
   1592 if any(c[1] is None for c in cast_chunks):
   1593     self.print('Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?', logging.ERROR)
-> 1594     raise ValueError('Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?')
   1596 # this should be the transducer to waterline, positive down
   1597 refpt = self.multibeam.return_prefix_for_rp()

ValueError: Unable to sound velocity correct, one of the data chunks was unable to find a cast, have you imported a cast yet?

Seems to have an issue with the sound velocity. In this test setup, the sound velocity was set to a fixed value of 1469 m/s.

[ericgyounkin]

Kluster requires a sound velocity profile. This is generally provided in the .all/.kmall file logged through SIS. I am going to guess that the software you used to get the point cloud above skips ray-tracing the corrected beam vectors, or just uses the surface sound velocity value that you mention.

You can import a surface sound velocity profile into Kluster, to get past the correct for sound velocity action. The easiest way to do that is to use Hydroffice (https://www.hydroffice.org/soundspeed/) and generate a synthetic cast, using the WOA. Export the cast to .svp, and drag it into Kluster, which should get you the right action.

https://kluster.readthedocs.io/en/latest/processing/index.html#sound-velocity

[barry-gallagher]

There is also a menu item for WOA casts which uses Sound Speed Manager (SSM) for generating the cast. I would suggest just installing SSM and making a cast this time but it is something to keep in mind if this would be a regular part of your workflow.