Update jsf_data library #60

xyp8023 · 2020-04-01T16:19:30Z

Hi, I added a few things in the jsf_data library, the main idea is to add python interface for jsf_dvl_ping, and match the sound velocity in jsf_dvl_ping to jsf_sss_ping, so that slant range can be calculated. Also a few attributes to jsf_sss_ping are added.

Up to date with Nils' auvlib

1. modified the xtf_reader.c to read Hugin files (copy John's code) 2. printed the slant range as float number 3. set the input to be parsed in test_xtf as non-hard-coded

copied John's code for preprocess here 1. Added find_nadir, normalize, and regularize sss 2. Updated the README.md for preprocess

1. Added sample_interval for jsf_sss_ping 2. Added python interface for jsf_dvl_ping

1. Added altitude, validity flag, sample interval for jsf_sss_ping 2. Added function match_sound_vel to match sound velocity from jsf_dvl_ping to jsf_sss_ping and calculate the max slant range

nilsbore · 2020-04-01T17:43:53Z

I don't know why travis is not showing, but that seems to have worked: https://travis-ci.org/github/nilsbore/auvlib/builds/669798473

nilsbore · 2020-04-01T17:48:18Z

The windows build failed, I'm trying to find out why

xyp8023 · 2020-04-01T18:04:34Z

I found someone says: "Adding the line ADD_COMPILE_FLAG("/D_ENABLE_EXTENDED_ALIGNED_STORAGE") to CMakeLists.txt" could fix it.

xyp8023 · 2020-04-01T18:09:03Z

Oh, that is actually for CMakeLists.txt of VS2017.
WebAssembly/binaryen@e8ac331?w=1#diff-af3b638bc2a3e6c650974192a53c7291

nilsbore · 2020-04-01T18:17:16Z

But I don't see the actual error in the appveyour output, do you? I see that you added some define for aligned storage, what is it used for?

nilsbore · 2020-04-01T18:19:45Z

On another note, I have to review #58 before I can merge this (since this is working from that branch). I don't really grasp all of John's changes yet, so that will take a bit more time. In the future, try to keep new stuff on as clean a branch as possible.

xyp8023 · 2020-04-01T18:20:18Z

The error in the appveuour output is:

[00:16:42] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1271): error C2338: You've instantiated std::aligned_storage<Len, Align> with an extended alignment (in other words, Align > alignof(max_align_t)). Before VS 2017 15.8, the member type would non-conformingly have an alignment of only alignof(max_align_t). VS 2017 15.8 was fixed to handle this correctly, but the fix inherently changes layout and breaks binary compatibility (only for uses of aligned_storage with extended alignments). Please define either (1) _ENABLE_EXTENDED_ALIGNED_STORAGE to acknowledge that you understand this message and that you actually want a type with an extended alignment, or (2) _DISABLE_EXTENDED_ALIGNED_STORAGE to silence this message and get the old non-conformant behavior. [C:\projects\auvlib\build\src\data_tools\jsf_data.vcxproj]

xyp8023 · 2020-04-01T18:23:36Z

On another note, I have to review #58 before I can merge this (since this is working from that branch). I don't really grasp all of John's changes yet, so that will take a bit more time. In the future, try to keep new stuff on as clean a branch as possible.

Right, I forgot to remove John's changes in the this branch. But I did try to use his code to parse the xtf files, it seemed to work.

xyp8023 · 2020-04-01T18:25:17Z

But I don't see the actual error in the appveyour output, do you? I see that you added some define for aligned storage, what is it used for?

The line I added was no good, I misunderstood how other people have fixed it.

nilsbore · 2020-04-01T18:25:21Z

Do you know which change of yours triggered this error with the aligned storage in the first place? I don't see anything that would have changed the behavior of the appveyor build, at least from a quick glance at your code.

xyp8023 · 2020-04-01T18:28:02Z

Do you know which change of yours triggered this error with the aligned storage?

I am not sure, but I would guess it is in the "match_sound_vel" when I try to assign the sound_vel to jsf_sss_ping

nilsbore · 2020-04-01T18:43:42Z

I think it might simply be that you are trying to serialize std::unordered_map with cereal, but there is no include of the cereal serialization function for that container, like here:

auvlib/src/data_tools/include/data_tools/std_data.h

Line 18 in 5b54b9f

#include <cereal/archives/binary.hpp>

. I would suggest trying to add that line (#include <cereal/types/unordered_map.hpp>) in std_data.h. And remove the define you added.

xyp8023 · 2020-04-01T18:44:34Z

#include <cereal/types/unordered_map.hpp>

I will try that.

xyp8023 · 2020-04-01T19:00:15Z

I think it might simply be that you are trying to serialize std::unordered_map with cereal, but there is no include of the cereal serialization function for that container, like here:

auvlib/src/data_tools/include/data_tools/std_data.h

Line 18 in 5b54b9f

#include <cereal/archives/binary.hpp>

. I would suggest trying to add that line (#include <cereal/types/unordered_map.hpp>) in std_data.h. And remove the define you added.

But before this PR, there is std::unordered_map in jsf_dvl_ping already.

nilsbore · 2020-04-01T19:04:41Z

Hmm, ok, there might be something else going on then...

nilsbore · 2020-04-01T19:27:30Z

Yes, the culprit seems to be something else. Let's continue tomorrow.

xyp8023 · 2020-04-01T19:29:41Z

Yes, the culprit seems to be something else. Let's continue tomorrow.

okay

…_ping.flag_ to see what happens

xyp8023 · 2020-04-02T09:37:40Z

The error seems to be in jsf_data.cpp between line 86 and line 448, because there is a warning at line 86 and another warning at line 448, but an error in between.

[00:07:04] jsf_data.cpp
[00:07:04] Unknown compiler version - please run the configure tests and report the results
[00:07:04] cl : Command line warning D9002: ignoring unknown option '-O3' [C:\projects\auvlib\build\src\pybathy_maps\pydraw_map.vcxproj]
[00:07:04] cl : Command line warning D9002: ignoring unknown option '-fPIC' [C:\projects\auvlib\build\src\pybathy_maps\pydraw_map.vcxproj]
[00:07:04] C:\projects\auvlib\src\eigen_cereal\include\eigen_cereal/eigen_cereal.h(17): warning C4244: 'initializing': conversion from '__int64' to 'int32_t', possible loss of data [C:\projects\auvlib\build\src\pydata_tools\pyall_data.vcxproj]
[00:07:04] C:\projects\auvlib\src\data_tools\src\jsf_data.cpp(50): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data [C:\projects\auvlib\build\src\data_tools\jsf_data.vcxproj]
[00:07:05] pydraw_map.cpp
[00:07:05] C:\projects\auvlib\src\data_tools\src\jsf_data.cpp(51): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data [C:\projects\auvlib\build\src\data_tools\jsf_data.vcxproj]
[00:07:05] C:\projects\auvlib\src\data_tools\src\jsf_data.cpp(56): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data [C:\projects\auvlib\build\src\data_tools\jsf_data.vcxproj]
[00:07:05] C:\projects\auvlib\src\cereal\include\cereal/cereal.hpp(505): note: see reference to function template instantiation 'void cereal::save<AA,double,3,1,0,3,1>(Archive &,const Eigen::Matrix<double,3,1,0,3,1> &)' being compiled
[00:07:05] with
[00:07:05] [
[00:07:05] AA=cereal::BinaryOutputArchive,
[00:07:05] Archive=cereal::BinaryOutputArchive
[00:07:05] ]
[00:07:05] C:\projects\auvlib\src\cereal\include\cereal/cereal.hpp(417): note: see reference to function template instantiation 'ArchiveType &cereal::OutputArchive<ArchiveType,1>::processImpl<Eigen::Matrix<double,3,1,0,3,1>,0>(const T &)' being compiled
[00:07:05] with
[00:07:05] [
[00:07:05] ArchiveType=cereal::BinaryOutputArchive,
[00:07:05] T=Eigen::Matrix<double,3,1,0,3,1>
[00:07:05] C:\projects\auvlib\src\data_tools\src\jsf_data.cpp(63): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data [C:\projects\auvlib\build\src\data_tools\jsf_data.vcxproj]
[00:07:05] ]
[00:07:05] C:\projects\auvlib\src\data_tools\src\jsf_data.cpp(86): warning C4018: '<': signed/unsigned mismatch [C:\projects\auvlib\build\src\data_tools\jsf_data.vcxproj]
[00:07:05] C:\projects\auvlib\src\cereal\include\cereal/cereal.hpp(417): note: see reference to function template instantiation 'ArchiveType &cereal::OutputArchive<ArchiveType,1>::processImpl<Eigen::Matrix<double,3,1,0,3,1>,0>(const T &)' being compiled
[00:07:05] with
[00:07:05] [
[00:07:05] ArchiveType=cereal::BinaryOutputArchive,
[00:07:05] T=Eigen::Matrix<double,3,1,0,3,1>
[00:07:05] ]
[00:07:05] C:\projects\auvlib\src\cereal\include\cereal/cereal.hpp(311): note: see reference to function template instantiation 'void cereal::OutputArchivecereal::BinaryOutputArchive,1::process<const Eigen::Matrix<double,3,1,0,3,1>&>(T)' being compiled
[00:07:05] with
[00:07:05] [
[00:07:05] T=const Eigen::Matrix<double,3,1,0,3,1> &
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1271): error C2338: You've instantiated std::aligned_storage<Len, Align> with an extended alignment (in other words, Align > alignof(max_align_t)). Before VS 2017 15.8, the member type would non-conformingly have an alignment of only alignof(max_align_t). VS 2017 15.8 was fixed to handle this correctly, but the fix inherently changes layout and breaks binary compatibility (only for uses of aligned_storage with extended alignments). Please define either (1) _ENABLE_EXTENDED_ALIGNED_STORAGE to acknowledge that you understand this message and that you actually want a type with an extended alignment, or (2) _DISABLE_EXTENDED_ALIGNED_STORAGE to silence this message and get the old non-conformant behavior. [C:\projects\auvlib\build\src\data_tools\jsf_data.vcxproj]
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1291): note: see reference to class template instantiation 'std::_Aligned<272,16,double,false>' being compiled
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1298): note: see reference to class template instantiation 'std::_Aligned<272,16,int,false>' being compiled
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1305): note: see reference to class template instantiation 'std::_Aligned<272,16,short,false>' being compiled
[00:07:05] ]
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1312): note: see reference to class template instantiation 'std::_Aligned<272,16,char,false>' being compiled
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1320): note: see reference to class template instantiation 'std::aligned_storage<272,16>' being compiled
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1352): note: see reference to alias template instantiation 'std::aligned_storage_t<272,16>' being compiled
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\type_traits(1357): note: see reference to class template instantiation 'std::aligned_union<0,_Ty>' being compiled
[00:07:05] with
[00:07:05] [
[00:07:05] _Ty=jsf_data::jsf_dvl_ping
[00:07:05] ]
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\algorithm(79): note: see reference to alias template instantiation 'std::aligned_union_t<0,jsf_data::jsf_dvl_ping>' being compiled
[00:07:05] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include\algorithm(4274): note: see reference to class template instantiation 'std::_Optimistic_temporary_buffer<jsf_data::jsf_dvl_ping>' being compiled
[00:07:05] C:\projects\auvlib\src\data_tools\src\jsf_data.cpp(448): note: see reference to function template instantiation 'void std::stable_sort<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<_Ty>>>,jsf_data::match_sound_vel::<lambda_282701942c233423c13c9ddf612ca1a3>>(_BidIt,_BidIt,_Pr)' being compiled
[00:07:05] with

nilsbore · 2020-08-10T14:42:07Z

Closing in favour of #62

xyp8023 added 8 commits March 18, 2020 14:13

Merge pull request #1 from nilsbore/master

693121d

Up to date with Nils' auvlib

Modified the xtf reader to read Hugin files

63eee26

1. modified the xtf_reader.c to read Hugin files (copy John's code) 2. printed the slant range as float number 3. set the input to be parsed in test_xtf as non-hard-coded

Added preprocess for xtf_pings

5af4339

copied John's code for preprocess here 1. Added find_nadir, normalize, and regularize sss 2. Updated the README.md for preprocess

reset to the commit without preprocess

43e74d0

Added small things for jsf_data library

025a9b6

1. Added sample_interval for jsf_sss_ping 2. Added python interface for jsf_dvl_ping

Added match_sound_vel for jsf_data

813c984

1. Added altitude, validity flag, sample interval for jsf_sss_ping 2. Added function match_sound_vel to match sound velocity from jsf_dvl_ping to jsf_sss_ping and calculate the max slant range

Updated docs

d6bc0fc

trying to fix the windows compiling error

5b54b9f

try to fix the windows building error again

e106c9b

try to fix the windows building error still: do not serialize jsf_sss…

15287f7

…_ping.flag_ to see what happens

nilsbore mentioned this pull request Aug 10, 2020

Only jsf parts of #60 #62

Open

nilsbore closed this Aug 10, 2020

nilsbore mentioned this pull request Aug 26, 2020

Add John's code in a way that can be merged with no conflicts #58

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update jsf_data library #60

Update jsf_data library #60

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020

nilsbore commented Apr 1, 2020 •

edited

Loading

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020

nilsbore commented Apr 1, 2020

xyp8023 commented Apr 1, 2020 •

edited by nilsbore

Loading

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020 •

edited

Loading

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020 •

edited

Loading

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020

nilsbore commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 2, 2020

nilsbore commented Aug 10, 2020

Update jsf_data library #60

Update jsf_data library #60

Conversation

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020

nilsbore commented Apr 1, 2020 • edited Loading

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020

nilsbore commented Apr 1, 2020

xyp8023 commented Apr 1, 2020 • edited by nilsbore Loading

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020 • edited Loading

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020 • edited Loading

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

nilsbore commented Apr 1, 2020

nilsbore commented Apr 1, 2020

xyp8023 commented Apr 1, 2020

xyp8023 commented Apr 2, 2020

nilsbore commented Aug 10, 2020

nilsbore commented Apr 1, 2020 •

edited

Loading

xyp8023 commented Apr 1, 2020 •

edited by nilsbore

Loading

nilsbore commented Apr 1, 2020 •

edited

Loading

nilsbore commented Apr 1, 2020 •

edited

Loading