Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test suite error on mips64el architecture #1469

Open
2 tasks done
tillea opened this issue Sep 17, 2020 · 70 comments
Open
2 tasks done

Test suite error on mips64el architecture #1469

tillea opened this issue Sep 17, 2020 · 70 comments
Labels
todo Triaged for implementation in some unspecified future version
Milestone
next

Comments

@tillea
Copy link

tillea commented Sep 17, 2020

  • This issue is for the C core of igraph.
  • This issue is a bug report or a feature request, not a support question.

Hi,
the Debian packaged version of igraph 0.8.2 has a build issue on mips64el architecture (and only on this architecture). You can find a full build log here. I recommend seeking for the string

224: Eigenvector centrality (igraph_eigenvector_centrality): FAILED (arpack.at:59)

in this longish build log. From my previous experience with igraph issues I know you usually are asking for the content of tests/testsuite.dir. If you need it in this case as well I could try to find a mips64el machine and fetch those information from that build but I have no way to access this "easily" for the moment.
Kind regards, Andreas.
@szhorvat
Copy link
Member

szhorvat commented Sep 17, 2020 •
edited

Since we don't have access to this architecture (Travis does not support it), would you be able to try the latest master branch? The test suite has been improved since 0.8.2, so this problem may already be fixed.

We are preparing for a 0.8.3 release, so this would be the perfect time to fix any remaining problems.

@tillea
Copy link
Author

tillea commented Sep 17, 2020 via email

@szhorvat
Copy link
Member

szhorvat commented Sep 17, 2020 •
edited

I think if you have good reasons to assume that 0.8.3 will fix the problem it would be the easiest (=less time consuming way) on my side to simply try 0.8.3 and see what happens on all Debian architectures.

I just double-checked, and unfortunately I could not find any changes that could have fixed this (though I may have missed some).

This is likely a numerical precision issue (i.e. more of a problem with the test than the library). If you can retrieve tests/testsuite.dir, it might give a hint about what we could change to make the problem go away.

@szhorvat
Copy link
Member

I do have one idea about how to improve this test but it would still need to be tested ... there's no guarantees that it fixes this issue.

@tillea
Copy link
Author

tillea commented Sep 18, 2020 via email

@szhorvat
Copy link
Member

szhorvat commented Sep 18, 2020 •
edited

Thank you so much for the help with this Andreas! There is no rush, if we don't fix it for 0.8.3, we'll fix it for the next release. I did spend some more time yesterday thinking about what may be going wrong, and I am actually quite puzzled: it seems unlikely that a numerical precision issue could cause this failure.

Still, there is something that doesn't quite look right in the test. Normalization (the scale argument) should be turned on to ensure consistent results, in theory. However, looking at how the function works, I don't believe that it was this that caused the failure on MPIS. However, I'll change that for 0.8.3.

@szhorvat
Copy link
Member

Just noting that the issue does not seem to be fully reproducible according to this message on the Debian-MIPS mailing list:

https://lists.debian.org/debian-mips/2020/09/msg00009.html

@tillea
Copy link
Author

tillea commented Sep 23, 2020 via email

@szhorvat szhorvat reopened this Oct 1, 2020
@szhorvat
Copy link
Member

szhorvat commented Oct 3, 2020

@tillea 0.8.3 is out now. When you have time, let us now how it fares on MIPS.

@tillea
Copy link
Author

tillea commented Oct 5, 2020 via email

@szhorvat
Copy link
Member

szhorvat commented Oct 5, 2020

I guess you might really need tests/testsuite.log now, right?

Yes, that would be quite useful. But there is no hurry.

@szhorvat
Copy link
Member

@tillea Did you manage to retrieve the testsuite.log directory?

@tillea
Copy link
Author

tillea commented Oct 23, 2020 via email

@szhorvat szhorvat added this to the 0.9.0 milestone Oct 26, 2020
@szhorvat
Copy link
Member

szhorvat commented Nov 11, 2020 •
edited

Just another small reminder @tillea :-)

If you don't have time now, I suggest closing this issue for now. MIPS seems to be a mostly defunct architecture, the issue is only intermittently reproducible, and we don't have a MIPS computer to test on. It seems it may not be worth the effort either on our or on your part. Still, if you can retrieve the logs, I'd like to take a look, in case they reveal a hidden issue that may affect other platforms too.

In the meantime, a lot of housecleaning has been done for igraph. If we're lucky, the next version (0.9) won't have this problem. If it does, the issue can be re-opened at that time.

@tillea
Copy link
Author

tillea commented Nov 16, 2020 via email

@tillea
Copy link
Author

tillea commented Nov 27, 2020

tests.zip
I've finally managed to reproduce the issue on a mips64el machine. Here are the logs.

@szhorvat
Copy link
Member

Thanks @tillea! That result is super-weird: some of the results are zeros when they shouldn't be. Something is definitely broken, but this is going to be very hard to debug without access to a machine where the issue is reproducible.

I also wonder if the issue is in igraph, or in ARPACK (which igraph uses for this specific calculation).

@szhorvat
Copy link
Member

szhorvat commented Nov 27, 2020 •
edited

Here's the diff so others don't have to download the archive:

#                             -*- compilation -*-
228. arpack.at:57: testing Eigenvector centrality (igraph_eigenvector_centrality): ...
./arpack.at:59: ${CC} ${CFLAGS} ${abs_top_srcdir}/examples/simple/eigenvector_centrality.c -I${abs_top_srcdir}/include -I${abs_top_builddir}/include -L${abs_top_builddir}/src/.libs -ligraph -lm  -o itest
./arpack.at:59: cat ${abs_top_srcdir}/examples/'simple/eigenvector_centrality.out' | sed "s/@VERSION@/$(cat ${abs_top_srcdir}/IGRAPH_VERSION)/g" > expout
./arpack.at:59: 
./arpack.at:59: DYLD_LIBRARY_PATH=${abs_top_builddir}/src/.libs${DYLD_LIBRARY_PATH+:$DYLD_LIBRARY_PATH} LD_LIBRARY_PATH=${abs_top_builddir}/src/.libs${LD_LIBRARY_PATH+:$LD_LIBRARY_PATH} ./itest ${SED_PIPE_CRLF2LF}
--- expout	2020-11-27 18:01:46.065124982 +0000
+++ /home/tille/igraph-0.8.3+ds/tests/testsuite.dir/at-groups/228/stdout	2020-11-27 18:01:46.113125752 +0000
@@ -1,3 +1,3 @@
- 1.0000 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005
+ 1.0000 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005 0.1005
  1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
  1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
228. arpack.at:57: 228. Eigenvector centrality (igraph_eigenvector_centrality): (arpack.at:57): FAILED (arpack.at:59)

@tillea
Copy link
Author

tillea commented Nov 27, 2020 via email

@ntamas
Copy link
Member

ntamas commented Dec 2, 2020

Some update: I managed to boot a mips64el Debian image with QEMU (thanks to these images) so I'll look into this soon.

@jgmbenoit
Copy link
Contributor

Apparently the issue is fixed in version 0.8.5.

@jgmbenoit
Copy link
Contributor

May be I was to fast: I confused ppc64el with mips64el . My mistake. Let see.

@szhorvat
Copy link
Member

No related code was touched in 0.8.5. I suspect that this isn't even an igraph problem, but an ARPACK one.

@jgmbenoit
Copy link
Contributor

igraph version 0.8.5 built well on mips64el arch: https://buildd.debian.org/status/package.php?p=igraph&suite=sid
So I guess that this issue can be closed.

@ntamas
Copy link
Member

ntamas commented Dec 14, 2020

This is consistent with what I've seen on my machine in a QEMU mips64el instance. Closing it for now; feel free to reopen if the issue resurfaces.

@ntamas ntamas closed this as completed Dec 14, 2020
@jgmbenoit
Copy link
Contributor

Actually the issue is still there. The test randomly failed.

@jgmbenoit
Copy link
Contributor

I finally isolated the place where the issue emerges. It is indeed arpack issue.

In src/arpack.c in function igraph_arpack_rssolve the dsaupd_ loop works fine
in the sense that the loop keeps giving the same output when the random seed is fixed.
The issue emerges at the postamble of the loop, dseupd_ gives random output in the same conditions.
I will submit a bugreport to the debian maintainer of arpack

Meanwhile the question is how to neutralize this issue.
Given that, are you still willing to implement a --without-scg option ?

@tillea
Copy link
Author

tillea commented Jan 29, 2021 via email

@jgmbenoit
Copy link
Contributor

I can extract a C code that reproduces the issue, but not a patch as I am not familiar with the arpack library.

@jgmbenoit
Copy link
Contributor

A patch for arpack is the best path. I will give it a try.

@jgmbenoit
Copy link
Contributor

Meanwhile, I have noticed #if HAVE_GFORTRAN ... #else ... #endif in src/arpack.c but I could not figure out where HAVE_GFORTRAN is acually set. Is it dead code ?

@tillea
Copy link
Author

tillea commented Jan 30, 2021 via email

@szhorvat
Copy link
Member

Meanwhile, I have noticed #if HAVE_GFORTRAN ... #else ... #endif in src/arpack.c but I could not figure out where HAVE_GFORTRAN is acually set. Is it dead code ?

I believe this is used by the R interface of igraph (just like all the #ifdef USING_R). It is not dead code.

@vtraag
Copy link
Member

vtraag commented Jan 30, 2021

Note also that src/arpack.c is part of igraph not of ARPACK (it provides the interface between igraph code and ARPACK). If there is any error in that part of the code, it should not be patched upstream but here in the igraph library.

@jgmbenoit
Copy link
Contributor

@vtraag The error emerges after a call to dseupd_ which is an arpack function.

@szhorvat
Copy link
Member

I believe @vtraag was responding to Andreas about the HAVE_GFORTRAN thing.

@ntamas
Copy link
Member

ntamas commented Jan 30, 2021

My two cents: if dseupd_ gives random output for the same input then it is most likely a bug in ARPACK, although it would be good to have an isolated test case that does not involve igraph. @jgmbenoit have you managed to produce such a test case, or do you need a hand with creating one? If you have submitted a bug report to ARPACK, can you add the link here so I can keep an eye on it?

Re the --without-scg option, @szhorvat noted that the bug does not arise in the spectral coarse graining routines (which could easily be made optional) but in eigenvector calculations (which are pretty central to igraph) so there is probably not much point in adding --without-scg.

@szhorvat szhorvat mentioned this issue Jan 30, 2021
Open
@jgmbenoit
Copy link
Contributor

I am working a test case.

@jgmbenoit
Copy link
Contributor

I finally get a test case. I submitted the bug to the arpack[-ng Debian maintainer (#981646). It is a regression as the test works for the previous package 3.7.0. It might be a gcc-10 or a arpack 3.8.0 issue, but not a numerical issue.

@ntamas
Copy link
Member

ntamas commented Feb 2, 2021

Awesome @jgmbenoit , thanks a lot, we are very grateful for your work on this!

I'll keep an eye on the Debian issue.

@jgmbenoit
Copy link
Contributor

I forwarded the issue to upstream: #294.
They are not willing to fix it. On my side, I have no idea to step forward. Any hint is welcome.

@szhorvat
Copy link
Member

szhorvat commented Feb 3, 2021 •
edited

Wow, that response ...

I am concerned about two things:

  1. What is needed for igraph to be included in Debian? Can you please clarify the rules? As I understand, the test failure was a problem.
  2. I don't want users to get broken software. In my opinion, the biggest sin scientific/research software can commit is to silently and knowingly returns wrong results ... Since we know about the problem, we should not let it stand.

Here are things we can do:

  1. It has been proposed to simply not include igraph in the mips64el version of Debian. Do I understand correctly that this would have no effect on other platforms? As far as I can tell, it is very unlikely that anyone would use igraph—or do any scientific work at all—on mips64. Thus this is a reasonable solution in my opinion. (Perhaps the exception is Loongson? Does anyone use that seriously? Opinions?)

  2. It was proposed to exclude the SCG stuff from the next igraph release. I am strongly against this: (1) the next release is happening in less than two weeks (2) the problem is with ARPACK, which is used by many parts of igraph, not just SCG (as Tamás pointed out)

  3. Build igraph with its bundled ARPACK version on that platform and see if that fixes the problem. This seems reasonable to me. @tillea brought up the concern of security issues. Realistically, security issues are not very close to the top of priority list for scientific software, and igraph should not be trusted to be secure (!). The priority is to return correct results for reasonable input, not to avoid crashing at all costs even when presented with input specifically crafted to crash the software. While bugs are bugs and should be fixed, one must pick priorities depending on the intended application.

Personally, I would recommend option 1. Thoughts @ntamas @jgmbenoit @tillea ?

Also, how does this situation affect ARPACK-NG's inclusion in Debian, especially on MIPS?

@szhorvat
Copy link
Member

szhorvat commented Feb 3, 2021

I am wondering, since ARPACK seems a bit unstable, and since sometimes it returns a completely wrong result (not the wrong eigenvector, but something that is not an eigenvector), would it make sense to verify the result in cases like eigenvector centrality?

Computing eigenvectors is hard, but verifying them is easy. If the result is not as expect to some precision, show a warning to the user.

Given the nature of numeric computation, there will always be pathological cases that it won't handle. So verification would be useful, regardless of this specific MIPS issue.

But: Does ARPACK itself not verify the result? That would be strange.

@jgmbenoit
Copy link
Contributor

I am in favour of 1, namely, to remove igraph from the mips64el Debian distribution, at least temporarily (The issue can disappear as it appears). For a scientific software, it is reasonable. For a administrative software, it will be necessary to be able to build it (build + test + no serious issue) on all the architectures supported by Debian. A big umbrella scientific software like SageMath does not fulfil this criteria (see its current build status), but at least it runs on amd64 and arm64 computers. Solution 2 is not reasonable. Solution 3 will eat precious time for nothing.
I am on my way to work on solution (1).

@szhorvat
Copy link
Member

szhorvat commented Feb 3, 2021

Thanks @jgmbenoit, sounds good! Hopefully this can be re-evaluated when a new ARPACK version (or Fortran compiler?) comes out.

@ntamas
Copy link
Member

ntamas commented Feb 4, 2021

would it make sense to verify the result in cases like eigenvector centrality?

I have never been really convinced that we are using ARPACK correctly and not missing something obvious. After all, Octave uses ARPACK and does not seem to double-check ARPACK's results either. (I've spent lots of time reading the C code of Octave when I was trying to track down ARPACK-related errors in igraph back in the old days).

Adding a check does not hurt (performance-wise) and it is simple enough to do, but I'd rather leave it for after 0.9; we still have enough stuff on our plate for that release.

@stale
Copy link

stale bot commented Apr 5, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Issues that have been inactive for a long time; will be closed in the absence of further activity label Apr 5, 2021
@tillea
Copy link
Author

tillea commented Apr 5, 2021 via email

@vtraag vtraag added the todo Triaged for implementation in some unspecified future version label Apr 6, 2021
@stale stale bot removed the stale Issues that have been inactive for a long time; will be closed in the absence of further activity label Apr 6, 2021
@szhorvat
Copy link
Member

szhorvat commented Jan 8, 2022

This problem seems to have appeared again in a different form, mentioned here by @jgmbenoit :

#1651 (comment)

This time, igraph_pagerank() gives wrong results with the ARPACK method. The symptom is exactly the same: some of the vector elements are (incorrectly) returned as zeros.

I extracted the values from the build log. The the "large unweighted graph", result are as at the end of this comment. Dividing both vectors by the first element (to have comparable normalization) shows that the only difference is where ARPACK returned zeros. The other vector elements are in fact the same.

PRPACK result:

{0.00314286, 0.00314286, 0.00314286, 0.00314286, 0.00314286, \
0.00314286, 0.00314286, 0.00314286, 0.00314286, 0.00314286, \
0.00314286, 0.00314286, 0.00314286, 0.00314286, 0.00314286, \
0.00314286, 0.00314286, 0.00314286, 0.00314286, 0.00314286, \
0.00314286, 0.00314286, 0.00314286, 0.00314286, 0.00314286, \
0.00314286, 0.00314286, 0.00314286, 0.00314286, 0.00314286, \
0.00314286, 0.00314286, 0.00314286, 0.00314286, 0.00314286, \
0.00314286, 0.00314286, 0.00314286, 0.00581428, 0.00403333, \
0.00403333, 0.00403333, 0.00403333, 0.00403333, 0.00403333, \
0.00403333, 0.00403333, 0.00403333, 0.00403333, 0.00403333, \
0.00403333, 0.00403333, 0.00403333, 0.00403333, 0.00409409, \
0.00409409, 0.00409409, 0.00409409, 0.00409409, 0.00409409, \
0.00409409, 0.00409409, 0.00409409, 0.00409409, 0.00409409, \
0.00409409, 0.00409409, 0.00409409, 0.00409409, 0.00409409, \
0.00409409, 0.00409409, 0.00409409, 0.00409409, 0.00409409, \
0.00409409, 0.00409409, 0.00409409, 0.00409409, 0.00409409, \
0.00409409, 0.00405359, 0.00405359, 0.00405359, 0.00405359, \
0.00405359, 0.00405359, 0.00405359, 0.00405359, 0.00405359, \
0.00405359, 0.00405359, 0.00405359, 0.00405359, 0.00405359, \
0.00405359, 0.00405359, 0.00405359, 0.00405359, 0.00426804, \
0.00426804, 0.00426804, 0.00426804, 0.00426804, 0.00426804, \
0.00426804, 0.00426804, 0.00426804, 0.00412507, 0.00412507, \
0.00412507, 0.00412507, 0.00412507, 0.00412507, 0.00488198, \
0.00488198, 0.00488198, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00437737, 0.00437737, 0.00437737, \
0.00437737, 0.00437737, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459, 0.00439459, 0.00439459, \
0.00439459, 0.00439459, 0.00439459}

ARPACK results are:

{0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, \
0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, \
0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, \
0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, 0.0123798, \
0.0123798, 0.0123798, 0.0123798, 0.0123798, 0, 0, 0, 0, 0, 0, 0, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0.0158874, 0.0158874, 0.0158874, 0.0158874, \
0.0158874, 0.0158874, 0.0158874, 0.0158874, 0.0158874, 0.0158874, \
0.0161267, 0.0161267, 0.0161267, 0.0161267, 0.0161267, 0.0161267, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \
0, 0, 0, 0.0172426, 0.0172426, 0.0172426, 0.0172426, 0.0172426, \
0.0172426, 0.0173104, 0.0173104, 0.0173104, 0.0173104, 0.0173104, \
0.0173104, 0.0173104, 0.0173104, 0.0173104, 0.0173104, 0, 0, 0, 0, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0173104, 0.0173104, \
0.0173104, 0.0173104, 0.0173104, 0.0173104, 0.0173104}

@szhorvat szhorvat mentioned this issue Jan 10, 2022
Closed
20 tasks
@jgmbenoit jgmbenoit mentioned this issue Oct 29, 2023
Closed
@szhorvat szhorvat mentioned this issue Dec 10, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
todo Triaged for implementation in some unspecified future version
Projects
None yet
Milestone
next
Development

No branches or pull requests
5 participants
@ntamas @szhorvat @vtraag @jgmbenoit @tillea