Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rankings do not consider cleanliness of reception #1

Closed
PKLoops opened this issue Nov 28, 2018 · 23 comments
Closed

Rankings do not consider cleanliness of reception #1

PKLoops opened this issue Nov 28, 2018 · 23 comments

Comments

@PKLoops
Copy link

PKLoops commented Nov 28, 2018

SDR's with overload, intermods or repetetive harmonic related carriers ruin results and score highly,

I will not leave any examples as this would be poor show, but the data rank cannot be trusted for any useful measurement.

Take some time to explore the higher ranked SDRs at http://rx.linkfanel.net/snr.html by using them.
@linkfanel
Copy link
Member

linkfanel commented Nov 28, 2018 •
edited

Hello, thanks for your interest in this issue. I'm very well aware of it, and there have been several discussions, on ValentF(x) forums and in private, about the limitations of this approach and possibilities for enhancement. Please note that this is only a proof-of-concept prototype, that unfortunately went long-term. My opinion, supported by technical reasons, is that a long-term solution should be developed and integrated upstream in the KiwiSDR project.

Please also note that technically, neither SNR scoring data nor this SNR score page are within the scope of the dyatlov software.

This SNR score page does not aim at establishing a fair global ranking of all receivers, as this is not a relevant approach to choosing a receiver to listen on, nor at objectively comparing receiver performance.

Besides the inherent unreliability of the measurements due to factors like those you mention, which still remains to be solved, what is your concern exactly?

@PKLoops
Copy link
Author

PKLoops commented Nov 28, 2018 via email •
edited

@linkfanel
Copy link
Member

linkfanel commented Dec 1, 2018 •
edited

My interest is from a purely perfectionist desire to have all my 24 Channels of KiwiSDR system working as best as they can be and recognised for doing so, which is only fair considering the investment involved.

Ah so you're the guy in Croydon, nice!

I have achieved many technical improvements over the average KiwiSDR set up, like local noise reduction, flat frequency response from 500KHz-30MHz and working on 0-500KHz, with BCB selective reduction/notches on multiple receivers ( 4+) from same antenna system, with more planned... so I think you get my drift...

That's great! Thanks for your contributions. My vision about this was that in the KiwiSDR admin page there would be a list of setup steps that you can tick off. The most advanced ones at the bottom of the list would be suggestions and guides for items like you just described.

If you write an advanced optimization guide showcasing your work, I could also link it from the score page since that's under my control.

I just hope for a fair measurement system, so that other providers can judge their own devices, and systems in a technically empirical / more accurately measured way so they can see how they are going, and give them something to aspire to...

Unfortunately it's hard to be really fair, because the scoring approach depends on factors extrinsic to the receiver: what signals are transmitted in the area, the local time at which the measurement is taken (or what methodology is used to smooth and average them). I also think it's important to keep viewing this as healthy emulation rather than a zero-sum ranking competition game, and - while they can be useful - not put too much importance on the numbers themselves. I can add wording in that sense on the score page.

I will help technically if you want or allow me to ?

As I said, the best approach in my opinion would be to take this development upstream to have KiwiSDR receivers generate and export SNR scores themselves. There are WPSR autorun sessions now so I imagine it should be simpler to reuse that logic to take a waterfall channel and make a measurement. But I don't operate a KiwiSDR board myself.

@linkfanel
Copy link
Member

My vision about this was that in the KiwiSDR admin page there would be a list of setup steps that you can tick off. The most advanced ones at the bottom of the list would be suggestions and guides for items like you just described.
If you write an advanced optimization guide showcasing your work, I could also link it from the score page since that's under my control.

Unfortunately it's hard to be really fair, because the scoring approach depends on factors extrinsic to the receiver: what signals are transmitted in the area, the local time at which the measurement is taken (or what methodology is used to smooth and average them). I also think it's important to keep viewing this as healthy emulation rather than a zero-sum ranking competition game, and - while they can be useful - not put too much importance on the numbers themselves. I can add wording in that sense on the score page.

Earlier this year in February, I've added on the score page a checklist of reception optimization steps, contributed by WA2ZKD on http://valentfx.com/vanilla/discussion/comment/9530/#Comment_9530 ; and also such wording to put the SNR numbers into proper perspective.

@darksidelemm
Copy link

Is there any further information on the SNR metrics used? (Or maybe a link to some source code?)

We've recently made our KiwiSDR Public (http://kiwisdr.areg.org.au:8073/ - #2 on the rankings at the moment!), and I'm interested in running these metrics more regularly to investigate the behaviour of some local power-line noise which we have noticed. (The site is fed from a single-wire-earth-return system).

@darksidelemm
Copy link

It's also worth noting that stations which block out certain portions of the spectrum score much higher than they should with this system. @PKLoops's stations for example, which only show the amateur radio bands, are likely right up the top of the listings due to the huge areas with zero signal.

@PKLoops
Copy link
Author

PKLoops commented Jan 9, 2021

I'm happy to join back in on the conversation on interference issues and ranking..

For the two of my 6 Kiwi cards that are permanently dedicated to all 14 x LF/MF/HF WSPR frequencies, they are fed with fan dipoles that only perform on ham bands, using spectrum masks helps blank out interference from VDSL Leakage and POE devices in neighbouring houses ( I have 4 noisy houses that back on to my yard as I am located in a heavily built up housing estate in suburbia )
On the other 4 x Kiwi Cards, there are only minimal spectrum masks in place...

I do not highpass filter above 2 MHz to block out the broadcast band, so each of the signals received in the <2MHz spectrum will be evaluated and contribute to the S/N rank calculations that Pierre makes.

The whole system is powered from DC / battery, all antennas are effectively balanced and have Choke Baluns to remove any common mode Noise currents. The network cable (30m long) is Shielded, goes underground and has choke cores each end to prevent LAN / IT gear noise.

The single GPS Antenna is actively split to the 6 cards and my Lightning Detector.

I will attempt to upload and attach a video of the equipment later on.

Kind Regards
Paul Karlstrand
VK3KHZ

@csylvain
Copy link

i'd like to suggest being able to queue a re-evaluation of a station. i've made changes but the last evaluation of my station was practically a month ago.

@darksidelemm
Copy link

Is there any further information available on the SNR calculation algorithms? Or even better... source code?

I've been making tweaks to the VK5ARG site (and there's now a second KiwiSDR up there), and I'm very interested to look at how the performance varies over time, and with various configurations of filters and splitters.

@linkfanel
Copy link
Member

Hello Mark,

Sorry for not replying earlier. But I've read all those messages and have been pondering about them for some time.

Is there any further information on the SNR metrics used? (Or maybe a link to some source code?)

As already stated on http://rx.linkfanel.net/snr.html : "These ratios are calculated between the 95th percentile of power levels across the whole spectrum (signal), and the median power level (noise)." That's pretty much the whole algorithm. The spectrum/waterfall data comes as an array of numbers: the whole frequency range is divided among small bins or slots, and each of them carries a signal power level for that frequency bin. The algorithm is to sort these bins by power level, with the weaker one on side, and the strongest at the other end. The signal power value in the middle at the separation between the weaker half and the stronger half - or median - is taken as noise floor; while the value at the separation between the weaker 95% and the top strongest 5% signals is taken as the signal value. Then getting the signal-to-noise ratio is a simple subtraction between these two values (subtraction and not division because the signal powers in dB are on a logarithmic scale).

It's also worth noting that stations which block out certain portions of the spectrum score much higher than they should with this system. @PKLoops's stations for example, which only show the amateur radio bands, are likely right up the top of the listings due to the huge areas with zero signal.

Actually, I had already changed my script last year to exclude all the blocked spectrum before picking the median value, otherwise blocked spectrum bins, with a lower signal power value than anything else, could indeed skew the calculated noise floor to much lower than it should have been. So that part of the problem has already been fixed. If band-restricted receivers are still at the top of the ranking, it could be that they're just that good, or also that the signal patterns and levels are very different on those common, heavily used bands, compared to mostly unused spectrum. Historically, receivers with a very strong MW broadcast band have also seen their scores skewed up. There's no straight answer.

We've recently made our KiwiSDR Public (http://kiwisdr.areg.org.au:8073/ - #2 on the rankings at the moment!), and I'm interested in running these metrics more regularly to investigate the behaviour of some local power-line noise which we have noticed. (The site is fed from a single-wire-earth-return system).

As stated on score page: "The Auto Scale button on the waterfall control tab will also give you instant, real-time signal and noise level metrics - as the WF max and WF min numbers - similar to these scores to evaluate the performance and impact of your improvements as you work on your receiver." These waterfall display calibration numbers are calculated along the same algorithm, so when fully zoomed out to show the full spectrum, the subtraction WF max - WF min will give you a score that you can use to measure and follow the performance of your receiver and the results of your tinkering. That's what I recommend using, because you can run these measurements yourself whenever and as often as you want or need! Also, my scores include smoothing over past measurements, so they wouldn't be the best to visualize the direct results of your changes.

The source code of my scripts is not public, and is not all that interesting anyway; it's just a piece of JavaScript I run in my web browser. I've been thinking that any better or new solution might be better off relying on the official Python KiwiClient command-line utility, which now supports such waterfall extraction and analysis out of the box.

i'd like to suggest being able to queue a re-evaluation of a station. i've made changes but the last evaluation of my station was practically a month ago.

I run these measurements manually, because doing it automatically comes with concerns about infrastructure strain, security, runtime requirements, and even meaning and interpretation of the results. I will certainly admit that I don't run measurements as often as people would like, because I only have so much effort to put into this free service. But I've been thinking that the least I can do is to offer more if some people want to pay. I don't really want to take your money, but I know I'm not going to do much otherwise.

  • If and when someone wants to ship a few bucks with their request, I could update the measurements and scores for everyone the next day, guaranteeing their particular receiver gets a new score (all receivers don't always do every time, for example if they're down or full when the measurements are run).

  • If people want to sponsor this work, I can rewrite this into an automatic system, updating scores much more often and regularly.

  • I still stand by my assessment that from a distributed system point of view, it would be better if each receiver measured, calculated and exported their score themselves, rather than having an external tool poll every node on the network for it. So I think it might be a better overall solution to sponsor jks, the author and owner of KiwiSDR, to implement this as a KiwiSDR feature instead.

Let me know if there's interest.

@csylvain
Copy link

csylvain commented Mar 15, 2021 via email

@jks-prv
Copy link

jks-prv commented Mar 16, 2021 •
edited

Since most of the difficult support code is already done I'm willing to implement this. You have to give me an SNR calculation algorithm though if you want something other than what Pierre does now. It might be interesting to compute the SNR over 0-30 MHz and also just >= 3 MHz (or some other value) to factor out the AM BCB.

Because of WSPR autorun the mechanism to start an internal process that connects to a free channel is fairly easy. The result can just be another field added to kiwisdr.com/public/index.html that rx.linkfanel.net, or anyone else, can poll/upload.

@jks-prv
Copy link

jks-prv commented Mar 22, 2021

Okay, v1.443 has a first cut at an internal SNR measurement function. See the explanation here: https://github.com/jks-prv/Beagle_SDR_GPS/blob/master/CHANGE_LOG

@jks-prv
Copy link

jks-prv commented Mar 22, 2021 •
edited

With v1.444 the two SNR values (all band, HF) are also reported in the information returned by a /status query, e.g. my_kiwi:8073/status

And since that data is the source of information used to build http://rx.kiwisdr.com/index.html it also appears there for every publicly listed Kiwi (if you look at index.html as a text file or do a "show page source" when viewing the page in a browser).

@csylvain
Copy link

csylvain commented Mar 23, 2021 via email

@linkfanel
Copy link
Member

I've added the internal scores reported by the KiwiSDRs on http://rx.linkfanel.net/snr.html In time I'll switch the map to use those new scores too.

@darksidelemm
Copy link

@linkfanel Might be worth showing the HF-only SNR too, since stations with very strong AM broadcast station signals and everything else blanked out skew the results hugely. (e.g. the VK3KHZ stations - 48 dB 'all band' SNR, 8 dB HF-only SNR!)

@linkfanel
Copy link
Member

Good point, I've added it as a third column. Let's see what this gives. Further comments about the new scores, how they could be improved or what score should be used for the map, are welcome.

@darksidelemm
Copy link

Thanks! I'm also kind of curious how the VK5ARG stations always managed to end up with a huge SNR on your measurements, when the SNR was clearly changing throughout the day (see plots at the bottom of this page: https://www.areg.org.au/areg-remote-hf-receiver-kiwisdr ).
Were you sampling throughout the day and picking the highest SNR? Or did we just get lucky :-)

@linkfanel
Copy link
Member

I always make the measurements for my scores (external column) around 1100z. According to the plot, that's when VK5ARG stations' SNR is close to highest. So I guess you just got lucky.

@darksidelemm
Copy link

OK, taking measurements at a fixed UTC time will certainly give preferential results to stations that are in local night. I would suggest using the peak SNR observed over whatever time period is available as the result for each station.

@linkfanel
Copy link
Member

I have just switched the markers on the map to use the new self-reported internal SNR scores. The logic is as follows: the HF-only self-reported score is used if available, if not the full-spectrum self-reported score is used instead, and if no self-reported score is available, the classic external score is used as fallback.

Users will benefit from up-to-date and instantly relevant scores, as by default self-reported scores are calculated every hour. HF-only scores are favored as it was pointed out that very strong MW bands tend to skew the scores. I don't especially like excluding signals below 1800 kHz, but a choice has to be made since it makes a significant difference; and this is a shortwave receiver map, not just a map of remotes for commercial MW band broadcasts, so this is our editorial line.

Now that the score calculations are not our doing anymore, unless there are further actionable comments, I'm going to close this issue; any further request for improvement of the scores should be taken directly to the KiwiSDR project or forums instead from now on.

@jks-prv
Copy link

jks-prv commented Apr 12, 2021

any further request for improvement of the scores should be taken directly to the KiwiSDR project or forums instead from now on.

That's fine. Thanks for all your hard work in creating and maintaining these resources. They are much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@darksidelemm @jks-prv @linkfanel @csylvain @PKLoops