HRIT Decoding (J2K and missing CPPDUs) #15
Comments
|
I haven't been able to test HRIT since goesrecv has not been modified for QPSK yet. What is your demodulator and decoder chain? Thanks for the VCDU files, I'll take a look! |
|
HRIT headers are intact but the JPEG/JFIF headers are broken for some reason. There does seem to be a bigger demuxing issue so I'll take a look at that first, then look at the JPEG issues.
|
This is a professional meteorological receiver, we used it to get HRIT / LRIT with Electro-L. It supports Viterbi and Reed-Solomon decoding and provides 1024 (full frame with sync word), 1020 (CVCDU) or 892 (VCDU) bytes packets over Ethernet. |
|
Nice! is it a Dartcom receiver? I have started work on a QPSK DSP chain using GNURadio source for reference. Using LRPT code as well is a good idea! I'll be getting back to that soon. |
No, this is a Russian model produced by http://www.etra-plus.ru/ (D8L, could not be found on the site, possibly EOL) |
|
Interesting, I thought it looked similar to the Comtech CDM modem series, then I found this on the D4L product page:
|
|
The JPEG headers in these images seem to be malformed and don't have enough info to recover the image data.
They are basically empty when compared to a normal JPEG/JFIF header. I'm going through the HRIT documentation at the moment to see if there's anything explaining this.
|
|
According to http://nmsc.kma.go.kr/resources/homepage/pdf/GK2A_HRIT_Mission_Specification_Documen_v1.0.pdf, p.24, |
|
This makes sense as Pillow was throwing exceptions when loading the image. It needs OpenJPEG installed and in PATH to handle J2K. It also mentions they're using J2K in a lossless configuration so that could explain the sparse headers. |
|
Got images from IR channels! Still having troubles with VIS channel ((
|
|
Nice work! I'll look at streamlining that process then integrate it into xrit-rx. |
|
I left the system running overnight, and today I discovered the following: (lines with [JPEG] in last section is output of my script). It is suspicious that the number of missing bytes is the same in all segments (2 CP_PDU) except the first and last, where it is one CP_PDU.
|
|
New info about CPPDU loss: after adding CPPDU counter to As you can see, every 442th from beginning of the TPFile CPPDU is loss. So, we have loss 8190 bytes for each 3619980 bytes (8190*442) of file length. For large file, there will be data loss after 7239960, 10859940 bytes and so on. As IR-channel's hrit files are much less than 3.6M, they have no data loss at all. |
|
Fantastic work tracking down all of these issues! I finally have some time to devote to HRIT support over the next few days. I'll be implementing your findings in the Side note: What is your RF setup for receiving HRIT? (dish size, feed, amplifiers, filters) |
sam210723
changed the title
|
For HRIT we use meteorological receiving station based on 3-meter dish and circular-polarized feed-horn with built-in LNA. I know that GK-2A uses linear polarization, but I cannot change antenna's construction. Described above ETRA receiver is used for demodulation. LRIT is received by 2-meter dish with self-built "cantenna" feed-horn (from drainpipe :) ) with linear polarization, Nooelec SAWBird GOES LNA/filter and Airspy R2 receiver. I have not tested yet HRIT on 2m dish (cannot powerup LNA from ETRA by coax cable, need to bring additional power to the roof), but signal on 2m dish and Airspy seems to be powerful enough: Also, your |
|
I agree, HRIT on the 2m does look strong enough. Do you have an IQ recording from the 2m dish I can download? I can try running it through my GNU Radio HRIT demodulator. You could try ETRA on the 2m dish using an external bias-tee between the ETRA and SAWbird to inject 5V DC. A cheap eBay one should do the job. This way you won't have to run separate power to the antenna. |
|
These are IQ-files recorded in Gqrx (freq=1695.4 MHz, rate=10MSPS). They are HUGE (approx. 450M each for 5-seconds recording), one is with VCID=63 (no data) and other is VCID=0 (FD). https://drive.google.com/file/d/1XQQAg5-DvX-Gz_TLhViHapV3tod4B8FJ/view?usp=sharing If you can not replay these files, please, give me a tip how to record. |
|
QPSK constellation is definitely visible in those recordings. I'm not sure if Viterbi + RS would be enough to recover any data from this signal.
Do you have a VCDU capture of VIS006 imagery? I'd like to tackle that CP_PDU issue after getting J2K image output working. |
Do you mean dumps by script |
|
Ah sorry, you are right. For some reason I was thinking they were not transmitted at night like old COMS-1 LRIT. In that case I'll try adding an APID filter to the demuxer so I can work on just VIS006, then move on to J2K stuff. |
|
Something strange is happening with APID. It seems that all the channels are spread across all APIDs: and so on |
|
KMA have a habit of not following their own specifications. I noticed this when working on LRIT. It could still be a demuxer issue since I'm completely ignoring APIDs at the moment. It was done this way because the APID was useless with COMS-1 LRIT when the demuxer core was written. It used to be the VCID*64 so was of no real benefit. I will need to track APIDs now with HRIT because the FD wavelength segments are interleaved which complicates product generation.
|
|
Confirmed CP_PDU #441 is having issues, and the APIDs are completely useless. The CP_PDU header is there but it never gets marked as completed which is needed to trigger the payload length and CRC check, then its meant to be appended to the current TP_File. Time to delve into the demuxer core... |
|
Found it! The I was foolishly using that header value to trigger the start of a new TP_File, which works for small files. But once the CP_PDU header lines up again with the start of the
I've re-arranged that section of the demuxer to now check if there's an unfinished TP_File rather than assume an offset of zero is the start of a new TP_File. All 50 HRIT files for that FD are now being saved correctly.
|
|
And we have complete Full Disks! I've re-written
Couldn't help myself. I had to render a false colour with GeoSatSignal
I'll adapt the changes in |
|
Wow! That's fantastic! :D |
|
I've now got xrit-rx saving assembled HRIT full disk images without saving
All that's left for HRIT support is to update the progress bar indicator to handle multiple wavelengths. I may add a UDP socket input option before next release so you can easily try it on the real-time downlink using the ETRA modem. I'll also need to work out licensing of |
|
Progress bar now shows each wavelength individually.
Changes merged into |
|
Just released |
|
Hello, My attempts have not yet been successful with the prime focus 2.3 m mesh antenna, although the signal looks quite sufficient in sdrsharp: |
|
Although xrit-rx is ready for HRIT, goesrecv is not yet. The HRIT downlink from GK-2A uses QPSK modulation and LRIT uses BPSK. I have been working on adding the necessary components to goesrecv for QPSK demodulation but have not finished this yet. I've been posting progress updates to Twitter: https://twitter.com/sam210723/status/1274632866526191616. Here I'm testing with FY-4A LRIT which is also QPSK. 2.3m may be large enough for HRIT but I won't be able to say for sure until we have a working receive chain. |
|
Thank you so much for your response and for your brilliant work!
|
|
Glad to hear xrit-rx and goesrecv-monitor are working well for you! Great looking signal there on your 2.3m dish. |
|
Hi Sam, and everyone who has contributed to this fantastic outcome. Great work and talent! Cheers |
I got a dump of an HRIT stream and am trying to decode it using xrit-rx. I get a lot of errors like
Product files (.bin) are written, but their size is too small.
I have checked VCDU counter field, and I does not see any packet loss from receiver, counter leaps only on transmission start/stop:
Links to dumps:
hrit03.dmp
hrit03.log
hrit04.dmp
hrit04.log
The text was updated successfully, but these errors were encountered: