Support for ITU-R P.1812-4

[1ntell1gentDB]

Hi Benjamin,

Thanks for this wonderful tool, it will get even better if you consider to addd support for propagation model ITU-R P.1812-4.

With thanks.

[bwinkel]

Hi @1ntell1gentDB,

I looked into this in more detail. The algorithms to calculate LoS, diffraction, troposcatter, and ducting are quite similar in P.452-16 and P.1812-4, but not exactly equal. For example, the latter completely neglects the atmospheric loss terms - which of course don't play a big role at the VHF/UHF frequencies. The biggest difference is in clutter handling. P.452 assumes no clutter in the complete height profile and applies end-point correction. P.1812 adds clutter heights to terrain heights for each position on the propagation path.

Unfortunately, some heavy API changes seem necessary, if one wanted to have several methods offered to the user within the pathprof sub-package. I think, I might do this at some point, but it will certainly take time. A quicker solution would probably be, to use the P.452 algorithm and apply some pre- and post-processing. This will not give the exact same results as P.1812 (at the moment there is no way to leave out the LoS focussing/multipath correction and atmospheric attenuation in pycraf's calculations), but it will be very close. I'm willing to help you implementing the clutter/building entry loss functions of P.1812 as well as the location-percent distribution.

Do you have some example, which we could use for that? I have especially in mind, what database you use for clutter. We'd also need to find a solution for the problem, that SRTM data are in part influenced by clutter (e.g., a forest usually increases the terrain height in SRTM).

[bwinkel]

If I'm not mistaken, P.1812 includes the clutter into the terrain heights of the path (plus some endpoint correction). Applying P.2108 would then do an additional clutter correction, which likely leads to completely overestimated clutter attenuation. Of course, if you don't add clutter heights to the terrain, you could do that.

However, I'd rather modify the output of the height_map_data function - effectively adding the clutter heights to the “height_profs” array. (I'll show you how to do that.) Of course, one needs a database for the clutter in the area of interest. However, as I said, one needs clean topographic data to start with - and SRTM is not clean. An alternative could be to assume that SRTM already contains topography + clutter and just work with that (no additional correction). I'll leave it up to you, which is better for your application. In any case, I don't have access to any other GIS data than SRTM.

Stay tuned.

[1ntell1gentDB]

Hi Benjamin,

Thanks for the clarifications, SRTM is our primary terrain dataset as well, if it already includes clutter, then there is no need to modify height_map_data function in order to add clutter details. Lets stick with SRTM with no additional correction at-least for now.

[bwinkel]

Dear @1ntell1gentDB,

I've finished another tutorial notebook ("03f_pre+postprocessing.ipynb") that shows how you could (roughly) estimate the P.1812 attenuations using pycraf's P.452 implementation. It is completely untested and for now it resides in the "p.1812" branch on the repository. If you want to make use of it, you'll have to install from that branch (pip-install won't work, until I've made another release).

Could you please have a look, and get back to me, if anything is unclear? Note again, the clutter handling is sub-optimal. For one, SRTM is not cleaned for clutter and the other thing is that the P.452 algorithm handles clutter in a different manner, which is not 100% compatible with P.1812. However, it may well be that the errors are negligible - don't know.

At least you get the location-percentage distributions ;-)

[1ntell1gentDB]

Thanks, Ben,

I pulled the 'p.1812' branch for testing as per your suggestions but i encountered some errors during testing. I used the following command for testing it: 'python setup.py test -P p.1812'

Please see the attached log file with the error for your review:
itup1812log.txt

There are also some minor updates on the P.1812-4 calculations froM the ITU, I will send to you the updated MATLAB code and documentations bit late today.

Thanks.

[bwinkel]

Hmm, in that log-file I can't see an error (only compilation/installation debug messages). Since you're using MacOS - have you followed the installation instructions? The default compiler (clang) won't work, but you have to use gcc or llvm. What's the output of gcc -v?

A comparison with MATLAB code would be useful, indeed.

[1ntell1gentDB]

Yes, i did follow all the installation instructions, I am not using the default compiler/clang, I have installed gcc-7.

Please see attached the output of gcc -v
gcc_version.txt

[bwinkel]

Indeed, this looks good.

I just realize that you use a very strange test command - with -P p.1812 you would test a package "p.1812" which doesn't exist. To make the branch active you have to do the following:

git clone https://github.com/bwinkel/pycraf
cd pycraf
git checkout -b p.1812  # make a local branch with name "p.1812"
git merge origin/p.1812  # merge remote branch (github) into local
python setup.py test --remote-data=any

Alternatively, you can download the zipped source distribution of the branch and work with that:

wget https://github.com/bwinkel/pycraf/archive/p.1812.zip
unzip p.1812.zip
cd pycraf-p.1812
python setup.py test --remote-data=any

[bwinkel]

PS: Once the tests were successful, you need to install the package from the source:

python setup.py install

[1ntell1gentDB]

Thanks, the command I am using to test is from your manual specifying testing of specific sub-package installed from the source...

https://bwinkel.github.io/pycraf/install.html

[bwinkel]

Yes, but in this case, the "p.1812" is not a sub-package of pycraf, but just a version of the software. (The modifications will be merged back to the main development branch, once we know, that the tutorial notebook provides more or less correct results.)

[1ntell1gentDB]

Ok, agree, I am also trying to test the same on a Windows machine but it seems the 'setup.py' is missing from the pycraf folder:-)

[bwinkel]

How did you download the sources, via git or downloading the zip?

[1ntell1gentDB]

via git on a console

[bwinkel]

Well, that's interesting, because it usually works on my Windows machine. In any case, installing from source on Windows is not fun, because you need to install a compiler; see the note in the installation instructions. I'd stick to the MacOS machine, if you can (or even better Linux ;-)).

[1ntell1gentDB]

with OSX I am still getting this import error:

ImportError: dlopen(/private/var/folders/f8/17vc30jn3619r5wtm_1m549h0000gn/T/pycraf-test-f386qsc9/lib/python3.6/site-packages/pycraf/pathprof/cyprop.cpython-36m-darwin.so, 2): Symbol not found: _GOMP_parallel

On Windows machine now it runs test then stop and complains about missing astropy package and to import table...

[bwinkel]

This is a well-known problem (without a good solution, unfortunately), see e.g. gprMax/gprMax#134. You could try their approach and change in the file pycraf/pathprof/setup_package.py:

'extra_link_args': ['-fopenmp', '-lgomp'],

to

'extra_link_args': ['-fopenmp', '-lgomp', '-Wl,-rpath,/usr/local/opt/gcc/lib/gcc/7/'],

From your gcc -v output, however, I fear that the path /usr/local/opt/gcc/lib/gcc/7/ has to be changed to something else (perhaps /opt/local/gcc/lib/gcc/7/?). Without access to the machine, it's really hard to debug. This is one of the reasons, by-the-way, why I seriously considered to drop MacOS support completely. Just to be on the safe side, did you brew link --overwrite gcc@7?

Another solution could be to use the LLVM compiler instead (again, see installation notes).

[bwinkel]

According to this thread it could also be related to using Python 3.6.

[1ntell1gentDB]

I see, its true that I am running it on Python 3.6.5, let me try to change the environment to Python 3.5 or below bit not Python 2.7 it doesn't work also...I have to install Visual studio in order to run on Windows:-)

[1ntell1gentDB]

The tests on Windows machine went very well after installing VSC, please see the attached text file:

tests.txt

[bwinkel]

Glad to hear that.

[1ntell1gentDB]

Hi Ben,

I did managed to test propagation losses using the 'pathprof' module example from here:
https://bwinkel.github.io/pycraf/pathprof/index.html#module-pycraf.pathprof

My inputs are in the following file:
ppropg.txt
The results are in the following file:
Test_results.txt
It seems that the complete path loss results are not correct, the program has substituted it with the tropospheric results which i didn't even considered.
Please advise if the 'pathprof' module is the correct one for testing the 'p.1812' implementation, I will test again to compare with results from MATLAB.
Thanks.

[bwinkel]

Hi,

the calculation was very likely correct, you only made a small mistake when printing the result:

tot_loss = pathprof.loss_complete(pprop, G_t, G_r)
print(
    'L_bfsg: {:5.2f} - Free-space loss\n'
    'L_bd:   {:5.2f} -  Basic Tx Loss\n' 
    'L_b:    {:5.2f} -  Complete Path Loss\n'
    'correction'.format(*tot_loss)
)

The return value tot_loss is a Python tuple (containing more than three elements). The .format(*tuple) syntax will is just a nice short form for .format(tuple[0], tuple[1], ..., tuple[n]). The problem with your code is, that you print the first 3 values only. The print statement doesn't know that you're only interested in the LoS, diffraction and total loss, it will just print the first three elements of the tot_loss tuple. What you could do is the following:

tot_loss = pathprof.loss_complete(pprop, G_t, G_r)
L_bfsg, L_bd, L_bs, L_ba, L_b, L_b_corr, L = tot_loss  
print(
    'L_bfsg: {:5.2f} - Free-space loss\n'
    'L_bd:   {:5.2f} -  Basic Tx Loss associated with diffraction\n' 
    'L_b:    {:5.2f} -  Complete Path Loss\n'
    'correction'.format(L_bfsg, L_bd, L_b)
)

[1ntell1gentDB]

Thanks for clarifications.You didn't answer my second question, is the p. 1812 implementation already incorporated into pathprop module or I should test another module?

[bwinkel]

As I wrote earlier, I didn't change the pathprof module itself, but made another Tutorial Notebook, which demonstrates how to do the calculations based on pycraf's P.452 implementation. Please have a look.

[1ntell1gentDB]

Okay, now after this suggested modification it produces strange error see below:
tot_loss = pathprof.loss_complete(pprop, G_t, G_r)
L_bfsg, L_bd, L_bs, L_ba, L_b, L_b_corr, L = tot_loss
print(
'L_bfsg: {:5.2f} - Free-space loss\n'
'L_bd: {:5.2f} - Basic Tx Loss associated with diffraction\n'
'L_b: {:5.2f} - Complete Path Loss\n'
'correction'.format(L_bfsg, L_bd, L_b)
)

"L_bfsg, L_bd, L_bs, L_ba, L_b, L_b_corr, L = tot_loss
valueError: too many values to unpack (Expected 5)"

[bwinkel]

No idea what's going on. Could you please run

import pycraf
print(pycraf.__version__)

[1ntell1gentDB]

it brings an error:
"...module 'pycraf' has no attribute 'version'...."

[bwinkel]

The underscores are important.

[1ntell1gentDB]

yes i did put them in the console, just forgot to type them here

[bwinkel]

Could you copy-paste the error message? Every bit is important here. I'm asking, because if you typed in the underscores, the error message should include them, as well.

[1ntell1gentDB]

here we go...

[bwinkel]

My gut feeling was right. You need double underscores, both at beginning and end.

[1ntell1gentDB]

mmmh, i see, its version 0.25.7dev237

[bwinkel]

Which is correct. What is the output of

print(tot_loss)

in the above example?

[1ntell1gentDB]

name 'tot_loss' is not defined

[bwinkel]

Of course you need to run your program first...

[1ntell1gentDB]

ok

[bwinkel]

OK, so there are really seven values in tot_loss. Therefore, I don't understand the error message that you posted. Have you copy-pasted the code, which I posted above or typed it in? In the latter case, are you sure, there is no typo? (Copy-pasting is always the preferred method.)

By-the-way, it would be nicer if you'd not attach files but copy-paste code and terminal output as text into GitHub. This way, other users can find it via search... GitHub has some nice mark-down features. For example, you can use triple-back-ticks (```) to syntax-highlight code, etc. See also https://guides.github.com/features/mastering-markdown/.

[1ntell1gentDB]

I understand, i am running the program in a separate windows machine while i am communicating with you here using a Mac which failed to compile pycraf thats why i am not able to directly copy the code:-)

[1ntell1gentDB]

Now what should i do to get the output of each terrain height in an array from pycraf, I want to use it in the MATLAB code to validate the two implementations...

[bwinkel]

For one path or for a map?

[1ntell1gentDB]

I just want to compare one path at a time with validated results from ITU.

[bwinkel]

You can use srtm_height_profile function for this. It's used internally by the path propagation functions.

[1ntell1gentDB]

Thanks, the other thing is, I don't see the Tx power (Ptx) among the input parameters as required by p.1812.

[bwinkel]

Yes. But if you take the path loss and subtract it from the Tx power, you directly get the Rx power, which you can convert to E-field, if necessary, using the conversions sub-package. Again, pycraf hasn't got a fully-featured implementation of P.1812.

[1ntell1gentDB]

Hi @bwinkel,
I was trying to run the heightprofile.py in both WIndows and Ubuntu machine but encountered the following import error:

$ python heightprofile.py
Traceback (most recent call last):
File "heightprofile.py", line 10, in
from . import cygeodesics
ImportError: cannot import name 'cygeodesics'

what could be the problem?

[bwinkel]

The modules (python files) of the pycraf package are not meant to be run stand-alone. You need to install and import the package in order to use functions. Please see the documentation and tutorials, how to work with pycraf.

[1ntell1gentDB]

Hi Ben,

I have tried my best but i found it difficult to extract height profiles perhaps an example could be handy. Regarding the P.1812-4 branch, I went through the notebook but the explanations are not clear on what one should do to compile and run it. an example of a running script would also be handy in this case.

[1ntell1gentDB]

If I'm not mistaken, P.1812 includes the clutter into the terrain heights of the path (plus some endpoint correction). Applying P.2108 would then do an additional clutter correction, which likely leads to completely overestimated clutter attenuation. Of course, if you don't add clutter heights to the terrain, you could do that.

However, I'd rather modify the output of the height_map_data function - effectively adding the clutter heights to the “height_profs” array. (I'll show you how to do that.) Of course, one needs a database for the clutter in the area of interest. However, as I said, one needs clean topographic data to start with - and SRTM is not clean. An alternative could be to assume that SRTM already contains topography + clutter and just work with that (no additional correction). I'll leave it up to you, which is better for your application. In any case, I don't have access to any other GIS data than SRTM.

Stay tuned.

Hi Ben,

I trust all is well, I have now obtained a clutter dataset which I would like to use together with the SRTM data. Will you please assist to to get this done as discussed in the above post.

Thnx.