Fondsweb: currency not always detected #179

rdorsch · 2021-03-07T11:26:01Z

Hi,

I just realized that the currency is not always correctly detected. Here is an example:

rd@h370:/usr/share/doc/libfinance-quote-perl/examples$ ./stockdump.pl Fondsweb LU0823414635
Use of uninitialized value in join or string at /usr/share/perl5/Finance/Quote/Fondsweb.pm line 90.
Use of uninitialized value in join or string at /usr/share/perl5/Finance/Quote/Fondsweb.pm line 90.
$VAR1 = {
          'LU0823414635type' => 'fund',
          'LU0823414635year_range' => ' - 303.18',
          'LU0823414635isin' => 'LU0823414635',
          'LU0823414635currency' => undef,
          'LU0823414635method' => 'fondsweb',
          'LU0823414635last' => '.',
          'LU0823414635name' => 'BNP Paribas Funds Energy Transition C',
          'LU0823414635success' => 1,
          'LU0823414635isodate' => '2021-03-04',
          'LU0823414635date' => '03/04/2021',
          'LU0823414635nav' => '.'
        };
rd@h370:/usr/share/doc/libfinance-quote-perl/examples$

I suspect it is because the price 1.237,88 EUR is >= 1.000,00 EUR and there is a dot separate for the thousands.

Looking at the regexp, this

root@h370:/usr/share/perl5/Finance/Quote# diff -u Fondsweb.pm.orig Fondsweb.pm
--- Fondsweb.pm.orig    2021-01-25 13:32:09.000000000 +0100
+++ Fondsweb.pm 2021-03-07 12:52:09.068270289 +0100
@@ -80,14 +80,14 @@
                        my $details = $te->table(0, 6);
                        my $lastRowIndex = @{$details->rows} - 1;
                        # extract data with re
-                       my @highest = $details->cell($lastRowIndex - 1, 1) =~ m/^(\d+),(\d+)\s.+/;
-                       my @lowest = $details->cell($lastRowIndex, 1) =~ m/^(\d+),(\d+)\s.+/;
-                       $info{ $symbol, "year_range" } = join('.', @highest) . " - " . join('.', @lowest);
+                       my @highest = $details->cell($lastRowIndex - 1, 1) =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s.+/;
+                       my @lowest = $details->cell($lastRowIndex, 1) =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s.+/;
+                       $info{ $symbol, "year_range" } = join('.', join('',@highest[0,1]),$highest[2]) . " - " . join('.', join('',@lowest[0,1]),$lowest[2]);
 
                        # nav, last, currency
                        my $raw_nav_currency = $tree->findvalue( '//div[@class="fw--fondDetail-price"]' );
-                       my @nav_currency = $raw_nav_currency =~ m/^(\d+),(\d+)\s(\w+)/;
-                       $info{ $symbol, 'nav' } = join('.', @nav_currency[0,1]);
+                       my @nav_currency = $raw_nav_currency =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s(\w+)/;
+                       $info{ $symbol, 'nav' } = join('.', join('',@nav_currency[0,1]),$nav_currency[2]);
                        $info{ $symbol, 'last' } = $info{ $symbol, 'nav' };
                        $info{ $symbol, 'currency' } = $nav_currency[-1];
 
root@h370:/usr/share/perl5/Finance/Quote#

fixes the problem for me:

rd@h370:/usr/share/doc/libfinance-quote-perl/examples$ ./stockdump.pl Fondsweb LU0823414635
$VAR1 = {
          'LU0823414635currency' => 'EUR',
          'LU0823414635isin' => 'LU0823414635',
          'LU0823414635date' => '03/04/2021',
          'LU0823414635year_range' => '1489.87 - 303.18',
          'LU0823414635name' => 'BNP Paribas Funds Energy Transition C',
          'LU0823414635isodate' => '2021-03-04',
          'LU0823414635method' => 'fondsweb',
          'LU0823414635last' => '1237.88',
          'LU0823414635success' => 1,
          'LU0823414635nav' => '1237.88',
          'LU0823414635type' => 'fund'
        };
rd@h370:/usr/share/doc/libfinance-quote-perl/examples$

Note, I have essentially zero Perl experience, it would be good if somebody reviews the changes (which for regexps is always a pain for me) before committing them.

Thanks
Rainer

The text was updated successfully, but these errors were encountered:

rdorsch · 2021-03-07T17:14:24Z

For gnucash the symbol needs to be returned, therefore I added the symbol line:

root@h370:/usr/share/perl5/Finance/Quote# diff -u Fondsweb.pm.orig Fondsweb.pm
--- Fondsweb.pm.orig    2021-01-25 13:32:09.000000000 +0100
+++ Fondsweb.pm 2021-03-07 18:07:20.650301418 +0100
@@ -80,18 +80,19 @@
                        my $details = $te->table(0, 6);
                        my $lastRowIndex = @{$details->rows} - 1;
                        # extract data with re
-                       my @highest = $details->cell($lastRowIndex - 1, 1) =~ m/^(\d+),(\d+)\s.+/;
-                       my @lowest = $details->cell($lastRowIndex, 1) =~ m/^(\d+),(\d+)\s.+/;
-                       $info{ $symbol, "year_range" } = join('.', @highest) . " - " . join('.', @lowest);
+                       my @highest = $details->cell($lastRowIndex - 1, 1) =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s.+/;
+                       my @lowest = $details->cell($lastRowIndex, 1) =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s.+/;
+                       $info{ $symbol, "year_range" } = join('.', join('',@highest[0,1]),$highest[2]) . " - " . join('.', join('',@lowest[0,1]),$lowest[2]);
 
                        # nav, last, currency
                        my $raw_nav_currency = $tree->findvalue( '//div[@class="fw--fondDetail-price"]' );
-                       my @nav_currency = $raw_nav_currency =~ m/^(\d+),(\d+)\s(\w+)/;
-                       $info{ $symbol, 'nav' } = join('.', @nav_currency[0,1]);
+                       my @nav_currency = $raw_nav_currency =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s(\w+)/;
+                       $info{ $symbol, 'nav' } = join('.', join('',@nav_currency[0,1]),$nav_currency[2]);
                        $info{ $symbol, 'last' } = $info{ $symbol, 'nav' };
                        $info{ $symbol, 'currency' } = $nav_currency[-1];
 
                        # Other metadata
+                       $info{ $symbol, "symbol" } = $symbol;
                        $info{ $symbol, 'method' } = 'fondsweb';
                        $info{ $symbol, "type" } = "fund";
                        $info{ $symbol, "success" } = 1;
root@h370:/usr/share/perl5/Finance/Quote#

See also

https://lists.gnucash.org/pipermail/gnucash-devel/2021-March/045777.html

rdorsch · 2021-03-07T22:23:44Z

There is another issue in the module: It does not support batched requests (as needed for e.g. gnucash):

rd@h370:/usr/share/doc/libfinance-quote-perl/examples$ ./stockdump.pl fondsweb LU0119891520 LU2027374805|grep last
          'LU0119891520last' => '708.99',
          'LU2027374805last' => '708.99',

Making tree local in the foreach loop fixes this:

root@h370:/usr/share/perl5/Finance/Quote# diff -u Fondsweb.pm.orig Fondsweb.pm
--- Fondsweb.pm.orig    2021-01-25 13:32:09.000000000 +0100
+++ Fondsweb.pm 2021-03-07 23:18:39.479740161 +0100
@@ -39,11 +39,11 @@
        my $quoter = shift;
        my @symbols  = @_;
        my $te = HTML::TableExtract->new( depth => 0, count => 6 );
-       my $tree = HTML::TreeBuilder::XPath->new;
        my %info;
 
        # Iterate over each symbol
        foreach my $symbol (@symbols) {
+               my $tree = HTML::TreeBuilder::XPath->new;
                my $url = $FONDSWEB_URL . $symbol;
                #~ debug_ua( $quoter->user_agent );
 
@@ -80,18 +80,19 @@
                        my $details = $te->table(0, 6);
                        my $lastRowIndex = @{$details->rows} - 1;
                        # extract data with re
-                       my @highest = $details->cell($lastRowIndex - 1, 1) =~ m/^(\d+),(\d+)\s.+/;
-                       my @lowest = $details->cell($lastRowIndex, 1) =~ m/^(\d+),(\d+)\s.+/;
-                       $info{ $symbol, "year_range" } = join('.', @highest) . " - " . join('.', @lowest);
+                       my @highest = $details->cell($lastRowIndex - 1, 1) =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s.+/;
+                       my @lowest = $details->cell($lastRowIndex, 1) =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s.+/;
+                       $info{ $symbol, "year_range" } = join('.', join('',@highest[0,1]),$highest[2]) . " - " . join('.', join('',@lowest[0,1]),$lowest[2]);
 
                        # nav, last, currency
                        my $raw_nav_currency = $tree->findvalue( '//div[@class="fw--fondDetail-price"]' );
-                       my @nav_currency = $raw_nav_currency =~ m/^(\d+),(\d+)\s(\w+)/;
-                       $info{ $symbol, 'nav' } = join('.', @nav_currency[0,1]);
+                       my @nav_currency = $raw_nav_currency =~ m/^(\d+)\.{0,1}(\d*),(\d+)\s(\w+)/;
+                       $info{ $symbol, 'nav' } = join('.', join('',@nav_currency[0,1]),$nav_currency[2]);
                        $info{ $symbol, 'last' } = $info{ $symbol, 'nav' };
                        $info{ $symbol, 'currency' } = $nav_currency[-1];
 
                        # Other metadata
+                       $info{ $symbol, "symbol" } = $symbol;
                        $info{ $symbol, 'method' } = 'fondsweb';
                        $info{ $symbol, "type" } = "fund";
                        $info{ $symbol, "success" } = 1;
root@h370:/usr/share/perl5/Finance/Quote#

Output looks good now:

rd@h370:/usr/share/doc/libfinance-quote-perl/examples$ ./stockdump.pl fondsweb LU0119891520 LU2027374805|grep last
          'LU2027374805last' => '68.39',
          'LU0119891520last' => '708.99',
rd@h370:/usr/share/doc/libfinance-quote-perl/examples$

In summary the patch above fixes three issues:

prices greater than or equal 1000
symbol needs to be returned for gnucash
support for batched requests

vincentl · 2021-03-10T22:21:28Z

Thanks for the patch. It will be in the next release candidate.

The changes only permit a single comma in the price. Does Fondsweb quote in other currencies where we might expect multiple commas?

rdorsch · 2021-03-11T21:44:08Z

Many thanks for your response.

You have a very good point. I browsed though fondsweb to find a fond with >= 1 Mio but could not find one. A few JPY fonds were in the order of 30.000 JPY, but nothing higher. I would expect that the correct format is

[...]xxx.xxx.xxx,xxxx

A fund in Indonesian rupiah (IDR) could be above one million IDR, but I did not find a fund in IDR at all.

A more robust fix might be to first remove all . characters and use then the original regexp. Since I have essentially zero Perl experience it probably does not make sense if I come up with a change proposal.

vincentl · 2021-03-19T11:58:34Z

Added a simple function to handle multiple '.' separators and included your fix for batching. Will land in rc2.

rdorsch · 2021-03-20T16:12:04Z

Many thanks :-)

I noticed that you dropped

                  $info{ $symbol, "symbol" } = $symbol;

Without that information, I see a warning:

rd@h370:/usr/share/doc/libfinance-quote-perl/examples$ gnc-fq-dump fondsweb LU0119891520 LU2027374805
Finance::Quote fields Gnucash uses:
symbol: LU0119891520 (deduced) <=== required
date: 03/19/2021 <=== recommended
currency: EUR <=== required
last: 732.55 <=\
nav: 732.55 <=== one of these
price: <=/
timezone: <=== optional

** This stock quote cannot be used by GnuCash!

=====

Finance::Quote fields Gnucash uses:
symbol: LU2027374805 (deduced) <=== required
date: 03/19/2021 <=== recommended
currency: USD <=== required
last: 71.48 <=\
nav: 71.48 <=== one of these
price: <=/
timezone: <=== optional

** This stock quote cannot be used by GnuCash!

rd@h370:/usr/share/doc/libfinance-quote-perl/examples$

which goes away if the symbol is returned.

I did not yet find a malfunction inside gnucash, so the warning could also be a false alarm.

vincentl added the next release label Mar 10, 2021

vincentl added a commit that referenced this issue Mar 19, 2021

Fix issue with comma/decimal separators. See issue #179.

578ffa2

vincentl closed this as completed Mar 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fondsweb: currency not always detected #179

Fondsweb: currency not always detected #179

rdorsch commented Mar 7, 2021 •

edited

rdorsch commented Mar 7, 2021

rdorsch commented Mar 7, 2021

vincentl commented Mar 10, 2021

rdorsch commented Mar 11, 2021

vincentl commented Mar 19, 2021

rdorsch commented Mar 20, 2021

Fondsweb: currency not always detected #179

Fondsweb: currency not always detected #179

Comments

rdorsch commented Mar 7, 2021 • edited

rdorsch commented Mar 7, 2021

rdorsch commented Mar 7, 2021

vincentl commented Mar 10, 2021

rdorsch commented Mar 11, 2021

vincentl commented Mar 19, 2021

rdorsch commented Mar 20, 2021

rdorsch commented Mar 7, 2021 •

edited