Allow update of all chart catalogs #21
Comments
|
Dirk, wouldn't 'update all' become a runaway download? I suppose you could just have "Stop", so I guess it might prove to be very useful. Like the idea of multiple catalog updates. |
|
I woukd prefer a batch download when having a good internet connection over:
|
|
Sorry I made a bad comment. I understand what you want. It may take a little more work to get that. There are lots of things this plugin needs. Most importantly it needs a way to get charts from places other than just the US. Once we see how some other chart databases work we can think about how to automate some of the more mundane tasks like updating charts. |
|
I have a version of the plugin with this feature almost working. If we think it is important (a lot of people will think that) I could use some expert help in finding what I think is the last bug in the code. How it works:
It almost works except for some reason it keeps deciding the local files are not present (some of the time). When manually scanning for new charts it finds all of them but not when looping. |
|
Pavel, I have a version of the plugin that implements this feature. It seems to work pretty well. The code is not complex. You can check into it at https://github.com/transmitterdan/chartdldr_pi |
|
Have been trying this Update all and it works very nicely. Recently when I added catalogs for all the European rivers and hit update all I get a message that says the catalog is not available, be sure the internet connection is working (when the internet actually is working). This comment causes "Update All" to pause until I hit OK. I guess the catalog is not being found for some reason so the catalog cannot be updated and the charts are then not updated and downloaded. You hit Continue and the Update All proceeds. It appears that chartdldr_pi.io no longer has these XML catalogs. etc |
|
The files exist but the wxCurl refused to try to download them for some reason. It may be due to the https url or something else. The Curl library source is no longer part of the plugin sources so I can't trace down into the wxCurl code to find out what it doesn't like. It never actually tries to download the .xml files. |
|
I wonder what changed? Is this a bug? or something specific with my setup? |
|
I don't know what is wrong but I'm beginning to think something has happened to the raw.githubusercontent.com server itself. The Austrian chart catalog request is returning 404 Not found. The other chart catalogs on the github server don't return anything at all. The NOAA catalogs work ok. The US Army Corp of Engineers chart server has apparently changed its URL. That ACE system has been a "work in progress" for a while. |
|
Yes, agreed. All the NOAA ones work for me, but all the ones using github do not. |
|
I'm wondering if this is the old redirect problem in wxCurl that we fixed a while back. If I had to guess I would say it's something along those lines. Best regards,
|
|
It has something to do with TLS protocol. For some unknown reason the wxCurl code cannot negotiate a secure connection and that is why it does not work with htts://raw.githubusercontent.com files. It never sends the TLSv1.2 CLIENT HELLO packet. The certificate Pavel has provided is valid so that should not be the problem but I tried to use a different client certificate anyway but that didn't help. I don't know why it doesn't work but I'll keep poking around and try to figure it out. |
|
Dan... Thanks Pavel |
|
I think you guys are wizards! Pavel hard to believe you are at the beach, you should see the snow we are removing from the roofs and rear yards in Boston, including huge 500 lb Icicle chunks hanging off of the gutters. (Sorry OT) |
|
Yes, I have wireshark captures of a good transaction using a web browser and the failed transaction using the plugin. The call to wxCurl that sets up the certificate file returns OK. But the initial negotiation doesn’t work. Attached are the wireshark captures from a good transaction and the plugin failure. Dan From: Pavel Kalian Dan... Thanks Pavel — |
|
Pavel, The error reported by curl is “SSL connect error”. So it is finding the certificate (if not there would be a different error) but something goes wrong in the attempt to connect. Dan From: Pavel Kalian Dan... Thanks Pavel — |
|
Pavel, This line: SetOpt(CURLOPT_SSL_VERIFYPEER, false); will make it work. But that defeats the purpose of the certificate in the first place. Dan From: Pavel Kalian Dan... Thanks Pavel — |
|
So I guess it is something with the Certificate or with SSL. Dan, tried compile of your new version with git checkout 06e4684 chartdldr_pi-0.7-0v4.0-p-06e4684-disable-cert-win32.exe It works Updating all the xml catalogs, I note that the name of the catalogs disappears in the listing, but I think that is intentional. ( Later- I notice that when I close the plugin and then open it again, the actual XML files are listed again. - seems fine to me.) The earlier Catalog or directory listings in the top section Chart Downloader Tab contained the name of the XML file. This shortening is simpler I think. I had tried Edit Catalog and change to the actual directory in several Czech listings to the actual location However this location pathname does work and all the other ones. Hope that helps. |
|
Rick, Give me an example that does not work with exact URL. Best regards, |
|
Dan, |
|
Rick, The xml files at github.com are not the same as the files at raw.githubusercontent.com. The downloader doesn’t understand the format of the files at https://github.com/chartcatalogs/catalogs so I don’t this this is a bug in the downloader. That must be why Pavel put the correct files at raw.githubusercontent.com. Dan |
|
Rick,
In reality what you get when you point the downloader to e.g. https://github.com/chartcatalogs/catalogs/blob/master/BR_RNC_Catalog.xml is really an html file. It starts like this:
<!DOCTYPE html>
<html lang="en" class="">
Thus the downloader does not read it as a chart catalog.
Dan
|
|
Ah Ah - Thanks Makes me want to know more about raw.githubusercontent.com and how it works. |
|
'Reboot' the issue
|
|
In the fork I made "transmitterdan/chartdldr_pi" this solution seems to work. I don't see any harm in not validating the server credentials. We don't execute and server scripts or run executables. Best regards,
|
|
Dan I can confirm that I have been using your solution and it does seem to work. |
|
All, I found that the US Army Corp of Engineers has been fiddling with their servers. The chart_sources.xml file in the nohal repository is no longer correct. I have “corrected” it in the transmitterdan/chartdldr_pi fork. Dan |
|
This morning I have been trying Catalog "Update" and "Download" of charts and it seemed to be working well, with every one of my very long list of catalogs (just about all of them) Most recently I am testing "Update All" and it is finding several Catalogs that have not been downloaded entirely. It is very useful feature I think. Its is doing this in the background now and seems to be pretty good with respect to staying in the background, however when it starts a new chart with RNC Product Catalog 01 CGD it pops to the foreground, and I can then get the browser back to the foreground. There are some other catalogs that have this behavior, but I can certainly live with it. I am now on 28 0f 135 charts downloading... I will test the US Army Corps of Engineers, next, -Is that the US Rivers and Great Lakes? |
|
Update of all Catalogs works have been testing all morning with many many catalogs - most of them that are available. (Using T Dan compile) |
|
So this feature caused exactly what I expected - confusion... What about hiding it unless enabled in preferences? |
|
What about intermediate window with upate cats first then option to update all carts? Shoeing numder of files and warning. |
|
NO!!!! No more complicating the stuff. It is not understandable already now, as you can see from G's reaction |
…fusion of beginner users. Enhances #21
|
OK, this is my take on the "problem". We have now one button to confuse new users less and the functionality is just one checkbox in the preferences away from the experienced ones. Objections? |
|
I'm not sure what you are suggesting. I think we leave the preferences as it is. The update all feature is what 90% of people will use I think. It takes too much work to go through each catalog one at a time then click another button to get the charts. Most people will not want to do that. I don't think Gille is confused. |
|
I'm suggesting not having the update all button visible by default (just build the master branch). And I can assure you that not just Gilletarom, but every new user will say "Too complicated, I don't understand it" - have been doing this for some time... |
|
Pavel, I merged your changes into my fork and built it for Windows. I am ok with the preferences change you made. I don't think it is too complicated but we will see how it is accepted. I also added the feature to my fork that Gille wanted which is to tell the count of charts along with new and need to be updated charts. |
|
Can wait to try it. Have build dev env again first though. |
|
Good solution pavel! |
with a single click and have all updated charts for all catalogs preselected for download.
Alternatively allow selection of multiple catalogs (shift-click) before click on "update"
There are ~20 RNC catalogs for US waters, plus as many ENC. Updating each is several clicks and waits...
Thanks!
Dirk
The text was updated successfully, but these errors were encountered: