Add better filtering for large data sources

[jflasher]

If we can add better filtering for ChinaAQIData and Arpalazio, I think it'd go a ways towards avoiding timeouts. My guess is that some of the timeouts (when fetch doesn't finish in time) are happening because we're trying to save a lot of measurements to the database (even if they're duplicates).

[MichalCz]

Joe, just thinking out loud, but on arpalazio I think it may be a small change in line 131 - from filter to until... I'll check that out. Ideally not the whole file would get loaded...

[MichalCz]

Well... that doesn't work well with request - sadly it crashes because of ending the piped stream before the data is consumed...

I must think of that scenario on newer versions. So this is not really an option. :(

[jflasher]

No problem! I just cut down the IL adapter which I think should cut out some needless insert attempts.

…

On November 21, 2018 at 13:08:04, Budleigh Salterton ***@***.******@***.***)) wrote: Well... that doesn't work well with request - sadly it crashes because of ending the piped stream before the data is consumed... I must think of that scenario on newer versions. So this is not really an option. :( — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub(#547 (comment)), or mute the thread(https://github.com/notifications/unsubscribe-auth/AAz0JlNPP5SsCKz-T5rYRIWpukd8yAOjks5uxZaEgaJpZM4YNgx_).

[majesticio]

I'm closing this issue because these adapters aren't currently being used and the fetch process has been updated since this issue was created.