Handle network retries in fetching remote entries

[ronaldtse]

Sometimes Relaton source data web sites are unreliable and may require retries.

@skalee suggests that Relaton can handle retries when fetching remote entries.

@andrew2net what do you think is the cleanest way to do so? I imagine we will need some configurable options, and we need to specify the timeouts/parallel/rate limits of each individual service so the user calls won't run into issues.

Originally from glossarist/iev-data#103.

[ronaldtse]

Please also refer to glossarist/iev-data#103 for an initial implementation.

[andrew2net]

@ronaldtse we can implement it in the relaton gem, so we don't have to implement the functionality in each flavor gem but in this case using the flavor gems separately won't handle network retries.

[ronaldtse]

@andrew2net yeah that's okay. However each flavor gem may have different endpoint limitations. How do we express them in the gems and ensure that the user complies with them? e.g. rate limit 6 per second, etc.

[andrew2net]

However each flavor gem may have different endpoint limitations. How do we express them in the gems and ensure that the user complies with them? e.g. rate limit 6 per second, etc.

@ronaldtse we can set the information about limitations in the flavor gems and get it as need

[ronaldtse]

Sounds good. Please help proceed, thanks!

[andrew2net]

@ronaldtse the relaton gem is used to fetch documents one by one, so it can't use threads to fetch documents simultaneously. Network retries without parallel querying could significantly slow down fetching documents.
I think we need either implement fetching a bunch of references or use thread pool outside the relaton gem.

[ronaldtse]

Perhaps it is wise to use a thread pool?

[andrew2net]

Yes, thread poll is that we need to speed up fetching a bunch of documents. But for now, relaton has fetch method which accepts only one reference. A caller has to wait until the relaton returns a result before fetching the next reference. We need to change the way how the other scripts use the relaton.
I suggest we leave the "fetch" method as is, for compatibility with currently existed use cases, and implement a new method "queue_fetch" which doesn't return a result immediately but allow to request all needed reference then wait until the references fetching.
The question is, do we need to implement network retries for the current "fetch" method (it could dramatically slow down fetching in some cases)? Shouldn't we implement it only in the new "queue_fetch" method?

[skalee]

How about adding retries keyword parameter with default value 0? It won't break existing users too, and I suppose that blocking #fetch will be much simpler to use than non-blocking #queue_fetch. Of course only multi-threaded clients will find this feature beneficial, but I guess this would be the case for other solutions too.

(Though I'm not sure yet if I'll be able to use that reliably. Jekyll doesn't seem to be multi-threaded. However #queue_fetch doesn't seem to be helpful in my case either.)

[ronaldtse]

@skalee you have a good point. Agree with the treatment.