issue95: SSLExceptions Handled #99
Conversation
| public class FetchedResultHandler implements HttpDownloader.Callback { | ||
|
|
||
| private static final Logger logger = LoggerFactory.getLogger(FetchedResultHandler.class); | ||
|
|
||
| private Storage targetStorage; | ||
| private LinkStorage linkStorage; | ||
| private final Set<String> sslExceptions = Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>()); |
There was a problem hiding this comment.
This data will be lost when the crawler is restarted.
|
|
||
| public FetchedResultHandler(Storage targetStorage) { | ||
| public FetchedResultHandler(Storage linkStorage, Storage targetStorage) { | ||
| this.linkStorage = (LinkStorage) linkStorage; |
There was a problem hiding this comment.
We should not do this cast until do some other changes in the crawler architecture, because it will break some other parts of the crawler. Until we make this all interactions with the storages need to be done via the method public Object insert(Object obj).
| logger.info("Failed to download URL: {}\n>Reason: {}", link.getURL().toString(), e.getMessage()); | ||
| } | ||
| } | ||
|
|
||
| private void handleSSLExceptions(LinkRelevance link){ |
There was a problem hiding this comment.
Instead of inserting a new URL in the link storage, fixing the Fetcher to work with HTTPS redirects would be a more robust solution.
Can you invest more time in finding a solution that involves changing only the fetcher, i.e, making sure that the fetcher is be able to follow redirects to https links automatically without involving the other parts of the crawler?
That may involve configuring correctly the Apache httpclient library inside the SimpleHttpFetcher class.
No description provided.