Dns prefetch #336

JemyCheung · 2019-09-09T12:50:48Z

In order to resolve DNS resolution errors

bachue · 2019-09-10T01:16:11Z

library/src/main/java/com/qiniu/android/common/AutoZone.java

    private final String ucServer;
    private Map<ZoneIndex, ZoneInfo> zones = new ConcurrentHashMap<>();
    private Client client = new Client();
+    private ZoneInfo zoneInfo;


@JemyCheung 这变量好像一直在被赋值而没有用到过？

bachue · 2019-09-10T01:17:32Z

library/src/main/java/com/qiniu/android/common/ZoneInfo.java

@@ -1,5 +1,7 @@
 package com.qiniu.android.common;

+import android.util.Log;


@JemyCheung 感觉增加了一个没意义的 import

bachue · 2019-09-10T01:25:17Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+            dnsPrefetcher = new DnsPrefetcher();
+            synchronized (dnsPrefetcher) {
+                if (dnsPrefetcher == null) {
+                    dnsPrefetcher = new DnsPrefetcher();


@JemyCheung 为什么这里赋值了两次？一次是 synchronized 一次还不是？

bachue · 2019-09-10T01:30:49Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+    public static DnsPrefetcher dnsPrefetcher = null;
+    private String token;
+
+    private HashMap<String, List<InetAddress>> mConcurrentHashMap = new HashMap<String, List<InetAddress>>();


@JemyCheung 我好像记得我说过这个变量应该是 static 的？

bachue · 2019-09-10T01:32:02Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+    public void preFetch() throws UnknownHostException {
+        for (int i = 0; i < mHosts.size(); i++) {
+            List<InetAddress> inetAddresses = getDnsBySystem().lookup(mHosts.get(i));
+            mConcurrentHashMap.put(mHosts.get(i), inetAddresses);


@JemyCheung 这里不能因为发生了 UnknownHostException 就退出的情况，发生任何异常都是预期的（不只是 UnknownHostException 吧）要全部 lookup 完才行

@JemyCheung 这里不能因为发生了 UnknownHostException 就退出的情况，发生任何异常都是预期的（不只是 UnknownHostException 吧）要全部 lookup 完才行

异常抛出我再全部重新过一遍

bachue · 2019-09-10T01:56:54Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+    }
+
+    public List<InetAddress> getInetAddressByHost(String host) {
+        if (mConcurrentHashMap != null) {


@JemyCheung 这变量不是 init 的时候初始化的吗？为什么会有 null 的可能性？

bachue · 2019-09-10T02:17:04Z

library/src/main/java/com/qiniu/android/utils/AndroidNetwork.java

+                while (ias.hasMoreElements()) {
+                    ia = ias.nextElement();
+                    if (ia instanceof Inet6Address) {
+                        continue;// skip ipv6


@JemyCheung 考虑下 ipv6 如何兼容，不要 skip 掉，接下来 ipv6 的使用场景会越来越多的。

bachue · 2019-09-11T07:32:23Z

library/src/main/java/com/qiniu/android/http/Client.java

    }

-    private static class ResponseTag {
+    public static class ResponseTag {


@JemyCheung 这个真的需要 public 出来吗？

bachue · 2019-09-11T07:43:07Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+                    mHosts.add(zone.upDomainsList.get(i));
+            }
+        }
+        mHosts.add(Config.preQueryHost);


这里你确定不需要去重吗？

bachue · 2019-09-11T07:45:28Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+    }
+
+
+    public void preHosts() {


@JemyCheung 这个接口应该是 private 的吧

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

bachue · 2019-09-11T08:02:21Z

library/src/main/java/com/qiniu/android/storage/Configuration.java

+        if (builder.useHttps) {
+            autoZone = new AutoZone();
+        } else {
+            autoZone = new AutoZone(false);


@JemyCheung 你这里直接写 new AutoZone(builder.useHttps) 不就好了？

bachue · 2019-09-11T08:07:39Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+    public static DnsPrefetcher dnsPrefetcher = null;
+    private static String token;
+
+    private static HashMap<String, List<InetAddress>> mConcurrentHashMap = new HashMap<String, List<InetAddress>>();


@JemyCheung 这里的 mConcurrentHashMap 不应该是一个本身就是线程安全的 Map 吗？为什么只使用普通的 HashMap？这样的话任何对 mConcurrentHashMap 的操作都应该加锁了？

Mei-Zhao · 2019-09-17T02:52:52Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+        ConcurrentHashMap<String, List<InetAddress>> concurrentHashMap = dnsPrefetcher.getConcurrentHashMap();
+        byte[] dnscache = StringUtils.toByteArray(concurrentHashMap);
+
+        recorder.set("lastcache", data.getBytes());


写2个文件容易有不一致性问题，可以考虑是否只有一个文件，文件名称为第一个文件内容；每次设置新的，都删除旧的，在固定目录保留有一个文件，如果旧文件删除失败，至多有2个文件，对比一下时间戳

写2个文件容易有不一致性问题，可以考虑是否只有一个文件，文件名称为第一个文件内容；每次设置新的，都删除旧的，在固定目录保留有一个文件，如果旧文件删除失败，至多有2个文件，对比一下时间戳

我优化一下

Mei-Zhao · 2019-09-17T03:21:33Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+            e.printStackTrace();
+        }
+        if (config.dns != null) {
+            DnsPrefetcher.getDnsPrefetcher().dnsPreByCustom(config.dns);


如果用户配置了dns会做2次域名解析？一次init系统dns,一次用户dns

如果用户配置了dns会做2次域名解析？一次init系统dns,一次用户dns

这里目前是的，这个地方可能考虑了很久会有一点复杂，系统dns是一个必备或者保险操作，如果用户自定义dns预取全部失败的话，起码通过系统dns解析结果还是能正常上传。
同时这里如果只做系统dns预取或者客户自定义dns预取，会在自定义预取时还是需要先获取所有待预取hosts和传token，操作上也没有简化很多。所以在init的时候做了系统预取

Mei-Zhao · 2019-09-17T03:37:50Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+     *
+     * @return true:重新预期并缓存, false:不需要重新预取和缓存
+     */
+    public static boolean checkRePrefetchDns(String token, Configuration config) {


这个方法有文件的读写，调用之前要确保文件并发读写的问题

Mei-Zhao · 2019-09-17T07:28:03Z

library/src/main/java/com/qiniu/android/http/Client.java

                        e.printStackTrace();
                    }
-                    return okhttp3.Dns.SYSTEM.lookup(hostname);
+                } else if (DnsPrefetcher.getDnsPrefetcher().getInetAddressByHost(hostname) != null) {


不优先使用DnsPrefetcher嘛

bachue · 2019-09-23T08:16:09Z

library/src/main/java/com/qiniu/android/common/Constants.java


 public final class Constants {
-    public static final String VERSION = "7.3.15";
+    public static final String VERSION = "7.3.18";


@JemyCheung 这个版本不合适叫 7.4.0 吧

bachue · 2019-09-23T08:55:43Z

library/src/main/java/com/qiniu/android/http/DnsPrefetcher.java

+        String currentTime = String.valueOf(System.currentTimeMillis());
+        String localip = AndroidNetwork.getHostIP();
+        String akAndScope = StringUtils.getAkAndScope(token);
+        String cacheKey = format(Locale.ENGLISH, "%s:%s:%s", currentTime, localip, akAndScope);


@JemyCheung 这里建议你把 CacheKey 单独作为一个类，实现序列化和反序列化接口，否则代码中到处都是针对 ":" 的处理，难以理解

bachue · 2019-09-23T09:02:40Z

library/src/androidTest/java/com/qiniu/android/DnsApiTest.java

+        } catch (IOException e) {
+            e.printStackTrace();
+        }
+        byte[] data = recorder.get(recorder.getFileName());


@JemyCheung 考虑 getFileName() 返回 null 的情况

bachue · 2019-09-23T09:03:27Z

library/src/androidTest/java/com/qiniu/android/DnsApiTest.java

+            e.printStackTrace();
+        }
+        byte[] data = recorder.get(recorder.getFileName());
+        DnsPrefetcher.recoverDnsCache(data);


@JemyCheung 考虑 data 返回 null 的情况

bachue · 2019-10-09T11:16:52Z

library/src/main/java/com/qiniu/android/storage/ResumeUploaderFast.java

            singleDomainRetry = 1;
-            retried += 1;
+            retried.getAndAdd(1);
            upHost = config.zone.upHost(token.token, config.useHttps, upHost);


这里把 singleDomainRetry 和 upHost 也改成 atomic 变量，synchronized 就可以移除了。

JemyCheung added 13 commits July 26, 2019 11:19

add threads and retry

ebaa5fb

multithreads_retry

9e8eaff

recode retry

0620a6c

update retry

e301472

updata multithreads

cb6c771

format code

1733b23

update threads and fix a record bug

15083c2

format code

d3d7561

update multithreads

3455c49

protect mkfile func

b6e3beb

Thread

053f72e

dnsPrefetch

ee7da7a

7.3.18

ab61e43

bachue suggested changes Sep 10, 2019

View reviewed changes

recoder

af00247

bachue suggested changes Sep 11, 2019

View reviewed changes

JemyCheung added 5 commits September 14, 2019 11:41

review and code protect

43c110d

concurrentHashMap

172e671

change code style

5ece256

change code

a1b2e9b

test

eee4645

Mei-Zhao reviewed Sep 17, 2019

View reviewed changes

recode

516a8fa

Mei-Zhao reviewed Sep 17, 2019

View reviewed changes

custom dns

8667835

bachue suggested changes Sep 23, 2019

View reviewed changes

bachue reviewed Sep 23, 2019

View reviewed changes

bachue suggested changes Sep 23, 2019

View reviewed changes

JemyCheung added 3 commits September 23, 2019 19:01

recode cacheKey

08c8605

format code test

2f03fb6

safety

3971a03

bachue reviewed Oct 9, 2019

View reviewed changes

JemyCheung added 7 commits October 9, 2019 19:43

safety

31fefbd

some func have been optimised

56ebbbb

rename atomic

a6c4ee5

checkIpv6

5d4ff72

recode get ip

786dab3

getHostIp

376f3f7

test

a262f76

bachue merged commit 8509d09 into qiniu:master Oct 12, 2019

		@@ -1,5 +1,7 @@
		package com.qiniu.android.common;

		import android.util.Log;

Dns prefetch #336

Dns prefetch #336

Uh oh!

Conversation

JemyCheung commented Sep 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants