fresh install with split-config breaks the addressbook

[tharvik]

I'm the maintainer for i2p on Gentoo and we found an issue when freshly installing the package. We are using there a split-install, having a directory in /usr/share/i2p for the install and a separated user directory in /var/lib/i2p. The launcher is started in the install dir, with -Di2p.dir.app=/var/lib/i2p/app -Di2p.dir.config=/var/lib/i2p/config -Di2p.dir.router=/var/lib/i2p/router.

Now, when starting the router for the first time, some install files are copied to the user directory. The hosts.txt is put by it in the config dir.
The default config of the addressbook points to some relative files (for eg defaultSettings.put("router_addressbook", "../hosts.txt")), which is only valid in a non-split setup. The addressbook files resides in the router directory, not in the config one. So we end up with an empty addressbook, which isn't even able to sync with subscriptions as no address can be resolved.

I'm unsure of how to best approach it, I see a few ways to do so but not a clear one (from what I gathered, there is not a dedicated way to know what is the known list of address)

• should the default config point to the install directory?
  • but files there are not writable, and router_addressbook wants to update it
• should the migration process actually copy the hosts.txt to the addressbook?
  • there is a bunch of references to "hosts.txt" throughout the code, so I'm scared of having it defined sometimes via the install dir and sometimes via the addressbook

@eyedeekay, btw I registered on git.idk.i2p around a month ago, I would happily continue the conversation there if you accept me :)

[eyedeekay]

Thanks for the report and thanks for pinging me about the i2pgit.org registration. The amount of spam registration is unbelievable, people need to get in touch with me to get the accounts enabled. I approved your account a moment ago.

I think, based on what you said, there's no way we get out of it without looking at all those references to hosts.txt in the code and figuring out why it might sometimes be defined via the install dir and sometimes via the address book. So I'll add auditing those to my list. Will get back to you with more soon.

[eyedeekay]

These are all the non-comment, non-translation occurences to the string hosts.txt in java files in i2p.i2p, flagging them here to keep track:

./apps/routerconsole/jsp/WEB-INF/classes/net/i2p/router/web/jsp/help_jsp.java:686:      out.print(intl._t("Note that subscribing to a hosts.txt service is an act of trust, as a malicious subscription could give you incorrect addresses, so be careful subscribing to lists from unknown sources."));
./apps/susidns/src/java/src/i2p/susi/dns/BaseBean.java:27:    private static final String DEFAULT_PRIVATE_BOOK = "../privatehosts.txt";
./apps/susidns/src/tmp/i2p/susi/dns/jsp/addressbook_jsp.java:318:            out.print(intl._t("Export in hosts.txt format"));
./apps/susidns/src/tmp/i2p/susi/dns/jsp/addressbook_jsp.java:765:      out.print(intl._t("Import from hosts.txt file"));
./apps/susidns/src/tmp/i2p/susi/dns/jsp/subscriptions_jsp.java:234:      out.print(intl._t("Those URLs refer to published hosts.txt files."));
./apps/susidns/src/tmp/i2p/susi/dns/jsp/subscriptions_jsp.java:236:      out.print(intl._t("The default subscription is the hosts.txt from {0}, which is updated infrequently.", "i2p-projekt.i2p"));
./apps/susidns/src/tmp/i2p/susi/dns/jsp/config_jsp.java:257:      out.print(intl._t("Your public hosts.txt file (choose a path within your webserver document root)"));
./apps/susidns/src/tmp/i2p/susi/dns/jsp/config_jsp.java:259:      out.print(intl._t("Your hosts.txt (don't change)"));
./apps/susidns/src/tmp/i2p/susi/dns/jsp/index_jsp.java:210:      out.print(intl._t("It regularly updates your hosts.txt file from distributed sources or \"subscriptions\"."));
./apps/ministreaming/java/test/junit/net/i2p/client/streaming/I2PSocketAddressTest.java:25:		FileOutputStream out = new FileOutputStream("hosts.txt");
./apps/ministreaming/java/test/junit/net/i2p/client/streaming/I2PSocketAddressTest.java:35:		File f = new File("hosts.txt");
./apps/ministreaming/java/test/junit/net/i2p/client/streaming/I2PSocketEepGetTest.java:53:        FileOutputStream out = new FileOutputStream("hosts.txt");
./apps/ministreaming/java/test/junit/net/i2p/client/streaming/I2PSocketEepGetTest.java:63:        File f = new File("hosts.txt");
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:127:    private static final String FALLBACK_LIST = "hosts.txt";
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:189:                                       "If you are using clients outside the router JVM, consider using the hosts.txt " +
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:298:                _log.logAlways(Log.WARN, "No hosts.txt files found, Initialized hosts database with zero entries");
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:2197:        String lname = "privatehosts.txt";
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:2200:        lname = "userhosts.txt";
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:2203:        lname = "hosts.txt";
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:2211:            DataHelper.loadProps(props, new File("hosts.txt"), true);
./apps/addressbook/java/src/net/i2p/router/naming/BlockfileNamingService.java:2215:            System.out.println("No hosts.txt to test with");
./apps/addressbook/java/src/net/i2p/addressbook/Daemon.java:137:            opts.setProperty("file", "hosts.txt");
./apps/addressbook/java/src/net/i2p/addressbook/Daemon.java:818:        NamingService ns = getNamingService("hosts.txt");
./apps/addressbook/java/src/net/i2p/addressbook/Daemon.java:844:        defaultSettings.put("local_addressbook", "../userhosts.txt");
./apps/addressbook/java/src/net/i2p/addressbook/Daemon.java:845:        defaultSettings.put("router_addressbook", "../hosts.txt");
./apps/addressbook/java/src/net/i2p/addressbook/Daemon.java:846:        defaultSettings.put("published_addressbook", "../eepsite/docroot/hosts.txt");
./apps/addressbook/java/src/net/i2p/addressbook/Daemon.java:855:        defaultSettings.put("naming_service", "hosts.txt");
./apps/addressbook/java/src/net/i2p/addressbook/HostTxtParser.java:314:        File f = new File("tmp-hosts.txt");
./apps/addressbook/java/test/junit/net/i2p/router/naming/BlockfileNamingServiceTest.java:32:        InputStream is = getClass().getResourceAsStream("/hosts.txt");
./apps/addressbook/java/test/junit/net/i2p/router/naming/BlockfileNamingServiceTest.java:41:        hostsTxt = new File(routerDir, "hosts.txt");
./apps/addressbook/java/test/junit/net/i2p/router/naming/BlockfileNamingServiceTest.java:43:        is = getClass().getResourceAsStream("/hosts.txt");
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/I2Ping.java:126:                hostListFile = "hosts.txt";
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/I2Ping.java:205:            "ping <opts> -h (pings all hosts in hosts.txt)\n" +
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/localServer/LocalHTTPServer.java:179:            String book = "privatehosts.txt";
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/localServer/LocalHTTPServer.java:181:                book = "userhosts.txt";
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/localServer/LocalHTTPServer.java:183:                book = "hosts.txt";
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/localServer/LocalHTTPServer.java:352:        if ("hosts.txt".equals(book))
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/localServer/LocalHTTPServer.java:354:        else if ("userhosts.txt".equals(book))
./apps/i2ptunnel/java/src/net/i2p/i2ptunnel/localServer/LocalHTTPServer.java:356:        else if ("privatehosts.txt".equals(book))
./router/java/src/net/i2p/router/startup/PortableWorkingDir.java:86:        File test = new File(oldDirf, "hosts.txt");
./router/java/src/net/i2p/router/startup/WorkingDir.java:178:        File test = new File(oldDirf, "hosts.txt");
./router/java/src/net/i2p/router/startup/WorkingDir.java:349:        "hosts.txt,i2psnark.config,i2ptunnel.config,jetty-i2psnark.xml," +
./core/java/src/net/i2p/client/naming/SingleFileNamingService.java:611:        NamingService ns = new SingleFileNamingService(I2PAppContext.getGlobalContext(), "hosts.txt");
./core/java/src/net/i2p/client/naming/EepGetAndAddNamingService.java:42:    private final static String DEFAULT_HOSTS_FILE = "hosts.txt";
./core/java/src/net/i2p/client/naming/HostsTxtNamingService.java:50:        "privatehosts.txt,userhosts.txt,hosts.txt";
./core/java/src/net/i2p/data/PrivateKeyFile.java:1092:        String[] filenames = new String[] {"privatehosts.txt", "userhosts.txt", "hosts.txt"};

[zzzi2p]

I think where you went astray here is having separate router, config, and app dirs. Nobody does that and probably not tested. In trunk they're the same by default and we kinda use them interchangeably. In our startup we just set config and everything else defaults to that. In particular the migration of files to those three (one) dirs at first startup (as implemented in WorkingDir.java) may not have the right files go to the right places, or maybe we're not consistent elsewhere in the code.

Yes we can work on cleanup/fixes but it won't be in time for our next release in December. As a workaround you may wish to set them all the same or do symlinks, I don't know how you want to deal with new vs. existing installs but that's your field of expertise. But I think you may hit more weird problems if they're not all effectively pointing to the same place. That's on us that we haven't really said that anywhere before.

As far as the ../ in addressbook/config.txt, it really doesn't matter, it should work split or non-split install as long as the dir settings are all the same. And it really doesn't matter, because hosts.txt is copied from install dir to config dir, and then imported into hostsdb.blockfile (BlockfileNamingService) on first run. The files to be imported are hardcoded to be in the router dir, we're not looking in the config.txt file to see what to import. Neither hosts.txt is never modified. I guess hosts.txt isn't where we think it should be so it doesn't get imported.

Hopefully that explains what's going on, sorry for the trouble.
Thanks for supporting I2P.