Ember v1.5.3

[untaimed18]

What's New

• Drag folders onto Ember to share them. Dropping folders on the window used to name what you dropped and then open the folder picker anyway, so dropping five folders still meant choosing one of them by hand. They are shared as they land now, however many you drop at once, and the folder picker accepts more than one selection too. Dropping a file offers to share the folder holding it, since sharing here works a folder at a time. We still ask first in the two cases worth asking about: when a drop would include your user folder, which would offer your documents, desktop and pictures to other users, and when it carries far more folders than anyone means to share in one go
• The downloads list does more without leaving it. Right-clicking an empty part of the list opens the actions that apply to the whole list: pause, resume, stop or cancel everything, copy every download link, paste a link, clear finished rows, select all, and open your Downloads folder. Selecting rows also puts a Copy Link button next to Cancel, which reads Copy All Links once you have selected more than one
• Ember search results appear as they are found. We used to hold everything a lookup gathered until the whole search finished, and an Ember lookup deliberately keeps asking further peers rather than stopping early, so on a cold start you watched Kad fill the list while Ember showed nothing until the very end. Results now arrive as they are found, the first one straight away and the rest in batches. A file both networks know about updates the row you already have instead of adding a second one
• The Ember Network page shows when answers are being cut short. A peer replies to a keyword query with as many records as fit in one packet, which is about five, so a busy word can hold far more than anyone can see. There was no way to tell that apart from a quiet network. The page now counts the answers we had to trim and the records that did not fit
• Four new figures for judging whether your node is healthy. An estimate of how many nodes are out there, how many of your contacts have actually answered you rather than just been mentioned by somebody else, how far behind republishing is, and how long it has been since any Ember packet arrived. Between them they separate still joining from joined and quiet from stuck

What's Fixed

We compared Ember's network against our own Kad implementation, constant for constant, and audited every change made since 1.5.2. Most of what follows is the result. The theme is silent failure: several of these switched a feature off completely and reported nothing.

Before you upgrade

• Nothing to do. We didn't touch the library database, so you can move between 1.5.0, 1.5.1, 1.5.2 and 1.5.3 in either direction. 1.5.3 also speaks the same Ember network protocol as 1.5.2, so the two find each other normally while everyone upgrades

Publishing and search

• Your files stop quietly leaving search. Keyword publishing could switch itself off entirely, and did so after every restart on a well-connected node. Sources kept publishing the whole time, so your files stayed downloadable by anyone who already had the link while their keyword records expired and they disappeared from search. Nothing reported it, because from the inside it looked like there was nothing due
• Searching your own library returns all of it. When Ember read your own node's index it applied the limit that exists only because a reply has to fit in one packet, so it offered about five of your files. That bit hardest on a small network, where your node holds much of the index and the local read is most of the search
• A search result can't leave a download impossible to finish. In 1.5.2 we made a content hash learned from the network need two publishers to agree before we hold a file to it, and covered the path that finds sources. A hash arriving with a search result took a different route and kept the old rule, so one wrong value still meant a download that failed its final check, reopened every part, and started again forever
• A popular word can't be taken over and locked. When a keyword filled up we displaced whichever publisher held the most entries, which reads as fairness until you notice that a publisher identity is a free keypair: an arrival holding nothing always outranked an established publisher, so a few hundred keys could strip a word bare and then keep it. A full word now turns new records away, and no single publisher may hold more than 45 of its 300 entries
• A full word frees up the moment its records expire. Every limit counts records that are still resident and the tidy-up only ran every five minutes, so a word at its limit turned away genuine records for that long after the records blocking it had already died

The Ember Network

• What you store for other people survives a restart. We dropped all of it on exit. Replication refills it within the hour and publishers re-announce on their own schedules, so nothing was lost for good, but on a young network, and especially when an update restarts many nodes at once, that leaves a window where content is simply missing. The saved file isn't trusted: every record goes back in through the ordinary checks
• Publishing addresses peers that are still there. We cached whole contacts for four hours and never rechecked them, so a publish kept talking to peers the routing table had already dropped. We now look up who is closest at the time we publish
• A contact that never existed can't end a lookup early. Contacts arrive unverified, so a peer could answer with an invented address one bit away from what we were looking for, pin the front of the queue, and make every later answer look like no progress. A lookup now only counts a peer that has actually replied
• Half the background traffic is gone. Records held on other publishers' behalf were re-sent to twenty peers every hour, which was the single largest thing Ember put on the wire, about double everything else it sends. Re-sending cannot extend a record's life, because expiry is computed from the publisher's signed timestamp and every copy is identical. What it genuinely buys is reaching nodes that joined recently, and two hours buys that just as well
• Changing address doesn't leave people dialling your old one. Source records carry the address to connect to and were only refreshed on a two-hour cycle, so a DHCP lease change or an ISP reconnect left every record we had placed pointing downloaders at whoever holds that address now
• Source records expire on their own clock. They borrowed the keyword record's twenty-four hours. A source names a peer to download from, so it stops being true the moment that peer leaves, while a keyword record stays true whoever is online. Six hours survives two missed refreshes and clears a departed peer four times sooner
• A node with Kad switched off can tell whether its port is open. Only Kad's probe could establish it, so an Ember-only node marked every source record it published as firewalled for as long as it ran. That is the safe direction, but it relays connections that never needed relaying and hides exactly the open nodes that make the best relays for everyone else
• A spoofed address can't get a real peer's session dropped. When we had to shed a session both rules picked the wrong one: a peer pushed aside while it was busy looks idle, so it always seemed the right one to drop. Refreshing a peer's liveness also outlived its session by half, so every scheduled check found a dead session and paid for a fresh handshake
• Your node paces itself by how big the network is. A node among two hundred peers and a node among two hundred thousand ran identical timers. Ember now estimates the size of the network from how tightly packed its neighbours are and scales what it checks and how often against it

Library

• Copy All Links copies in the order you are looking at. The button took files in the order they were scanned rather than the order on screen, so sorting by name, size or folder changed the list in front of you and made no difference at all to what landed on the clipboard

Transfers

• Pause All and Resume All follow the filter box. Stop All and Cancel All already applied to the downloads you could actually see, but these two reached every download regardless. With a filter typed in, two commands sitting in the same menu meant different things by All, and the pair that reached rows you could not see was the dangerous half
• A long upload stops freezing at 100%. An upload counts the bytes actually sent, and a peer that re-requests data pushes that past the size of the file, routinely and by tens of megabytes over a long session. We trimmed the figure to the file size before it reached the speed calculation, so from that moment every sample looked identical and the row froze: no speed, the counter stuck exactly at the file size, and a status of Complete. One session here carried on serving at over a megabyte a second for another half hour behind a row that said it had finished

Updating

• Ember tells you when an update handed off to an installer that never ran. Installing ends by launching the installer and exiting, so nothing of ours is left to see whether it started. Our installer isn't Authenticode-signed yet, which makes every release an unknown program to SmartScreen, and a refusal there is indistinguishable from success from inside Ember: the app closes and nothing happens. The verified installer is now kept on disk and the attempt recorded, so the next launch can see the update didn't land, say so, and offer to run it
• That offer survives the routine update check. The check a few seconds after startup wiped the notice, then found the staged version again and offered to download the very same bytes and repeat the hand-off that had just failed, so the feature disabled itself in the exact situation it was built for
• A staged installer is still held to the rollback floor. Rechecking the signature answers whether these are the bytes we verified, which is a different question from whether that version is still one we will install. A staged build now has to beat the running one, and anything below the floor is deleted rather than offered

Full Changelog: v1.5.2...v1.5.3

---

This discussion was created from the release Ember v1.5.3.