The 'publish' output directory is removed on every run (while copying templates). This prevents the docsetutil command from properly updating the existing .atom file and just generates a new one instead.
Use --keep-intermediate-files cmd line switch to prevent deletion.
That doesn't work. In the GBDocSetPublishGenerator at line 41 which precedes execution of docsetutil, the 'copy or move' operation deletes the destination path.
Hm, it's been a while since I checked this code, but it was originally designed so that as long as atom file was present at output path, it was updated (you could delete the package though) - as described here. If it's not working like this, it might have been broken with one of recent patches. Does it work if you comment out line 41?
Another thing - line 41 is only executed if docset wasn't installed, perhaps the whole thing was faulty from start in regard it expected installation phase before hand... You can also try installing docset to see what happens.
Doesn't work either with --install-docset as the publish is removed as well by copyTemplateFilesToOutputPath which is called unconditionally on every output generator.