See #IQSS/dataverse.harvard.edu/issues/73 for the details on the investigation. AuthenticationServiceBean being a @Singleton, without any @Lock or @ConcurrencyManagement annotations, defaults to being a locking time bomb, with the capacity to paralyze the entire application. The simple solution is to move the methods that don't need to be locking out of the service, and into a regular @Stateless bean. Though it'll probably be cleaner to create a new @Singleton bean - AuthenticationProviderRegistrationService? - and move the provider registration methods there (the original intended purpose of the service bean); and leave the methods that do "normal" work in AuthenticatonService, changing it to @Stateless.

Part 2: We were exceptionally lucky that this wasn't happening with any regularity before, because we are more or less ASKING FOR IT. If publishing a dataset, in the synchronous mode takes any time at all, the spinner goes away very quickly, and then "nothing happens" - which all but encourages the user to click the publish button again. Making the service methods above non-locking will not stop these multiple Publish commands from blocking each other. But it will of course prevent them from locking up the entire application.
2 things:

1. The "spinner disappearing too quickly" appears to be unique to our prod. setup (a minute or so?) It goes on for much longer on my dev. box; so it must be something specific to how our production server is (mis)configured behind Apache. Is there an issue for that? - I couldn't find one right away. But let's investigate that - it's likely messing up other updates and such.
2. Publish NEEDS A LOCK!! Even if the spinner kept going for an hour - a user could still reload the page and try again; or end up doing it from a different window, etc. We only lock the dataset when we go into the async. mode for the file registration. We just need to lock it every time we publish, the same way we do with EditInProgress when saving changes.
   The dataset that was causing havoc yesterday has 10 files - right under the cutoff, so it was being done synchronously, and they appear to have tried to publish it 4 times, in parallel.

I suggest we address both the evil singleton and the "publish lock" part above in one PR - because why not.