Releases: FoxxMD/multi-scrobbler
Release list
0.17.0
What's New?
Preview UI Feature Parity
The new UI has feature parity with the existing UI, plus so many more improvements. 🎉
0.17.0 will be the last release before the new UI replaces the existing one. If you have design feedback or things aren't working as expected please report them in this issue. Also would be nice if you left a comment letting me know that everything is working as expected and you like the new design! Feedback either way is good.
Tip
Switch to the new UI by clicking Try New UI in the existing dashboard
or navigate to http://yourMsDomain/next
Listenbrainz User-Agent Changes
Listenbrainz has updated their User-Agent requirements to require a point of contact, similar to how Musicbrainz works.
The 0.16.5 release that addressed Listenbrainz User-Agent updates is no longer sufficient. Use the updated Listenbrainz for multi-scrobbler to update your configs to include a contact.
Important
This update is optional but strongly recommended as your traffic may be blocked by LZ when no contact is included. You should add this new config property as soon as possible.
A future release of multi-scrobbler will require contact in your configuration.
Improved Source Play Processing
The base Source class has been restructured to use a queue workflow, similar to the base Client class, that enables new functionality and improves all existing Play processing. Some highlights:
- Backlog play processing no longer blocks Source startup
- Ingress based Sources can now send batch scrobbles and get an immediate response without needing to wait for all Plays to process
- Source plays that fail to process can now be retried (same way as Client plays)
- Sources can process existing, queued Plays even if polling has failed
Optimized Existing Play Checks
When polling Listening History based Sources MS parses all returned history, every time, to look for changes in the list.
Previously, if you had preCompare transforms these needed to run for all history before MS could check if each Play already existed.
Now, MS stores a hash of the original data, before transforms, and uses this to check for existing Plays before transforming. This optimization can potentially save hundreds of API calls to musicbrainz and wasted CPU cycles transforming Plays that would then be thrown away immediately.
Play Events
The events shown on the Timeline for a Play in the Preview UI were previously hacked together from many different locations in a Play's data.
This data has now been restructured as a first-class data structure persisted to the database. What this means for you is that the Timeline will now show a more complete, user-friendly list of events and that future event types are trivial to create.
Important
Migrating the old data structures was more trouble that it was worth. Therefore, Plays created before this release will not show any events on the Timeline. The same is true if you decide to rollback (new Plays from this release will not render any timeline events).
Other than the above caveat, Plays are still compatible between versions and new timeline events will be created if you re-process a Play.
Api Validation and Docs
Thanks to @Mini-Sylar 's excellent express-typed-router library MS now has fully-typed api routes.
Ingress-based Sources like Webscrobbler and Listenbrainz/Last.fm Endpoints get exhaustive request body/query validation with same kind of descriptive errors configuration reports.
And all api endpoints, including ingress, are now self-documenting and added to the main docs website. If you've ever wanted to build on top of the data MS creates, you now can!
Note: Other than ingress-based endpoints, the APIs are documented but not fully stable yet. They will be stabilized once the preview ui is released.
BrainzMash Musicbrainz Server
For users who are using Musicbrainz for enhancing their scrobbles: a new, public Musicbrainz mirror is detailed in the docs under the BrainzMash Musicbrainz Server expandable section.
BrainzMash is a community-run pool of self-hosted, read-only hybrid Musicbrainz/Lidarr metadata servers. Requests are made to one, public endpoint that load-balances traffic out to all participating, private servers.
Supplementing your config with this additional Musicbrainz server can increase your transformer speed and throughput.
Security Documentation
A Security section has been added to the docs that details the risks of public access to your Multi-Scrobbler and what you can do to secure it.
Thank to @gnosticJade for the Caddy example and corrections on protected routes!
Other Changelog Highlights
⛰️ Features
- (lastfm) Use p-retry for request retries
- (librefm) Implement rate limiting
- (No Category) Use better rate limiting implementation for musicbrainz
🐛 Bug Fixes
- (atproto) Guard against null rate limit header parsing #689
- (atproto) Detect dupe rkey in batch before persist #689
- (atproto) Implement proper error handling for xrpc calls #689
- (source) Check for player repeat using closed listening session fallback #691
- (tests) Improve network mocking to compensate for libraries with frozen imports
- (ui) Log settings route schema and middleware #693
- (transform): Catch and log errors during console diff generation #684
📚 Documentation
- (No Category) Add security guidance
- (No Category) Adjust caddy snippet with better guidance
- (No Category) Cleanup atproto admonitions
- (No Category) Add BrainzMash musicbrainz guidance
- (No Category) Add openapi docs
🚜 Refactor
- (lastfm) Simplify api error reporting
🧪 Testing
- (lastfm) Test rate limiting and error handling
Contributors
Assets 2
0.17.0-RC1
0.16.5
This is a small release due to needing to get Listenbrainz's new request requirements out to stable! A larger release with new UI functionality will be coming soon.
What's New?
Listenbrainz User-Agent
A recent botnet attack has forced Listenbrainz to require all traffic to include a User-Agent and implement retry backoffs.
Multi-Scrobbler already implements backoffs but did not have a proper User-Agent in place. This release introduces a UA for all requests that should restore Listenbrainz connectivity.
teal.fm production lexicon
teal.fm has moved their lexicon to the production namespace and so this release of multi-scrobbler updates the repository records to use the new namespace.
Other Changes
- @halkeye fixed the code for generating "Edit this page" links in the docs 🎉
Contributors
Assets 2
0.16.4
The jumble of patches released this week in the aftermath of 0.16.0 were indicative of a poor release.
I take pride in making sure Multi-scrobbler is not one of those projects you are hesitant to upgrade because you don't know if "this will be the one that breaks unexpectedly". 0.16.0 was not up to that quality and for that, I apologize.
This release should hopefully make things right.
What's New?
ENV Testing and Stability
The ENV improvements made in 0.16.0 are here to stay but to address the quality issues mentioned above this release focused on building a proper test suite to validate and test all ENV keys with real, valid data:
- tests were added for all sources/clients that
- generate all ENVs for each source/client with real data derived from the ENV schema shape, using zocker
- assign these ENVs to the actual environment and then build the actual source/client, verifying no config errors and that exactly one source/client of that type exist
This now puts ENV testing on the same level of testing as file/aio config and should catch any errors, like those fixed in 0.16.1 - 0.16.3, before they get to the release stage.
Note
I have done my best to ensure backwards compatibility while also improving validation for certain types of config (ex URLs) but it is possible, since there was no test data beforehand, I have over-narrowed a usecase that now results in validation errors.
If your Multi-Scrobbler was working before 0.16.0 (or this release) and now results in config errors please open an issue. This is not your fault, I need to do a better job documenting these edge cases and will fix any of these in another release.
Full Changelog
⛰️ Features
- (No Category) Relax configureAs requirements for AIO configs
- (No Category) Strongly-type internal events #666
- (No Category) Add ENV config test suite
- (No Category) Update json schema to draft-2020-12
- (No Category) Normalize domain:port url input
- (No Category) Better validation for normalized config urls
🐛 Bug Fixes
- (discord) Fix discord ENV parsing validation
- (No Category) Check for ENV prefix with startsWith instead of includes #668
- (No Category) Narrow env config key collection to include prefix underscore #668
- (No Category) Hand-rolled env data fixtures for some edge cases
- (No Category) Hand-roll applemusic env fixture data
📚 Documentation
- (No Category) Add more guidance for File config example/editor
- (No Category) Hand-roll discord ENV updates
0.16.3
0.16.2
0.16.1
0.16.0
Important
Release 0.14.0 contains breaking changes. Read the Upgrade Path docs and 0.14.0 Release Notes before upgrading from Multi-Scrobbler earlier than 0.14.0!
Warning
BREAKING CHANGE: This release fully enforces ID requirements that were strongly recommended in 0.14.0. If you already followed the upgrade guide, or started with MS 0.14.0+ then no interaction is needed. Otherwise, upgrade to 0.14.0 before upgrading to this release.
What's New?
Reduced Memory Usage
Moving to Zod in 0.15.0 provided a big reduction in memory footprint that has been further improved in this release by moving individual Source/Client validation to be dynamically loaded. This means MS will not load validation/types for Source/Clients you do not use.
Users with a minimal number of Sources/Clients should see the largest impact from this.
Benchmarks for MS with no Sources/Clients
Startup
| Version | Runtime | Validation Context | % Memory Usage Change |
|---|---|---|---|
| 0.14.2 | tsx | ts gen + ajv | - |
| 0.15.0 | node | zod + eager validation | -13% |
| 0.16.0 | node | zod + dynamic validation | -14% |
Stabilized (after 4 minutes of running)
| Version | Runtime | Validation Context | % Memory Usage Change |
|---|---|---|---|
| 0.14.2 | tsx | ts gen + ajv | - |
| 0.15.0 | node | zod + eager validation | -22% |
| 0.16.0 | node | zod + dynamic validation | -27% |
ENV Config UX Improvements
The move to Zod in 0.15.0 also opened the door for big improvements to ENV Config experience for the end-user:
Self-Documenting
ENV variables now share, or inherit, schema from file/aio config so ENV descriptions match those found elsewhere. The ENV config for Source/Clients in the docs is now self-documenting (generated from schema) to prevent typos and make common ENV easier to extend.
Improved Validation
ENV validation at startup is now more straightforward and useful. Instead of mapping variables to file (KOITO_TOKEN => data.token) and then printing an error
✖ Invalid input: expected string, received undefined
→ at data.token
Variables are now directly validated to give you a clearer idea of what is missing or wrong
✖ Invalid input: expected string, received undefined
→ at KOITO_TOKEN
Source/Client ID Enforcement
Warning
BREAKING CHANGE but only for users who have not completed ID migration from 0.14.0
To further improve DX/UX the ID requirements first introduced in 0.14.0 are now mandatory.
Tip
If you have already added an ID to all your Sources/Clients then no action is required.
Previously, MS would log a warning and use your Source/Client name as the ID, as noted in the 0.14.0 upgrade guide. Now, it reports a configuration error if no ID is present.
To migrate, fully complete the Configuring IDs section from the 0.14.0 migration guide.
Other Changelog Highlights
⛰️ Features
- (lfm endpoint) Implement better api emulation #656
- (source) Add Ingress SOT type
- (now playing) Buffer valid updates for endpoint sources #659
- (ui) Improve UX for theme switcher
- (ui) Implement basic settings page
- (No Category) Improve location wording for config errors
🐛 Bug Fixes
📚 Documentation
- (No Category) Use italicize-bold text style for required env properties instead of separate column
- (No Category) Update env descriptions for id, name, and enable
- (No Category) Update now playing for clients
- (No Category) Fix config.json example
- (No Category) Add lastfm endpoint alternative guidance
- (No Category) Update configuration types language
0.15.0
Important
Release 0.14.0 contained breaking changes and deprecations that have been removed in 0.15.0. Read the Upgrade Path docs and 0.14.0 Release Notes before upgrading from Multi-Scrobbler earlier than 0.14.0!
What's New?
Apple Music Source
Thanks to the efforts of @Exerra , Apple Music is now implemented as a Source for Multi-Scrobbler!
Exerra, the primary developer of node-musickit-api, discovered that the API access required for MS to monitor listening activity only needs credentials that can be acquired from the browser of a logged in user. Critically, this bypasses the long-standing assumption that a paid apple developer account was required to access these APIs.
Exerra also contributed the entire Apple Music Source implementation to Multi-Scrobbler 👑 If you end up using this Source please consider sponsoring them as a thank you for all this hard work!
Improved Subsonic Playback Reporting
The OpenSubsonic API recently finalized a new spec for Playback Reporting that, when implemented by both the client and server, enables realtime player position reporting/states. This allows MS to track Subsonic players with the same level of accuracy as Spotfy, Jellyfin, and other first-class Sources!
Huge thanks to @JannisPohle for the initial PR on cleaning up subsonic players ( #640 ) but then stepping up to also implement this playback functionality ( #590 ) 👑
As emphasized above, this new functionality requires that both the Subsonic server and the client you use to play music from that server both implement Playback Reporting. An informational tooltip in the UI Preview will indicate if enhanced reporting is available.
Data Directories Standardization
Warning
This is a breaking change that mainly affects local (Nodejs) installations.
If you are using a Docker installation with default volume mounting (...:/config) then you are not affected.
Previously, all persistent data was stored in the directory defined by the CONFIG_DIR ENV, or defaulted to {projectDir}/config.
Now, these locations are configurable by data type and fallback to OS-standard locations:
- Configuration (
config.json,jellyfin.json, etc...) usesCONFIG_DIRENV or OS-standard locations like$XDG_CONFIG_HOME/multi-scrobbler - Data (
ms.db, logs folder, stored credentials, etc...) usesDATA_DIRENV or OS-standard locations like$XDG_DATA_HOME/multi-scrobbler
Upgrading
If you are using a Docker installation with default volume mounting to /config then no change is required. For backwards compatibility, the docker image installation defaults both of these ENVs to /config.
For users using the nodejs/local installation, you will likely want to explicitly set both ENVs for your start command. For instance, if you are currently storing everything in the config folder within the cloned repository and the working directory is also the repository folder then add these ENVs to the beginning of your command:
CONFIG_DIR=./config DATA_DIR=./config ... npm run startSee the Config and Data Directories section in the installation docs to find the OS-standard locations MS will use when these ENVs are not set.
Monitoring Behavior
In preparation for wider usage, the feature known as "Should Scrobble Behavior" has been renamed to "Monitoring Behavior". The configuration naming has been changed to reflect this.
Upgrading
If you are an Icecast or Azuracast user who has systemScrobble in your File/AIO config options, rename this property to autoMonitor. The behavior for this property remains the same.
Example
Old:
[
{
"name": "Station 1",
"id": "myIceStation1",
"data": {
"url": "http://myIcecast.com/radio"
},
"options": {
"systemScrobble": true
}
}
]New:
[
{
"name": "Station 1",
"id": "myIceStation1",
"data": {
"url": "http://myIcecast.com/radio"
},
"options": {
"autoMonitor": true
}
}
]Config Improvements
Warning
Contains a breaking change that potentially affects User Stage Transforms and removes deprecated caching configuration from pre 0.14.0.
Internally, config has undergone a complete overhaul and is now entirely built using Zod. This is a huge improvement over the hacky typescript-types-to-json-schema generator that was being used because the config shape now fully dictates validation, rather than there being a reflection middle step. It additionally removes the ajv library and saves almost 30MB in the docker image when uncompressed.
With the overhaul, a simplification of the config reduced development complexity and made the config explorer in docs clearer, at the expensive of a few, potential breaking changes.
Upgrading
User Stage Transforms
If you have User Stage transforms and they do not already have a "type": "user" property, you need to now add this property or validation will throw an error.
Example
Previous User Transform config could be untyped like
"options": {
"playTransform": {
"preCompare": [
{
"title": [
"badTerm",
]
}
],
}Now, all user transforms must be typed like all other transforms or config parsing will throw an error:
"options": {
"playTransform": {
"preCompare": [
{
"type": "user",
"title": [
"badTerm",
]
}
],
}Removed Deprecated Cache Config
Cache Configuration was simplified in 0.14.0 but using pre-0.14.0 configuration only caused a deprecation warning. Now, these deprecations have been removed and using the old configuration will cause an error.
Node Native Runtime
Multi-Scrobbler has been refactored to remove the dependency on tsx as the runtime. Now, MS uses Nodejs' type stripping so that it can directly run the source typescript files without needing the typescript runtime or any compilation steps.
This reduces packaging, tooling, and local run complexity:
- environments do not need to rely on
npmscripts for running MS- or needing to specify the tsx binary with node for commands
- Improves process monitoring readability
- Reduces memory usage since typescript does not need to be loaded into memory
- Reduces disk space since typescript does not need to be installed to run in production
- Tooling calls (mocha, vscode extensions) can directly use node without a shim
If you are currently running MS without using npm run start and have questions about how to migrate please open a discussion.
UI Preview Improvements
The New UI Preview introduced in 0.14.2 has seen several rounds of refinement thanks to user feedback. It is still in "read-only" mode for this release but the next release will begin to introduce write functionality. Check out the preview on your instance:
http://yourMsDomain/next
and share your feedback in the github issue!
Other Changelog Highlights
🐛 Bug Fixes
- (client) Correct create/update timestamps for queues and scrobbles
- (client) Correct SOT scrobble api requests to not have the same from/to value
- (listenbrainz) Fix play fetch more indicator
- (musicbrainz) Catch scenario where all search prereqs fail
- (rocksky) Fix missing albumArtist field #632
- (ui) Stabilize play timeline item order
- (mpd) Assign empty array if artists not defined #621
- (No Category) Fix play fetch options logging data for pageless functions
🧪 Testing
- (No Category) Fix db error test using transient db
Changes Since RC1
⛰️ Features
- (No Category) Reduce db write frequency when idle by buffering date updates #653
🐛 Bug Fixes
- (webscrobbler) Fix bad access of PlayerState #650
- (No Category) Fix missing logPath base dir argument
- (No Category) Guard against tracks/track in mb recording medium #649
📚 Documentation
- (No Category) Add missing user type to user transform examples
- (No Category) Clarify when activity is scrobbled
Contributors
Assets 2
0.15.0-RC1
Important
Release 0.14.0 contained breaking changes and deprecations that have been removed in 0.15.0. Read the Upgrade Path docs and 0.14.0 Release Notes before upgrading from Multi-Scrobbler earlier than 0.14.0!
Important
This RC contains potential breaking changes. If you have the time, please test this image and report if everything works for you! Good or bad, I need user feedback on the stability of this release.