v3.2.0

packard pack

This command is the whole reason I started writing this tool last February. You give it a set of files and file tree roots, and it assembles all of the found files into albums. Then it figures out the block size and the blocks free on the destination path you point it towards, and then figures how it can most efficiently pack a subset of the albums into the available space on that device. It uses a number of tricks, some of which are probably too tricky for its own good, but I'm using it now, and while it needs refinement, it's pretty much exactly what I hoped it would be. More complete docs are in README.md.

• f1f4a16 pack: Add a command to fill a volume with releases, optimally. (@othiym23)
• 18db393 Calculate the space free on a volume by calling df (TBD: figure out how to do this from the CLI on Windows). (@othiym23)

packard audit

• 9a2ccea audit: Add new genres used in my archive. (@othiym23)
• 3b1a96f Add packard audit to summary in README.md. (@othiym23)

packard optimize

• 430c46c optimize: Use src/utils/knapsack-albums.js, just like packard pack. (@othiym23)

packard pls

• 71c345c pls: Use less stilted language when indicating how many tracks were included in the playlist. (@othiym23)
• 906c086 pls: Explicitly sort by date now that the underlying album-assembly method returns the album list unsorted. (@othiym23)

packard unpack

• 2e08f4a unpack: Don't report anything when no albums are extracted. (@othiym23)

simplify how file scanning works and fix the progress bar

• d794bab src/flatten-tracks.js doesn't need to exist, and DRY up the rest of filesystem scanning a bit. (@othiym23)
• 4cd9f7b Merge src/read-fs-{artists,albums,tracks}.js into src/read-fs-artists.js. This is the first step towards completely revamping how packard builds up the list of files to scan, as well as centralizing cruft handling in a more useful way. (@othiym23)
• 48416fa Simplify path list generation. Not wired in, but a lot simpler than src/read-fs-artists.js, even as it reuses a lot of the same code. (@othiym23)

clean up Promise usage

• af1d92f Merge src/albums.js into src/command/albums.js, and simplify how Promises are used. Extract sort functions into src/utils/sort.js. (@othiym23)
• 12b3dea Simplify code by simplifying how Promises are used. Also, new versions of tap allow you to return Promises inside tests, and tap will figure out what to do with them. (@othiym23)
• a760fee Bluebird has .mapSeries. Use it to call out operations that need to remain serialized. (@othiym23)

improve metadata testing

• d89c170 Simplify the tag reader interface by eliminating the need for "extras". (@othiym23)
• 916aa85 Test tag readers to make sure they don't crash when audio files don't contain tags. (@othiym23)
• 63f88a6 Add yet more frame / atom / field types found in tags in the wild. (@othiym23)
• 59e63c0 Add ID3v2 writing that's good enough for writing tests for ID3v2 reading. Include CONTRIBUTING.md so people can see that they need to install eyeD3 in order to run the tests that write tags to MP3 files (sorry about that, but nothing available for Node.js is up to the job, and I don't have the time to port my own library right now). Also, eyeD3 is very nice. (@othiym23)
• 53776aa Sometimes I'm not so good about making sure that different tests don't reuse directories. (@othiym23)
• 0988c3c Make the FLAC tag reading tests roughly equivalent to the MP3 tests. (@othiym23)
• 4edb01a Add QuickTime atom writing that's good enough for writing tests for .m4a file reading. This time you have to install AtomicParsley, which is a C++ program, but still the only thing out there that's trustworthy for writing tags that are interoperable with iTunes. At least both eyeD3 and AtomicParsley run on Windows (I think). (@othiym23)
• c063a00 Put the right releases in the right directories, so that it's easier to understand the output. (@othiym23)

The FLAC metadata writing saga

There's a long and convoluted story behind this:

Problem: about 1 out of every 4 test runs, the unpack tests would fail due to finding the wrong number of albums in a zipfile. The albums are assembled from metadata, and the extra album would have all the default values from the names, indicating that the FLAC metadata wasn't being set on the extracted tracks.

Hypothesis: in failing cases, the FLAC reader is getting only part of a chunk and is terminating early because it gets a finish event before it's finished looking at chunks. (Or at least this is what I spent several hours thinking.)

          / parserGauge - parser
zipStream
          \ writeGauge - extracted

          / Through - Tokenizr
fd_slicer
          \ Through - fs.WritableStream

• stream events are the same on failing and succeeding reads
• nothing unusual happening in the stream buffering (checked with stream)
• see what's getting written to the FLAC parser by monkeypatching parser.write:

on success:

002 - 01 FLACParser 8390
002 - 02 FLACParser 8390
005 - 01 FLACParser 8386
005 - 02 FLACParser 8383

on failure:

002 - 01 FLACParser 8282
002 - 02 FLACParser 8390
005 - 01 FLACParser 8386
005 - 02 FLACParser 8282

• from this, it looks like the zipstream isn't sending all the chunks to the FLAC parser.
• see if all the chunks are getting written to the output files by monkeypatching extracted.write:

on success:

002 - 01 extracted 8390 FLACParser 8390
002 - 02 extracted 8390 FLACParser 8390
005 - 01 extracted 8386 FLACParser 8386
005 - 02 extracted 8383 FLACParser 8383

on failure:

002 - 01 extracted 8282 FLACParser 8282
002 - 02 extracted 8390 FLACParser 8390
005 - 01 extracted 8386 FLACParser 8386
005 - 02 extracted 8383 FLACParser 8383

• because there aren't any errors, and the length of the failed files is the same as empty.flac, it looks like the test code that writes the FLAC tags is incorrect, which is both reassuring and super irritating.

CONCLUSION: The reading code was fine, but there was a race condition in the code that populated stub FLAC tracks with metadata. Once that was fixed, the problem went away.

• 17a8787 Fix FLAC metadata writing race, part 1. (@othiym23)
• 091ef34 Fix FLAC metadata writing race, part 2. flac-metadata cannot deal with concurrency at all. (@othiym23)
• 39e58d2 Write a script to stress-test packing and unpacking audio-file archives. (@othiym23)

tweaks

• 0273dee Explain why I've put so much time into this tool. (@othiym23)
• d3ce767 Update TODOS.md. (@othiym23)
• [b24470c](b24470c0c2455b7518aeaaa9bb68b...

v3.1.0

more metadata!

It was always my plan to add support for MP3 and AAC files to packard, it's just that for DJing and listening, I've been buying nothing but FLAC files (or buying WAVs that I then transcode to FLAC before archiving them – thanks, Beatport!). It turns out the state of the art as far as tag reading isn't quite where I'd like it to be, but with some creative hacking, I was able to get the streaming ID3v2 frame and QuickTime atom parsers I needed to make this happen.

In principle, packard should now be agnostic to audio file types, and adding additional file types, should someone want to (MQA? ALAC? Monkey's Audio? shn?), should be easy at this point, assuming a streaming parser for the tag frames exist in the npm ecosystem. At some point, I'd like to add the ability to clean up tags, but that's a little stickier, given how complicated writing the different kinds of tags are.

• c68c35b ID3v2 frames: use musicmetadata to read tags from MP3 files, and create an abstract tag reader function that maps file types to specific kind of tag reader needed. (@othiym23)
• 0637077 QuickTime atoms: use mp4-parser to read tags from AAC files, and plug it into the abstract tag reader. mp4-parser assumes that everything's UTF-8, so fix up the standard iTunes atom names (which are encoded in... something else) using hax. (@othiym23)
• 460018f src/flac/* → src/metadata/flac/*. (@othiym23)
• 5525f51 src/mp3/* → src/metadata/mp3/*. (@othiym23)
• 56a1956 Add ability to extract .m4a files and scan them for metadata from zipfiles. (@othiym23)
• 664c6c4 Extract iTunes helpers from .m4a reader. (@othiym23)
• 9910efd Rename functions to reflect that packard can handle MP3 and AAC files as well as FLAC files now. (@othiym23)
• 8033d16 Extract tag internal-name-to-packard-name mappings from metadata readers. (@othiym23)
• fcc9444 Improve ID3v2 tag reading by incorporating more frame types and simplifying how mappings are used. (@othiym23)
• 27ee550 Update to @packard/model@3, which uses tags instead of flacTags. (@othiym23)
• 8da151e Add more tag types based on what's out there in the wild. (@othiym23)
• b44f31c Warn when a metadata reader encounters an unfamiliar tag type – that's the quickest way to ensure that the unknown tag type gets incorporated into the tag-reading framework. (@othiym23)

tweaks

• 180470d Burn the boats! Commit to standard! Which I already did! But I forgot to delete .eslintrc! (@othiym23)
• 5638219 albums and optimize were missing from the command-line summary included in README.md and test/cli-basic.js. (@othiym23)
• fc9c697 Use files and file tree roots consistently across packard commands. (@othiym23)
• f7855df Hack around issue with single-track albums as a way of stalling until proper cuesheet parsing can be added. (@othiym23)

v3.0.0

BREAKING

• ad096b6 Extract options into config/options.js and config/default.js. In the process, noticed that the -S option was defined twice. One of them had to be remapped, so packard optimize -S (aka --block-size) got changed to -B. Because this is a breaking change to how the CLI is invoked, this version bump must be semver-major. (@othiym23)
• 593c1bf Fix two small bugs in packard optimize option handling. (@othiym23)

CLI scaffolding redesign

• ef2eb03 src/show-albums.js → src/commands/albums.js. (@othiym23)
• eedafe6 artists: Move into src/command/artists.js, with report. (@othiym23)
• 23cb54b audit: Move into src/command/audit.js, with report. (@othiym23)
• 0e4b58f inspect: Move into src/command/inspect.js. (@othiym23)
• b684ffc Move pls and unpack into src/command, finishing the process of extracting commands from src/cli.js. (@othiym23)
• defa12e artists: Move progress bar setup from CLI scaffold into command. (@othiym23)
• 3133845 Incorporate src/metadata/index.js back in to src/command/unpack.js. The resulting module is huge and unwieldy, but it's no longer experiencing feature envy. (@othiym23)
• 58ec294 Include src/flac/scan.js into src/command/unpack.js, because it was largely redundant already. (@othiym23)

cleaning up configuration

• bcd0d65 Extract saveConfig into src/config/save.js. (@othiym23)
• f0c6908 Move log configuration to src/config/default.js (@othiym23)
• 26fd9cb Add top-level trapping of errors in commands, as well as centralized logic to make it so that every command can use --save-config. (@othiym23)
• 32083e8 Make option handling shorter. (@othiym23)
• 243f0c7 Make option handling tidier. (@othiym23)

tweaks

• 1979af4 albums: Untildify each root. (@othiym23)
• e0359a0 Always disable the progress bar on error. (@othiym23)
• ecd3aca Hoist are-we-there-yet debugging so that it's logged for every command. (@othiym23)
• cecc395 A first step towards simplifying progress group handling. It's going to take a few more. (@othiym23)
• 0f98607 When printing out banner when there are no files for packard unpack to process, disable the progress bar first, so the message doesn't get overwritten. (@othiym23)
• 2d9ffcf Handle video files in zipfiles, and just treat them like Files for now. (@othiym23)
• e691b54 Share the set of cruft names between the filesystem and zipfile scanners. (@othiym23)

build

• b328af3 Include babel-polyfill at the top level to keep Node.js 0.10 happy. (@othiym23)
• 373399f Remove lib/ before transpiling to clear out dead code. (@othiym23)
• c2105e4 cruft is a Set, so use a polyfill for Node 0.10. (@othiym23)

v2.4.0

packard optimize

Ohhhh gooddddd it's gettting so clooose! packard optimize is the last piece before being able to write packard pack – it takes a given set of directories and files, reads the metadata of the contained audio files, assembles them into albums, calculates the sizes of the albums, and then runs them through a knapsack optimization to fill the most of a specified space (with provided allocation block size) with complete releases.

• e6d4cae optimize: Figure out how many releases will fit into a given capacity. (@othiym23)

packard albums

packard albums is the next small step towards being able to have the workflow described in the release notes for packard@2.0.0-0, below. For now, it just prints a path to a folder containing an album, along with its size in blocks. I haven't really figured out how to deal with albums that are split across multiple directories (i.e. multi-disc sets), but I'm working on it.

• 1f85093 albums: Add command to assemble albums from audio file trees and then summarize the albums, with sizes. (@othiym23)
• 58e3783 Tweak output of packard albums to include only one line per album, and append total blocks used by the whole set. (@othiym23)

tweaks

• e404140 Clean up command-line option handling. Add functional tests for all of the top-level commands. (@othiym23)
• c371f02 Get off the happy path in tests to improve useful coverage. (@othiym23)
• ac8f6c1 Split LICENSE out of README. (@othiym23)

v2.3.0

packard audit

I'm super thingy about metadata quality. Almost all of my music has been tagged using Picard to ensure that all of the stuff I'm listening to is in Musicbrainz, I try to make sure I'm using (my own) standard vocabulary of musical subgenres, I'm picky about how remixes are offset in track names (square brackets not round, plz) – there's a ton of this stuff I care about, and I'm constantly dropping the ball when trying to finalize releases before putting them in my archive. This is the first spike towards automating metadata auditing in packard (a previous version of my audio file management tools did all this, but it also had a bug that caused it to corrupt ID3v2 tags, so this version just flags problem files and lets you fix them with a metadata editor you trust).

• 49f3c78 audit: Add a command to run metadata quality checks against both releases and tracks. (@othiym23)

tweaks

• bfeddf9 Assemble albums across filesystem trees. (@othiym23)
• d579c51 Where possible, treat trees and files as interchangeable when generating lists of files to scan. (@othiym23)
• dd27660 Add logging. (@othiym23)

build

• 7c45025 Add Node.js 5 to Travis build matrix. (@othiym23)
• b6ea310 Update standard and tap. (@othiym23)

v2.2.2

• 8825d2c Because yauzl and the FLAC metadata reader are both streaming, combine extracting an audio file from a zipfile with scanning it for FLAC metadata, saving a whole bunch of time. (@othiym23)
• fbf6519 Be more robust in the face of missing tag metadata. (@othiym23)
• b4aba56 Instead of grouping tracks by artist and album, which does bad things to compilations, group either by the album artist and album, or by the containing directory. (@othiym23)
• 8d66885 Warn instead of throwing when there are files left over after assembling albums. (@othiym23)
• 7ad684c Update dependencies to latest, including handling the babel@6 apocalypse and switching from babel/polyfill to babel-polyfill. (@othiym23)
• f816b88 To reduce ambiguity, Promise → Bluebird. (@othiym23)
• 1be9878 When sorting by (potentially partial) ISO date strings, use moment with explicit templates, because it gets bitter otherwise. (@othiym23)
• 0f895f2 src/utils/zip.js wasn't using graceful-fs (@othiym23)
• 209909e Only when there are covers to be copied will the progress tracker track cover copying. Satisfyingly tautological. (@othiym23)
• fa52134 Dump are-we-there-yet state at the end of commands. (@othiym23)

v2.2.1

where the hell am I going with all this?

• 44a7ebf Create TODO.md. (@othiym23)
• 890531c I use FoldingText to manage TODO.md, and it has weird whitespace requirements. (@othiym23)
• d695e1c Mark some tasks finished in TODO.md. (@othiym23)

fleshing out the (meta)data model

• 4311bcc Add a File model to wrap stats and paths. (@othiym23)
• 05d6b53 Track inherits from File, and Track.fromFLAC now sets more Track values based on FLAC tags. (@othiym23)
• 00f9ce9 Tracks have a File, instead of being a file. (Sometimes Track data will come along without being tied to a physical file.) (@othiym23)
• 127bf63 Cover inherits from File. (@othiym23)
• 607e847 Add a Cuesheet that inherits from File. (@othiym23)
• 74e5d2c Add an Archive and an AudioFile that inherits from File so all the commonly-used entity types are included in the model. (@othiym23)
• 02c84e4 Add a function to convert a Track into a SingletrackAlbum. (@othiym23)
• 11ad51e Update Albums to initialize more values based on optional passed-in values. (@othiym23)
• 24d1f8c Switch to model references from simple strings for album and track artists and track references to albums. (@othiym23)
• 57438ed Move Album tracklist sorting onto the model. (@othiym23)
• b96fca8 Extract the model into @packard/model. (@othiym23)
• 0a0622d Instead of throwing an error when encountering an unfamiliar file type, log a warning and map it to a File. (@othiym23)
• 5d97150 unpack: sourceArchive and destArchive are Archives now. (@othiym23)

refactoring & redesign

• c336dcd Extract makePlaylist into src/utils/make-playlist.js, and add the ability for packard unpack --playlist to write a playlist for that unpack run to a file. (@othiym23)
• 4019ab9 Extract playlist generation into src/utils/make-playlist.js. (@othiym23)
• 37c5caf Write functional tests for readRoot, because it didn't have any and was constantly crashing whenever anything feeding into it changed. (@othiym23)
• 7f5a450 Extract flatten to src/flatten-tracks.js. (@othiym23)
• ced5ea8 Rework how filesystem metadata is read to be based on reading the paths, rather than a recursive-descent traversal of Artist -> Album -> Track. (@othiym23)
• 0dd770b Switch to the new audio file tree reader. (@othiym23)
• 74e5d2c The code mentions "bundles" in a few places, without there being any clear concept of what they are (except as a convenient way to attach bits of data to a set of entities as various functions decorate files with metadata). Try to make that more concrete by replacing bundles with more concrete entities wherever possible. (@othiym23)
• 75e0ab3 Simplify assembling tracks into albums (@othiym23)
• cd11f1b flac.albumsFromFS mostly duplicated src/read-fs-artists.js, so consolidate on the latter. (@othiym23)
• dd325c7 Separate track metadata scanning from album assembly. (@othiym23)
• 752deb3 src/read-tree.js → src/read-fs-artists.js. (@othiym23)

tweaks

• 27ce091 Add more entries to the list of filesystem cruft. (@othiym23)
• d80e342 (Re-)Flatten audio file trees when scanning albums. (@othiym23)
• 4eb912f Don't explode when confronted with zipfiles containing audio files in subdirectories. (@othiym23)
• bf302ba Upgrade dependencies, and remove eslint pragmas rendered superfluous by new version of standard. (@othiym23)
• 99ad30f Fix up spacing in two tests to make standard happy. (@othiym23)
• a6cba9a Since this program does pretty much nothing but work with files, use graceful-fs to make it better at working with files. (@othiym23)
• 593ac69 Got rid of an extraneous .then() in a test. (@othiym23)
• 297b427 Use .finally() to simplify test cleanup. (@othiym23)
• 67b2c33 Convert from CJS to ES2015 imports everywhere. (@othiym23)
• eada3a6 More CJS into ES2015 imports. (@othiym23)
• 9b01086 Update playlist generation to accommodate changes to model. (@othiym23)
• 17f2564 Use only ES2015 module syntax for both exporting and importing. (@othiym23)
• 5b805dc The tests are still ES5.1, though. (@othiym23)
• 66e38bd tap@1.3 gathers coverage information natively, which considerably simplifies coverage gathering. Verify that the code passes standard before running test suite. (@othiym23)
• 2c165fc Run coverage instead of coveralls for Travis. (@othiym23)
• a04a559 Change test name to match the name of the module under test within the source tree. (@othiym23)
• e0cedfa es6-shim → babel/polyfill. (@othiym23)
• 9f6b772 Update to a version of tap that tolerates cyclical object references thanks to [only-shallow](http://...

v2.2.0

packard pls

• 27730ca Document packard pls. (@othiym23)

packard inspect

• 2b74c10 Document packard inspect. (@othiym23)

continuous integration

• 2be2104 Continuous integration is a good idea, so add packard to Travis. (@othiym23)
• 0f725d7 Containerized builds are faster, so enable them (by turning off the need for sudo?). (@othiym23)
• 894f86c Add istanbul to measure coverage of transpiled JavaScript, which is imperfect (how will I ever get to 100% coverage ;_;), but better than nothing. (@othiym23)
• 768d172 Only transpile once when running with istanbul. (@othiym23)
• 839316b Send coverage information to coveralls.io. (@othiym23)
• 37a054a Report to coveralls.io on successful Travis builds. (@othiym23)
• 5cb1cf7 Only run coveralls from Travis. (@othiym23)
• 5dd5fb1 Depend directly on standard. (@othiym23)

test improvements

• 8024cb7 Add tests for Albums, which now must be passed a name and album artist at creation time. (@othiym23)

TOML support

• 6f7f8c9 WIP: Add a streaming TOML generator. (@othiym23)

tweaks

• fb98cb6 Convert ES6 classes to use ES6 module syntax. (@othiym23)
• 09e915b Update all the deps, unpinning rimraf because glob@5.0.3 fixes the bad interaction between the two. (@othiym23)
• dd64e9b yargs no longer prefixes the binary name with 'node '. (@othiym23)
• f2957d7 standard is pickier about returning from module.main, so replace returns from switch with break. (@othiym23)
• 0bcb117 standard no longer requires eslint pragmas to overwrite global Promise with imported Bluebird. (@othiym23)
• d62ebd9 src/zip-utils.js → src/utils/zip.js (@othiym23)
• e4ecf3f It's nice to keep sample output around, but it doesn't need to be part of the repo. (@othiym23)

v2.1.2

• db778fb unpack: Adding better functional tests flushed out a bug in album assembly from metadata! (@othiym23)
• ceee209 standard@3.0.0-beta.0 is slightly more stringent about some style choices. (@othiym23)
• 3a7ca54 Update dependencies, but pin rimraf because rimraf@2.3 interacts badly with glob@5.0.0. (@othiym23)

v2.1.1

• 2605a92 Write functional tests for src/unpack.js and src/zip-utils.js. (@othiym23)
• 2ee2915 Add yargs summary for npm pls. (@othiym23)
• ef1f9f0 Use Bluebird's Promise.map() instead of Promise.resolve().map(). (@othiym23)
• 50c04a3 Clean up logging a little. (@othiym23)