Skip to content

v3.2.0

Latest
Latest

Choose a tag to compare

View all tags
@othiym23 othiym23 released this 11 Jan 03:38
· 58 commits to master since this release
v3.2.0
c26c6c4

packard pack

This command is the whole reason I started writing this tool last February. You give it a set of files and file tree roots, and it assembles all of the found files into albums. Then it figures out the block size and the blocks free on the destination path you point it towards, and then figures how it can most efficiently pack a subset of the albums into the available space on that device. It uses a number of tricks, some of which are probably too tricky for its own good, but I'm using it now, and while it needs refinement, it's pretty much exactly what I hoped it would be. More complete docs are in README.md.

  • f1f4a16 pack: Add a command to fill a volume with releases, optimally. (@othiym23)
  • 18db393 Calculate the space free on a volume by calling df (TBD: figure out how to do this from the CLI on Windows). (@othiym23)

packard audit

packard optimize

  • 430c46c optimize: Use src/utils/knapsack-albums.js, just like packard pack. (@othiym23)

packard pls

  • 71c345c pls: Use less stilted language when indicating how many tracks were included in the playlist. (@othiym23)
  • 906c086 pls: Explicitly sort by date now that the underlying album-assembly method returns the album list unsorted. (@othiym23)

packard unpack

  • 2e08f4a unpack: Don't report anything when no albums are extracted. (@othiym23)

simplify how file scanning works and fix the progress bar

  • d794bab src/flatten-tracks.js doesn't need to exist, and DRY up the rest of filesystem scanning a bit. (@othiym23)
  • 4cd9f7b Merge src/read-fs-{artists,albums,tracks}.js into src/read-fs-artists.js. This is the first step towards completely revamping how packard builds up the list of files to scan, as well as centralizing cruft handling in a more useful way. (@othiym23)
  • 48416fa Simplify path list generation. Not wired in, but a lot simpler than src/read-fs-artists.js, even as it reuses a lot of the same code. (@othiym23)

clean up Promise usage

  • af1d92f Merge src/albums.js into src/command/albums.js, and simplify how Promises are used. Extract sort functions into src/utils/sort.js. (@othiym23)
  • 12b3dea Simplify code by simplifying how Promises are used. Also, new versions of tap allow you to return Promises inside tests, and tap will figure out what to do with them. (@othiym23)
  • a760fee Bluebird has .mapSeries. Use it to call out operations that need to remain serialized. (@othiym23)

improve metadata testing

  • d89c170 Simplify the tag reader interface by eliminating the need for "extras". (@othiym23)
  • 916aa85 Test tag readers to make sure they don't crash when audio files don't contain tags. (@othiym23)
  • 63f88a6 Add yet more frame / atom / field types found in tags in the wild. (@othiym23)
  • 59e63c0 Add ID3v2 writing that's good enough for writing tests for ID3v2 reading. Include CONTRIBUTING.md so people can see that they need to install eyeD3 in order to run the tests that write tags to MP3 files (sorry about that, but nothing available for Node.js is up to the job, and I don't have the time to port my own library right now). Also, eyeD3 is very nice. (@othiym23)
  • 53776aa Sometimes I'm not so good about making sure that different tests don't reuse directories. (@othiym23)
  • 0988c3c Make the FLAC tag reading tests roughly equivalent to the MP3 tests. (@othiym23)
  • 4edb01a Add QuickTime atom writing that's good enough for writing tests for .m4a file reading. This time you have to install AtomicParsley, which is a C++ program, but still the only thing out there that's trustworthy for writing tags that are interoperable with iTunes. At least both eyeD3 and AtomicParsley run on Windows (I think). (@othiym23)
  • c063a00 Put the right releases in the right directories, so that it's easier to understand the output. (@othiym23)

The FLAC metadata writing saga

There's a long and convoluted story behind this:

Problem: about 1 out of every 4 test runs, the unpack tests would fail due to finding the wrong number of albums in a zipfile. The albums are assembled from metadata, and the extra album would have all the default values from the names, indicating that the FLAC metadata wasn't being set on the extracted tracks.

Hypothesis: in failing cases, the FLAC reader is getting only part of a chunk and is terminating early because it gets a finish event before it's finished looking at chunks. (Or at least this is what I spent several hours thinking.)

          / parserGauge - parser
zipStream
          \ writeGauge - extracted

          / Through - Tokenizr
fd_slicer
          \ Through - fs.WritableStream
  • stream events are the same on failing and succeeding reads
  • nothing unusual happening in the stream buffering (checked with stream)
  • see what's getting written to the FLAC parser by monkeypatching parser.write:

on success:

002 - 01 FLACParser 8390
002 - 02 FLACParser 8390
005 - 01 FLACParser 8386
005 - 02 FLACParser 8383

on failure:

002 - 01 FLACParser 8282
002 - 02 FLACParser 8390
005 - 01 FLACParser 8386
005 - 02 FLACParser 8282
  • from this, it looks like the zipstream isn't sending all the chunks to the FLAC parser.
  • see if all the chunks are getting written to the output files by monkeypatching extracted.write:

on success:

002 - 01 extracted 8390 FLACParser 8390
002 - 02 extracted 8390 FLACParser 8390
005 - 01 extracted 8386 FLACParser 8386
005 - 02 extracted 8383 FLACParser 8383

on failure:

002 - 01 extracted 8282 FLACParser 8282
002 - 02 extracted 8390 FLACParser 8390
005 - 01 extracted 8386 FLACParser 8386
005 - 02 extracted 8383 FLACParser 8383
  • because there aren't any errors, and the length of the failed files is the same as empty.flac, it looks like the test code that writes the FLAC tags is incorrect, which is both reassuring and super irritating.

CONCLUSION: The reading code was fine, but there was a race condition in the code that populated stub FLAC tracks with metadata. Once that was fixed, the problem went away.

  • 17a8787 Fix FLAC metadata writing race, part 1. (@othiym23)
  • 091ef34 Fix FLAC metadata writing race, part 2. flac-metadata cannot deal with concurrency at all. (@othiym23)
  • 39e58d2 Write a script to stress-test packing and unpacking audio-file archives. (@othiym23)

tweaks

  • 0273dee Explain why I've put so much time into this tool. (@othiym23)
  • d3ce767 Update TODOS.md. (@othiym23)
  • b24470c TODO.md needed some cleanup. (@othiym23)
  • 46d23e1 Fix typos in CONTRIBUTING.md (@othiym23)
  • 6d46b61 Refactor zipfile unpacking for comprehensibility, including testing it a bunch more. (@othiym23)
  • b93299b Further clarify the zipfile unpacking code, including adding more useful logging. (@othiym23)
  • 6a5fff3 Update to @packard/model@3.0.1, which uses a less draconian strategy for sanitizing file and directory names. (@othiym23)
  • 4d238a1 yauzl will occasionally emit a zipfile entry or two when Bluebird.promisify is running the dezalgoed yauzl.open, which says bad things about yauzl and the V8 microtask queue, simultaneously. (@othiym23)
  • cf18f6a Use Set.prototype.size instead of relying on casting being present at runtime (because it won't be when running tests in older versions of Node.js). (@othiym23)
  • 1778724 Added logging. (@othiym23)
  • 4fe1003 Remove closures when calling untildify – they're unnecessary. (@othiym23)
  • bb0e1a6 Tell yargs to read config values from the environment with a PACKARD_ prefix. (@othiym23)
  • 31391c3 Log cover images found when assembling albums. (@othiym23)

Travis

  • 1ded5c1 Tell Travis to use g++ 4.8 when building native modules for newer versions of Node.js and nan. (@othiym23)
  • 536c29b Maybe I need to set the compiler version before install time? (@othiym23)
  • 9847521 The Travis docs tell me I should try using clang. (@othiym23)
  • 6717823 It seems that clang is problematic. Use g++ for sure. (@othiym23)
  • e2bced5 Temporarily opt out of Travis containerization so that the tests can use sudo to install eyeD3 and AtomicParsley. (@othiym23)
  • db6bed1 eyeD3 0.6, the version installed by Ubuntu Precise, doesn't support the --quiet option. (@othiym23)
  • eb441a3 Support both 0.6 and 0.7 of eyeD3 to unblock Travis. (@othiym23)
  • f108765 Find a more robust way to get the eyeD3 version so that both Travis and my dev environment can be happy at the same time. (@othiym23)
  • b8f6bb6 Travis added eyeD3 and AtomicParsley to their package whitelist, so I don't need to use sudo to install them anymore. Yay for containerized builds! (@othiym23)
  • 6147ac4 Linux df fails differently than Darwin's. (@othiym23)
  • e2ab948 Notify Slack when a Travis build finishes. (@othiym23)

build

  • f9b599c Fix package.json to point to the current main entry point. (@othiym23)
  • 1ce65bc Buffer.prototype.indexOf was only added in Node.js 5, so depend on buffertools to get that functionality in older Nodes. (@othiym23)
  • d74525c Configure babel to emit sourcemaps when transpiling, which nyc will use to transparently map back to the ES6 code for coverage measurement. So cool! (@othiym23)
  • d65ece2 Babel grinds on: update dependencies. (@othiym23)
  • c7be511 Update to latest tap. (@othiym23)
  • 6017cc8 Update dependencies. (@othiym23)
Assets 2
Loading