Rewriting the folder structure in a hook

[krotscheck]

Heya! I have a rather odd use case - we've got a multi-cloud environment, and each of our mirrors is actually an AFS slave in each cloud, with two masters with all the data on them. The upside of this is that we only have two enormous mirrors, and each slave only keeps those things in cache that are frequently used. Downside... well, AFS has a practical folder size limit of ~64K files.

To resolve this with pip, rpm, etc, we've rewritten the folder structure, and hidden it behind mod_rewrite. For example: /foo becomes /f/foo, /bar becomes /b/bar, /q becomes /q/q. I'm currently trying to build a set of registry-static hooks which automatically generate this structure, and I need a bit of advice and/or troubleshooting help.

Firstly, relocating the tarballs is quick and simple. Yay! I can't, unfortunately, figure out how to relocate the various index.json files. Digging into follow-registry and patch-package-json showed me that the folder name for the json output is derived from the package name. Modifying that permanently changes the name of the package in the mirror, so that's not an option.

My current thought, is that I can update registry-static to provide an additional parameter in the data that gets sent to the hooks (peer to tarballs), which includes the actual output directory, and permit modifications to persist. This would propagate to where the tarballs live, etc.

I'd like to know if that's an approach you'd be comfortable with, and/or whether there's a different way to approach this problem, and if not - where the best place to apply those changes would be.

Thanks in advance :)

[davglass]

My current thought, is that I can update registry-static to provide an additional parameter in the data that gets sent to the hooks (peer to tarballs), which includes the actual output directory, and permit modifications to persist. This would propagate to where the tarballs live, etc.

I'm totally fine with this idea, @bengl thoughts?

[bengl]

If I'm reading this right, it may actually be easier to implement a shim blob-store over fs-blob-store that does whatever directory changes and modifications to JSON that are necessary.

If you could even do a rewrite rule on nginx to insert the appropriate directories so that it all looks normal to your npm clients, but I don't think that's strictly necessary.

[krotscheck]

@bengl Ya know, I think that just might work. Let me see what I can come up with and get back to this thread :)

[krotscheck]

And we have a winner! https://github.com/krotscheck/js-sorting-fs-blob-store
I'm still writing tests to make sure it's well covered, thanks for your help!

[davglass]

Awesome! 🍻

[krotscheck]

A couple more questions: Is it safe to kill the process and resume? Example:

#!/bin/bash

set -e

CMD="/usr/bin/registry-static"
CMD_ARGS="-d localhost -o /afs/.openstack.org/mirror/npm --blobstore afs-blob-store"

echo "Obtaining npm tokens and running registry-static."
k5start -t -f /etc/npm.keytab service/npm -- timeout -k 2m 30m $CMD $CMD_ARGS

echo "registry-static completed successfully, running vos release."
k5start -t -f /etc/afsadmin.keytab service/afsadmin -- vos release -v mirror.npm

Also, have either of you tried replacing the absolute tarball paths with relative URL's? Instead of http://foo.com/npm/my_mirror_package, use /npm/my_mirror_package? I can use mod_substitute to change things, but if it just works with folder paths, that'd be easier.

[davglass]

It should be safe to kill it, it keeps a sequence file for the module that it's processing and will resume at that spot when it starts back up.

As for the relative path, I don't think the npm client can handle that. It should fetch the tarball exactly from the URL that it sees it in the json file that it parses.

[krotscheck]

Gotcha, so mod_substitute's the way to go. Thanks for the insight.

[krotscheck]

Closing, as our mirrors are up and running! (mostly- second discussion pending)