Documentation unclear on how to use iiif-tile-seed

[cmahnke]

I'm trying to evaluate iiif-tile-seed as a drop in replacement for iiif_static.py but can't get it to work. To get forward in my experiments I would like to propose a short and a long term solution:

• Short term: Please provide a complete and working example on how to use it with Docker - the current one doesn't provide the actual config.json. Please provide a generic solution without hard coded path names
• Long term: For easier scripting make the -config-source and -config options optional and allow basic operations like generation of tiles and info.json by just passing an input file and an output directory.

BTW, the documentation is also unclear about using the tool without Docker, in the example section there is a parameter -options, that's not in the listing of switches and that isn't recognized.

Background: I'm currently using iiif_static.py from my scripts to generate tiles and info.json that can be delivered by Hugo, a static site generator, but I'm not really satisfied by it's performance and trying find replacement with parallel processing capabilities. It would be very helpful, if iiif-tile-seed can fullfil the task quicker but with a similar simple and elegant (as in: no fake URIs) command line interface.

[thisisaaronland]

Thanks for this. I'll have a look at things more closely, soon.

[cmahnke]

Great, I'm still on this, I'll add some more findings on the first on this:
To make sure it's not related to space characters in the path name I've created a new directory /tmp/neu.
Side note: URIs with %20 get translated into ' ' according to the logging output of -verbose -loglevel debug

My config.json resides in tmp/neu/etc:

{
    "level": {
        "compliance": "2"
    },
    "graphics": {
        "source": {
            "name": "native"
        }
    },
    "images": {
        "source": {
            "name": "Blob",
            "path": "file:"
        }
    }
}

In the following examples only images.source is changed.

Two examples with the configuration above:
Trying a relative file URI file:/img54.jpg

docker run -v /tmp/neu/etc:/etc/go-iiif go-iiif /bin/iiif-tile-seed -verbose -loglevel debug -config-source file:////etc/go-iiif -scale-factors 1,2,4,8 file:/page54.jpg
15:38:25.673478 [iiif-tile-seed][] DEBUG Tile waiting to seed 'page54.jpg': 339ns
15:38:25.674258 [iiif-tile-seed][] WARNING Failed to seed tiles for 'page54.jpg', blob (key "page54.jpg") (code=NotFound): stat /page54.jpg: no such file or directory
15:38:25.674828 [iiif-tile-seed][] DEBUG Time to seed tiles for 'page54.jpg': 548.585µs

This seems right, since the is no /img54.jpg - at least not on the local file system

Trying an absolute file URI file:///tmp/neu/img54.jpg

docker run -v /tmp/neu/etc:/etc/go-iiif go-iiif /bin/iiif-tile-seed -verbose -loglevel debug -config-source file:////etc/go-iiif -scale-factors 1,2,4,8 file:///tmp/neu/page58.jpg
15:37:11.760639 [iiif-tile-seed][] DEBUG Tile waiting to seed 'tmp/neu/page58.jpg': 295ns
15:37:11.761562 [iiif-tile-seed][] WARNING Failed to seed tiles for 'tmp/neu/page58.jpg', blob (key "tmp/neu/page58.jpg") (code=NotFound): stat /tmp/neu/page58.jpg: no such file or directory
15:37:11.762083 [iiif-tile-seed][] DEBUG Time to seed tiles for 'tmp/neu/page58.jpg': 694.6µs

This is clearly wrong:

 stat -s /tmp/neu/page58.jpg
st_dev=16777220 st_ino=13070654989 st_mode=0100644 st_nlink=1 st_uid=501 st_gid=0 st_rdev=0 st_size=23874313 st_atime=1599228472 st_mtime=1599228472 st_ctime=1599228472 st_birthtime=1599228472 st_blksize=4096 st_blocks=46632 st_flags=0

stat on OS level finds the file - since I don't suspect the stat syscall of Go broken this might be a mismatch of variables in the logger...

Now with images.source.path in config.json changed to file:///tmp/neu - no trailing slash:

docker run -v /tmp/neu/etc:/etc/go-iiif go-iiif /bin/iiif-tile-seed -verbose -loglevel debug -config-source file:////etc/go-iiif -scale-factors 1,2,4,8 page58.jpg
15:44:10.430842 [iiif-tile-seed][] DEBUG Tile waiting to seed 'page58.jpg': 649ns
15:44:10.431665 [iiif-tile-seed][] WARNING Failed to seed tiles for 'page58.jpg', stat /tmp/neu: no such file or directory
15:44:10.432088 [iiif-tile-seed][] DEBUG Time to seed tiles for 'page58.jpg': 537.284µs

stat is wrong again, but path variable and image name aren't concated.

Now with images.source.path in config.json changed to file:///tmp/neu/ - trailing slash

docker run -v /tmp/neu/etc:/etc/go-iiif go-iiif /bin/iiif-tile-seed -verbose -loglevel debug -config-source file:////etc/go-iiif -scale-factors 1,2,4,8 page58.jpg
15:45:41.812173 [iiif-tile-seed][] DEBUG Tile waiting to seed 'page58.jpg': 45.192µs
15:45:41.812900 [iiif-tile-seed][] WARNING Failed to seed tiles for 'page58.jpg', stat /tmp/neu: no such file or directory
15:45:41.813325 [iiif-tile-seed][] DEBUG Time to seed tiles for 'page58.jpg': 582.828µs

Trailing slash isn't passed

With empty ("'') in images.source.path:

docker run -v /tmp/neu/etc:/etc/go-iiif go-iiif /bin/iiif-tile-seed -verbose -loglevel debug -config-source file:////etc/go-iiif -scale-factors 1,2,4,8 file:///tmp/neu/page58.jpg
15:47:01.557637 [iiif-tile-seed][] DEBUG Tile waiting to seed 'tmp/neu/page58.jpg': 336ns
15:47:01.558201 [iiif-tile-seed][] WARNING Failed to seed tiles for 'tmp/neu/page58.jpg', open blob.Bucket: no scheme in URL ""
15:47:01.558471 [iiif-tile-seed][] DEBUG Time to seed tiles for 'tmp/neu/page58.jpg': 358.184µs

I'm not sure what this implies, maybe the tries with images.source.path in config.json set to somthing with the prefix file: and an argument starting with file: resulted in something like file://file: but that didn't got logged?
Probably not, there is certainly some logic to detect that - or the logging is really broken ;)

The next possible solution is the Disksource type:

            "name": "Disk",
            "path": "/tmp/neu"

Result:

docker run -v /tmp/neu/etc:/etc/go-iiif go-iiif /bin/iiif-tile-seed -verbose -loglevel debug -config-source file:////etc/go-iiif -scale-factors 1,2,4,8 page58.jpg
15:53:10.424941 [iiif-tile-seed][] DEBUG Tile waiting to seed 'page58.jpg': 438ns
15:53:10.425585 [iiif-tile-seed][] WARNING Failed to seed tiles for 'page58.jpg', stat /tmp/neu: no such file or directory
15:53:10.426033 [iiif-tile-seed][] DEBUG Time to seed tiles for 'page58.jpg': 592.51µs

But since the stat is a plain lie again, I'll stop here, probably those source definitions get converted into some URIs before passed to another layer which does the actual processing and the wrong code path (or my complete misunderstanding) is down there...

[cmahnke]

I've found one possible error in the example:

$> docker run -v /usr/local/go-iiif-vips/docker:/usr/local/go-iiif \
	go-iiif /bin/iiif-tile-seed \
	-config-source file:////usr/local/go-iiif/config \
	-scale-factors 1,2,4,8 \
	file:///zuber.jpg

The directory containing zuber.jpg isn't explicitly mounted into Docker, if I run the following line with settings as in the previous comment (context directory mounted in the first -v argument):

docker run -v /tmp/neu/etc:/etc/go-iiif -v /tmp/neu/:/tmp/neu/ go-iiif /bin/iiif-tile-seed -verbose -loglevel debug -config-source file:////etc/go-iiif -scale-factors 1,2,4,8 page58.jpg 

I'll get:

16:47:28.478877 [iiif-tile-seed][] DEBUG Tile waiting to seed 'page58.jpg': 485ns

But then the program doesn't return and don't produces any output....
After a few minutes I hit Ctrl+C

---

• I'm on OS X, /tmp/ is a symlink on /private/tmp, getting rid of the symlink (by using /private/tmp in config.json and on the command line) doesn't help.
• I've upgraded the Docker base image to golang:1.15-alpine for builder, doesn't help either.

[cmahnke]

I dug a bit deeper, the missing mount was the main problem, the missing out put if the program is running is probably just unimplemented.
Another issue ist that some configuration sections are named misleading:

    "derivatives": {
        "cache": {
            "name": "Blob",
            "path": "file:"
        }
    }

Sounds like caching, but isn't, at least in context of my use case: I's responsible that results (tiles) are actually stored to disk.
Next thing i'm going to investigate is how to set the tile size.

BTW: I've got the feeling that performance worse theniiif_static.py at least not in the default configuration, maybe it's just generating to much tiles...