Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crawl Images issue #16

Open
altomator opened this issue Nov 18, 2023 · 2 comments
Open

Crawl Images issue #16

altomator opened this issue Nov 18, 2023 · 2 comments

Comments

@altomator
Copy link

Hi all,

I'm trying out Vikus after seeing it demonstrated at the IIIF Napoli conference last June.

When I run the "Generate instance" on the default collection:
https://iiif.wellcomecollection.org/presentation/collections/genres/Stickers

or any of mine, like this one:
https://mirador.bnf.fr/?iiif-content=https://raw.githubusercontent.com/altomator/IIIF/main/collection/collection-bd.json

it failed at the CrawlImages step:
[2023-11-18 17:24:00 +0000] [685] [INFO] connection open
vikus-iiif-generator-main-vikusdocker-1 | this function is a coroutine: crawlCollection
vikus-iiif-generator-main-redis-1 | 1:M 18 Nov 2023 17:24:33.956 * 1 changes in 3600 seconds. Saving...
vikus-iiif-generator-main-redis-1 | 1:M 18 Nov 2023 17:24:33.960 * Background saving started by pid 20
vikus-iiif-generator-main-redis-1 | 20:C 18 Nov 2023 17:24:33.980 * DB saved on disk
vikus-iiif-generator-main-redis-1 | 20:C 18 Nov 2023 17:24:33.981 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
vikus-iiif-generator-main-redis-1 | 1:M 18 Nov 2023 17:24:34.067 * Background saving terminated with success
vikus-iiif-generator-main-vikusdocker-1 | crawlCollection took 2.0 seconds
vikus-iiif-generator-main-vikusdocker-1 | this function is a coroutine: crawlImages
vikus-iiif-generator-main-vikusdocker-1 | crawlImages took 0.0023 seconds
vikus-iiif-generator-main-vikusdocker-1 | this function is a coroutine: makeSpritesheets
vikus-iiif-generator-main-vikusdocker-1 | ['/modules/sharpsheet/bin/sharpsheet', '/data/juicy-omelette/images/thumbs/.jpg', '--outputPath', '/data/juicy-omelette/images/sprites', '--outputFormat', 'jpg', '--outputQuality', '60', '--sheetDimension', '2048', '--spriteSize', '224']
vikus-iiif-generator-main-vikusdocker-1 | 17:24:36 INFO INFO:rich:stdout: sharpsheet.py:42
vikus-iiif-generator-main-vikusdocker-1 | looking for images at
vikus-iiif-generator-main-vikusdocker-1 | /data/juicy-omelette/images/thumbs/
.jpg
vikus-iiif-generator-main-vikusdocker-1 | done
vikus-iiif-generator-main-vikusdocker-1 |
vikus-iiif-generator-main-vikusdocker-1 | ERROR ERROR:rich:error generating spritesheet: no sharpsheet.py:45
vikus-iiif-generator-main-vikusdocker-1 | images found
vikus-iiif-generator-main-vikusdocker-1 |
vikus-iiif-generator-main-vikusdocker-1 | makeSpritesheets took 0.18 seconds
vikus-iiif-generator-main-vikusdocker-1 | this function is a coroutine: makeFeatures
vikus-iiif-generator-main-vikusdocker-1 | 0/0 images found in cache
vikus-iiif-generator-main-vikusdocker-1 | makeFeatures took 1.2 seconds
vikus-iiif-generator-main-vikusdocker-1 | this function is a coroutine: makeUmap
vikus-iiif-generator-main-vikusdocker-1 | 17:24:39 INFO INFO:umaper:Fit UMAP dimensionReduction.py:30
vikus-iiif-generator-main-vikusdocker-1 | INFO INFO:umaper:X shape: (0,) dimensionReduction.py:32
vikus-iiif-generator-main-vikusdocker-1 | [2023-11-18 17:24:39 +0000] [572] [ERROR] Exception in ASGI application
vikus-iiif-generator-main-vikusdocker-1 | Traceback (most recent call last):
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
vikus-iiif-generator-main-vikusdocker-1 | result = await app( # type: ignore[func-returns-value]
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
vikus-iiif-generator-main-vikusdocker-1 | return await self.app(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 1106, in call
vikus-iiif-generator-main-vikusdocker-1 | await super().call(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 122, in call
vikus-iiif-generator-main-vikusdocker-1 | await self.middleware_stack(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in call
vikus-iiif-generator-main-vikusdocker-1 | raise exc
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in call
vikus-iiif-generator-main-vikusdocker-1 | await self.app(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 91, in call
vikus-iiif-generator-main-vikusdocker-1 | await self.simple_response(scope, receive, send, request_headers=headers)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 146, in simple_response
vikus-iiif-generator-main-vikusdocker-1 | await self.app(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in call
vikus-iiif-generator-main-vikusdocker-1 | raise exc
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in call
vikus-iiif-generator-main-vikusdocker-1 | await self.app(scope, receive, sender)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in call
vikus-iiif-generator-main-vikusdocker-1 | raise e
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in call
vikus-iiif-generator-main-vikusdocker-1 | await self.app(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 718, in call
vikus-iiif-generator-main-vikusdocker-1 | await route.handle(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
vikus-iiif-generator-main-vikusdocker-1 | await self.app(scope, receive, send)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
vikus-iiif-generator-main-vikusdocker-1 | response = await func(request)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 274, in app
vikus-iiif-generator-main-vikusdocker-1 | raw_response = await run_endpoint_function(
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
vikus-iiif-generator-main-vikusdocker-1 | return await dependant.call(**values)
vikus-iiif-generator-main-vikusdocker-1 | File "/scripts/main.py", line 411, in run
vikus-iiif-generator-main-vikusdocker-1 | await make_umap(
vikus-iiif-generator-main-vikusdocker-1 | File "/scripts/main.py", line 351, in make_umap
vikus-iiif-generator-main-vikusdocker-1 | await makeUmap(
vikus-iiif-generator-main-vikusdocker-1 | File "/scripts/helpers.py", line 37, in wrapper
vikus-iiif-generator-main-vikusdocker-1 | result = await helper(func, *args, **kwargs)
vikus-iiif-generator-main-vikusdocker-1 | File "/scripts/helpers.py", line 29, in helper
vikus-iiif-generator-main-vikusdocker-1 | return await func(*args, **kwargs)
vikus-iiif-generator-main-vikusdocker-1 | File "/scripts/vikus.py", line 208, in makeUmap
vikus-iiif-generator-main-vikusdocker-1 | embedding = umaper.fit_transform(features)
vikus-iiif-generator-main-vikusdocker-1 | File "/scripts/dimensionReduction.py", line 33, in fit_transform
vikus-iiif-generator-main-vikusdocker-1 | embeddings = self.umap.fit_transform(X)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/umap/umap
.py", line 2887, in fit_transform
vikus-iiif-generator-main-vikusdocker-1 | self.fit(X, y, force_all_finite)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/umap/umap_.py", line 2354, in fit
vikus-iiif-generator-main-vikusdocker-1 | X = check_array(X, dtype=np.float32, accept_sparse="csr", order="C", force_all_finite=force_all_finite)
vikus-iiif-generator-main-vikusdocker-1 | File "/usr/local/lib/python3.8/site-packages/sklearn/utils/validation.py", line 938, in check_array
vikus-iiif-generator-main-vikusdocker-1 | raise ValueError(
vikus-iiif-generator-main-vikusdocker-1 | ValueError: Expected 2D array, got 1D array instead:
vikus-iiif-generator-main-vikusdocker-1 | array=[].
vikus-iiif-generator-main-vikusdocker-1 | Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

@cpietsch
Copy link
Owner

Hi @altomator,
I will have a look at this issue. It looks like the api at wellcomecollection differs from the time the code was created.

@altomator
Copy link
Author

altomator commented Nov 25, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants