Exporting IPED files to database

[thiagocaparelli]

I want to export all the processed itens to a data warehouse. Is there a way to export the result of the processesing to a DB?

[lfcnassif]

While processing, It's possible to export all items content to a S3 MinIO object storage, there is an option in IPEDConfig.txt for that. There is another option to index all results to an ElasticSearch/OpenSearch cluster.

After processing, you can use python scripts for that too, see #875

[thiagocaparelli]

Hello Nassif!
I've tried to export to a MinIO server but I got this error. How can I pass those credentials (login/password)?

2024-04-11 14:21:56 [INFO] [engine.core.Worker] Starting MinIOTask
java.lang.RuntimeException: 'MinioCredentials' not set by ENV var, sys prop or cmd line param.
at iped.engine.task.MinIOTask.loadCredentials(MinIOTask.java:190)
at iped.engine.task.MinIOTask.init(MinIOTask.java:146)
at iped.engine.core.Worker.initTasks(Worker.java:128)
at iped.engine.core.Worker.(Worker.java:110)
at iped.engine.core.Manager.initWorkers(Manager.java:542)
at iped.engine.core.Manager.process(Manager.java:264)
at iped.app.processing.Main.startManager(Main.java:178)
at iped.app.processing.Main.execute(Main.java:240)
at iped.app.processing.Main.main(Main.java:308)
2024-04-11 14:21:56 [ERROR] [app.processing.Main] Processing Error:
java.lang.RuntimeException: 'MinioCredentials' not set by ENV var, sys prop or cmd line param.
at iped.engine.task.MinIOTask.loadCredentials(MinIOTask.java:190) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.MinIOTask.init(MinIOTask.java:146) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.initTasks(Worker.java:128) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.(Worker.java:110) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Manager.initWorkers(Manager.java:542) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Manager.process(Manager.java:264) ~[iped-engine-4.1.6.jar:?]
at iped.app.processing.Main.startManager(Main.java:178) [iped.jar:?]
at iped.app.processing.Main.execute(Main.java:240) [iped.jar:?]
at iped.app.processing.Main.main(Main.java:308) [iped.jar:?]
Thanks in advance.

[thiagocaparelli]

I've also tried to create the server without credentials, but it seems that they can't accept.
PS C:\Windows\system32> $env:MINIO_ROOT_USER=""
PS C:\Windows\system32> $env:MINIO_ROOT_PASSWORD=""
PS C:\Windows\system32> C:\minio.exe server C:\Data --console-address ":9001"
MinIO Object Storage Server
Copyright: 2015-2024 MinIO, Inc.
License: GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
Version: RELEASE.2024-04-06T05-26-02Z (go1.21.9 windows/amd64)

API: http://10.235.4.247:9000 http://127.0.0.1:9000
RootUser: minioadmin
RootPass: minioadmin

WebUI: http://10.235.4.247:9001 http://127.0.0.1:9001
RootUser: minioadmin
RootPass: minioadmin

CLI: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
$ mc alias set 'myminio' 'http://10.235.4.247:9000' 'minioadmin' 'minioadmin'

Docs: https://min.io/docs/minio/linux/index.html
Status: 1 Online, 0 Offline.
STARTUP WARNINGS:

• Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
• The standard parity is set to 0. This can lead to data loss

[lfcnassif]

You need to specify credentials using this syntax as env var:
set MinioCredentials=accesskey:xxxxxx;secretkey:yyyyyyyyyy

Or as a java sys prop:
iped.exe -DMinioCredentials=accesskey:xxxxxx;secretkey:yyyyyyyyyy -d input -o output

Or as an IPED extra property:
iped.exe -XMinioCredentials=accesskey:xxxxxx;secretkey:yyyyyyyyyy -d input -o output

[lfcnassif]

Just fixed the message above.

[thiagocaparelli]

It worked!

I'm trying now do the same in Opencluster but I got this error:

PS C:\IPED-4.1.6_plus_java_plugins\iped-4.1.6> .\iped.exe -d S:\Caparelli\SP_543_2024\1_Dados_Originais\SP_543_2024.E01 -o c:\saida
2024-04-16 17:14:05 [ERROR] [app.processing.Main] Processing Error:
org.apache.http.ConnectionClosedException: Connection is closed
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:910) ~[opensearch-rest-client-2.1.0.jar:2.1.0]
at org.opensearch.client.RestClient.performRequest(RestClient.java:307) ~[opensearch-rest-client-2.1.0.jar:2.1.0]
at org.opensearch.client.RestClient.performRequest(RestClient.java:295) ~[opensearch-rest-client-2.1.0.jar:2.1.0]
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[opensearch-rest-high-level-client-2.1.0.jar:2.1.0]
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1747) ~[opensearch-rest-high-level-client-2.1.0.jar:2.1.0]
at org.opensearch.client.RestHighLevelClient.ping(RestHighLevelClient.java:682) ~[opensearch-rest-high-level-client-2.1.0.jar:2.1.0]
at iped.engine.task.index.ElasticSearchIndexTask.init(ElasticSearchIndexTask.java:181) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.initTasks(Worker.java:128) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.(Worker.java:110) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Manager.initWorkers(Manager.java:542) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Manager.process(Manager.java:264) ~[iped-engine-4.1.6.jar:?]
at iped.app.processing.Main.startManager(Main.java:178) [iped.jar:?]
at iped.app.processing.Main.execute(Main.java:240) [iped.jar:?]
at iped.app.processing.Main.main(Main.java:308) [iped.jar:?]
Caused by: org.apache.http.ConnectionClosedException: Connection is closed
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
at java.lang.Thread.run(Unknown Source) ~[?:?]

Is it something related to credentials? I configured ENV OPENSEARCH_INITIAL_ADMIN_PASSWORD and using the docker opencluster with 2 nodes default.

Thanks in advance.

[lfcnassif]

Sorry for the delay, please see #2195.