attempting to install first time

[EMCP]

I had a attempt to install this in my meltano project but.. wasnt quite hitting the mark unfortunately..

https://meltano.slack.com/archives/C01PKLU5D1R/p1647388209417079

I guess usually loaders are using pip because they are written in python.. in this case.. we're using golan so.. all I tried so far was

meltano add --custom loader target-jsonl-blob

but was met with errors

Installing loader 'target-jsonl-blob'...
Loader 'target-jsonl-blob' could not be installed: failed to install plugin 'target-jsonl-blob'.
ERROR: Could not find a version that satisfies the requirement target-jsonl-blob (from versions: none)
ERROR: No matching distribution found for target-jsonl-blob

Failed to install plugin(s)

Perhaps I need to simply

git clone < this project >

go build -o target-jsonl-blob

EDIT:

just for kicks I also tried

meltano add loader target-jsonl-blob

got

Loader 'target-jsonl-blog' is not known to Meltano

[EMCP]

I built the blob and now just.. taking a look how to run it properly.. wire in the config.. so thats cool.. one issue is at first run the help menu is expecting files which.. might be counter intuitive

 ./target-jsonl-blob --help
JSONLines Singer target for blob storages

Usage:
  target-jsonl-blob [flags]

Flags:
  -c, --config string   Config file
  -h, --help            help for target-jsonl-blob
  -i, --input string    Input file
2022/03/19 10:24:42 unable to decode into struct, Config File "config" Not Found in "[]"

[EMCP]

ah I see now in meltano I have the following in my config

loaders:
  - name: target-jsonl-blog
    namespace: target_jsonl_blog
    pip_url: target-jsonl-blog
    executable: target-jsonl-blog

i will move the executable somewhere and wire it into this loader .. and start debugging the docs.. for wiring this into S3.. but it would be good to maybe confirm what I should do installation wise (like are those errors expected and I should ignore them ? )

[EMCP]

okay nice,

I wired in the full target path.. and now seeing something that makes sense

$  meltano invoke  target-jsonl-blob
2022/03/19 11:02:49 Using config file /home/.meltano/run/target-jsonl-blob/target.60f9783b-b941-45ba-993a-c32a0d0eb7e4.config.json
2022/03/19 11:02:49 unable to decode into struct, bucket is required

EDIT: I now have a config that i think is valid .. but wondering where/how to wire in AWS credentials? or is this something I need to add in source?

  - name: target-jsonl-blob
    namespace: target_jsonl_blob
    executable: /home/target-jsonl-blob/target-jsonl-blob
    config:
      bucket: file:///home/meltano-project-ib/load

[edgarrmondragon]

Hi @EMCP sorry for the delay, I was out for a couple of weeks and this got buried below other notifications.

one issue is at first run the help menu is expecting files which.. might be counter intuitive

That looks like a 🐛: #4

I wired in the full target path.. and now seeing something that makes sense

Yup, the executable has to be an absolute path. I don't think you can plug the project path but that would be helpful, e.g. $MELTANO_PROJECT_ROOT/loaders/target-jsonl-blob.

wondering where/how to wire in AWS credentials? or is this something I need to add in source?

The options are

* Environment Variables
* Shared Credentials file
* Shared Configuration file (if SharedConfig is enabled)
* EC2 Instance Metadata (credentials only)

from https://docs.aws.amazon.com/sdk-for-go/api/aws/session/

[EMCP]

im going to close this as I cannot dedicate time to sorting this out more... and focusing instead on using parquet