Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make --endpoint-url configurable by environment, or move it before the subcommand #52

Closed
smousa opened this issue Mar 17, 2024 · 1 comment

Comments

@smousa
Copy link
Contributor

smousa commented Mar 17, 2024

Hello!

I have tried quite a few parquet cli's out there, and I think this one is the best!

Here is my use case:
I am running this tool in a local environment in docker compose against a local s3 emulator/server. However, because the --endpoint-url occurs after the subcommands and not the main command, it's makes my requests a little more verbose to work with. Here is what I mean...

This is my docker-compose:

services:
    s3:
    image: scireum/s3-ninja
    pull_policy: always
    ports:
      - 9000:9000
    labels:
      - traefik.enable=true
      - traefik.http.routers.s3.rule=Host(`s3`) && PathPrefix(`/`)
      - traefik.http.services.s3.loadbalancer.server.port=9000

  parquet-tools:
    profiles:
      - script
    build:
      dockerfile_inline: | 
        ARG PYTHON_TOOLS_VERSION=0.2.15
        FROM apache/arrow-dev:amd64-conda-python-3.10
        RUN python -m ensurepip --upgrade
        RUN pip install parquet-tools==0.2.15
    environment:
      AWS_ACCESS_KEY_ID:  ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    entrypoint:
      - parquet-tools
    depends_on:
      - s3

  aws-cli:
    profiles:
      - script
    image: public.ecr.aws/aws-cli/aws-cli
    pull_policy: always
    environment:
      AWS_ACCESS_KEY_ID:  ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    entrypoint:
      - aws
      - --endpoint-url
      - http://s3:9000
    depends_on:
      - s3

volumes:
  s3:

Now in comparing the aws-cli service and the parquet-tools service, you notice that I can add the --endpoint-url to the entrypoint such that when I run it from the cli it looks like this:

$ docker compose run --rm aws-cli s3 ls s3://BUCKET/...

When using the parquet-tools binary, I have to do something like:

$ docker compose run --rm parquet-tools show --endpoint-url http://s3:9000 s3://BUCKET/...

I would love if I could call the command like this against my local s3 (either by moving the flag or having the flag supported by env var):

$ docker compose run --rm parquet-tools show s3://BUCKET/...

Thanks!

@ktrueda
Copy link
Owner

ktrueda commented Mar 19, 2024

closed by #53

@ktrueda ktrueda closed this as completed Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants