[BUG] DB downloading failed (new MaxMind policy 2024) #853

cwilson1776 · 2024-02-25T00:11:36Z

Is there an existing issue for this?

I have searched the existing issues

#833
but it was closed without resolution

Name of mod

swag-maxmind

Name of base container

swag

Current Behavior

swag | Downloading GeoIP2 City database.
swag | tar: short read

Expected Behavior

swag | Downloading GeoIP2 City database.
(success)

Steps To Reproduce

Configured maxmind as directed in compose.yml:
DOCKER_MODS: linuxserver/mods:swag-maxmind
MAXMINDDB_LICENSE_KEY: xxxxxx
docker compose up -d swag
docker compose logs -f swag
shows that the download failed. It appears that MaxMind has recently (2024.02.07) changed their download policy, at least for new users(?):

We will begin enforcing our API endpoint policies which could cause breaking changes if you you have not configured your integration correct. Read more on our release note

From there, an additional link says that you now need to use a curl command similar to this:

curl -O -J -L -u YOUR_ACCOUNT_ID:YOUR_LICENSE_KEY 'https://download.maxmind.com/geoip/databases/GeoIP2-City-CSV/download?suffix=zip'

rather than what swag-maxmind uses:

curl --silent https://download.maxmind.com/app/geoip_download?edition_id=${EDITION_ID}&license_key=YOUR_LICENSE_KEY&suffix=tar.gz" -o "$TMPDIR/${filename}.tar.gz"

I'm not sure how to update the repo here to effect the following changes, but by making these within the container I was able to successfully download:

Modified the following lines in /etc/libmaxminddb.cron.conf

# MAXMINDDB_URL="https://download.maxmind.com/geoip/databases"

MAXMINDDB_USER_ID="..."

Modified /etc/periodic/weekly/libmaxminddb

[ -z "$MAXMINDDB_URL" ] && MAXMINDDB_URL="https://download.maxmind.com/geoip/databases"

[ -z "$MAXMINDDB_USER_ID" ] && { echo "No MaxMind user id found; exiting. Please enter your user id into /etc/libmaxminddb.cron.conf"; exit 1; }

...

curl --silent -L -J -u "${MAXMINDDB_USER_ID}:${MAXMINDDB_LICENSE_KEY}" "${MAXMINDDB_URL}/${EDITION_ID}/download?suffix=tar.gz" -o "$TMPDIR/${filename}.tar.gz"

Then, running /etc/periodic/weekly/libmaxminddb from within the container successfully downloaded the db.

Environment

- OS: VERSION="22.04.4 LTS (Jammy Jellyfish)"
- How docker service was installed: distro package manager

CPU architecture

x86-64

Docker creation

services:
  swag:
    image: lscr.io/linuxserver/swag:latest
    container_name: swag
    networks:
      - frontend
      - frontend_ip6
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    environment:
      <<: *default-tz-puid-pgid
      URL: XXXXXXXXXXXXXXX
      SUBDOMAINS: XXXXXXXXXXXXX
      VALIDATION: dns
      DNSPLUGIN: cloudflare
      PROPAGATION: 60
      DOCKER_MODS: linuxserver/mods:swag-maxmind
      MAXMINDDB_LICENSE_KEY: XXXXXXXXXXXXXXXXXXXXXXXX
    volumes:
      - $DOCKERDIR/appdata/swag/config:/config
    ports:
      - "443:443"
      - "80:80"
      - "25565:25565/tcp"

Container logs

[mod-init] Running Docker Modification Logic
[mod-init] Adding linuxserver/mods:swag-maxmind to container
[mod-init] Downloading linuxserver/mods:swag-maxmind from lscr.io
[mod-init] Installing linuxserver/mods:swag-maxmind
[mod-init] linuxserver/mods:swag-maxmind applied to container
[migrations] started
[migrations] 01-nginx-site-confs-default: skipped
[migrations] done
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝

   Brought to you by linuxserver.io
───────────────────────────────────────

To support the app dev(s) visit:
Certbot: https://supporters.eff.org/donate/support-work-on-certbot

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    1000
User GID:    1000
───────────────────────────────────────

using keys found in /config/keys
Variables set:
PUID=1000
PGID=1000
TZ=US/Eastern
URL=XXXXXXXXX
SUBDOMAINS=XXXXXXX
EXTRA_DOMAINS=
ONLY_SUBDOMAINS=false
VALIDATION=dns
CERTPROVIDER=
DNSPLUGIN=cloudflare
EMAIL=
STAGING=

Using Let's Encrypt as the cert provider
SUBDOMAINS entered, processing
Sub-domains processed are: XXXXXXXXXXXXXXXXXXXXXXXXX
No e-mail address entered or address invalid
dns validation via cloudflare plugin is selected
Certificate exists; parameters unchanged; starting nginx
The cert does not expire within the next day. Letting the cron script handle the renewal attempts overnight (2:08am).
**** adding libmaxminddb to package install list ****
[mod-init] **** Installing all mod packages ****
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/1) Installing libmaxminddb (1.7.1-r2)
Executing busybox-1.36.1-r15.trigger
OK: 201 MiB in 216 packages
Applying the maxmind mod...
Downloading GeoIP2 City database.
tar: short read
Applied the maxmind mod
[custom-init] No custom files found, skipping...
[ls.io-init] done.
Server ready

The text was updated successfully, but these errors were encountered:

aptalca · 2024-02-25T02:30:02Z

Please report upstream.

Alpine controls the db updates

https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/main/libmaxminddb/libmaxminddb.cron?ref_type=heads

cwilson1776 · 2024-02-26T03:16:14Z

Reported: aports#15813

cwilson1776 · 2024-02-29T17:12:06Z

PR committed upstream !61320

I'm not sure what the process is from here: how long until a package with this fix is available on the alpine mirrors, how the swag-maxmind mod picks up that new package... and if any documentation updates are needed in this README.md...

LinuxServer-CI · 2024-03-30T23:18:58Z

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

DougEdey · 2024-04-10T02:13:37Z

I hit this too, as a workaround, I manually downloaded the GeoLite2-City database to the shared config directory (/config/geoip2db) and made sure it was extracted as GeoLite2-City.mmdb

That at least got my swag-maxmind setup working

tanmaychimurkar · 2024-04-24T15:41:52Z

I hit this too, as a workaround, I manually downloaded the GeoLite2-City database to the shared config directory (/config/geoip2db) and made sure it was extracted as GeoLite2-City.mmdb

Hey I did the same, however, I noticed that the GeoLite2-City.mmdb does not auto reload after the duration mentioned in auto_reload parameter of maxmind.conf. Can you please confirm if this is the case for you as well?

Thanks

tanmaychimurkar · 2024-04-25T06:59:50Z

Hey just found out from Maxmind's documentation that they have put up a section for automated updating of their databases: https://dev.maxmind.com/geoip/updating-databases

Linking maxmind's GitHub repository for automatic database updates if anyone stumbles upon the same issue: https://github.com/maxmind/geoipupdate

We can map the volume to wherever swag maps /config/geoip2db/ folder on host machine. Hope this helps!!

LinuxServer-CI · 2024-05-25T23:20:15Z

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

Crash1602 · 2024-06-06T07:33:35Z

With the current swag container 2.10.0 and maxmind-mod, the problem (tar: short read) still exists. The only thing that helps is the workaround, the manual download.

libmaxminddb
**** goaccess already installed, skipping ****
**** swag-auto-reload deps already installed, skipping ****
**** libmaxminddb already installed, skipping ****
**** libmaxminddb already installed, skipping ****
**** Applied the SWAG dashboard mod ****
Applying the maxmind mod...
Downloading GeoIP2 City database.
tar: short read
Applied the maxmind mod
[custom-init] No custom files found, skipping...
MOD Auto-reload: Watching the following files/folders for changes (excluding .sample and .md files):
/config/nginx
[ls.io-init] done.
nginx: [emerg] MMDB_open("/config/geoip2db/GeoLite2-City.mmdb") failed - Error opening the specified MaxMind DB file in /config/nginx/maxmind.conf:1
�
Server ready
nginx: [emerg] MMDB_open("/config/geoip2db/GeoLite2-City.mmdb") failed - Error opening the specified MaxMind DB file in /config/nginx/maxmind.conf:1

aptalca · 2024-06-06T12:33:29Z

Alpine's libmaxmind package was only fixed in 3.20. The issue will persist until SWAG is rebased on alpine 3.20, which will be soon

LinuxServer-CI · 2024-07-07T23:23:05Z

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

IBeTheBlueCat · 2024-07-21T13:25:01Z

I can confirm that this issue is resolved in the latest version of SWAG as they rebased to Alpine 3.2, this resolved my issue with the exact same symptoms as above

Crash1602 · 2024-07-22T13:49:22Z

I can also confirm that the error has been fixed with version 2.11.0-ls312 (Alpine 3.20).

tanmaychimurkar · 2024-07-22T13:50:37Z

Can confirm the same.

DougEdey · 2024-07-22T13:56:37Z

@cwilson1776 can you confirm this works for you too and close this issue?

cwilson1776 · 2024-07-27T03:38:42Z

@cwilson1776 can you confirm this works for you too and close this issue?

Confirmed. Updated to the latest docker image, restarted my container, and manually executed /etc/periodic/weekly/libmaxminddb from inside the running container -- successfully downloaded the db.

LinuxServer-CI added the no-issue-activity label Mar 30, 2024

LinuxServer-CI removed the no-issue-activity label Apr 10, 2024

LinuxServer-CI added the no-issue-activity label May 25, 2024

LinuxServer-CI removed the no-issue-activity label Jun 6, 2024

LinuxServer-CI added the no-issue-activity label Jul 7, 2024

LinuxServer-CI removed the no-issue-activity label Jul 21, 2024

raldone01 mentioned this issue Jul 23, 2024

Bump Alpine to Alpine 3.20 to fix tar: invalid tar magic GilbN/geoip2influx#17

Closed

cwilson1776 closed this as completed Jul 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] DB downloading failed (new MaxMind policy 2024) #853

[BUG] DB downloading failed (new MaxMind policy 2024) #853

cwilson1776 commented Feb 25, 2024 •

edited

Loading

aptalca commented Feb 25, 2024

cwilson1776 commented Feb 26, 2024 •

edited

Loading

cwilson1776 commented Feb 29, 2024

LinuxServer-CI commented Mar 30, 2024

DougEdey commented Apr 10, 2024

tanmaychimurkar commented Apr 24, 2024

tanmaychimurkar commented Apr 25, 2024

LinuxServer-CI commented May 25, 2024

Crash1602 commented Jun 6, 2024 •

edited

Loading

aptalca commented Jun 6, 2024

LinuxServer-CI commented Jul 7, 2024

IBeTheBlueCat commented Jul 21, 2024

Crash1602 commented Jul 22, 2024

tanmaychimurkar commented Jul 22, 2024

DougEdey commented Jul 22, 2024

cwilson1776 commented Jul 27, 2024

[BUG] DB downloading failed (new MaxMind policy 2024) #853

[BUG] DB downloading failed (new MaxMind policy 2024) #853

Comments

cwilson1776 commented Feb 25, 2024 • edited Loading

Is there an existing issue for this?

Name of mod

Name of base container

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

CPU architecture

Docker creation

Container logs

aptalca commented Feb 25, 2024

cwilson1776 commented Feb 26, 2024 • edited Loading

cwilson1776 commented Feb 29, 2024

LinuxServer-CI commented Mar 30, 2024

DougEdey commented Apr 10, 2024

tanmaychimurkar commented Apr 24, 2024

tanmaychimurkar commented Apr 25, 2024

LinuxServer-CI commented May 25, 2024

Crash1602 commented Jun 6, 2024 • edited Loading

aptalca commented Jun 6, 2024

LinuxServer-CI commented Jul 7, 2024

IBeTheBlueCat commented Jul 21, 2024

Crash1602 commented Jul 22, 2024

tanmaychimurkar commented Jul 22, 2024

DougEdey commented Jul 22, 2024

cwilson1776 commented Jul 27, 2024

cwilson1776 commented Feb 25, 2024 •

edited

Loading

cwilson1776 commented Feb 26, 2024 •

edited

Loading

Crash1602 commented Jun 6, 2024 •

edited

Loading