ipfs get doesn't work on files with "]" in the name

[Arlodotexe]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.tech

Version

Kubo version: 0.16.0
Repo version: 12
System version: amd64/windows
Golang version: go1.19.1

Config

Click to expand

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Origin": [
        "http://example.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": true,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWGCLGARP16J2Q67FeykYRxoAagJBVKG2c18ggkXQy7zgX"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": [
      {
        "Addrs": [
          "/dnsaddr/node-1.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcFf2FH3CEgTNHeMRGhN7HNHU1EXAxoEk6EFuSyXCsvRE"
      },
      {
        "Addrs": [
          "/dnsaddr/node-2.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcFmLd5ySfk2WZuJ1mfSWLDjdmHZq7rSAua4GoeSQfs1z"
      },
      {
        "Addrs": [
          "/dnsaddr/node-3.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfFmzSDVbwexQ9Au2pt5YEXHK5xajwgaU6PpkbLWerMa"
      },
      {
        "Addrs": [
          "/dnsaddr/node-4.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfJeB3Js1FG7T8YaZATEiaHqNKVdQfybYYkbT1knUswx"
      },
      {
        "Addrs": [
          "/dnsaddr/node-5.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfVvzK4tMdFmpJjEKDUoqRgP4W9FnmJoziYX5GXJJ8eZ"
      },
      {
        "Addrs": [
          "/dnsaddr/node-6.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfZD3VKrUxyP9BbyUnZDpbqDnT7cQ4WjPP8TRLXaoE7G"
      },
      {
        "Addrs": [
          "/dnsaddr/node-7.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfZP2LuW4jxviTeG8fi28qjnZScACb8PEgHAc17ZEri3"
      },
      {
        "Addrs": [
          "/dnsaddr/node-8.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfgsJsMtx6qJb74akCw1M24X1zFwgGo11h1cuhwQjtJP"
      },
      {
        "Addrs": [
          "/dnsaddr/node-9.ingress.cloudflare-ipfs.com"
        ],
        "ID": "Qmcfr2FC7pFzJbTSDfYaSy1J8Uuy8ccGLeLyqJCKJvTHMi"
      },
      {
        "Addrs": [
          "/dnsaddr/node-10.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfR3V5YAtHBzxVACWCzXTt26SyEkxdwhGJ6875A8BuWx"
      },
      {
        "Addrs": [
          "/dnsaddr/node-11.ingress.cloudflare-ipfs.com"
        ],
        "ID": "Qmcfuo1TM9uUiJp6dTbm915Rf1aTqm3a3dnmCdDQLHgvL5"
      },
      {
        "Addrs": [
          "/dnsaddr/node-12.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfV2sg9zaq7UUHVCGuSvT2M2rnLBAPsiE79vVyK3Cuev"
      }
    ]
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "6h",
    "Strategy": "all"
  },
  "Routing": {
    "Methods": null,
    "Routers": null,
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": [],
    "ConnMgr": {
      "GracePeriod": "1m",
      "HighWater": 75,
      "LowWater": 45,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "foo": "foobar"
}

Description

Repro steps:

1. Create an empty folder
2. Create an empty file in that folder, and name it anything. Include ] in the name.
3. Add the folder to Kubo with ipfs add -r ./path
4. Copy the folder CID from the terminal
5. Run ipfs get Qmfoo on the copied CID.
6. Observe the error: Error: invalid platform path: path components cannot contain any of [<>:"\|?*] : "test file].txt"

It's worth noting that this does not happen with the [ character.

[CsterKuroi]

reproduce in old version v0.9.1

[after-egypt]

On the Linux version 0.16.0 I didn't get any errors.

[lidel]

This is because this problem is specific to MS Windows.

Ref. https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file:

The following reserved characters:

    < (less than)
    > (greater than)
    : (colon)
    " (double quote)
    / (forward slash)
    \ (backslash)
    | (vertical bar or pipe)
    ? (question mark)
    * (asterisk)

User-friendly way to resolve this is to

• easy fix: provide user with meaningful error ("Microsoft Windows does not support" instead of vague "invalid platform path".
• proper fix: add ipfs get --with-unsupported-filenames as opt-in (and mention it in the error returned to the user) – passing this parameter will allow Windows users to get data but replace unsupported characters in filenames with underscore (_).

[franklovefrank]

hi @Arlodotexe I am looking at this issue but cannot reproduce it. can you please post a stack trace?

[susarlanikhilesh]

@lidel this is not reproducible on windows.

[Jorropo]

@susarlanikhilesh files do not have names they are just bytes, directories have a list of name cid pairs.

You need to try to download the folder that contains a file with a ] not a file.

[phillmac]

@lidel the problem is, neither the right square bracket or left square bracket are restricted, and you can in fact have files and folder names contain those.
the link nor the list of characters you referenced forbid using right and left square bracket.

Can we please get them removed from the filter list?

I specifically want the brackets preserved and not replaced with underscores because they are an important part of folder naming schemas that will break with them removed or replaced.

[lidel]

Ah, indeed. Tracked down the Windows-specific sanitization, it is in go-libipfs/tar (ipfs get RPC uses TAR behind the scenes):
https://github.com/ipfs/go-libipfs/blob/302b2799386dea7afb72ba0b4c32a5c427215d06/tar/sanitize_windows.go#L12

Just like others noted the [ and ] are allowed in file names on Windows, but we try to sanitize them for some reason (was unable to find out why, code is 5 years old).

If anyone with Windows has time to test allowing them does not break ipfs get,
please submit a fix (open a PR in go-libipfs and here)