Bug: Migrating repos with a lot of issues and attachments corrupts attachments on upload to s3

[bijelicaleksandar]

Hello ! Great tool, helped me migrate several repos when my company stopped the internal GitLab instaance.

First several migrations were fine, but when migrating the last (largest) repo with around 2500 issues and many more attachments, I ran into an issue where all files/images would show like this:

(Blacked the file name)

It was strange, because using the same session cookie, I was able to migrate images in other, test, repos.

So, I suspected either a file size issue or a file count limit.

Searching the code and internet, ran into this, which helped me resolve the issue:

https://stackoverflow.com/questions/58655532/increasing-maxcontentlength-and-maxbodylength-in-axios

Modified the src/githubHelper.ts file, the following section:

  /**
   * Gets attachment using http get
   */
  async getAttachment(relurl: string) {
    try {
      const attachmentUrl = this.host + '/' + this.projectPath + relurl;
      const data = (
        await axios.get(attachmentUrl, {
          responseType: 'arraybuffer',
          headers: {
            // HACK: work around GitLab's API lack of GET for attachments
            // See https://gitlab.com/gitlab-org/gitlab/-/issues/24155
            Cookie: `_gitlab_session=${this.sessionCookie}`,
          },
          maxContentLength: Infinity,
          maxBodyLength: Infinity
        })
      ).data;
      return Buffer.from(data, 'binary');
    } catch (err) {
      console.error(`Could not download attachment #${relurl}.`);
      return null;
    }
  }

The two lines added were:

          maxContentLength: Infinity,
          maxBodyLength: Infinity

After this it worked.

As far as I can tell there's no down side for this change. Wanted to open this issue to manily document my problem and how I solved it in case someone else runs into this