Skip to content

Conversation

jan-auer
Copy link
Member

@jan-auer jan-auer commented Feb 9, 2018

This PR adds support for the new chunked upload endpoint refactors the debug information file upload implemented in:

Also, the now-legacy archive dSYM upload is refactored into the same module and fixes some prior issues. All related code is located in the utils::dif_upload and api modules. The upload follows this general schema:

  1. Check whether GET /api/chunk-upload is available. If so, use the provided configuration for chunked uploading, otherwise fall back to the ZIP archive upload. This applies to all commands that previously used the files/dsyms endpoint: upload-dsym, upload-dbg and upload-breakpad
  2. Walk the file system and collect all matching DIFs. Previously, this iteration would yield batches immediately; now batching is deferred.
  3. Apply a conditional processing step. For now, this step only checks for Apple dSYMs with hidden ITC symbols (__hidden#<num>_) and resolves them using dsymutil if BCSymbolMaps are provided. This replaces the original dSYMs for subsequent steps.
  4. Calculate SHA1 checksums for each files. Additionally, if chunked uploads are available, chunk the files and calculate checksums for each chunk. Previously, this was performed while walking the file tree, but having a separate step might yield better UX.
  5. Check for missing DIFs on the server, either using the new DIF assemble endpoint or the legacy missing dSYMs endpoint.
  6. Upload missing chunks (or files) in batches. The batch sizes are determined either by the chunk upload configuration, or in the fallback case by configuration defaults. The chunk upload relies on libcurl's gzipping to compress the upload, the legacy uploads creates a ZIP archive.
  7. If the chunk upload is used, start polling for assembling and symbol processing in 5 second intervals. This will block until symcaches (etc...) have been computed for all uploaded DIFs. In the fallback case, no such polling is performed.
  8. Display the full list of uploaded DIFs with metadata (UUID, architecture, symbol name) to the user and return the list of uploaded DIFs (struct DsymFile). The caller can then determine missing dSYMs or apply post processing.

This is the list of changes and features included in this PR:

  • Use sha1's Digest directly instead of Strings
  • Support FAT objects in the file walker
  • Search ZIPs for symbols while walking
  • Resolve BCSymbolMaps before uploading
  • Calculate checksums and chunks (of resolved objects)
  • Fix legacy ZIP archive upload (checksums and batch sizes)
  • Implement chunk upload and prefer it over archive upload
  • Activate GZIP transfer encoding for all uploads
  • Make super nice progress bars that are actually useful (!!111!elf)
  • Provide a constructor from CLI arguments (::from_cli)
  • Support --require-all
  • Use utils::dif_upload in upload-dsym
  • Delete old utils::upload module.

NOTE: A new unified upload-sym(s) command will be added along with some further refactoring in a follow-up PR.

@jan-auer jan-auer self-assigned this Feb 9, 2018
@jan-auer jan-auer requested a review from mitsuhiko February 9, 2018 13:31
@jan-auer
Copy link
Member Author

jan-auer commented Feb 9, 2018

The build failures are due to an unreleased symbolic version.

@jan-auer
Copy link
Member Author

jan-auer commented Feb 9, 2018

Just noticed that I'm missing support for require_all; updated PR description.

@jan-auer
Copy link
Member Author

@mitsuhiko I will run a symbolic release once the linked PR is merged and merge this once tests pass again.

Was thinking about how to test this, but the only way I could come up with was sourcing a test repository with test files.

The new upload command and some more refactorings will happen in follow up PRs, this is already large enough.

@jan-auer jan-auer changed the title ref(upload): Add support for the chunked DIF upload [WIP] ref(upload): Add support for the chunked DIF upload Feb 12, 2018
@jan-auer
Copy link
Member Author

@mitsuhiko This is ready for review now.

@jan-auer jan-auer merged commit 029ccb9 into master Feb 12, 2018
@jan-auer jan-auer deleted the ref/chunk-upload branch February 12, 2018 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants