Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

master branch is creating massive files #469

Closed
tthayer opened this issue Apr 4, 2023 · 14 comments · Fixed by #476
Closed

master branch is creating massive files #469

tthayer opened this issue Apr 4, 2023 · 14 comments · Fixed by #476
Labels
bug Something isn't working

Comments

@tthayer
Copy link

tthayer commented Apr 4, 2023

Every song it is downloading is 400-500MB so 4-5 albums are currently using about 50GB. Is this the expected behavior?
@miraclx
Copy link
Owner

miraclx commented Apr 4, 2023 •
edited
Loading

Can you include a link for the tracks that you observed this behaviour with? Or include logs?

Seems to be a duplicate of #437.

@tthayer
Copy link
Author

tthayer commented Apr 4, 2023 •
edited
Loading

Here are the logs:

% freyr https://open.spotify.com/album/4I8BEAiq1kPqRRlvWLTo5B\?si\=2sV1WWTyTRae5bVjhzM8ig                                                                                                                                                                                      23-04-04 - 13:44:24
    ____
   / __/_______  __  _______
  / /_/ ___/ _ \/ / / / ___/
 / __/ /  /  __/ /_/ / /
/_/ /_/   \___/\__, /_/
              /____/ v0.9.0

freyr - (c) Miraculous Owonubi <omiraculous@gmail.com>
------------------------------------------------------
Checking directory permissions...[done]
[https://open.spotify.com/album/4I8BEAiq1kPqRRlvWLTo5B?si=2sV1WWTyTRae5bVjhzM8ig]
 [•] Identifying service...[Spotify]
 [•] Checking authentication...[unauthenticated]
 [Spotify Login]
  [•] Logging in...[done]
 Detected [album]
 Obtaining album metadata...[done]
  ➤ Album Name: I Need to Start a Garden
  ➤ Artist: Haley Heynderickx
  ➤ Tracks: 8
  ➤ Type: Album
  ➤ Year: 2018
 [•] Collating [I Need to Start a Garden]...
  [•] Inquiring tracks...[done]
   • [01 No Face]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 3 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
   • [02 The Bug Collector]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 2 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
   • [03 Jo]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 4 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
   • [04 Worth It]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 3 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
   • [05 Show You a Body]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 3 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
   • [06 Untitled God Song]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 5 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
   • [07 Oom Sha La La]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 2 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
   • [08 Drinking Song]
      | ➤ Collating sources...
      |  ➤ [•] YouTube Music...[success, found 4 sources]
      | ➤ Awaiting audiofeeds...[done]
      | [✓] Got album art
      | [✓] Got raw track file
      | [•] Post Processing...
 [•] Download Complete
 [•] Embedding Metadata...
  • [✓] 01 No Face
  • [✓] 02 The Bug Collector
  • [✓] 03 Jo
  • [✓] 04 Worth It
  • [✓] 05 Show You a Body
  • [✓] 06 Untitled God Song
  • [✓] 07 Oom Sha La La
  • [✓] 08 Drinking Song
[•] Collation Complete
============ Stats ============
 [•] Runtime: [31.3s]
 [•] Total queries: [01]
 [•] Total tracks: [08]
     » Skipped: [00]
     ✓ Passed:  [08]
     ✕ Failed:  [00]
 [•] Output directory: [.]
 [•] Total Output size: 4.05 GB
 [•] Total Network Usage: 31.88 MB
     ♫ Media: 31.09 MB
     ➤ Album Art: 799.26 KB
 [•] Output bitrate: 320k
===============================

I did notice that while watching the filesystem the original file is an acceptable size and then after adding metadata the file size balloons.

@tthayer
Copy link
Author

tthayer commented Apr 4, 2023

Here's a screenshot of what I'm seeing, too:
image

@tthayer
Copy link
Author

tthayer commented Apr 4, 2023

Also, I tried freyr get --no-cover https://blahblah and it still adds the cover art. I'm thinking that might be where most of the file size is.

@miraclx
Copy link
Owner

miraclx commented Apr 4, 2023

Interesting, gimme a sec to look at this..

@miraclx miraclx added the bug Something isn't working label Apr 4, 2023
@tthayer
Copy link
Author

tthayer commented Apr 4, 2023

Here are a couple of tests using processing tools:

% AtomicParsley "01 No Face-temp-65676.m4a" -T                                                                                                                                                                                                                                 23-04-04 - 14:17:27
APar_readX read failed, expect 4294967294, got 500728030: end of file

exiftool -v1 output

...
  | | | | CoverArt = .....JFIF...HH..C.................................................[snip]
  | | | | - Tag 'covr', Type='data', Flags=0xd, Lang=0x0000 (99907 bytes)
  MediaDataSize = 4598717
  MediaDataOffset = 121873
  MediaData = ..Lavc58.91.100B ...8!..`..!..`..!..`..!..`..!..`..!..`..!..`..!..`..!..`[snip]
  - Tag 'mdat' (4598717 bytes)
  Tag '\x00\x00\x00\x00' extends to end of file

@tthayer
Copy link
Author

tthayer commented Apr 5, 2023

The exiftool output shows the cover art taking up the appropriate amount of space (~100kb) and that last line indicates that weird tag name taking up the rest of the file.

@miraclx
Copy link
Owner

miraclx commented Apr 5, 2023

Thanks for all this info, are you on Mac maybe?

Also, do you observe this behavior with the docker images? Try freyrcli/freyrjs-git:master.

@tthayer
Copy link
Author

tthayer commented Apr 5, 2023

Yes, I'm on a Mac. I'll give the docker container a try.

@tthayer
Copy link
Author

tthayer commented Apr 5, 2023

The docker image does the thing correctly!

@tthayer
Copy link
Author

tthayer commented Apr 5, 2023

I installed the version of AtomicParsley that is used in the container locally (20210715.151551.e7ad03a) and that resolved the large file issues on my side. It appears homebrew only provides the HEAD version (20221229.172126.d813aa6).

@tthayer tthayer closed this as completed Apr 5, 2023
@miraclx
Copy link
Owner

miraclx commented Apr 5, 2023

Interesting, can you try building from this branch to test?

wez/atomicparsley#63

I'm about to make this the base branch we're vendored off of because we could use the patch in there.

@tthayer
Copy link
Author

tthayer commented Apr 5, 2023

The atomicparsley binary built from that repo fork worked:

% ls -lh .                                                                                     23-04-05 - 16:01:29
total 139696
-rw-r--r--  1 xxx  xxx   4.5M Apr  5 16:00 01 No Face.m4a
-rw-r--r--  1 xxx  xxx   8.8M Apr  5 16:00 02 The Bug Collector.m4a
-rw-r--r--  1 xxx  xxx   7.0M Apr  5 16:00 03 Jo.m4a
-rw-r--r--  1 xxx  xxx    17M Apr  5 16:00 04 Worth It.m4a
-rw-r--r--  1 xxx  xxx   7.9M Apr  5 16:00 05 Show You a Body.m4a
-rw-r--r--  1 xxx  xxx   8.1M Apr  5 16:00 06 Untitled God Song.m4a
-rw-r--r--  1 xxx  xxx   6.8M Apr  5 16:00 07 Oom Sha La La.m4a
-rw-r--r--  1 xxx  xxx   7.5M Apr  5 16:00 08 Drinking Song.m4a
-rw-r--r--  1 xxx xxx    98K Apr  5 16:00 cover.jpg

% AtomicParsley --version                                                                                                                      23-04-05 - 16:01:41
AtomicParsley version: 20230114.175602.0 21bde60444065af8496833391a958b2938259810 (utf8)

@miraclx
Copy link
Owner

miraclx commented Apr 5, 2023

Beautiful! Thanks for the info. I'll reopen this and close once we've rebased on that patch. Thanks again!

@miraclx miraclx reopened this Apr 5, 2023
@miraclx miraclx mentioned this issue Apr 6, 2023
Merged
@miraclx miraclx closed this as completed Apr 6, 2023
@miraclx miraclx mentioned this issue Jan 5, 2024
Closed
@miraclx miraclx mentioned this issue Jan 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chore: update AtomicParsley in Docker, update docs miraclx/freyr-js
2 participants
@tthayer @miraclx