Youtube import subscription instructions is not working anymore

[simula67]

Description

Following the instructions at https://newpipe.schabi.org/FAQ/tutorials/import-export-data/#import-youtube to download subscriptions no longer works as it seems like Youtube has changed its interface. Going to https://www.youtube.com/subscription_manager?action_takeout=1 no longer starts a download, but it redirects to https://www.youtube.com/feed/channels. This prevents importing of subscriptions list which could have been then loaded to Newpipe app.

Expected behavior

It should be possible to import subscriptions from youtube account using instructions at https://newpipe.schabi.org/FAQ/tutorials/import-export-data/#import-youtube.

Actual Behavior

Instructions given to import subscriptions from Youtube not working anymore. The instructions ask you to download subscriptions using the URL: https://www.youtube.com/subscription_manager?action_takeout=1, but this merely redirects to https://www.youtube.com/feed/channels and does not start a download.

Steps to reproduce

1. Read import instructions from https://newpipe.schabi.org/FAQ/tutorials/import-export-data/#import-youtube

2. Follow the link at https://www.youtube.com/subscription_manager?action_takeout=1

3. Gets redirected to https://www.youtube.com/feed/channels. No download starts.

4. Cannot import subscriptions from youtube

Platform

All platforms

[Stypox]

Could you provide the new link youtube uses to download subscriptions?

[opusforlife2]

@Stypox #1751 (comment)

[simula67]

@Stypox

Seems like a multi-step process now:

1. Go to something like : https://takeout.google.com/takeout/custom/youtube?continue=https%3A%2F%2Fmyaccount.google.com%2Fyourdata%2Fyoutube%3Fhl%3Den_GB%26authuser%3D0

2. Click on "All data included"

3. Select "Deselect all" and select only "subscriptions" and click on OK

4. Go to "Next step"

5. Click "Create export"

6. Download link will be sent to email by default

7. Use the emailed link to download to obtain subscriptions.json

[opusforlife2]

Download link will be sent to email by default

When you say "by default", does that mean there are other options?

[simula67]

Yes there are other options like "Add to Drive", "Add to Dropbox", "Add to OneDrive" and "Add to Box" besides the default delivery method, which is "Send download link via email".

[inigosola]

@Stypox

Seems like a multi-step process now:

1. Go to something like : https://takeout.google.com/takeout/custom/youtube?continue=https%3A%2F%2Fmyaccount.google.com%2Fyourdata%2Fyoutube%3Fhl%3Den_GB%26authuser%3D0
2. Click on "All data included"
3. Select "Deselect all" and select only "subscriptions" and click on OK
4. Go to "Next step"
5. Click "Create export"
6. Download link will be sent to email by default
7. Use the emailed link to download to obtain subscriptions.json

I have followed these steps, I get a subscriptions.json file, but when I import the file into NewPipe, I get an error and it doesn't work.

[Stypox]

@simula67 great, thank you. I will try to implement this

[Djay86]

I also tried importing the .json to newpipe but it doesn't know what to do with it unless there is some way to convert it for newpipe.

[saecki]

I've converted my subscriptions.json file in a quite naive shell script to a newpipe fomat. You'll need jq and ripgrep but if you just want a quick and dirty fix:

subscriptions=$(cat $1 | jq '.[]' | jq '.snippet.title,.snippet.resourceId.channelId' | rg --multiline --no-filename -e '"(.*?)"\n"(.*?)"\n' -r '{
    "service_id": 0,
    "url": "https://www.youtube.com/channel/$2",
    "name": "$1"
},')

json="{
\"app_version\": \"0.20.2\",
\"app_version_int\": 956,
\"subscriptions\": [
${subscriptions::-1}
]
}"

echo "$json" > newpipe_subscriptions.json

[IvanProgramming]

I've converted my subscriptions.json file in a quite naive shell script to a newpipe fomat. You'll need jq and ripgrep but if you just want a quick and dirty fix:

subscriptions=$(cat $1 | jq '.[]' | jq '.snippet.title,.snippet.resourceId.channelId' | rg --multiline --no-filename -e '"(.*?)"\n"(.*?)"\n' -r '{
    "service_id": 0,
    "url": "https://www.youtube.com/channel/$2",
    "name": "$1"
},')

json="{
\"app_version\": \"0.20.2\",
\"app_version_int\": 956,
\"subscriptions\": [
${subscriptions::-1}
]
}"

echo "$json" > newpipe_subscriptions.json

Sorry for some dummy question, but can you give an example of usage this script in bash command line?

[saecki]

So basically you could create a convert.sh file and copy the entire script in there.

#!/bin/sh
subscriptions=$(cat $1 | jq '.[]' | jq '.snippet.title,.snippet.resourceId.channelId' | rg --multiline --no-filename -e '"(.*?)"\n"(.*?)"\n' -r '{
    "service_id": 0,
    "url": "https://www.youtube.com/channel/$2",
    "name": "$1"
},')

json="{
\"app_version\": \"0.20.2\",
\"app_version_int\": 956,
\"subscriptions\": [
${subscriptions::-1}
]
}"

echo "$json" > $2

To make it executable:
chmod +x convert.sh
Then it can be used like:
./convert.sh <input_file> <output_file>
In this case:
./convert.sh subscriptions.json newpipe_subscriptions.json

[IvanProgramming]

@saecki Thank You for such detailed description. And excuse for some troubling

[Stypox]

Oh no. I just tried the steps you provided above and everything seems to be localized, even the path inside the .zip file provided... We would have to ask the user to extract the json file themselves, making everything really complicated

[saecki]

Maybe it's easier to parse the website after all.

[Stypox]

Could you all test the apk provided in #4759

[simula67]

The apk works correctly to import subscriptions. Thanks for doing this.

[ghost]

@saecki

./convert.sh: 14: ./convert.sh: Bad substitution

[pavulonx]

@Ammako
One-liner using only jq

jq '{app_version: "0.20.2", app_version_int: 956, subscriptions: [.[] | {service_id: 0, url: ("https://www.youtube.com/channel/" + .snippet.resourceId.channelId), name: .snippet.title }]}' [subscriptions filename here] > newpipe_subscriptions.json

[hkvongit]

@Ammako
One-liner using only jq

jq '{app_version: "0.20.2", app_version_int: 956, subscriptions: [.[] | {service_id: 0, url: ("https://www.youtube.com/channel/" + .snippet.resourceId.channelId), name: .snippet.title }]}' [subscriptions filename here] > newpipe_subscriptions.json

I used this command and i was able to import all my youtube subscription to my newPipe and newPipe Legacy as well.

Steps i followed:

1. Go to the Google takeout manager.
2. Log in if asked.
3. Click on "All data included", then on "Deselect all", then select only "subscriptions" and click "OK".
4. Click on "Next step" and then on "Create export".
5. Click on the "Download" button after it appears.
   From the downloaded takeout zip extract the .json file. It is usually located under YouTube and YouTube Music/subscriptions/subscriptions.json
   (Steps from 1 to 5 copied from NewPipe FAQ)
6. Open the terminal and navigate to the destination where you have your subscription.json
7. Run command :
   jq '{app_version: "0.20.2", app_version_int: 956, subscriptions: [.[] | {service_id: 0, url: ("https://www.youtube.com/channel/" + .snippet.resourceId.channelId), name: .snippet.title }]}' [subscriptions filename here] > newpipe_subscriptions.json
8. Then you will get newpipe_subscriptions.json
9. Copy that file to your device.
10. Open your NewPipe app and go to subscriptions page.
11. Under subscriptions click on Previous export
12. Choose your files , here newpipe_subscriptions.json
13. Then the app will ask "Keep in mind this operation can be network expensive. Do you want to continue ?. Click on OK
14. It will start importing.

:)

[1-am-r00t]

I had to add string interpolation for channelId and title to make @rozenj one-liner work for me:
jq '{app_version: "0.20.2", app_version_int: 956, subscriptions: [.[] | {service_id: 0, url: ("https://www.youtube.com/channel/" + "\(.snippet.resourceId.channelId)"), name: "\(.snippet.title)" }]}' subscriptions.json > newpipe_subscriptions.json

Without string interpolation I got this error message:

jq: error (at subscriptions.json:0): string ("https://ww...) and number (null) cannot be added

I have to mention, that I used a quite cumbersome solution to export my subscriptions, since I used to keep all my subscriptions on a "Brand-Account" and I could not export them the regular way as explained in NewPipe, because YouTube "could not export data". Ended up harvesting the YT Subscription API (https://developers.google.com/youtube/v3/docs/subscriptions/list?apix_params=%7B%22part%22%3A%5B%22snippet%2CcontentDetails%22%5D%2C%22maxResults%22%3A5000%2C%22mine%22%3Atrue%7D#usage).

[ghost]

I had to add string interpolation for channelId and title to make @rozenj one-liner work for me:
jq '{app_version: "0.20.2", app_version_int: 956, subscriptions: [.[] | {service_id: 0, url: ("https://www.youtube.com/channel/" + "\(.snippet.resourceId.channelId)"), name: "\(.snippet.title)" }]}' subscriptions.json > newpipe_subscriptions.json

Without string interpolation I got this error message:

jq: error (at subscriptions.json:0): string ("https://ww...) and number (null) cannot be added

I have to mention, that I used a quite cumbersome solution to export my subscriptions, since I used to keep all my subscriptions on a "Brand-Account" and I could not export them the regular way as explained in NewPipe, because YouTube "could not export data". Ended up harvesting the YT Subscription API (https://developers.google.com/youtube/v3/docs/subscriptions/list?apix_params=%7B%22part%22%3A%5B%22snippet%2CcontentDetails%22%5D%2C%22maxResults%22%3A5000%2C%22mine%22%3Atrue%7D#usage).

Thank you for the tip on brand accounts!

Though I still get a error with jq
jq: error (at subscriptions.json:1561): Cannot index string with string "snippet"