Backup fails #47
Comments
|
+1. I am experiencing the exact same behavior. @MatthieuBizien, I expect this is the same as #46 but has a complete bug report. Probably safe to assume #46 is a duplicate. You can see the output described by #46 here in line just above the full Traceback.
|
|
@quickfold, any suggestions on getting past this? i tried to see if there was a previous commit has that might work that wasn’t apparent immediately. |
|
@tylerwince No, I have no idea how to troubleshoot this. I am hoping @MatthieuBizien or someone else will get a chance to respond sometime soon. |
|
Mine started having similar failure today. Is there a way to skip markdown backup? All I really want is the JSON file, and markdown has caused issues before (such as if a page title starts with |
|
Having the same issue also - I've only discovered this tool as of yesterday so never had a working back up. |
|
I successfully back up my roam database for twice in the beginning, bt not it can not work normally any longer. It shows: .github#L1 I have tried: |
The cause actually turned out to be because I was using the Google Account authentication rather than just a plain email and password. In order to set it to the email and password, you have to go through the "forgotten password" process. Currently there's no way of setting your password in your account setting, which is something that's expected in most UI's with account details nowadays. Hope this helps someone! 😅 |
|
I was having a similar issue while using github actions, so I tried running roam-to-git on my local machine, and I received an error because one of my page names was too long (got the hint from @everruler12's comment). I removed the page, and then was able to get a successful backup working. If your backup is also failing at the export step, I would suggest trying it on a local machine to see if there is a similar reason why the markdown isn't saving. There was additional debugging information than seems available in github actions logs. |
|
@maggiedelano Thanks for commenting on this (abandoned?) issue. I am on Windows and am not a programmer. What does it mean to run roam-to-git on your local machine? I don't think I have any really long page names - can I just check by looking at Roam 'All Pages' page? Of course you have no obligation, but if you can read my debug log in the first post, I would really appreciate any ideas for what's gone wrong. |
I don't know where your log is. It will show |
|
Thanks, @everruler12. @cobblepot1: I was able to find the OSError in my own logs from a while ago to verify this was the issue, and it had the same info as if I ran it on my own computer (in this case I installed it on my mac). I had to go into the raw logs to find the error; I didn't see it in the main logs. Here's how to find yours and see more information about what the issue may be. EDIT: actually, it looks like there may be two different things going wrong. The OSError does show up in my usual logs, so there must be multiple things going on that are causing the backups to fail occasionally.
|
|
I'm sorry (?) to report that it was not a file name length issue. Will wait for other ideas, thanks. |
|
OK, anyone following this thread, I got things working on Windows 10. I deleted the existing backup repo and created a new one following the instructions here: https://www.roamstack.com/automatic-backups/ I think the problem was that for my |
|
@cobblepot1 I had a similar issue with my guide not being clear about only copying the graph name, instead of the URL. If you don't mind, could you check my guide and give feedback if my current explanation is easier to understand? https://eriknewhard.com/blog/backup-roam-in-github |
|
Nice of you to respond! Actually, I think the explanation on the roamstack page is clearer - to use the database name that is at the top of the left side panel. I'd suggest adding that just to direct all attention away from the URL when creating this variable. |
|
Got it, thanks! |
|
The backup errors should be fixed by #92 🍾. I'm closing that issue, but feel free to re-open it if you get new errors. |
Describe the bug
I've setup the Action to run but it fails.
To Reproduce
Steps to reproduce the behavior:
Followed the instructions to set up the action and secrets, but backup fails.
Expected behavior
Backup to succeed.
Traceback
##[section]Starting: Request a runner to run this job
Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'ubuntu-latest'
Can't find any online and idle self-hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
Found online and idle hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
##[section]Finishing: Request a runner to run this job
Current runner version: '2.273.0'
##[group]Operating System
Ubuntu
18.04.5
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-18.04
Version: 20200817.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20200817.1/images/linux/Ubuntu1804-README.md
##[endgroup]
Prepare workflow directory
Prepare all required actions
Download action repository 'actions/checkout@v2'
Download action repository 'actions/setup-python@v1'
Download action repository 'elstudio/actions-js-build@v3'
##[group]Build container for action use: '/home/runner/work/_actions/elstudio/actions-js-build/v3/commit/Dockerfile'.
##[command]/usr/bin/docker build -t 3b3ac6:5d85631f2edb4c238a2db0c2dc76337b -f "/home/runner/work/_actions/elstudio/actions-js-build/v3/commit/Dockerfile" "/home/runner/work/_actions/elstudio/actions-js-build/v3/commit"
Sending build context to Docker daemon 77.31kB
Step 1/13 : FROM alpine:3.10
---> be4e4bea2c2e
Step 2/13 : LABEL version="1.0.0"
---> Running in 0adffe704988
Removing intermediate container 0adffe704988
---> 64b635ca3eb0
Step 3/13 : LABEL repository="https://github.com/elstudio/actions-js-build"
---> Running in 37ac4670e420
Removing intermediate container 37ac4670e420
---> 5fd55af56296
Step 4/13 : LABEL homepage="https://github.com/elstudio/actions-js-build"
---> Running in 448b9c199a7f
Removing intermediate container 448b9c199a7f
---> 5f6e95f76b47
Step 5/13 : LABEL maintainer="el-studio Actions servers+actions@el-studio.com"
---> Running in 0eaceba85109
Removing intermediate container 0eaceba85109
---> b6e56198e83b
Step 6/13 : LABEL com.github.actions.name="GitHub Action for git commit"
---> Running in a92546333669
Removing intermediate container a92546333669
---> ca210162d4f1
Step 7/13 : LABEL com.github.actions.description="Commits any changed files and pushes the result back to origin."
---> Running in 3ce79b11e3b2
Removing intermediate container 3ce79b11e3b2
---> df5e3cee6c86
Step 8/13 : LABEL com.github.actions.icon="git-commit"
---> Running in e81ffe676943
Removing intermediate container e81ffe676943
---> 232836aa8db4
Step 9/13 : LABEL com.github.actions.color="green"
---> Running in 351320b5c397
Removing intermediate container 351320b5c397
---> 416e50091176
Step 10/13 : COPY LICENSE README.md THIRD_PARTY_NOTICE.md /
---> e38a8c328057
Step 11/13 : RUN apk --update --no-cache add git
---> Running in 3cb66673d15d
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/6) Installing ca-certificates (20191127-r2)
(2/6) Installing nghttp2-libs (1.39.2-r1)
(3/6) Installing libcurl (7.66.0-r0)
(4/6) Installing expat (2.2.8-r0)
(5/6) Installing pcre2 (10.33-r0)
(6/6) Installing git (2.22.4-r0)
Executing busybox-1.30.1-r3.trigger
Executing ca-certificates-20191127-r2.trigger
OK: 21 MiB in 20 packages
Removing intermediate container 3cb66673d15d
---> 53efe912012e
Step 12/13 : COPY "entrypoint.sh" "/entrypoint.sh"
---> e66cab7335ac
Step 13/13 : ENTRYPOINT ["/entrypoint.sh"]
---> Running in c5193f57d696
Removing intermediate container c5193f57d696
---> 3625bb579e7f
Successfully built 3625bb579e7f
Successfully tagged 3b3ac6:5d85631f2edb4c238a2db0c2dc76337b
##[endgroup]
##[group]Run actions/checkout@v2
with:
repository: quickfold/roam-backup
token: ***
ssh-strict: true
persist-credentials: true
clean: true
fetch-depth: 1
lfs: false
submodules: false
##[endgroup]
Syncing repository: quickfold/roam-backup
##[group]Getting Git version info
Working directory is '/home/runner/work/roam-backup/roam-backup'
[command]/usr/bin/git version
git version 2.28.0
##[endgroup]
Deleting the contents of '/home/runner/work/roam-backup/roam-backup'
##[group]Initializing the repository
[command]/usr/bin/git init /home/runner/work/roam-backup/roam-backup
Initialized empty Git repository in /home/runner/work/roam-backup/roam-backup/.git/
[command]/usr/bin/git remote add origin https://github.com/quickfold/roam-backup
##[endgroup]
##[group]Disabling automatic garbage collection
[command]/usr/bin/git config --local gc.auto 0
##[endgroup]
##[group]Setting up auth
[command]/usr/bin/git config --local --name-only --get-regexp core.sshCommand
[command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
[command]/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader
[command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http.https://github.com/.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
[command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
##[endgroup]
##[group]Fetching the repository
[command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +01363b876c7eca53f45ee1ac353e03ce16fa2b09:refs/remotes/origin/master
remote: Enumerating objects: 7, done.
remote: Counting objects: 14% (1/7)
remote: Counting objects: 28% (2/7)
remote: Counting objects: 42% (3/7)
remote: Counting objects: 57% (4/7)
remote: Counting objects: 71% (5/7)
remote: Counting objects: 85% (6/7)
remote: Counting objects: 100% (7/7)
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 25% (1/4)
remote: Compressing objects: 50% (2/4)
remote: Compressing objects: 75% (3/4)
remote: Compressing objects: 100% (4/4)
remote: Compressing objects: 100% (4/4), done.
remote: Total 7 (delta 0), reused 0 (delta 0), pack-reused 0
From https://github.com/quickfold/roam-backup
[new ref] 01363b876c7eca53f45ee1ac353e03ce16fa2b09 -> origin/master
##[endgroup]
##[group]Determining the checkout info
##[endgroup]
##[group]Checking out the ref
[command]/usr/bin/git checkout --progress --force -B master refs/remotes/origin/master
Reset branch 'master'
Branch 'master' set up to track remote branch 'master' from 'origin'.
##[endgroup]
[command]/usr/bin/git log -1
commit 01363b876c7eca53f45ee1ac353e03ce16fa2b09
Author: quickfold nemo@ihtfp.net
Date: Sat Aug 29 17:22:06 2020 +0800
Create main.yml
##[group]Run actions/setup-python@v1
with:
python-version: 3.8
architecture: x64
##[endgroup]
Successfully setup CPython (3.8.5)
##[group]Run pip install git+https://github.com/MatthieuBizien/roam-to-git.git
�[36;1mpip install git+https://github.com/MatthieuBizien/roam-to-git.git�[0m
shell: /bin/bash -e {0}
env:
pythonLocation: /opt/hostedtoolcache/Python/3.8.5/x64
##[endgroup]
Collecting git+https://github.com/MatthieuBizien/roam-to-git.git
Cloning https://github.com/MatthieuBizien/roam-to-git.git to /tmp/pip-req-build-7g6retmo
Collecting gitpython>=3.1.*
Downloading GitPython-3.1.7-py3-none-any.whl (158 kB)
Collecting loguru==0.4.*
Downloading loguru-0.4.1-py3-none-any.whl (54 kB)
Collecting pyppeteer>=0.0.25
Downloading pyppeteer-0.2.2-py3-none-any.whl (145 kB)
Collecting python-dotenv>=0.10.*
Downloading python_dotenv-0.14.0-py2.py3-none-any.whl (17 kB)
Collecting psutil>=5.6.0
(460 kB)
Collecting gitdb<5,>=4.0.1
Downloading gitdb-4.0.5-py3-none-any.whl (63 kB)
Collecting websockets<9.0,>=8.1
Downloading websockets-8.1-cp38-cp38-manylinux2010_x86_64.whl (78 kB)
Collecting tqdm<5.0.0,>=4.42.1
Downloading tqdm-4.48.2-py2.py3-none-any.whl (68 kB)
Collecting pyee<8.0.0,>=7.0.1
Downloading pyee-7.0.2-py2.py3-none-any.whl (12 kB)
Collecting appdirs<2.0.0,>=1.4.3
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting urllib3<2.0.0,>=1.25.8
Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting smmap<4,>=3.0.1
Downloading smmap-3.0.4-py2.py3-none-any.whl (25 kB)
Using legacy 'setup.py install' for roam-to-git, since package 'wheel' is not installed.
Using legacy 'setup.py install' for psutil, since package 'wheel' is not installed.
Installing collected packages: smmap, gitdb, gitpython, loguru, websockets, tqdm, pyee, appdirs, urllib3, pyppeteer, python-dotenv, psutil, roam-to-git
Running setup.py install for psutil: started
Running setup.py install for psutil: finished with status 'done'
Running setup.py install for roam-to-git: started
Running setup.py install for roam-to-git: finished with status 'done'
Successfully installed appdirs-1.4.4 gitdb-4.0.5 gitpython-3.1.7 loguru-0.4.1 psutil-5.7.2 pyee-7.0.2 pyppeteer-0.2.2 python-dotenv-0.14.0 roam-to-git-0.1 smmap-3.0.4 tqdm-4.48.2 urllib3-1.25.10 websockets-8.1
##[group]Run roam-to-git --skip-git .
�[36;1mroam-to-git --skip-git .�[0m
shell: /bin/bash -e {0}
env:
pythonLocation: /opt/hostedtoolcache/Python/3.8.5/x64
ROAMRESEARCH_USER: ***
ROAMRESEARCH_PASSWORD: ***
ROAMRESEARCH_DATABASE: ***
##[endgroup]
2020-08-29 12:08:55.093 | INFO | roam_to_git.main:main:50 - Loading secrets from /home/runner/work/roam-backup/roam-backup/.env
2020-08-29 12:08:55.095 | DEBUG | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
[W:pyppeteer.chromium_downloader] start chromium download.
Download may take a few minutes.
0%| | 0/108773488 [00:00<?, ?it/s]
1%| | 624640/108773488 [00:00<00:17, 6192838.93it/s]
1%|▏ | 1546240/108773488 [00:00<00:15, 6860969.46it/s]
3%|▎ | 2867200/108773488 [00:00<00:13, 8006397.85it/s]
4%|▍ | 4689920/108773488 [00:00<00:10, 9604925.84it/s]
7%|▋ | 7086080/108773488 [00:00<00:08, 11690501.80it/s]
9%|▉ | 9789440/108773488 [00:00<00:07, 14075623.74it/s]
13%|█▎ | 13998080/108773488 [00:00<00:05, 17576645.10it/s]
19%|█▊ | 20275200/108773488 [00:00<00:03, 22416747.68it/s]
27%|██▋ | 29777920/108773488 [00:00<00:02, 29068466.44it/s]
39%|███▊ | 41953280/108773488 [00:01<00:01, 35251238.88it/s]
46%|████▋ | 50339840/108773488 [00:01<00:01, 40919085.99it/s]
54%|█████▍ | 58726400/108773488 [00:01<00:01, 47055718.48it/s]
65%|██████▌ | 70932480/108773488 [00:01<00:00, 57625668.93it/s]
73%|███████▎ | 79063040/108773488 [00:01<00:00, 58795778.75it/s]
85%|████████▍ | 92282880/108773488 [00:01<00:00, 69242396.98it/s]
93%|█████████▎| 101120000/108773488 [00:01<00:00, 73387058.43it/s]
100%|██████████| 108773488/108773488 [00:01<00:00, 59685707.90it/s]
[W:pyppeteer.chromium_downloader]
chromium download done.
[W:pyppeteer.chromium_downloader] chromium extracted to: /home/runner/.local/share/pyppeteer/local-chromium/588429
2020-08-29 12:09:03.369 | DEBUG | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-29 12:09:03.931 | DEBUG | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /tmp/tmps7r9ciu6
2020-08-29 12:09:03.943 | DEBUG | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /tmp/tmp1q33m0i7
2020-08-29 12:09:03.977 | DEBUG | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-29 12:09:03.988 | DEBUG | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-29 12:09:09.431 | DEBUG | roam_to_git.scrapping:signin:187 - Fill email ''
2020-08-29 12:09:09.460 | DEBUG | roam_to_git.scrapping:signin:187 - Fill email ''
2020-08-29 12:09:13.202 | DEBUG | roam_to_git.scrapping:signin:192 - Fill password
2020-08-29 12:09:13.922 | DEBUG | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-29 12:09:13.981 | DEBUG | roam_to_git.scrapping:signin:192 - Fill password
2020-08-29 12:09:14.688 | DEBUG | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-29 12:09:16.159 | DEBUG | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/'
2020-08-29 12:09:16.913 | DEBUG | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-29 12:09:16.942 | DEBUG | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/'
2020-08-29 12:09:17.723 | DEBUG | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-29 12:16:01.849 | DEBUG | roam_to_git.scrapping:download_rr_archive:76 - Closing browser markdown
2020-08-29 12:16:01.882 | DEBUG | roam_to_git.scrapping:download_rr_archive:78 - Closed browser markdown
2020-08-29 12:16:01.883 | ERROR | main::11 - An error has been caught in function '', process 'MainProcess' (3110), thread 'MainThread' (140230760871744):
Traceback (most recent call last):
AssertionError: All roads leads to Roam, but that one is too long. Try again when Roam servers are faster.
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.8.5/x64/bin/roam-to-git", line 11, in
load_entry_point('roam-to-git==0.1', 'console_scripts', 'roam-to-git')()
File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/loguru/_logger.py", line 1149, in catch_wrapper
return function(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/roam_to_git/main.py", line 76, in main
scrap(markdown_zip_path, json_zip_path, config)
File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/roam_to_git/scrapping.py", line 252, in scrap
asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks))
File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/roam_to_git/scrapping.py", line 69, in download_rr_archive
return await _download_rr_archive(document, output_type, output_directory, config)
File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/roam_to_git/scrapping.py", line 121, in _download_rr_archive
assert dot_button is not None, "All roads leads to Roam, but that one is too long. Try "
AssertionError: All roads leads to Roam, but that one is too long. Try again when Roam servers are faster.
2020-08-29 12:16:01.893 | DEBUG | roam_to_git.scrapping:_kill_child_process:215 - Terminate child process [psutil.Process(pid=3368, name='chrome', status='sleeping', started='12:09:02'), psutil.Process(pid=3379, name='chrome', status='sleeping', started='12:09:02'), psutil.Process(pid=3401, name='chrome', status='sleeping', started='12:09:02'), psutil.Process(pid=3411, name='chrome', status='sleeping', started='12:09:02'), psutil.Process(pid=3381, name='chrome', status='sleeping', started='12:09:02'), psutil.Process(pid=3400, name='chrome', status='sleeping', started='12:09:02')]
##[error]Process completed with exit code 1.
Post job cleanup.
[command]/usr/bin/git version
git version 2.28.0
[command]/usr/bin/git config --local --name-only --get-regexp core.sshCommand
[command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
[command]/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader
http.https://github.com/.extraheader
[command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
[command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http.https://github.com/.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
Cleaning up orphan processes
Please complete the following information:
Additional context
The text was updated successfully, but these errors were encountered: