Problems with dvc push

[LarryThermo]

Bug Report

Please provide information about your setup

Output of dvc version:

> dvc version
DVC version: 1.1.10
Python version: 3.7.5
Platform: Windows-10-10.0.17763-SP0
Binary: True
Package: exe
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Cache: reflink - not supported, hardlink - supported, symlink - not supported
Filesystem type (cache directory): ('NTFS', 'C:\\')
Repo: dvc, git
Filesystem type (workspace): ('NTFS', 'C:\\')

Additional Information (if any):
I am having problems with dvc push no longer working after I upgraded to the latest dvc.

My steps in a Windows Powershell window (if it appears a little bit disjointed its because I copied and pasted the output verbatim):

dvc init

You can now commit the changes to git.

+---------------------------------------------------------------------+
| |
| DVC has enabled anonymous aggregate usage analytics. |
| Read the analytics documentation (and how to opt-out) here: |
| https://dvc.org/doc/user-guide/analytics |
| |
+---------------------------------------------------------------------+

What's next?

• Check out the documentation: https://dvc.org/doc
• Get help and share ideas: https://dvc.org/chat
• Star us on GitHub: https://github.com/iterative/dvc

dvc remote add -d myremote s3://analysis-training-sets/masu-autofocus-models2
Setting 'myremote' as a default remote.
dvc add "model-all-00038-0.70911.pt"
100% Add|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:02, 2.99s/file]

To track the changes with git, run:

    git add model-all-00038-0.70911.ptj.dvc

dvc push
ERROR: unexpected error - unable to open database file

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[efiop]

Hi @LarryThermo . Could you please show full dvc push -v log?

[LarryThermo]

Thanks for the response Ruslan. MasuAutoFocusTraining.Simple.ManufacturingVariability.Beads> dvc push -v log 2020-07-20 09:12:12,240 DEBUG: Trying to spawn '['C:\\Program Files (x86)\\Data Version Control\\dvc.exe', 'daemon', '-q', 'updater']' 2020-07-20 09:12:18,739 DEBUG: Spawned '['C:\\Program Files (x86)\\Data Version Control\\dvc.exe', 'daemon', '-q', 'updater']' 2020-07-20 09:12:18,748 DEBUG: fetched: [(3,)] 2020-07-20 09:12:18,753 DEBUG: Checking if stage 'log' is in 'dvc.yaml' 2020-07-20 09:12:18,771 DEBUG: Assuming 'log' to be a stage inside 'dvc.yaml' 2020-07-20 09:12:18,772 DEBUG: fetched: [(6,)] 2020-07-20 09:12:18,789 ERROR: failed to push data to the cloud - 'log' does not exist as an output or a stage name in 'dvc.yaml': 'dvc.yaml' does not exist. ------------------------------------------------------------ Traceback (most recent call last): File "dvc\repo\__init__.py", line 318, in collect_granular File "dvc\repo\__init__.py", line 255, in collect File "dvc\repo\__init__.py", line 215, in get_stages File "dvc\repo\__init__.py", line 207, in get_stage File "dvc\dvcfile.py", line 210, in stages File "dvc\dvcfile.py", line 92, in _load dvc.stage.exceptions.StageFileDoesNotExistError: 'dvc.yaml' does not exist. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "dvc\command\data_sync.py", line 59, in run File "dvc\repo\__init__.py", line 36, in wrapper File "dvc\repo\push.py", line 32, in push File "dvc\repo\__init__.py", line 377, in used_cache File "dvc\repo\__init__.py", line 373, in <genexpr> File "dvc\repo\__init__.py", line 325, in collect_granular dvc.exceptions.NoOutputOrStageError: 'log' does not exist as an output or a stage name in 'dvc.yaml' ------------------------------------------------------------ Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! MasuAutoFocusTraining.Simple.ManufacturingVariability.Beads> dvc push -v 2020-07-20 09:13:39,123 DEBUG: Trying to spawn '['C:\\Program Files (x86)\\Data Version Control\\dvc.exe', 'daemon', '-q', 'updater']' 2020-07-20 09:13:45,665 DEBUG: Spawned '['C:\\Program Files (x86)\\Data Version Control\\dvc.exe', 'daemon', '-q', 'updater']' 2020-07-20 09:13:45,671 DEBUG: fetched: [(3,)] 2020-07-20 09:13:45,941 DEBUG: fetched: [(6,)] 2020-07-20 09:13:45,953 ERROR: unexpected error - unable to open database file ------------------------------------------------------------ Traceback (most recent call last): File "dvc\main.py", line 53, in main File "dvc\command\data_sync.py", line 59, in run File "dvc\repo\__init__.py", line 36, in wrapper File "dvc\repo\push.py", line 35, in push File "dvc\data_cloud.py", line 66, in push File "dvc\remote\base.py", line 78, in wrapper File "dvc\remote\index.py", line 87, in __enter__ File "dvc\remote\index.py", line 153, in load File "dvc\state.py", line 484, in _connect_sqlite sqlite3.OperationalError: unable to open database file ------------------------------------------------------------ Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! From: Ruslan Kuprieiev [mailto:notifications@github.com] Sent: Friday, July 17, 2020 9:45 PM To: iterative/dvc <dvc@noreply.github.com> Cc: Rystrom, Larry <Larry.Rystrom@thermofisher.com>; Mention <mention@noreply.github.com> Subject: Re: [iterative/dvc] Problems with dvc push (#4234) CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to SOC@thermofisher.com. Hi @LarryThermo<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=8RO59mJXM9K4E8utvZpf9QRI_ZZkP47sKP0Irq1v3O8&s=K0tzKPRXcIvKDCy6ip7jePX9j3U2mCgGtIRUAJIUsi8&e=> . Could you please show full dvc push -v log? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D660424482&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=8RO59mJXM9K4E8utvZpf9QRI_ZZkP47sKP0Irq1v3O8&s=AkFi6HDS05yhxqtHdyEMC4s8jHl6hGIdamZUEQXRm1A&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HHQV6EFK2IQMASZLK3R4ESDRANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=8RO59mJXM9K4E8utvZpf9QRI_ZZkP47sKP0Irq1v3O8&s=Krc9ZttJGoMCZYqj7-q8LWdRLtSldSvZ92GSFXxp3lQ&e=>.

[efiop]

@LarryThermo Thanks. Looks very strange, could you please show the contents of model-all-00038-0.70911.ptj.dvc ?

[LarryThermo]

Hi Ruslan, I had to regenerate it due to wanting to put it under version control in the interim. Complete sequence of regeneration commands:

dvc add "model-all-00038-0.70911.ptj"

Adding... ERROR: output 'model-all-00038-0.70911.ptj' is already tracked by SCM (e.g. Git). You can remove it from Git, then add to DVC. To stop tracking from Git: git rm -r --cached 'model-all-00038-0.70911.ptj' git commit -m "stop tracking model-all-00038-0.70911.ptj" Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

git rm -r --cached 'model-all-00038-0.70911.ptj'

rm 'model-all-00038-0.70911.ptj'

git commit -m "stop tracking model-all-00038-0.70911.ptj"

[Simple.ManufacturingVariability.Beads 9a2bc32] stop tracking model-all-00038-0.70911.ptj 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 model-all-00038-0.70911.ptj

dvc add "model-all-00038-0.70911.ptj"

100% Add|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:02, 2.88s/file] To track the changes with git, run: git add model-all-00038-0.70911.ptj.dvc .gitignore

dvc push

ERROR: unexpected error - unable to open database file Contents of model-all-00038-0.70911.ptj.dvc: outs: - md5: da3d1f8746a142ba52706dab6c6fed53 path: model-all-00038-0.70911.ptj From: Ruslan Kuprieiev [mailto:notifications@github.com] Sent: Monday, July 20, 2020 9:27 AM To: iterative/dvc <dvc@noreply.github.com> Cc: Rystrom, Larry <Larry.Rystrom@thermofisher.com>; Mention <mention@noreply.github.com> Subject: Re: [iterative/dvc] Problems with dvc push (#4234) CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to SOC@thermofisher.com. @LarryThermo<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=7PLItGQH1dOW24iNE935NzE8Ws7fR4SvUR2iRzElpyA&s=CvKQpVl3y5cxTNLMfNaooUgGDXax4G1HyfxdRwzples&e=> Thanks. Looks very strange, could you please show the contents of model-all-00038-0.70911.ptj.dvc ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D661154788&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=7PLItGQH1dOW24iNE935NzE8Ws7fR4SvUR2iRzElpyA&s=_wGn2TYI0ZUsAXP3lEBF9eZY_9XRcNwG_GBNF_dnZUE&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HAYM5UPTP4DNBW73UTR4RV4XANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=7PLItGQH1dOW24iNE935NzE8Ws7fR4SvUR2iRzElpyA&s=17z6C_n-GV1iUaP7rsdR5mImEw8DAqoYeu2TtYvekRY&e=>.

[efiop]

@LarryThermo Thanks!

@pmrowla Looks like something is up with the sqlite index db (though the very first log is different and confusing), maybe you have some ideas about it?

[pmrowla]

@LarryThermo can you try deleting the .dvc/tmp/index folder from your repo, and then re-run dvc push -v?

[LarryThermo]

dvc push -v

2020-07-21 09:26:45,576 DEBUG: Trying to spawn '['C:\\Program Files (x86)\\Data Version Control\\dvc.exe', 'daemon', '-q', 'updater']' 2020-07-21 09:26:52,107 DEBUG: Spawned '['C:\\Program Files (x86)\\Data Version Control\\dvc.exe', 'daemon', '-q', 'updater']' 2020-07-21 09:26:52,114 DEBUG: fetched: [(3,)] 2020-07-21 09:26:52,382 DEBUG: fetched: [(6,)] 2020-07-21 09:26:52,391 ERROR: unexpected error - unable to open database file ------------------------------------------------------------ Traceback (most recent call last): File "dvc\main.py", line 53, in main File "dvc\command\data_sync.py", line 59, in run File "dvc\repo\__init__.py", line 36, in wrapper File "dvc\repo\push.py", line 35, in push File "dvc\data_cloud.py", line 66, in push File "dvc\remote\base.py", line 78, in wrapper File "dvc\remote\index.py", line 87, in __enter__ File "dvc\remote\index.py", line 153, in load File "dvc\state.py", line 484, in _connect_sqlite sqlite3.OperationalError: unable to open database file ------------------------------------------------------------ Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! From: Peter Rowlands (변기호) [mailto:notifications@github.com] Sent: Monday, July 20, 2020 5:47 PM To: iterative/dvc <dvc@noreply.github.com> Cc: Rystrom, Larry <Larry.Rystrom@thermofisher.com>; Mention <mention@noreply.github.com> Subject: Re: [iterative/dvc] Problems with dvc push (#4234) CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to SOC@thermofisher.com. @LarryThermo<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=G849SKpEPXS7kSOsdNrj8kHDt0HoSbOLb8zNcUG8WAU&e=> can you try deleting the .dvc/tmp/index folder from your repo, and then re-run dvc push -v? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D661501707&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=XDP_21UwbanwBtQDv7vWa05wzZABjE1HMYUFbMvsp9A&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HECYF2MC45VS2JLMJDR4TQR5ANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=o3TnaZkqf31DZrYcVPp3cJnQNYCx6p2ZgYz9fbTEAgo&e=>.

[LarryThermo]

Actually I might have self-diagnosed the problem. I think the problem has do with a long directory or path name. The full path to the dvc repository is: C:\Projects-Git\MasuAutoFocusTraining.Hybrid\MasuAutoFocusTraining.Simple\MasuAutoFocusTraining.Simple.ManufacturingVariability\MasuAutoFocusTraining.Simple.ManufacturingVariability.Beads If I shorten it to: C:\Projects-Git\MasuAutoFocusTraining.Hybrid\MasuAutoFocusTraining.Simple\MasuAutoFocusTraining.Simple.ManufacturingVariability\a then the push succeeds. Best, Larry From: Peter Rowlands (변기호) [mailto:notifications@github.com] Sent: Monday, July 20, 2020 5:47 PM To: iterative/dvc <dvc@noreply.github.com> Cc: Rystrom, Larry <Larry.Rystrom@thermofisher.com>; Mention <mention@noreply.github.com> Subject: Re: [iterative/dvc] Problems with dvc push (#4234) CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to SOC@thermofisher.com. @LarryThermo<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=G849SKpEPXS7kSOsdNrj8kHDt0HoSbOLb8zNcUG8WAU&e=> can you try deleting the .dvc/tmp/index folder from your repo, and then re-run dvc push -v? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D661501707&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=XDP_21UwbanwBtQDv7vWa05wzZABjE1HMYUFbMvsp9A&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HECYF2MC45VS2JLMJDR4TQR5ANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=o3TnaZkqf31DZrYcVPp3cJnQNYCx6p2ZgYz9fbTEAgo&e=>.

[pmrowla]

I don't think there's much we can do about the path length limitation on Windows, but we could at least check for this and give a better error message if we fail to open the index DB.

[LarryThermo]

Yes, it really has nothing to do with dvc and improving the error message is about what you can do. Please note that I am really enjoying using dvc and thank you. From: Peter Rowlands (변기호) [mailto:notifications@github.com] Sent: Thursday, July 23, 2020 8:15 AM To: iterative/dvc <dvc@noreply.github.com> Cc: Rystrom, Larry <Larry.Rystrom@thermofisher.com>; Mention <mention@noreply.github.com> Subject: Re: [iterative/dvc] Problems with dvc push (#4234) CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to SOC@thermofisher.com. I don't think there's much we can do about the path length limitation on Windows, but we could at least check for this and give a better error message if we fail to open the index DB. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D663065050&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=__9ePWo8DZTHM_AgliBR8tlk_1DVuTImEPDEIbBBZs0&s=pkQjuZ99GVvBcM6eDjdkqJ3MggxINHqzdpebk2UW7TQ&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HDZCZRBK4QMUNRLGILR5BHV5ANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=__9ePWo8DZTHM_AgliBR8tlk_1DVuTImEPDEIbBBZs0&s=Z6Mmj_INo8iFpc9PE6WPad5lkLV_2RmU9gLsxSNbL9A&e=>.

[efiop]

Hm, btw, we use an absolute path when opening the db, so we probably could've avoided this issue if we were to use a relative path instead. This is something that we are constantly talking about in many places in dvc, and is prone to hitting MAX_PATH on other platforms as well. So the best solution will be if we make that switch, but that won't happen in the nearest future :(

As to the error(or warning), a meaningful one will require some sysconf magic (MAX_PATH might not be 260, it might be more or might even be ignored by the system) and will bring us questionable results. Shame sqlite doesn't raise a meaningful exception for this 🙁 So considering this, I think we should just close this issue for now, since there is an easy workaround. If we get more reports - we'll reopen and will probably try to use relpaths at least when dealing with sqlite, but that will likely make the issue arise in a different place 🙁

If anyone else is running into this issue, please leave a comment so we are aware. Thank you.