Added configuration option for not doing chunked upload for HTTP remotes#6117
Added configuration option for not doing chunked upload for HTTP remotes#6117scotty3005 wants to merge 8 commits intotreeverse:masterfrom
Conversation
for HTTP remotes Changes to be committed: modified: .codecov.yml modified: .coveragerc modified: .dvc/.gitignore modified: .dvc/config modified: .github/FUNDING.yml modified: .github/ISSUE_TEMPLATE/bug_report.md modified: .github/ISSUE_TEMPLATE/config.yml modified: .github/ISSUE_TEMPLATE/feature_request.md modified: .github/PULL_REQUEST_TEMPLATE.md modified: .github/dependabot.yml modified: .github/mergify.yml modified: .github/release-drafter.yml modified: .github/workflows/packages.yaml modified: .github/workflows/tests.yaml modified: .gitignore modified: .mailmap modified: .pre-commit-config.yaml modified: .pre-commit-hooks.yaml modified: .pylintrc modified: .zenodo.json modified: CODE_OF_CONDUCT.md modified: CONTRIBUTING.md modified: LICENSE modified: MANIFEST.in modified: README.rst modified: bin/dvc.bat modified: dvc/__init__.py modified: dvc/__main__.py modified: dvc/_debug.py modified: dvc/analytics.py modified: dvc/api.py modified: dvc/checkout.py modified: dvc/cli.py modified: dvc/command/__init__.py modified: dvc/command/add.py modified: dvc/command/base.py modified: dvc/command/cache.py modified: dvc/command/check_ignore.py modified: dvc/command/checkout.py modified: dvc/command/commit.py modified: dvc/command/completion.py modified: dvc/command/config.py modified: dvc/command/daemon.py modified: dvc/command/dag.py modified: dvc/command/data_sync.py modified: dvc/command/destroy.py modified: dvc/command/diff.py modified: dvc/command/experiments.py modified: dvc/command/freeze.py modified: dvc/command/gc.py modified: dvc/command/get.py modified: dvc/command/get_url.py modified: dvc/command/git_hook.py modified: dvc/command/imp.py modified: dvc/command/imp_url.py modified: dvc/command/init.py modified: dvc/command/install.py modified: dvc/command/live.py modified: dvc/command/ls/__init__.py modified: dvc/command/ls/ls_colors.py modified: dvc/command/metrics.py modified: dvc/command/move.py modified: dvc/command/params.py modified: dvc/command/plots.py modified: dvc/command/remote.py modified: dvc/command/remove.py modified: dvc/command/repro.py modified: dvc/command/root.py modified: dvc/command/run.py modified: dvc/command/stage.py modified: dvc/command/status.py modified: dvc/command/unprotect.py modified: dvc/command/update.py modified: dvc/command/version.py modified: dvc/compare.py modified: dvc/config.py modified: dvc/config_schema.py modified: dvc/daemon.py modified: dvc/dagascii.py modified: dvc/data_cloud.py modified: dvc/dependency/__init__.py modified: dvc/dependency/base.py modified: dvc/dependency/param.py modified: dvc/dependency/repo.py modified: dvc/dvcfile.py modified: dvc/env.py modified: dvc/exceptions.py modified: dvc/external_repo.py modified: dvc/fs/__init__.py modified: dvc/fs/_metadata.py modified: dvc/fs/azure.py modified: dvc/fs/base.py modified: dvc/fs/dvc.py modified: dvc/fs/fsspec_wrapper.py modified: dvc/fs/gdrive.py modified: dvc/fs/git.py modified: dvc/fs/gs.py modified: dvc/fs/hdfs.py modified: dvc/fs/http.py modified: dvc/fs/https.py modified: dvc/fs/local.py modified: dvc/fs/memory.py modified: dvc/fs/oss.py modified: dvc/fs/pool.py modified: dvc/fs/repo.py modified: dvc/fs/s3.py modified: dvc/fs/ssh/__init__.py modified: dvc/fs/ssh/connection.py modified: dvc/fs/webdav.py modified: dvc/fs/webdavs.py modified: dvc/fs/webhdfs.py modified: dvc/hash_info.py modified: dvc/ignore.py modified: dvc/info.py modified: dvc/istextfile.py modified: dvc/lock.py modified: dvc/logger.py modified: dvc/main.py modified: dvc/objects/__init__.py modified: dvc/objects/db/__init__.py modified: dvc/objects/db/base.py modified: dvc/objects/db/gdrive.py modified: dvc/objects/db/local.py modified: dvc/objects/db/ssh.py modified: dvc/objects/errors.py modified: dvc/objects/file.py modified: dvc/objects/stage.py modified: dvc/objects/tree.py modified: dvc/output.py modified: dvc/parsing/__init__.py modified: dvc/parsing/context.py modified: dvc/parsing/interpolate.py modified: dvc/parsing/versions.py modified: dvc/path_info.py modified: dvc/pathspec_math.py modified: dvc/progress.py modified: dvc/prompt.py modified: dvc/remote/__init__.py modified: dvc/remote/base.py modified: dvc/remote/index.py modified: dvc/remote/local.py modified: dvc/remote/slow_link_detection.py modified: dvc/repo/__init__.py modified: dvc/repo/add.py modified: dvc/repo/brancher.py modified: dvc/repo/checkout.py modified: dvc/repo/collect.py modified: dvc/repo/commit.py modified: dvc/repo/destroy.py modified: dvc/repo/diff.py modified: dvc/repo/experiments/__init__.py modified: dvc/repo/experiments/apply.py modified: dvc/repo/experiments/base.py modified: dvc/repo/experiments/branch.py modified: dvc/repo/experiments/diff.py modified: dvc/repo/experiments/executor/__init__.py modified: dvc/repo/experiments/executor/base.py modified: dvc/repo/experiments/executor/local.py modified: dvc/repo/experiments/gc.py modified: dvc/repo/experiments/ls.py modified: dvc/repo/experiments/pull.py modified: dvc/repo/experiments/push.py modified: dvc/repo/experiments/remove.py modified: dvc/repo/experiments/run.py modified: dvc/repo/experiments/show.py modified: dvc/repo/experiments/utils.py modified: dvc/repo/fetch.py modified: dvc/repo/freeze.py modified: dvc/repo/gc.py modified: dvc/repo/get.py modified: dvc/repo/get_url.py modified: dvc/repo/graph.py modified: dvc/repo/imp.py modified: dvc/repo/imp_url.py modified: dvc/repo/init.py modified: dvc/repo/install.py modified: dvc/repo/live.py modified: dvc/repo/ls.py modified: dvc/repo/metrics/__init__.py modified: dvc/repo/metrics/diff.py modified: dvc/repo/metrics/show.py modified: dvc/repo/move.py modified: dvc/repo/params/__init__.py modified: dvc/repo/params/diff.py modified: dvc/repo/params/show.py modified: dvc/repo/plots/__init__.py modified: dvc/repo/plots/data.py modified: dvc/repo/plots/diff.py modified: dvc/repo/plots/template.py modified: dvc/repo/pull.py modified: dvc/repo/push.py modified: dvc/repo/remove.py modified: dvc/repo/reproduce.py modified: dvc/repo/run.py modified: dvc/repo/scm_context.py modified: dvc/repo/stage.py modified: dvc/repo/status.py modified: dvc/repo/trie.py modified: dvc/repo/update.py modified: dvc/rwlock.py modified: dvc/schema.py modified: dvc/scheme.py modified: dvc/scm/__init__.py modified: dvc/scm/base.py modified: dvc/scm/git/__init__.py modified: dvc/scm/git/backend/__init__.py modified: dvc/scm/git/backend/base.py modified: dvc/scm/git/backend/dulwich.py modified: dvc/scm/git/backend/gitpython.py modified: dvc/scm/git/backend/pygit2.py modified: dvc/scm/git/objects.py modified: dvc/scm/git/stash.py modified: dvc/stage/__init__.py modified: dvc/stage/cache.py modified: dvc/stage/decorators.py modified: dvc/stage/exceptions.py modified: dvc/stage/imports.py modified: dvc/stage/loader.py modified: dvc/stage/monitor.py modified: dvc/stage/params.py modified: dvc/stage/run.py modified: dvc/stage/serialize.py modified: dvc/stage/utils.py modified: dvc/state.py modified: dvc/system.py modified: dvc/types.py modified: dvc/ui/__init__.py modified: dvc/ui/table.py modified: dvc/updater.py modified: dvc/utils/.gitignore modified: dvc/utils/__init__.py modified: dvc/utils/cli_parse.py modified: dvc/utils/collections.py modified: dvc/utils/conversions.py modified: dvc/utils/diff.py modified: dvc/utils/flatten.py modified: dvc/utils/fs.py modified: dvc/utils/html.py modified: dvc/utils/http.py modified: dvc/utils/humanize.py modified: dvc/utils/pager.py modified: dvc/utils/pkg.py modified: dvc/utils/serialize/__init__.py modified: dvc/utils/serialize/_common.py modified: dvc/utils/serialize/_json.py modified: dvc/utils/serialize/_py.py modified: dvc/utils/serialize/_toml.py modified: dvc/utils/serialize/_yaml.py modified: dvc/utils/stream.py modified: dvc/utils/table.py modified: dvc/version.py modified: fastentrypoints.py modified: pyproject.toml modified: scripts/build-requirements.txt modified: scripts/build.py modified: scripts/fpm/.gitignore modified: scripts/fpm/after-install.sh modified: scripts/fpm/after-remove.sh modified: scripts/fpm/build.py modified: scripts/fpm/notarize.py modified: scripts/fpm/sign.py modified: scripts/innosetup/.gitignore modified: scripts/innosetup/addSymLinkPermissions.ps1 modified: scripts/innosetup/addsymlink.iss modified: scripts/innosetup/build.py modified: scripts/innosetup/dvc.ico.dvc modified: scripts/innosetup/dvc_left.bmp.dvc modified: scripts/innosetup/dvc_up.bmp.dvc modified: scripts/innosetup/modpath.iss modified: scripts/innosetup/setup.iss modified: scripts/pyinstaller/.gitignore modified: scripts/pyinstaller/build.py modified: scripts/pyinstaller/entitlements.plist modified: scripts/pyinstaller/hooks/hook-dvc.py modified: scripts/pyinstaller/hooks/hook-dvc.system.py modified: scripts/pyinstaller/hooks/hook-dvc.tree.gs.py modified: scripts/pyinstaller/hooks/hook-dvc.utils.flatten.py modified: scripts/pyinstaller/hooks/hook-google_compute_engine.logger.py modified: scripts/pyinstaller/hooks/hook-googleapiclient.model.py modified: scripts/pyinstaller/hooks/hook-pydrive2.py modified: scripts/pyinstaller/hooks/hook-win32file.py modified: scripts/pyinstaller/sign.py modified: scripts/schema/dvc-yaml.json modified: setup.cfg modified: setup.py modified: test_requirements.txt modified: tests/__init__.py modified: tests/__main__.py modified: tests/basic_env.py modified: tests/conftest.py modified: tests/dir_helpers.py modified: tests/docker-compose.yml modified: tests/func/__init__.py modified: tests/func/conftest.py modified: tests/func/experiments/__init__.py modified: tests/func/experiments/conftest.py modified: tests/func/experiments/test_checkpoints.py modified: tests/func/experiments/test_diff.py modified: tests/func/experiments/test_experiments.py modified: tests/func/experiments/test_gc.py modified: tests/func/experiments/test_remote.py modified: tests/func/experiments/test_show.py modified: tests/func/metrics/__init__.py modified: tests/func/metrics/test_diff.py modified: tests/func/metrics/test_show.py modified: tests/func/params/__init__.py modified: tests/func/params/test_diff.py modified: tests/func/params/test_show.py modified: tests/func/parsing/__init__.py modified: tests/func/parsing/test_errors.py modified: tests/func/parsing/test_foreach.py modified: tests/func/parsing/test_interpolated_entry.py modified: tests/func/parsing/test_resolver.py modified: tests/func/plots/__init__.py modified: tests/func/plots/conftest.py modified: tests/func/plots/test_diff.py modified: tests/func/plots/test_modify.py modified: tests/func/plots/test_show.py modified: tests/func/plots/utils.py modified: tests/func/remote/__init__.py modified: tests/func/remote/test_index.py modified: tests/func/test_add.py modified: tests/func/test_analytics.py modified: tests/func/test_api.py modified: tests/func/test_check_ignore.py modified: tests/func/test_checkout.py modified: tests/func/test_cli.py modified: tests/func/test_commit.py modified: tests/func/test_config.py modified: tests/func/test_data_cloud.py modified: tests/func/test_diff.py modified: tests/func/test_dvcfile.py modified: tests/func/test_external_repo.py modified: tests/func/test_fs.py modified: tests/func/test_gc.py modified: tests/func/test_get.py modified: tests/func/test_get_url.py modified: tests/func/test_ignore.py modified: tests/func/test_import.py modified: tests/func/test_import_url.py modified: tests/func/test_init.py modified: tests/func/test_install.py modified: tests/func/test_live.py modified: tests/func/test_lock.py modified: tests/func/test_lockfile.py modified: tests/func/test_ls.py modified: tests/func/test_merge_driver.py modified: tests/func/test_move.py modified: tests/func/test_odb.py modified: tests/func/test_remote.py modified: tests/func/test_remove.py modified: tests/func/test_repo.py modified: tests/func/test_repro.py modified: tests/func/test_repro_multistage.py modified: tests/func/test_root.py modified: tests/func/test_run_cache.py modified: tests/func/test_run_multistage.py modified: tests/func/test_run_single_stage.py modified: tests/func/test_scm.py modified: tests/func/test_stage.py modified: tests/func/test_stage_load.py modified: tests/func/test_state.py modified: tests/func/test_status.py modified: tests/func/test_system.py modified: tests/func/test_unprotect.py modified: tests/func/test_update.py modified: tests/func/test_utils.py modified: tests/func/test_version.py modified: tests/func/utils/__init__.py modified: tests/func/utils/test_fs.py modified: tests/func/utils/test_html.py modified: tests/pylint_plugin_disable.py modified: tests/remotes/__init__.py modified: tests/remotes/azure.py modified: tests/remotes/base.py modified: tests/remotes/gdrive.py modified: tests/remotes/gs.py modified: tests/remotes/hdfs.py modified: tests/remotes/http.py modified: tests/remotes/local.py modified: tests/remotes/oss.py modified: tests/remotes/s3.py modified: tests/remotes/ssh.py modified: tests/remotes/user.key modified: tests/remotes/user.key.pub modified: tests/remotes/webdav.py modified: tests/remotes/webhdfs.py modified: tests/remotes_env.sample modified: tests/unit/__init__.py modified: tests/unit/command/__init__.py modified: tests/unit/command/ls/__init__.py modified: tests/unit/command/ls/test_ls.py modified: tests/unit/command/ls/test_ls_colors.py modified: tests/unit/command/test_add.py modified: tests/unit/command/test_cache.py modified: tests/unit/command/test_checkout.py modified: tests/unit/command/test_config.py modified: tests/unit/command/test_dag.py modified: tests/unit/command/test_data_sync.py modified: tests/unit/command/test_diff.py modified: tests/unit/command/test_experiments.py modified: tests/unit/command/test_get.py modified: tests/unit/command/test_get_url.py modified: tests/unit/command/test_git_hook.py modified: tests/unit/command/test_imp.py modified: tests/unit/command/test_imp_url.py modified: tests/unit/command/test_live.py modified: tests/unit/command/test_metrics.py modified: tests/unit/command/test_params.py modified: tests/unit/command/test_plots.py modified: tests/unit/command/test_repro.py modified: tests/unit/command/test_run.py modified: tests/unit/command/test_stage.py modified: tests/unit/command/test_status.py modified: tests/unit/command/test_update.py modified: tests/unit/dependency/__init__.py modified: tests/unit/dependency/test_dependency.py modified: tests/unit/dependency/test_params.py modified: tests/unit/fs/__init__.py modified: tests/unit/fs/test_azure.py modified: tests/unit/fs/test_base.py modified: tests/unit/fs/test_dvc.py modified: tests/unit/fs/test_fs.py modified: tests/unit/fs/test_hdfs.py modified: tests/unit/fs/test_repo.py modified: tests/unit/fs/test_repo_metadata.py modified: tests/unit/fs/test_s3.py modified: tests/unit/fs/test_tree.py modified: tests/unit/objects/__init__.py modified: tests/unit/objects/db/__init__.py modified: tests/unit/objects/db/test_local.py modified: tests/unit/objects/test_tree.py modified: tests/unit/output/__init__.py modified: tests/unit/output/test_load.py modified: tests/unit/output/test_local.py modified: tests/unit/output/test_output.py modified: tests/unit/remote/__init__.py modified: tests/unit/remote/ssh/__init__.py modified: tests/unit/remote/ssh/test_connection.py modified: tests/unit/remote/ssh/test_pool.py modified: tests/unit/remote/ssh/test_ssh.py modified: tests/unit/remote/test_base.py modified: tests/unit/remote/test_gdrive.py modified: tests/unit/remote/test_http.py modified: tests/unit/remote/test_index.py modified: tests/unit/remote/test_oss.py modified: tests/unit/remote/test_remote.py modified: tests/unit/remote/test_remote_tree.py modified: tests/unit/remote/test_slow_link_detection.py modified: tests/unit/remote/test_webdav.py modified: tests/unit/remote/test_webhdfs.py modified: tests/unit/repo/__init__.py modified: tests/unit/repo/plots/__init__.py modified: tests/unit/repo/plots/test_data.py modified: tests/unit/repo/plots/test_diff.py modified: tests/unit/repo/test_repo.py modified: tests/unit/repo/test_reproduce.py modified: tests/unit/repo/test_run.py modified: tests/unit/scm/__init__.py modified: tests/unit/scm/test_git.py modified: tests/unit/scm/test_scm.py modified: tests/unit/stage/test_cache.py modified: tests/unit/stage/test_loader_pipeline_file.py modified: tests/unit/stage/test_run.py modified: tests/unit/stage/test_serialize_pipeline_file.py modified: tests/unit/stage/test_serialize_pipeline_lock.py modified: tests/unit/stage/test_stage.py modified: tests/unit/stage/test_utils.py modified: tests/unit/test_analytics.py modified: tests/unit/test_collect.py modified: tests/unit/test_compare.py modified: tests/unit/test_config.py modified: tests/unit/test_context.py modified: tests/unit/test_daemon.py modified: tests/unit/test_dvcfile.py modified: tests/unit/test_external_repo.py modified: tests/unit/test_ignore.py modified: tests/unit/test_imports.py modified: tests/unit/test_info.py modified: tests/unit/test_interpolate.py modified: tests/unit/test_lockfile.py modified: tests/unit/test_logger.py modified: tests/unit/test_metrics.py modified: tests/unit/test_params.py modified: tests/unit/test_path_info.py modified: tests/unit/test_pathspec_math.py modified: tests/unit/test_plots.py modified: tests/unit/test_progress.py modified: tests/unit/test_prompt.py modified: tests/unit/test_run.py modified: tests/unit/test_rwlock.py modified: tests/unit/test_state.py modified: tests/unit/test_tabular_data.py modified: tests/unit/test_updater.py modified: tests/unit/test_version.py modified: tests/unit/ui/__init__.py modified: tests/unit/ui/test_console.py modified: tests/unit/ui/test_pager.py modified: tests/unit/ui/test_table.py modified: tests/unit/utils/__init__.py modified: tests/unit/utils/serialize/__init__.py modified: tests/unit/utils/serialize/test_yaml.py modified: tests/unit/utils/test_cli_parse.py modified: tests/unit/utils/test_collections.py modified: tests/unit/utils/test_conversions.py modified: tests/unit/utils/test_fs.py modified: tests/unit/utils/test_http.py modified: tests/unit/utils/test_humanize.py modified: tests/unit/utils/test_stream.py modified: tests/unit/utils/test_utils.py modified: tests/utils/__init__.py modified: tests/utils/asserts.py modified: tests/utils/httpd.py
for HTTP remotes Changes to be committed: modified: .codecov.yml modified: .coveragerc modified: .dvc/.gitignore modified: .dvc/config modified: .github/FUNDING.yml modified: .github/ISSUE_TEMPLATE/bug_report.md modified: .github/ISSUE_TEMPLATE/config.yml modified: .github/ISSUE_TEMPLATE/feature_request.md modified: .github/PULL_REQUEST_TEMPLATE.md modified: .github/dependabot.yml modified: .github/mergify.yml modified: .github/release-drafter.yml modified: .github/workflows/packages.yaml modified: .github/workflows/tests.yaml modified: .gitignore modified: .mailmap modified: .pre-commit-config.yaml modified: .pre-commit-hooks.yaml modified: .pylintrc modified: .zenodo.json modified: CODE_OF_CONDUCT.md modified: CONTRIBUTING.md modified: LICENSE modified: MANIFEST.in modified: README.rst modified: bin/dvc.bat modified: dvc/__init__.py modified: dvc/__main__.py modified: dvc/_debug.py modified: dvc/analytics.py modified: dvc/api.py modified: dvc/checkout.py modified: dvc/cli.py modified: dvc/command/__init__.py modified: dvc/command/add.py modified: dvc/command/base.py modified: dvc/command/cache.py modified: dvc/command/check_ignore.py modified: dvc/command/checkout.py modified: dvc/command/commit.py modified: dvc/command/completion.py modified: dvc/command/config.py modified: dvc/command/daemon.py modified: dvc/command/dag.py modified: dvc/command/data_sync.py modified: dvc/command/destroy.py modified: dvc/command/diff.py modified: dvc/command/experiments.py modified: dvc/command/freeze.py modified: dvc/command/gc.py modified: dvc/command/get.py modified: dvc/command/get_url.py modified: dvc/command/git_hook.py modified: dvc/command/imp.py modified: dvc/command/imp_url.py modified: dvc/command/init.py modified: dvc/command/install.py modified: dvc/command/live.py modified: dvc/command/ls/__init__.py modified: dvc/command/ls/ls_colors.py modified: dvc/command/metrics.py modified: dvc/command/move.py modified: dvc/command/params.py modified: dvc/command/plots.py modified: dvc/command/remote.py modified: dvc/command/remove.py modified: dvc/command/repro.py modified: dvc/command/root.py modified: dvc/command/run.py modified: dvc/command/stage.py modified: dvc/command/status.py modified: dvc/command/unprotect.py modified: dvc/command/update.py modified: dvc/command/version.py modified: dvc/compare.py modified: dvc/config.py modified: dvc/config_schema.py modified: dvc/daemon.py modified: dvc/dagascii.py modified: dvc/data_cloud.py modified: dvc/dependency/__init__.py modified: dvc/dependency/base.py modified: dvc/dependency/param.py modified: dvc/dependency/repo.py modified: dvc/dvcfile.py modified: dvc/env.py modified: dvc/exceptions.py modified: dvc/external_repo.py modified: dvc/fs/__init__.py modified: dvc/fs/_metadata.py modified: dvc/fs/azure.py modified: dvc/fs/base.py modified: dvc/fs/dvc.py modified: dvc/fs/fsspec_wrapper.py modified: dvc/fs/gdrive.py modified: dvc/fs/git.py modified: dvc/fs/gs.py modified: dvc/fs/hdfs.py modified: dvc/fs/http.py modified: dvc/fs/https.py modified: dvc/fs/local.py modified: dvc/fs/memory.py modified: dvc/fs/oss.py modified: dvc/fs/pool.py modified: dvc/fs/repo.py modified: dvc/fs/s3.py modified: dvc/fs/ssh/__init__.py modified: dvc/fs/ssh/connection.py modified: dvc/fs/webdav.py modified: dvc/fs/webdavs.py modified: dvc/fs/webhdfs.py modified: dvc/hash_info.py modified: dvc/ignore.py modified: dvc/info.py modified: dvc/istextfile.py modified: dvc/lock.py modified: dvc/logger.py modified: dvc/main.py modified: dvc/objects/__init__.py modified: dvc/objects/db/__init__.py modified: dvc/objects/db/base.py modified: dvc/objects/db/gdrive.py modified: dvc/objects/db/local.py modified: dvc/objects/db/ssh.py modified: dvc/objects/errors.py modified: dvc/objects/file.py modified: dvc/objects/stage.py modified: dvc/objects/tree.py modified: dvc/output.py modified: dvc/parsing/__init__.py modified: dvc/parsing/context.py modified: dvc/parsing/interpolate.py modified: dvc/parsing/versions.py modified: dvc/path_info.py modified: dvc/pathspec_math.py modified: dvc/progress.py modified: dvc/prompt.py modified: dvc/remote/__init__.py modified: dvc/remote/base.py modified: dvc/remote/index.py modified: dvc/remote/local.py modified: dvc/remote/slow_link_detection.py modified: dvc/repo/__init__.py modified: dvc/repo/add.py modified: dvc/repo/brancher.py modified: dvc/repo/checkout.py modified: dvc/repo/collect.py modified: dvc/repo/commit.py modified: dvc/repo/destroy.py modified: dvc/repo/diff.py modified: dvc/repo/experiments/__init__.py modified: dvc/repo/experiments/apply.py modified: dvc/repo/experiments/base.py modified: dvc/repo/experiments/branch.py modified: dvc/repo/experiments/diff.py modified: dvc/repo/experiments/executor/__init__.py modified: dvc/repo/experiments/executor/base.py modified: dvc/repo/experiments/executor/local.py modified: dvc/repo/experiments/gc.py modified: dvc/repo/experiments/ls.py modified: dvc/repo/experiments/pull.py modified: dvc/repo/experiments/push.py modified: dvc/repo/experiments/remove.py modified: dvc/repo/experiments/run.py modified: dvc/repo/experiments/show.py modified: dvc/repo/experiments/utils.py modified: dvc/repo/fetch.py modified: dvc/repo/freeze.py modified: dvc/repo/gc.py modified: dvc/repo/get.py modified: dvc/repo/get_url.py modified: dvc/repo/graph.py modified: dvc/repo/imp.py modified: dvc/repo/imp_url.py modified: dvc/repo/init.py modified: dvc/repo/install.py modified: dvc/repo/live.py modified: dvc/repo/ls.py modified: dvc/repo/metrics/__init__.py modified: dvc/repo/metrics/diff.py modified: dvc/repo/metrics/show.py modified: dvc/repo/move.py modified: dvc/repo/params/__init__.py modified: dvc/repo/params/diff.py modified: dvc/repo/params/show.py modified: dvc/repo/plots/__init__.py modified: dvc/repo/plots/data.py modified: dvc/repo/plots/diff.py modified: dvc/repo/plots/template.py modified: dvc/repo/pull.py modified: dvc/repo/push.py modified: dvc/repo/remove.py modified: dvc/repo/reproduce.py modified: dvc/repo/run.py modified: dvc/repo/scm_context.py modified: dvc/repo/stage.py modified: dvc/repo/status.py modified: dvc/repo/trie.py modified: dvc/repo/update.py modified: dvc/rwlock.py modified: dvc/schema.py modified: dvc/scheme.py modified: dvc/scm/__init__.py modified: dvc/scm/base.py modified: dvc/scm/git/__init__.py modified: dvc/scm/git/backend/__init__.py modified: dvc/scm/git/backend/base.py modified: dvc/scm/git/backend/dulwich.py modified: dvc/scm/git/backend/gitpython.py modified: dvc/scm/git/backend/pygit2.py modified: dvc/scm/git/objects.py modified: dvc/scm/git/stash.py modified: dvc/stage/__init__.py modified: dvc/stage/cache.py modified: dvc/stage/decorators.py modified: dvc/stage/exceptions.py modified: dvc/stage/imports.py modified: dvc/stage/loader.py modified: dvc/stage/monitor.py modified: dvc/stage/params.py modified: dvc/stage/run.py modified: dvc/stage/serialize.py modified: dvc/stage/utils.py modified: dvc/state.py modified: dvc/system.py modified: dvc/types.py modified: dvc/ui/__init__.py modified: dvc/ui/table.py modified: dvc/updater.py modified: dvc/utils/.gitignore modified: dvc/utils/__init__.py modified: dvc/utils/cli_parse.py modified: dvc/utils/collections.py modified: dvc/utils/conversions.py modified: dvc/utils/diff.py modified: dvc/utils/flatten.py modified: dvc/utils/fs.py modified: dvc/utils/html.py modified: dvc/utils/http.py modified: dvc/utils/humanize.py modified: dvc/utils/pager.py modified: dvc/utils/pkg.py modified: dvc/utils/serialize/__init__.py modified: dvc/utils/serialize/_common.py modified: dvc/utils/serialize/_json.py modified: dvc/utils/serialize/_py.py modified: dvc/utils/serialize/_toml.py modified: dvc/utils/serialize/_yaml.py modified: dvc/utils/stream.py modified: dvc/utils/table.py modified: dvc/version.py modified: fastentrypoints.py modified: pyproject.toml modified: scripts/build-requirements.txt modified: scripts/build.py modified: scripts/fpm/.gitignore modified: scripts/fpm/after-install.sh modified: scripts/fpm/after-remove.sh modified: scripts/fpm/build.py modified: scripts/fpm/notarize.py modified: scripts/fpm/sign.py modified: scripts/innosetup/.gitignore modified: scripts/innosetup/addSymLinkPermissions.ps1 modified: scripts/innosetup/addsymlink.iss modified: scripts/innosetup/build.py modified: scripts/innosetup/dvc.ico.dvc modified: scripts/innosetup/dvc_left.bmp.dvc modified: scripts/innosetup/dvc_up.bmp.dvc modified: scripts/innosetup/modpath.iss modified: scripts/innosetup/setup.iss modified: scripts/pyinstaller/.gitignore modified: scripts/pyinstaller/build.py modified: scripts/pyinstaller/entitlements.plist modified: scripts/pyinstaller/hooks/hook-dvc.py modified: scripts/pyinstaller/hooks/hook-dvc.system.py modified: scripts/pyinstaller/hooks/hook-dvc.tree.gs.py modified: scripts/pyinstaller/hooks/hook-dvc.utils.flatten.py modified: scripts/pyinstaller/hooks/hook-google_compute_engine.logger.py modified: scripts/pyinstaller/hooks/hook-googleapiclient.model.py modified: scripts/pyinstaller/hooks/hook-pydrive2.py modified: scripts/pyinstaller/hooks/hook-win32file.py modified: scripts/pyinstaller/sign.py modified: scripts/schema/dvc-yaml.json modified: setup.cfg modified: setup.py modified: test_requirements.txt modified: tests/__init__.py modified: tests/__main__.py modified: tests/basic_env.py modified: tests/conftest.py modified: tests/dir_helpers.py modified: tests/docker-compose.yml modified: tests/func/__init__.py modified: tests/func/conftest.py modified: tests/func/experiments/__init__.py modified: tests/func/experiments/conftest.py modified: tests/func/experiments/test_checkpoints.py modified: tests/func/experiments/test_diff.py modified: tests/func/experiments/test_experiments.py modified: tests/func/experiments/test_gc.py modified: tests/func/experiments/test_remote.py modified: tests/func/experiments/test_show.py modified: tests/func/metrics/__init__.py modified: tests/func/metrics/test_diff.py modified: tests/func/metrics/test_show.py modified: tests/func/params/__init__.py modified: tests/func/params/test_diff.py modified: tests/func/params/test_show.py modified: tests/func/parsing/__init__.py modified: tests/func/parsing/test_errors.py modified: tests/func/parsing/test_foreach.py modified: tests/func/parsing/test_interpolated_entry.py modified: tests/func/parsing/test_resolver.py modified: tests/func/plots/__init__.py modified: tests/func/plots/conftest.py modified: tests/func/plots/test_diff.py modified: tests/func/plots/test_modify.py modified: tests/func/plots/test_show.py modified: tests/func/plots/utils.py modified: tests/func/remote/__init__.py modified: tests/func/remote/test_index.py modified: tests/func/test_add.py modified: tests/func/test_analytics.py modified: tests/func/test_api.py modified: tests/func/test_check_ignore.py modified: tests/func/test_checkout.py modified: tests/func/test_cli.py modified: tests/func/test_commit.py modified: tests/func/test_config.py modified: tests/func/test_data_cloud.py modified: tests/func/test_diff.py modified: tests/func/test_dvcfile.py modified: tests/func/test_external_repo.py modified: tests/func/test_fs.py modified: tests/func/test_gc.py modified: tests/func/test_get.py modified: tests/func/test_get_url.py modified: tests/func/test_ignore.py modified: tests/func/test_import.py modified: tests/func/test_import_url.py modified: tests/func/test_init.py modified: tests/func/test_install.py modified: tests/func/test_live.py modified: tests/func/test_lock.py modified: tests/func/test_lockfile.py modified: tests/func/test_ls.py modified: tests/func/test_merge_driver.py modified: tests/func/test_move.py modified: tests/func/test_odb.py modified: tests/func/test_remote.py modified: tests/func/test_remove.py modified: tests/func/test_repo.py modified: tests/func/test_repro.py modified: tests/func/test_repro_multistage.py modified: tests/func/test_root.py modified: tests/func/test_run_cache.py modified: tests/func/test_run_multistage.py modified: tests/func/test_run_single_stage.py modified: tests/func/test_scm.py modified: tests/func/test_stage.py modified: tests/func/test_stage_load.py modified: tests/func/test_state.py modified: tests/func/test_status.py modified: tests/func/test_system.py modified: tests/func/test_unprotect.py modified: tests/func/test_update.py modified: tests/func/test_utils.py modified: tests/func/test_version.py modified: tests/func/utils/__init__.py modified: tests/func/utils/test_fs.py modified: tests/func/utils/test_html.py modified: tests/pylint_plugin_disable.py modified: tests/remotes/__init__.py modified: tests/remotes/azure.py modified: tests/remotes/base.py modified: tests/remotes/gdrive.py modified: tests/remotes/gs.py modified: tests/remotes/hdfs.py modified: tests/remotes/http.py modified: tests/remotes/local.py modified: tests/remotes/oss.py modified: tests/remotes/s3.py modified: tests/remotes/ssh.py modified: tests/remotes/user.key modified: tests/remotes/user.key.pub modified: tests/remotes/webdav.py modified: tests/remotes/webhdfs.py modified: tests/remotes_env.sample modified: tests/unit/__init__.py modified: tests/unit/command/__init__.py modified: tests/unit/command/ls/__init__.py modified: tests/unit/command/ls/test_ls.py modified: tests/unit/command/ls/test_ls_colors.py modified: tests/unit/command/test_add.py modified: tests/unit/command/test_cache.py modified: tests/unit/command/test_checkout.py modified: tests/unit/command/test_config.py modified: tests/unit/command/test_dag.py modified: tests/unit/command/test_data_sync.py modified: tests/unit/command/test_diff.py modified: tests/unit/command/test_experiments.py modified: tests/unit/command/test_get.py modified: tests/unit/command/test_get_url.py modified: tests/unit/command/test_git_hook.py modified: tests/unit/command/test_imp.py modified: tests/unit/command/test_imp_url.py modified: tests/unit/command/test_live.py modified: tests/unit/command/test_metrics.py modified: tests/unit/command/test_params.py modified: tests/unit/command/test_plots.py modified: tests/unit/command/test_repro.py modified: tests/unit/command/test_run.py modified: tests/unit/command/test_stage.py modified: tests/unit/command/test_status.py modified: tests/unit/command/test_update.py modified: tests/unit/dependency/__init__.py modified: tests/unit/dependency/test_dependency.py modified: tests/unit/dependency/test_params.py modified: tests/unit/fs/__init__.py modified: tests/unit/fs/test_azure.py modified: tests/unit/fs/test_base.py modified: tests/unit/fs/test_dvc.py modified: tests/unit/fs/test_fs.py modified: tests/unit/fs/test_hdfs.py modified: tests/unit/fs/test_repo.py modified: tests/unit/fs/test_repo_metadata.py modified: tests/unit/fs/test_s3.py modified: tests/unit/fs/test_tree.py modified: tests/unit/objects/__init__.py modified: tests/unit/objects/db/__init__.py modified: tests/unit/objects/db/test_local.py modified: tests/unit/objects/test_tree.py modified: tests/unit/output/__init__.py modified: tests/unit/output/test_load.py modified: tests/unit/output/test_local.py modified: tests/unit/output/test_output.py modified: tests/unit/remote/__init__.py modified: tests/unit/remote/ssh/__init__.py modified: tests/unit/remote/ssh/test_connection.py modified: tests/unit/remote/ssh/test_pool.py modified: tests/unit/remote/ssh/test_ssh.py modified: tests/unit/remote/test_base.py modified: tests/unit/remote/test_gdrive.py modified: tests/unit/remote/test_http.py modified: tests/unit/remote/test_index.py modified: tests/unit/remote/test_oss.py modified: tests/unit/remote/test_remote.py modified: tests/unit/remote/test_remote_tree.py modified: tests/unit/remote/test_slow_link_detection.py modified: tests/unit/remote/test_webdav.py modified: tests/unit/remote/test_webhdfs.py modified: tests/unit/repo/__init__.py modified: tests/unit/repo/plots/__init__.py modified: tests/unit/repo/plots/test_data.py modified: tests/unit/repo/plots/test_diff.py modified: tests/unit/repo/test_repo.py modified: tests/unit/repo/test_reproduce.py modified: tests/unit/repo/test_run.py modified: tests/unit/scm/__init__.py modified: tests/unit/scm/test_git.py modified: tests/unit/scm/test_scm.py modified: tests/unit/stage/test_cache.py modified: tests/unit/stage/test_loader_pipeline_file.py modified: tests/unit/stage/test_run.py modified: tests/unit/stage/test_serialize_pipeline_file.py modified: tests/unit/stage/test_serialize_pipeline_lock.py modified: tests/unit/stage/test_stage.py modified: tests/unit/stage/test_utils.py modified: tests/unit/test_analytics.py modified: tests/unit/test_collect.py modified: tests/unit/test_compare.py modified: tests/unit/test_config.py modified: tests/unit/test_context.py modified: tests/unit/test_daemon.py modified: tests/unit/test_dvcfile.py modified: tests/unit/test_external_repo.py modified: tests/unit/test_ignore.py modified: tests/unit/test_imports.py modified: tests/unit/test_info.py modified: tests/unit/test_interpolate.py modified: tests/unit/test_lockfile.py modified: tests/unit/test_logger.py modified: tests/unit/test_metrics.py modified: tests/unit/test_params.py modified: tests/unit/test_path_info.py modified: tests/unit/test_pathspec_math.py modified: tests/unit/test_plots.py modified: tests/unit/test_progress.py modified: tests/unit/test_prompt.py modified: tests/unit/test_run.py modified: tests/unit/test_rwlock.py modified: tests/unit/test_state.py modified: tests/unit/test_tabular_data.py modified: tests/unit/test_updater.py modified: tests/unit/test_version.py modified: tests/unit/ui/__init__.py modified: tests/unit/ui/test_console.py modified: tests/unit/ui/test_pager.py modified: tests/unit/ui/test_table.py modified: tests/unit/utils/__init__.py modified: tests/unit/utils/serialize/__init__.py modified: tests/unit/utils/serialize/test_yaml.py modified: tests/unit/utils/test_cli_parse.py modified: tests/unit/utils/test_collections.py modified: tests/unit/utils/test_conversions.py modified: tests/unit/utils/test_fs.py modified: tests/unit/utils/test_http.py modified: tests/unit/utils/test_humanize.py modified: tests/unit/utils/test_stream.py modified: tests/unit/utils/test_utils.py modified: tests/utils/__init__.py modified: tests/utils/asserts.py modified: tests/utils/httpd.py
for more information, see https://pre-commit.ci
|
|
||
| def _upload_fobj_not_chunked(self, fobj, to_info): | ||
| response = self.request(self.method, to_info.url, files={"file": fobj}) | ||
| if response.status_code not in (200, 201): |
There was a problem hiding this comment.
Duplicated code with what in _upload_fobj_chunked maybe we can move them to _upload_fobj
| # Context: https://github.com/iterative/dvc/issues/4131 | ||
| with self.request("GET", url, stream=True) as r: | ||
| if r.ok: | ||
| if r.ok or r.status_code == 404: |
There was a problem hiding this comment.
Would this cause some problems when the server was actually not found (404). Some additional check are needed.
|
Thanks for your contributions, so this PR is related to treeverse/dvc-http#9 ? I recommend You can add a link to it. And I'm curious why it shows 513 file changes when they actually change nothing, never see this before, we need to solve it. |
karajan1001
left a comment
There was a problem hiding this comment.
And, some tests are needed for this PR.
|
Hi, thanks for the feedback. I am willing to work on implementing some tests where needed. However I would be glad if I could have some help as it is not clear to me how to test remotes. |
|
I also wonder if https://github.com/intake/filesystem_spec/blob/master/fsspec/implementations/http.py is already able to do what you need. We are migrating to fsspec and will be considering replacing our http implementation with their soon, so might be a good time to give it a shot. |
|
It seems like |
|
@scotty3005 Note that your PR now includes perm changes to a lot of files, which obviously shouldn't be there. There is something wrong with your environment, most likely. Also looks like there is something wrong with commit authorship, likely your email was not added to https://github.com/settings/emails Regarding the contents of the PR, looks like we need to dive deeper to properly review this. That will probably happen once we get to migrating httpfs to fsspec. |
|
Hey @efiop, yes I noted the issue with the changes. I do not what happen. W.r.t. the authorship I believe it is because I worked from two different machines, and forgot to set the local user name and email. |
|
@scotty3005 As @karajan1001 noted, the current logic will cause problems for other servers, so we need more research here to see if we can actually detect this and handle the fallback properly. Unfortunately, I would probably recommend closing this PR for now and going through the research to implement proper logic. 🙁 |
|
Hey @efiop, sure I understand. I will close this and keep maintaining the fork for my particular use case. Thanks for the time you guys dedicated to this issue. I am looking forward to contribute to DVC in other occasions. |
Changes to be committed:
modified: dvc/config_schema.py
modified: dvc/fs/http.py
Draft proposal for feature request treeverse/dvc-http#9
❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
Thank you for the contribution - we'll try to review it as soon as possible. 🙏