Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cli: Improve http/https dichotomy #93

Merged
merged 1 commit into from Nov 17, 2020
Merged

cli: Improve http/https dichotomy #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 6 additions & 2 deletions cernopendata_client/cli.py
View file
Open in desktop
Expand Up @@ -115,7 +115,7 @@ def get_metadata(server, recid, doi, title, output_value):
@click.option(
"--protocol",
default="http",
type=click.Choice(["http", "https", "xrootd"]),
type=click.Choice(["http", "xrootd"]),
help="Protocol to be used in links.",
)
@click.option(
Expand Down Expand Up @@ -147,6 +147,8 @@ def get_file_locations(server, recid, doi, title, protocol, expand, verbose):
$ cernopendata-client get-file-locations --recid 5500 --verbose
"""
validate_server(server)
if server != SERVER_HTTP_URI:
protocol = server.split(":")[0]
if recid is not None:
validate_recid(recid)
record_json = get_record_as_json(server, recid, doi, title)
Expand All @@ -165,7 +167,7 @@ def get_file_locations(server, recid, doi, title, protocol, expand, verbose):
@click.option(
"--protocol",
default="http",
type=click.Choice(["http", "https", "xrootd"]),
type=click.Choice(["http", "xrootd"]),
help="Protocol to be used in links.",
)
@click.option(
Expand Down Expand Up @@ -229,6 +231,8 @@ def download_files(
$ cernopendata-client download-files --recid 5500 --filter-regexp py --filter-range 1-2
"""
validate_server(server)
if server != SERVER_HTTP_URI:
protocol = server.split(":")[0]
if recid is not None:
validate_recid(recid)
record_json = get_record_as_json(server, recid, doi, title)
Expand Down
2 changes: 1 addition & 1 deletion cernopendata_client/downloader.py
View file
Open in desktop
Expand Up @@ -87,7 +87,7 @@ def download_single_file(path=None, file_location=None, protocol=None):
if not xrootd_available:
display_message(
msg_type="error",
msg="xrootd is not installed on system. Please use the 'http' or 'https' protocol instead.",
msg="xrootd is not installed on system. Please use the 'http' protocol instead.",
)
sys.exit(1)
file_src = file_location.split("root://eospublic.cern.ch/")[-1]
Expand Down
2 changes: 1 addition & 1 deletion cernopendata_client/searcher.py
View file
Open in desktop
Expand Up @@ -180,7 +180,7 @@ def get_files_list(

:param server: CERN Open Data server to query
:param record_json: Record content in JSON
:protocol: Protocol to be used in links(http | https | xrootd)
:protocol: Protocol to be used in links(http | xrootd)
:expand: Flag for expanding file indexes
:verbose: Flag for showing size and checksum of file
:type server: str
Expand Down
66 changes: 2 additions & 64 deletions docs/usage.rst
View file
Open in desktop
Expand Up @@ -111,25 +111,7 @@ the **get-file-locations** command:
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.png

This command will output URIs for all the files associated with the record ID 5500, using the HTTP protocol.

**HTTPS protocol**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is good to remove this section, but then we may not be having any HTTPS note anywhere. So you can add here:

Note that you can specify `--server https://opendata.cern.ch` if you would like to use the HTTPS protocol instead.

in those places where we spoke about HTTPS... In this way when people search the docs for HTTPS they would find at least a note on this.


.. code-block:: console

$ cernopendata-client get-file-locations --recid 5500 --protocol https
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/BuildFile.xml
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/HiggsDemoAnalyzer.cc
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/List_indexfile.txt
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall.cc
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall_lvl3.cc
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3MC.py
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3data.py
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4MC.py
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4data.py
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.pdf
https://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.png

This command will output URIs for all the files associated with the record ID 5500, using the HTTPS protocol.
Note that you can specify `--server https://opendata.cern.ch` if you would like to use the HTTPS protocol instead.

**XRootD protocol**

Expand Down Expand Up @@ -271,51 +253,7 @@ In order to download data files belonging to a record, please use the
==> Success!

The command will download files into a `5500` directory.

**HTTPS protocol**

Note that you can use ``--protocol https`` command-line option if you
would rather want to download files with https protocol:

.. code-block:: console

$ cernopendata-client download-files --recid 5500 --protocol https
==> Downloading file 1 of 11
-> File: ./5500/BuildFile.xml
-> Progress: 0/0 kiB (100%)
==> Downloading file 2 of 11
-> File: ./5500/HiggsDemoAnalyzer.cc
-> Progress: 81/81 kiB (100%)
==> Downloading file 3 of 11
-> File: ./5500/List_indexfile.txt
-> Progress: 1/1 kiB (100%)
==> Downloading file 4 of 11
-> File: ./5500/M4Lnormdatall.cc
-> Progress: 14/14 kiB (100%)
==> Downloading file 5 of 11
-> File: ./5500/M4Lnormdatall_lvl3.cc
-> Progress: 15/15 kiB (100%)
==> Downloading file 6 of 11
-> File: ./5500/demoanalyzer_cfg_level3MC.py
-> Progress: 3/3 kiB (100%)
==> Downloading file 7 of 11
-> File: ./5500/demoanalyzer_cfg_level3data.py
-> Progress: 3/3 kiB (100%)
==> Downloading file 8 of 11
-> File: ./5500/demoanalyzer_cfg_level4MC.py
-> Progress: 3/3 kiB (100%)
==> Downloading file 9 of 11
-> File: ./5500/demoanalyzer_cfg_level4data.py
-> Progress: 3/3 kiB (100%)
==> Downloading file 10 of 11
-> File: ./5500/mass4l_combine.pdf
-> Progress: 17/17 kiB (100%)
==> Downloading file 11 of 11
-> File: ./5500/mass4l_combine.png
-> Progress: 90/90 kiB (100%)
==> Success!

The command will download files into a `5500` directory.
Note that you can specify `--server https://opendata.cern.ch` if you would like to use the HTTPS protocol instead.

**XRootD protocol**

Expand Down
5 changes: 3 additions & 2 deletions tests/test_cli_download_files.py
View file
Open in desktop
Expand Up @@ -14,6 +14,7 @@

from click.testing import CliRunner
from cernopendata_client.cli import download_files
from cernopendata_client.config import SERVER_HTTPS_URI


def test_dry_run_from_recid():
Expand Down Expand Up @@ -94,7 +95,7 @@ def test_download_files_https_pycurl():
os.remove(test_file)
test_download_files = CliRunner()
test_result = test_download_files.invoke(
download_files, ["--recid", 3005, "--protocol", "https"]
download_files, ["--recid", 3005, "--server", SERVER_HTTPS_URI]
)
assert test_result.exit_code == 0
assert os.path.isfile(test_file) is True
Expand All @@ -112,7 +113,7 @@ def test_download_files_https_requests(mocker):
os.remove(test_file)
test_download_files = CliRunner()
test_result = test_download_files.invoke(
download_files, ["--recid", 3005, "--protocol", "https"]
download_files, ["--recid", 3005, "--server", SERVER_HTTPS_URI]
)
assert test_result.exit_code == 0
assert os.path.isfile(test_file) is True
Expand Down