Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFC: [vcpkg-downloads] Helper script to parse download.json #38656

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

RFC: [vcpkg-downloads] Helper script to parse download.json #38656

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fe2b3f6
[vcpkg-downloads] Helper script to parse download.json
Neumann-A May 9, 2024
0829512
add vcpkg_download_distfile support and fix patches
Neumann-A May 9, 2024
555a341
Use - instead of _
Neumann-A May 9, 2024
1e0bc30
add "if" field
Neumann-A May 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
55 changes: 55 additions & 0 deletions ports/vcpkg-downloads/download.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"downloads" : [
{
"output_variable" : "extracted_src_github",
"from": "github",
"repository" : "facebook/zstd",
"ref": "v1.5.6",
"sha512" : "ca12dffd86618ca008e1ecc79056c1129cb4e61668bf13a3cd5b2fa5c93bc9c92c80f64c1870c68b9c20009d9b3a834eac70db72242d5106125a1c53cccf8de8",
"head_ref": "dev",
"patches" : [],
"target_dir" : "${other_src_dir}/zstd"
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could also be:

Suggested change
{
"output_variable" : "extracted_src_github",
"from": "github",
"repository" : "facebook/zstd",
"ref": "v1.5.6",
"sha512" : "ca12dffd86618ca008e1ecc79056c1129cb4e61668bf13a3cd5b2fa5c93bc9c92c80f64c1870c68b9c20009d9b3a834eac70db72242d5106125a1c53cccf8de8",
"head_ref": "dev",
"patches" : [],
"target_dir" : "${other_src_dir}/zstd"
},
"extracted_src_github" : {
"from": "github",
"repository" : "facebook/zstd",
"ref": "v1.5.6",
"sha512" : "ca12dffd86618ca008e1ecc79056c1129cb4e61668bf13a3cd5b2fa5c93bc9c92c80f64c1870c68b9c20009d9b3a834eac70db72242d5106125a1c53cccf8de8",
"head_ref": "dev",
"patches" : [],
"target_dir" : "${other_src_dir}/zstd"
},

Basically dropping the output_variable field

Copy link
Contributor

@Thomas1664 Thomas1664 May 9, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer "github": "output-variable": "...", ...} because otherwise the set of allowed fields would depend on the value of from which is way harder to parse compared to this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't quite understand this comment: I still need from here and the fields always depend on where it comes from. (doesn't matter if that is depending on from or some other mechanism )

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You suggested here to make the output variable the key of the object instead of having an array of keyless objects. I propose to make the value of from the key:

"github" : {
      "output-variable": "extracted_src_github",
      "repository" : "facebook/zstd",
      "ref": "v1.5.6",
      "sha512" : "ca12dffd86618ca008e1ecc79056c1129cb4e61668bf13a3cd5b2fa5c93bc9c92c80f64c1870c68b9c20009d9b3a834eac70db72242d5106125a1c53cccf8de8",
      "head_ref": "dev",
      "patches" : [],
      "target_dir" : "${other_src_dir}/zstd"
    }

This way you can decide more easily the download method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the main benefit for that is schema validation, however you might end up with something like:

[
 "github" : { .... },
 "github" : { .... },
 "github" : { .... },
 "github" : { .... },
 "gitlab" : { .... },
 "git" : { .... },
]

Leading to something.0.github.--- something.1.github.--- something.2.github.---. The is no meaning in that selection if there is duplicates.

While:

[
 "src1" : { .... },
 "src2" : { .... },
 "src3" : { .... },
 "src4" : { .... },
]

Actually has meaning since srcX has to be an unique identifier since otherwise outputs get overwritten. I also see the json keys as a right to left assignment in this case (data on the right creates the key on the left).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leading to something.0.github.--- something.1.github.--- something.2.github.---. The is no meaning in that selection if there is duplicates.

Why is this a problem? To my understanding, JSON arrays are allowed to have multiple elements (objects) with the same key, e.g. "{"key": "value", "key": "value"} is not allowed whereas ["key": {"subkey": "value"}, "key": {"subkey": "value"}] is allowed.

data on the right creates the key on the left

The output variable itself is data. According to this, it should be on the right whereas github or gitlab are less data-ish. I think JSON should go from general to specific. The output variable is a lot more specific than github or gitlab.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

["key": {"subkey": "value"}, "key": {"subkey": "value"}]

This is no valid JSON...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would need to be [ { "key": {"subkey": "value"} }, { "key": {"subkey": "value"} } ]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the output_variable as key variant, because you prevent collisions by design with that.

{
"output_variable" : "extracted_src_gitlab",
"from": "gitlab",
"host": "https://gitlab.com",
"repository" : "libeigen/eigen",
"ref": "3.4.0",
"sha512" : "ba75ecb760e32acf4ceaf27115468e65d4f77c44f8d519b5a13e7940af2c03a304ad433368cb6d55431f307c5c39e2666ab41d34442db3cf441638e51f5c3b6a",
"head_ref": "master",
Neumann-A marked this conversation as resolved.
Show resolved Hide resolved
"patches" : [],
"target_dir" : "${other_src_dir}/eigen"
},
{
"output_variable" : "extracted_src_bitbucket",
"from": "bitbucket",
"repository" : "blaze-lib/blaze",
"ref": "v3.8.2",
"sha512" : "9786628159991f547902ceb44a159f0ba84d08be16ccc45bfb9aad3cfbf16eaede4ea43d2d4981d420a8a387a07721b113754f6038a6db2d9c7ed2ea967b5361",
"head_ref": "master",
"patches" : [],
"target_dir" : "${other_src_dir}/blaze"
},
{
"output_variable" : "extracted_src_sourceforge",
"from": "sourceforge",
"repository" : "cunit/CUnit",
"ref": "2.1-3",
"filename": "CUnit-2.1-3.tar.bz2",
"sha512" : "547b417109332446dfab8fda17bf4ccd2da841dc93f824dc90a20635bcf1fb80fb2176500d8a0906940f3f3d3e2f77b2d70a71090c9ab84ad9af43f3582bc487",
"patches" : [],
"target_dir" : "${other_src_dir}/cunit"
},
{
"output_variable" : "extracted_src_git",
"from": "git",
"url" : "https://repo.or.cz/libtar.git",
"ref": "6d0ab4c78e7a8305c36a0c3d63fd25cd1493de65",
"sha512" : "0",
"patches" : [],
"target_dir" : "${other_src_dir}/libtar"
}

]
}
20 changes: 20 additions & 0 deletions ports/vcpkg-downloads/portfile.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
file(INSTALL
"${CMAKE_CURRENT_LIST_DIR}/vcpkg-downloads.cmake"
"${CMAKE_CURRENT_LIST_DIR}/vcpkg-port-config.cmake"
DESTINATION "${CURRENT_PACKAGES_DIR}/share/${PORT}")

vcpkg_install_copyright(FILE_LIST "${VCPKG_ROOT_DIR}/LICENSE.txt")
set(VCPKG_POLICY_CMAKE_HELPER_PORT enabled)

block()
set(other_src_dir "${CURRENT_BUILDTREES_DIR}/src2")
include("${CMAKE_CURRENT_LIST_DIR}/vcpkg-downloads.cmake")
vcpkg_download_from_json(JSONS "${CMAKE_CURRENT_LIST_DIR}/download.json")
message(STATUS "extracted_src_github:${extracted_src_github}")
message(STATUS "extracted_src_gitlab:${extracted_src_gitlab}")
message(STATUS "extracted_src_bitbucket:${extracted_src_bitbucket}")
message(STATUS "extracted_src_sourceforge:${extracted_src_sourceforge}")
message(STATUS "extracted_src_git:${extracted_src_git}")
endblock()

message(FATAL_ERROR)
222 changes: 222 additions & 0 deletions ports/vcpkg-downloads/vcpkg-downloads.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
include_guard(GLOBAL)

function(vcpkg_download_from_json)
cmake_parse_arguments(PARSE_ARGV 0 "arg" "" "" "JSONS")
if(DEFINED arg_UNPARSED_ARGUMENTS)
message(FATAL_ERROR "vcpkg_cmake_install was passed extra arguments: ${arg_UNPARSED_ARGUMENTS}")
endif()

if(NOT arg_JSONS)
list(APPEND arg_JSONS "${CMAKE_CURRENT_LIST_DIR}/download.json")
endif()

foreach(json IN LISTS arg_JSONS)
z_vcpkg_download_from_json(vars "${json}")
endforeach()

foreach(o IN LISTS vars)
set("${o}" "${${o}}" PARENT_SCOPE)
endforeach()
endfunction()

function(z_vcpkg_download_from_json outs json_file)
file(READ "${json_file}" json_data)
z_read_json_member(outvars "${json_data}" "downloads")
foreach(o IN LISTS outvars)
message("${o}=${${o}}")
endforeach()

set(out "")

foreach(download_item IN LISTS downloads.index_list)
set(prefix downloads.${download_item})
foreach(member IN LISTS ${prefix}.members) # Expand all CMake variables in members
string(CONFIGURE "${${prefix}.${member}}" "${prefix}.${member}" ESCAPE_QUOTES)
endforeach()

cmake_language(CALL z_vcpkg_download_from_${${prefix}.from} "${prefix}")
list(APPEND out "${${prefix}.output_variable}")

if(${prefix}.target_dir) # Check --editable?
file(REMOVE_RECURSE ${${prefix}.target_dir})
# Would like to use rename here ?
file(COPY "${${${prefix}.output_variable}}/" DESTINATION "${${prefix}.target_dir}")
set("${${prefix}.output_variable}" "${${prefix}.target_dir}" PARENT_SCOPE)
endif()

endforeach()
set(${outs} ${${outs}} ${out} PARENT_SCOPE)
endfunction()

##### Download stuff

function(z_vcpkg_download_from_github data_prefix)
set(opts "")
if(${data_prefix}.host)
list(APPEND opts GITHUB_HOST "${${data_prefix}.host}")
endif()

vcpkg_from_github(
OUT_SOURCE_PATH "${${data_prefix}.output_variable}"
REPO "${${data_prefix}.repository}"
REF "${${data_prefix}.ref}"
SHA512 "${${data_prefix}.sha512}"
HEAD_REF "${${data_prefix}.head_ref}"
PATCHES ${${data_prefix}.patches}
${opts}
)
set("${${data_prefix}.output_variable}" "${${${data_prefix}.output_variable}}" PARENT_SCOPE)
endfunction()

function(z_vcpkg_download_from_gitlab data_prefix)
set(opts "")
if(${data_prefix}.host)
list(APPEND opts GITLAB_URL "${${data_prefix}.host}")
endif()

vcpkg_from_gitlab(
OUT_SOURCE_PATH "${${data_prefix}.output_variable}"
REPO "${${data_prefix}.repository}"
REF "${${data_prefix}.ref}"
SHA512 "${${data_prefix}.sha512}"
HEAD_REF "${${data_prefix}.head_ref}"
PATCHES ${${data_prefix}.patches}
${opts}
)
set("${${data_prefix}.output_variable}" "${${${data_prefix}.output_variable}}" PARENT_SCOPE)
endfunction()

function(z_vcpkg_download_from_bitbucket data_prefix)
vcpkg_from_bitbucket(
OUT_SOURCE_PATH "${${data_prefix}.output_variable}"
REPO "${${data_prefix}.repository}"
REF "${${data_prefix}.ref}"
SHA512 "${${data_prefix}.sha512}"
HEAD_REF "${${data_prefix}.head_ref}"
PATCHES ${${data_prefix}.patches}
)
set("${${data_prefix}.output_variable}" "${${${data_prefix}.output_variable}}" PARENT_SCOPE)
endfunction()

function(z_vcpkg_download_from_git data_prefix)
vcpkg_from_git(
OUT_SOURCE_PATH "${${data_prefix}.output_variable}"
URL "${${data_prefix}.url}"
REF "${${data_prefix}.ref}"
#SHA512 "${${data_prefix}.sha512}"
HEAD_REF "${${data_prefix}.head_ref}"
PATCHES ${${data_prefix}.patches}
# [LFS [<url>]]
)
set("${${data_prefix}.output_variable}" "${${${data_prefix}.output_variable}}" PARENT_SCOPE)
endfunction()

function(z_vcpkg_download_from_sourceforge data_prefix)
vcpkg_from_sourceforge(
OUT_SOURCE_PATH "${${data_prefix}.output_variable}"
REPO "${${data_prefix}.repository}"
REF "${${data_prefix}.ref}"
SHA512 "${${data_prefix}.sha512}"
#HEAD_REF "${${data_prefix}.head_ref}"
PATCHES ${${data_prefix}.patches}
FILENAME "${${data_prefix}.filename}"
#[NO_REMOVE_ONE_LEVEL]
)
set("${${data_prefix}.output_variable}" "${${${data_prefix}.output_variable}}" PARENT_SCOPE)
endfunction()

function(z_vcpkg_download_from_file data_prefix)
endfunction()

##### JSON Parser

function(z_read_json_array out_vars json_data)
set(out "")
set(argn_list ${ARGN})
list(JOIN argn_list "." argn_list_dot)

string(JSON ${argn_list_dot}_length LENGTH "${json_data}" ${ARGN})
set(${argn_list_dot}.size "${${argn_list_dot}_length}")
list(APPEND out ${argn_list_dot}.size)
set(${argn_list_dot}.index_list "")
list(APPEND out ${argn_list_dot}.index_list)

math(EXPR ${argn_list_dot}_length "${${argn_list_dot}_length}-1" OUTPUT_FORMAT DECIMAL)

if(NOT ${argn_list_dot}_length LESS 0)
foreach(${argn_list_dot}_index RANGE ${${argn_list_dot}_length})
set(out_index "")
z_read_json_member(out_index "${json_data}" ${ARGN} "${${argn_list_dot}_index}")
list(APPEND out ${out_index})
list(APPEND ${argn_list_dot}.index_list "${${argn_list_dot}_index}")
unset(out_index)
endforeach()
endif()

foreach(o IN LISTS out)
set("${o}" "${${o}}" PARENT_SCOPE)
endforeach()
set("${out_vars}" ${out} PARENT_SCOPE)
endfunction()

function(z_read_json_obj out_vars json_data)
set(out "")
set(argn_list ${ARGN})
list(JOIN argn_list "." argn_list_dot)

string(JSON ${argn_list_dot}_length LENGTH "${json_data}" ${ARGN})
set(${argn_list_dot}.size "${${argn_list_dot}_length}")
list(APPEND out ${argn_list_dot}.size)
set(${argn_list_dot}.members "")
list(APPEND out ${argn_list_dot}.members)

math(EXPR ${argn_list_dot}_length "${${argn_list_dot}_length}-1" OUTPUT_FORMAT DECIMAL)

if(NOT ${argn_list_dot}_length LESS 0)
foreach(${argn_list_dot}_index RANGE ${${argn_list_dot}_length})
set(out_index "")
string(JSON member_name MEMBER "${json_data}" ${ARGN} "${${argn_list_dot}_index}")
z_read_json_member(out_index "${json_data}" ${ARGN} "${member_name}")
list(APPEND out ${out_index})
list(APPEND ${argn_list_dot}.members "${member_name}")
unset(out_index)
unset(member_name)
endforeach()
endif()
foreach(o IN LISTS out)
set("${o}" "${${o}}" PARENT_SCOPE)
endforeach()
set("${out_vars}" ${out} PARENT_SCOPE)
endfunction()

function(z_read_json_member out_vars json_data)
set(out "")
set(argn_list ${ARGN})
message(STATUS "${ARGN}")
list(JOIN argn_list "." argn_list_dot)
string(JSON "z_${argn_list_dot}.json_type" TYPE "${json_data}" ${ARGN})

if("${z_${argn_list_dot}.json_type}" MATCHES "^(NULL|NUMBER|STRING|BOOLEAN)$")
string(JSON "${argn_list_dot}" GET "${json_data}" ${ARGN})
# Cannot expand CMake variables here if you want output variables to be used in the same JSON file
# string(CONFIGURE "${${argn_list_dot}}" "${argn_list_dot}" ESCAPE_QUOTES)
list(APPEND out "${argn_list_dot}")
elseif("${z_${argn_list_dot}.json_type}" STREQUAL "ARRAY")
set(array_out "")
z_read_json_array(array_out "${json_data}" ${ARGN})
list(APPEND out ${array_out})
unset(array_out)
elseif("${z_${argn_list_dot}.json_type}" STREQUAL "OBJECT")
set(object_out "")
z_read_json_obj(object_out "${json_data}" ${ARGN})
list(APPEND out ${object_out})
unset(object_out)
else()
message(FATAL_ERROR "Unknown json type found")
endif()

foreach(o IN LISTS out)
set("${o}" "${${o}}" PARENT_SCOPE)
endforeach()
set("${out_vars}" ${out} PARENT_SCOPE)
endfunction()
1 change: 1 addition & 0 deletions ports/vcpkg-downloads/vcpkg-port-config.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include("${CMAKE_CURRENT_LIST_DIR}/vcpkg-downloads.cmake")
5 changes: 5 additions & 0 deletions ports/vcpkg-downloads/vcpkg.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"name": "vcpkg-downloads",
"version-date": "2024-05-07",
"license": "MIT"
}